Prune done work from the docs: fold the ingest review's open findings into open.md and delete it, cut the hosts wave-2 plan to what is unbuilt, drop the smaller-hosts survey
The plugins README no longer describes a run-time robots.txt check or boards that never close past the page cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a44bf5dacb
commit
802656aca4
8 files changed
+70
-472
No files matched your search
@@ -64,5 +64,7 @@ The reader, `/roles/{uid}`, shows the full posting: the same vote and Save contr
|
||||
|
||||
- [`docs/decisions.md`](docs/decisions.md): what was decided for roledesk and why.
|
||||
- [`docs/open.md`](docs/open.md): what is left to do, for whoever picks this up next.
|
||||
- [`plugins/README.md`](plugins/README.md): every reader, its limits and the Watchlist; [`docs/filter-survey.md`](docs/filter-survey.md): which source-side filters each host takes.
|
||||
- `docs/plan-*.md`: plans in progress, deleted when their work lands.
|
||||
- [`desk/README.md`](desk/README.md), [`pipeline/README.md`](pipeline/README.md), [`auth/README.md`](auth/README.md), [`deploy/README.md`](deploy/README.md): the copied modules, where each comes from and how to refresh it.
|
||||
- [`docs/hosting.md`](docs/hosting.md), [`docs/release.md`](docs/release.md), [`docs/server.md`](docs/server.md), [`docs/password-protection.md`](docs/password-protection.md), [`docs/llm-backends.md`](docs/llm-backends.md): the boilerplate's deployment docs.
|
||||
+28
-19
@@ -1,26 +1,35 @@
|
||||
# Open
|
||||
|
||||
What is left after the phone rework of 2026-09-25, for whoever picks the repo up next, in the order it bites.
|
||||
What is known and not yet fixed, checked against the code on 2026-09-30. What landed is in `decisions.md`; the unbuilt readers are planned in `plan-hosts-wave-2.md`.
|
||||
|
||||
- **The checkout moved.** `CLAUDE.md`, `README.md`, `desk/README.md` and the refresh commands assume `../boilerplate` and `../news` beside the checkout. Keep them beside it in the sandbox or change the paths. `.venv` holds absolute paths, so `uv sync --all-extras` again after the move.
|
||||
- **The boilerplate moved on.** Since roledesk's last refresh (`77052a8`, the schedules fix) it has `4573f6d` (the kit: the layout patterns every app built for itself, as classes) and `1679e88` (deploy from a WSL checkout, the sandbox setup). Refresh `deploy/` and `pipeline/` with `git archive`, and see whether `app/static/app.css` should use the kit's classes instead of its own copies of the same rules.
|
||||
- **Summarising is off** until the switch under Runs (default `summarise.enabled: false`) is used, and needs a model backend, as extract does. The Summarise button on a read page works either way once a backend is configured.
|
||||
- **The owner's agent has no rule on seniority.** Fit compares subject, not level, so an individual contributor's posting ranks beside a head's. Since 2026-09-30 the level is a rung on a ladder with two tracks (`LADDER` in `app/models.py`), filled for half of open roles; a prefer gate still names words (e.g. `principal`, `senior_manager`, `distinguished`, `director`, `exec`). A gate on "at least a rung" would read as the rail's filter does. Extract fills the rest.
|
||||
- **Votes do not train fit.** A vote only moves the rank at read time. Feeding votes into the prompt vectors is worth doing once there are a few hundred.
|
||||
- **76 companies are read by hand**, on the Watchlist with a careers link opened on 2026-09-29: 35 `unread`, 10 `robots`, 9 Teamtailor ones without a career host, 8 `group`, 7 `blocked`, 4 `gone`, 3 `key`. The remaining readers are planned in `docs/plan-hosts-wave-2.md`.
|
||||
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
|
||||
- **An unknown SmartRecruiters or Gem id answers an empty listing, not 404**, so a mistyped slug reads as a complete board with no roles and would close a company's roles. `set_board` reads the board first, which catches a typo only when the person notices "0 roles".
|
||||
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the shared reader waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
|
||||
- **Eightfold returns 10 roles a page and `MAX_PAGES` is 100**, so Infineon (1,289 roles) and Microsoft (about 2,300) would never be read complete and are not attached. The endpoint takes a location filter; a target has nowhere to state one.
|
||||
- **A SuccessFactors site on a company's own domain is not discovered from its own pages**, which link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
|
||||
- **HR Manager's portal disallows reading in robots.txt, its feed host has no robots.txt.** The module checks the feed's host, so a board would be read. Seaborg, the one company on it, is not attached until the owner decides whether the feed counts as allowed.
|
||||
- **Six new hosts' dates are not trusted** (Eightfold, Jibe, Gem, Deel, HiBob, HR Manager; Pinpoint states none), so their backlog counts as posted when first seen. Check each date on a live board and add the host to `TRUSTED_POSTED_HOSTS`.
|
||||
- **SmartRecruiters and Workable roles have no text.** Their lists carry none and their modules have no `detail`, so 7,391 open roles (Bosch, Wise, Ubisoft, Delivery Hero) are ranked on the title alone. Both hosts have a per-posting endpoint; a `detail` in each module fills the text 200 roles per board per pull.
|
||||
- **A posting fetched by hand feeds the reader only.** "Fetch the posting" stores `article_text`, which `summarise` reads and `embed` and `extract` do not.
|
||||
- **A list's teaser is not stored.** Phenom's and Oracle's lists carry a short description, which is dropped so that it cannot replace the full text; until the role's page is read the role has no text at all.
|
||||
## Text
|
||||
|
||||
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only `summarise` reads. `embed` and `extract` read `description or teaser`, so a Jobindex role keeps being ranked on its teaser. The fix is one text rule, `article_text or description or teaser`, in embed and extract, and queuing embed and score after a fetch.
|
||||
- **An edited posting is never embedded again.** The vector is dropped only when a teaser is replaced by a description. A hash of the embedded text on `vectors` would re-embed on any change.
|
||||
- **Extract marks a title-only role done for good.** `upsert_role` does not reset `extracted_at` when text arrives. Extract is on, so this bites. `_EXTRACT_WAITING` re-reads roles whose `seniority_model` is empty, which covers level and not pay.
|
||||
- **Lever keeps only the intro paragraph.** `lever.py` reads `description` and drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way.
|
||||
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 16.5% of open roles. Extract can return `salary_min` and `salary_max` but has read about 500 roles so far.
|
||||
- **A title-only role has no mark in the feed.** 1,304 open roles have no text, 869 of them Eightfold. A mark would show which fits are guesses.
|
||||
|
||||
## Ranking
|
||||
|
||||
- **The owner's agent has no rule on seniority.** Fit compares subject, not level. The ladder and the rail's From/To/Track filter exist (`LADDER` in `app/models.py`), and the level is filled for about half of open roles. A gate still names words (`principal`, `senior_manager`, `director`, `exec`); a gate on "at least a rung" would read as the rail's filter does.
|
||||
- **Votes train `taste` only.** The liked-minus-passed centroid weighs 0.2 on both agents. Feeding votes into the prompt vectors is worth doing once there are a few hundred.
|
||||
- **A starred company's roles crowd the top of Best first.** With the companies weight at 0.2 a favourite's roles gain about 0.13; after starring Databricks it held 33 of the top 60 rows. Lower the weight on the agent's card, or star fewer companies.
|
||||
- **The owner's agent passes any role in Sweden and Germany.** Its `any` group holds `country` DK, SE, DE, which is why SAP in Garching ranks. For Denmark, remote and Skåne only, the group is `country` DK, `remote` remote and `place` `region:skane`.
|
||||
|
||||
## Sources
|
||||
|
||||
- **The boilerplate moved on.** `deploy/` is at boilerplate `1679e88` and lacks `wsl-setup.sh` and `wsl-setup.ps1`; `pipeline/` and `auth/` differ from boilerplate HEAD (three languages, installable phone). Refresh them with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
|
||||
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
|
||||
- **Six hosts' dates are not trusted.** Eightfold, Jibe, Gem, Deel, HiBob and HR Manager declare `posted="listed"`, so their backlog counts as posted when first seen. Eightfold now maps `creationTs`, Jibe's `posted_date` equals the page's `datePosted`, and HR Manager's `pubDate` rises with the id; each needs one check on a live board before it declares `posted="trusted"`. Platsbanken's local time is read as UTC.
|
||||
- **A mistyped SmartRecruiters or Gem slug attaches as an empty board.** Both answer an empty listing for an unknown id. `empty="suspect"` and the 10% guard keep a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
|
||||
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the runner waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
|
||||
- **A SuccessFactors site on a company's own domain is not discovered.** Its pages link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
|
||||
- **HR Manager: decisions and data disagree.** `decisions.md` and `plugins/README.md` say no attached company uses it, but the live database has `hrmanager:teknologisk` attached with 0 open roles. Seaborg stays unattached until the owner decides whether the feed counts as allowed.
|
||||
- **Stripe's places carry a "Locations" suffix.** 282 open roles have place text like "Singapore Locations" or "Dublin; Ireland Locations".
|
||||
|
||||
## The pages
|
||||
|
||||
- **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite.
|
||||
- **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it.
|
||||
- **`company_key` keeps `ø`**, so "Orsted" and "Ørsted" are two companies. Folding it changes the primary key of `companies`.
|
||||
- **`company_key` keeps `ø`.** Aliases make "Orsted" and "Ørsted" one company; folding the key itself would change the primary key of `companies`.
|
||||
+19
-15
@@ -1,51 +1,55 @@
|
||||
# Plan: the readers left after the hosts of 2026-09-29
|
||||
|
||||
Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped.
|
||||
Planned 2026-09-29. None of the packages below is built. Each builds against the reader contract in `plugins/readers/__init__.py`. What is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages land or are dropped.
|
||||
|
||||
WP9 (the contract part) and WP11 were superseded by the readers rework (see `decisions.md`, Ingestion); WP10 and WP12 to WP15 build against the reader contract in `plugins/readers/__init__.py`.
|
||||
The board filter this plan had as WP11 landed as the agents' reach (`decisions.md`, Ingestion), which also brought Infineon and Microsoft under Eightfold's page cap.
|
||||
|
||||
## Where it stands
|
||||
|
||||
286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
|
||||
76 companies are on the Watchlist, each with the careers page opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
|
||||
|
||||
| Route | Companies | Roles seen |
|
||||
|---|---|---|
|
||||
| Sitemap and structured data | Arm, Mambu, Odoo, Aiven, SOPHiA GENETICS, AUTO1, Apple, ASML | about 2,000 |
|
||||
| Own JSON API | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | about 570 |
|
||||
| Own HTML list | Zalando, Netcompany, Bolt, CHECK24, Google with DeepMind, Lucca, Shopify, Framer, Bird, Ableton | about 950 |
|
||||
| A host that is read, with a gap | Infineon, Microsoft (Eightfold's page cap), Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
|
||||
| A host that is read, with a gap | Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
|
||||
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
|
||||
|
||||
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md` and the smaller hosts in `otherhosts.md`. The raw responses were not kept; a builder fetches its own fixtures.
|
||||
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures.
|
||||
|
||||
## Decisions to take first
|
||||
|
||||
**A listing that knows no title.** A sitemap lists URLs. `detail` may not set a title, since the listing rewrites it on every pull, so a sitemap reader would show a title made from the URL. Two ways out: the module reads the role pages inside `page`, which costs one request per role on every pull, or the contract gains a row key that marks the title as provisional, and `upsert_role` keeps a stored title over a provisional one. The second is the smaller cost per pull and touches `app/storage/store.py`.
|
||||
**A listing that knows no title.** A sitemap lists URLs, and the listing rewrites the title on every pull, so a sitemap reader would show a title made from the URL. The contract already has `Reads(title="provisional")` (`plugins/readers/_contract.py`), but nothing honours it: `upsert_role` still overwrites a title that `detail` filled. WP9 adds that rule.
|
||||
|
||||
**A filter in a target.** Eightfold takes a location, Workday and SuccessFactors a country facet. A target is `host:slug` and has nowhere to state one. `subscriptions.params` exists for searches; a board could use it, and `page` would receive it. This brings Infineon and Microsoft under the page cap and cuts a pull of Bosch (4,812 roles), Philips or SAP to the countries the agents accept.
|
||||
|
||||
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. Fourteen hosts landed as fourteen files without a shared edit, so the package takes a module per company as it is; `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
|
||||
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
|
||||
|
||||
## Work packages
|
||||
|
||||
Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The common rules of wave 1 apply: the live request limits, robots.txt read first, no key from a script, a final report with the targets to attach.
|
||||
Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The live request limits apply, robots.txt is read first, no key is taken from a script, and each package ends with a report of the targets to attach.
|
||||
|
||||
| WP | Model | Builds | Needs |
|
||||
|---|---|---|---|
|
||||
| 9 Provisional title | Opus | the row key, `upsert_role`'s rule and its tests; owns `plugins/ats.py`, `plugins/readers/__init__.py`, `app/storage/store.py`, `tests/test_readers.py`, `tests/test_storage.py` | the first decision |
|
||||
| 9 Provisional title | Opus | `upsert_role` keeps a stored title over a provisional one, and its tests; owns `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | |
|
||||
| 10 Sitemap reader | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
|
||||
| 11 Board filters | Opus | `params` on a board subscription, handed to `page`; owns `plugins/ats.py`, `app/jobs/pull.py`, `app/jobs/sources.py`, `app/web/api.py`, `app/static/companies.js` and their tests | the second decision |
|
||||
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
|
||||
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
|
||||
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
|
||||
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
|
||||
|
||||
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9, and WP9 and WP11 both own `plugins/ats.py`, so they run one after the other.
|
||||
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9.
|
||||
|
||||
### WP10 notes: Mambu and Arm
|
||||
|
||||
Both were read live on 2026-09-29 without a browser or a block.
|
||||
|
||||
- **Mambu, classic iCIMS.** `https://careers-mambu.icims.com/sitemap.xml` has one `<loc>` per job (`/jobs/{id}/{slug}/job`) with `lastmod`, plus `/jobs/intro`; robots.txt allows it and lists the sitemap. The detail `.../jobs/{id}/x/job?in_iframe=1` carries a JSON-LD `JobPosting` with `datePosted`, `description`, `jobLocation[].address.addressCountry`, `validThrough` and `employmentType`. Tenants use other subdomain prefixes, so the target stores the whole host. 8 roles.
|
||||
- **Arm, TalentBrew (Radancy).** The `*-arm.icims.com` hosts only redirect to `careers.arm.com`. `https://careers.arm.com/sitemap.xml` lists all 399 `/job/...` URLs in one request, without titles. The detail's JSON-LD has `datePosted` (unpadded, "2026-6-25"), `description`, `identifier` and `jobLocation[].address`. The paged list at `/search-jobs/results` is HTML inside JSON and only answers the full parameter set; its `data-total-results` counts content pages, `data-total-job-results` counts roles.
|
||||
|
||||
## Planner, after the merges
|
||||
|
||||
- Attach each board with `app companies set-board`, which reads it first and clears the reason.
|
||||
- Bring `seed/companies.json` in line with the live database: the board of every company that has one shown, the reason of every company that has none.
|
||||
- Add a host to `TRUSTED_POSTED_HOSTS` where its date is a posting date, with the evidence in the comment.
|
||||
- Bring `seed/companies.json` in line with the live database: the board of every company that has one, the reason of every company that has none.
|
||||
- Declare `Fields(posted="trusted")` on a reader whose date is a posting date, with the evidence in the docstring.
|
||||
- Restart the server by its PID after a backup in `../jobboard-backups/`.
|
||||
- Find the career host of the nine Teamtailor companies by opening each careers page.
|
||||
@@ -1,320 +0,0 @@
|
||||
# Career sites on smaller ATS hosts: can they be read without a browser?
|
||||
|
||||
Probed 2026-09-29 with `curl -m 20`, a Chrome User-Agent, at most one request per second per host, no login. "Verified" means I made the request and parsed the response. "Inferred" means I did not observe it. No host blocked me; no captcha or challenge was met anywhere.
|
||||
|
||||
Fixtures (one raw listing per host that worked, plus a detail sample where useful) are in the same directory as this file:
|
||||
|
||||
| Host | Listing fixture | Extra fixtures |
|
||||
|---|---|---|
|
||||
| oracle | out-otherhosts-oracle.json | out-otherhosts-oracle-detail.json |
|
||||
| bamboohr | out-otherhosts-bamboohr.json | out-otherhosts-bamboohr-detail.json |
|
||||
| pinpoint | out-otherhosts-pinpoint.json | out-otherhosts-pinpoint-detail.html |
|
||||
| icims | out-otherhosts-icims.html | out-otherhosts-icims-detail.html, out-otherhosts-icims-sitemap.xml |
|
||||
| jibe (Booking.com) | out-otherhosts-jibe.json | |
|
||||
| talentbrew (Arm) | out-otherhosts-talentbrew.json | |
|
||||
| gem | out-otherhosts-gem.json | out-otherhosts-gem-detail.json |
|
||||
| eightfold | out-otherhosts-eightfold.json | out-otherhosts-eightfold-detail.json |
|
||||
| phenom | out-otherhosts-phenom.json | out-otherhosts-phenom-detail.html |
|
||||
| wttj | out-otherhosts-wttj.json | |
|
||||
|
||||
## Corrections to the app's data
|
||||
|
||||
- Booking.com: not classic iCIMS. `careers.booking.com` links to `jobs.booking.com/booking/jobs`, which is iCIMS Jibe (a separate product with a JSON API). iCIMS itself sits behind it only for applying.
|
||||
- Arm: not a classic iCIMS list. `careers.arm.com` is TalentBrew (Radancy). The `*-arm.icims.com` hosts only serve login and redirect to `careers.arm.com/search-jobs`.
|
||||
- Philips: the stored `careers_url` (`/na/en/`) is the North America site with 306 roles. The global site `/global/en` has 865.
|
||||
- Yousign: renamed Youtrust. `yousign.com` now serves `youtrust.com`. On Welcome to the Jungle the organization slug is `youtrust` (12 roles); the stored slug `yousign` yields 0 roles.
|
||||
- Native Instruments: the ATS is Pinpoint but the board is the shared one of the parent, `inmusicbrands.pinpointhq.com` (21 roles across inMusic brands, only 1 for NI Berlin).
|
||||
- Just Eat Takeaway: Phenom shows the list, but applying goes to Workday (`applyUrl` on `wd3.myworkdaysite.com`). The list is still readable on Phenom.
|
||||
- Xeneta: the board is small (2 roles, both in Bucharest, none in Oslo).
|
||||
- Infineon: the suggested Eightfold endpoint `/api/apply/v2/jobs` returns 403 `Not authorized for PCSX`. The working one is `/api/pcsx/search`.
|
||||
|
||||
## Per-company table
|
||||
|
||||
| Name | Host confirmed | Identifier | Human-facing jobs URL | Listing endpoint | Roles seen | Verified | Note |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Milestone Systems | Oracle Recruiting Cloud (yes) | host `fa-ewto-saasfaprod1.fa.ocs.oraclecloud.com`, siteNumber `CX_1001` | https://fa-ewto-saasfaprod1.fa.ocs.oraclecloud.com/hcmUI/CandidateExperience/en/sites/CX_1001/jobs | `.../hcmRestApi/resources/latest/recruitingCEJobRequisitions?onlyData=true&expand=requisitionList&finder=findReqs;siteNumber=CX_1001,limit=200,offset=0` | 20 (TotalJobsCount 20) | yes | region segment is `ocs`, not `us2` etc.; the host is linked from milestonesys.com/careers |
|
||||
| Configit | BambooHR (yes) | subdomain `configit` | https://configit.bamboohr.com/careers | `https://configit.bamboohr.com/careers/list` | 2 (meta.totalCount 2) | yes | one is an unsolicited-application placeholder; embedded on configit.com via `embed.js` |
|
||||
| Xeneta | Pinpoint (yes) | subdomain `xeneta` | https://xeneta.pinpointhq.com/ | `https://xeneta.pinpointhq.com/postings.json` | 2 | yes | both roles Bucharest; list has no posted date |
|
||||
| Native Instruments | Pinpoint (yes, parent board) | subdomain `inmusicbrands`, filter `job.division.name == "inMusic NI GmbH"` | https://inmusicbrands.pinpointhq.com/ (linked from native-instruments.com/pages/careers) | `https://inmusicbrands.pinpointhq.com/postings.json` | 21 total, 1 for NI (Berlin, DE) | yes | shared board: must filter by division; department is null |
|
||||
| Mambu | iCIMS (yes, classic) | subdomain `careers-mambu` | https://careers-mambu.icims.com/jobs/search?ss=1&in_iframe=1 (linked from mambu.com/en/careers) | `https://careers-mambu.icims.com/sitemap.xml` (IDs, lastmod) and `.../jobs/search?ss=1&in_iframe=1&pr=0` (cards) | 8 | yes | sitemap lists 8 jobs plus an intro URL; `pr=1` returns no cards |
|
||||
| Booking.com | iCIMS Jibe (host differs from app's guess) | Jibe path `booking`, filter `brand=Booking.com` | https://jobs.booking.com/booking/jobs | `https://jobs.booking.com/api/jobs?page=1&limit=100&brand=Booking.com` | 98 for Booking.com, 146 with the Booking Holdings brand | yes | limit 200 returns 422, 100 is the cap; `brand=` filters, `brands=` does not |
|
||||
| Arm | TalentBrew (Radancy), not iCIMS list | site `careers.arm.com`, org id 33099 | https://careers.arm.com/search-jobs | `https://careers.arm.com/search-jobs/results?...RecordsPerPage=100&CurrentPage={n}...` (full query in section below) and `https://careers.arm.com/sitemap.xml` | 399 (page 4 has 99) | yes | JSON envelope carrying HTML; list has no date |
|
||||
| sennder | Gem (yes) | board `senndertechnologies-gmbh` | https://jobs.gem.com/senndertechnologies-gmbh (sennder.com/de/open-positions embeds it) | `POST https://jobs.gem.com/api/public/graphql` (JobBoardList) | 33 | yes | list has no date or description; detail query gives both |
|
||||
| Productboard | Gem (yes) | board `productboard` | https://jobs.gem.com/productboard | same POST | 11 | yes | 12 CZ + 4 US locations across 11 roles |
|
||||
| Infineon | Eightfold PCSX (yes, other endpoint) | host `jobs.infineon.com`, domain `infineon.com` | https://jobs.infineon.com/careers | `https://jobs.infineon.com/api/pcsx/search?domain=infineon.com&query=&location=&sort_by=relevance&start=0` | 1292 total, 10 per page | yes | `num` is ignored; 130 requests for a full poll; `/api/apply/v2/jobs` is 403 |
|
||||
| Just Eat Takeaway.com | Phenom (yes) | refNum `TAKEGLOBAL`, path `/global/en`, pageId `page11` | https://careers.justeattakeaway.com/global/en/search-results | `POST https://careers.justeattakeaway.com/widgets` (refineSearch) | 103 (totalHits 103; sitemap also 103 job URLs) | yes | `size` 200 returned all 103 in one call; apply is on Workday |
|
||||
| Philips | Phenom (yes) | refNum `PHILUS`, path `/global/en`, pageId `page6` | https://www.careers.philips.com/global/en/search-results | `POST https://www.careers.philips.com/widgets` (refineSearch) | 865 global (306 on `/na/en`) | yes | `size` 500 worked (500 of 865 in 1.7 MB); app's URL is the NA site only |
|
||||
| Yousign (now Youtrust) | Welcome to the Jungle (yes) | org slug `youtrust` | https://www.welcometothejungle.com/en/companies/youtrust/jobs | Algolia `POST https://CSEKHVMS53-dsn.algolia.net/1/indexes/wttj_jobs_production_en/query` | 12 | yes | app slug `yousign` returns 0; the same 12 appear in the `_fr` index |
|
||||
|
||||
## What one role gives, per host
|
||||
|
||||
| Host | Title | URL | Location with country | Remote indication | Date posted | Description | Pay |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Oracle | list `Title` | build: `{base}/hcmUI/CandidateExperience/en/sites/{site}/job/{Id}` (inferred, standard pattern; not fetched) | list `PrimaryLocation` ("Copenhagen, Denmark") and `PrimaryLocationCountry` ("DK"); `secondaryLocations` array | list `WorkplaceType` ("Hybrid", "On-site", or empty) | list `PostedDate` (date); detail `ExternalPostedStartDate` (timestamp) | detail only: `ExternalDescriptionStr` (plus responsibilities/qualifications strings, often empty) | none seen |
|
||||
| BambooHR | list `jobOpeningName` | `https://{sub}.bamboohr.com/careers/{id}` (detail returns it as `jobOpeningShareUrl`) | list `location.city`; country only in detail `location.addressCountry` ("Denmark") | list `isRemote` (null in sample) and `locationType` ("2" for a hybrid-looking role; mapping 0/1/2 is inferred) | detail only: `datePosted` | detail only: `description` HTML | detail `compensation` (null in sample) |
|
||||
| Pinpoint | list `title` | list `url` | list `location.name`/`city`; `province` holds country or region inconsistently; ISO country only in the detail JSON-LD `addressCountry` | list `workplace_type` (onsite/hybrid/remote) | detail JSON-LD `datePosted` only | list: `description`, `key_responsibilities`, `skills_knowledge_expertise`, `benefits` as HTML | list `compensation_*` fields, shown when `compensation_visible` is true (false in both samples) |
|
||||
| iCIMS classic | card `h3` and anchor title | card anchor `href` (`/jobs/{id}/{slug}/job`) | card "NL-Amsterdam" (country code prefix); detail JSON-LD `jobLocation[].address.addressCountry` | none observed | detail JSON-LD `datePosted`; sitemap has `lastmod` | card gives a truncated snippet; full text in the detail JSON-LD `description` | none seen |
|
||||
| Jibe (Booking.com) | `title` | `https://jobs.booking.com/booking/jobs/{slug}` (page returned 200); `apply_url` is the iCIMS apply link | `city`, `country`, `country_code`, `full_location`, lat/long | none obvious; `location_type` was "LAT_LNG" (a geo type, not remote) | `posted_date` | in the list: `description` HTML (about 1.7 MB per 100 roles) | `salary_value`, `salary_min_value`, `salary_max_value` (all "0" in the sample) |
|
||||
| TalentBrew (Arm) | card anchor text | card `href` (`/job/{city}/{slug}/33099/{jobId}`) | card `location` ("Cambridge, United Kingdom"); detail JSON-LD `addressCountry` | none | detail JSON-LD `datePosted` ("2026-6-25", unpadded); list has none | card teaser only; full in detail JSON-LD `description` | JSON-LD `baseSalary` present but empty |
|
||||
| Gem | `title` | build: `https://jobs.gem.com/{board}/{extId}` (detail page path in the bundle; `extId` may be numeric or base64-like) | `locations[]` with `name`, `city`, `isoCountry` (ISO alpha-3: ESP, DEU) | `locations[].isRemote`; `job.locationType` (IN_OFFICE, HYBRID; REMOTE inferred) | detail `firstPublishedTsSec` (epoch seconds); list has none | detail only: `descriptionHtml` | detail `compensationHtml` (null in sample) |
|
||||
| Eightfold | `name` | `https://jobs.infineon.com` + `positionUrl` (`/careers/job/{id}`); detail gives `publicUrl` | `locations`, `standardizedLocations` ("Shanghai, Shanghai, CN" ends with ISO alpha-2) | `workLocationOption` (onsite/hybrid; remote inferred), and facet "Remote" | list `postedTs` (epoch seconds); `creationTs` too | detail only: `jobDescription` HTML | none seen |
|
||||
| Phenom | `title` | `https://{host}/global/en/job/{jobSeqNo}/{Title-Slug}` (sitemap has these); list `applyUrl` points at the apply system | `country`, `city`, `cityStateCountry`, `multi_location` | none in the list fields I saw | list `postedDate`; detail JSON-LD `datePosted` | list `descriptionTeaser` only; full text in the detail JSON-LD `description` (HTML-escaped) | none seen |
|
||||
| WTTJ | `name` | `https://www.welcometothejungle.com/en/companies/{orgSlug}/jobs/{slug}` (fetched, 200) | `offices[]` with `city`, `country`, `country_code` | `remote` ("punctual", "partial", "fulltime", "unknown") and `has_remote` | `published_at` | list: `summary`, `key_missions`, `profile`; full HTML in the job page JSON-LD | `salary_minimum`, `salary_maximum`, `salary_currency`, `salary_period` in the list |
|
||||
|
||||
## Host sections
|
||||
|
||||
### Oracle Recruiting Cloud
|
||||
|
||||
Verified for Milestone Systems only.
|
||||
|
||||
One method for all companies on the host: yes, the REST resource is the same on every tenant. The identifiers differ (host, siteNumber). Both come from the human URL `https://{host}/hcmUI/CandidateExperience/{lang}/sites/{siteNumber}/jobs`. The site number is not always `CX_1001` (on other tenants it can be `CX`, `CX_1`, ...; inferred).
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `GET https://{host}/hcmRestApi/resources/latest/recruitingCEJobRequisitions?onlyData=true&expand=requisitionList&finder=findReqs;siteNumber={site},limit={n},offset={o}`
|
||||
2. `items[0].TotalJobsCount` is the total. `items[0].requisitionList[]` holds the roles. The outer `hasMore` / `count` describe the single search wrapper (count 1), so ignore them.
|
||||
3. Without `expand=requisitionList` (or `requisitionList.secondaryLocations`) the roles are missing and the response is only facets. This is the easy mistake.
|
||||
4. Page with `offset += limit` until `offset >= TotalJobsCount`. `limit=200` returned all 20 in one call; `limit=5&offset=15` returned the last 5 (verified).
|
||||
5. Detail: `GET .../recruitingCEJobRequisitionDetails?expand=all&onlyData=true&finder=ById;Id="{Id}",siteNumber={site}` (quote the Id as `%22..%22`). Returns `ExternalDescriptionStr`, `ExternalPostedStartDate`, `workLocation`, `secondaryLocations`.
|
||||
|
||||
Identifier to store: `host` + `siteNumber`; per role `Id`.
|
||||
|
||||
Requests per poll: `ceil(total / 200)` (1 here), plus one detail call per new role.
|
||||
|
||||
Bot protection: none met. Anonymous, no cookies, no headers beyond User-Agent.
|
||||
|
||||
Worth a parser: yes if any target uses it, because it is the cleanest API in this set. Oracle Fusion HCM is common among large enterprises and mid-caps; among European tech and Nordic companies it is uncommon (inferred from general knowledge), though Milestone shows it exists. One company in this list.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://fa-ewto-saasfaprod1.fa.ocs.oraclecloud.com/hcmRestApi/resources/latest/recruitingCEJobRequisitions?onlyData=true&expand=requisitionList&finder=findReqs;siteNumber=CX_1001,limit=200,offset=0'
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://fa-ewto-saasfaprod1.fa.ocs.oraclecloud.com/hcmRestApi/resources/latest/recruitingCEJobRequisitionDetails?expand=all&onlyData=true&finder=ById;Id=%2224345%22,siteNumber=CX_1001'
|
||||
```
|
||||
|
||||
### BambooHR
|
||||
|
||||
Verified for Configit only.
|
||||
|
||||
One method: yes (the endpoints are per-subdomain and identical). Human page `https://{sub}.bamboohr.com/careers` is a JS app calling the JSON endpoints; company sites embed `https://{sub}.bamboohr.com/js/embed.js`.
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `GET https://{sub}.bamboohr.com/careers/list` returns `{"meta":{"totalCount":N},"result":[...]}` with all roles, no paging (2 of 2 here; whether larger boards page is not tested).
|
||||
2. `GET https://{sub}.bamboohr.com/careers/{id}/detail` returns JSON `result.jobOpening` with `description`, `datePosted`, `location.addressCountry`, `compensation`, `jobOpeningShareUrl`. Send `Accept: application/json` (I did; the `.../careers/{id}` URL without it returns HTML).
|
||||
|
||||
Identifier: subdomain; per role `id`.
|
||||
|
||||
Requests per poll: 1 for the list, plus one detail call per new role (needed for date, country, description).
|
||||
|
||||
Bot protection: none met.
|
||||
|
||||
Worth a parser: probably yes. Two trivial endpoints, no auth. BambooHR is widespread among small and mid-size companies, mostly US, with a fair number of European and Nordic scale-ups (inferred); one company in this list.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://configit.bamboohr.com/careers/list
|
||||
curl -m 20 -A "Mozilla/5.0 ..." -H 'Accept: application/json' https://configit.bamboohr.com/careers/37/detail
|
||||
```
|
||||
|
||||
### Pinpoint
|
||||
|
||||
Verified for Xeneta and inMusic Brands (Native Instruments).
|
||||
|
||||
One method: yes, but a board may be shared by several brands (Native Instruments), so the mapping from company to board is many-to-one and a filter is needed (`job.division.name`, here "inMusic NI GmbH"). The company's own site only links the board.
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `GET https://{sub}.pinpointhq.com/postings.json` returns `{"data":[...]}` with every open posting including full HTML text. No paging keys, no total (2 and 21 roles; a large board might page, not tested).
|
||||
2. Per posting: `id`, `title`, `url`, `location{name,city,province}`, `workplace_type`, `employment_type`, `compensation_*`, `description`, `key_responsibilities`, `skills_knowledge_expertise`, `benefits`, `job{requisition_id, department, division}`. `deadline_at` is present.
|
||||
3. Date posted is missing from the list. The detail page `{url}` embeds JSON-LD JobPosting with `datePosted`, `identifier.value` (the UUID) and `addressCountry`.
|
||||
4. `HEAD` on `/postings.json` returns 404 (only GET works).
|
||||
|
||||
Identifier: subdomain (plus division filter if shared); per role `id` or the UUID in `url`.
|
||||
|
||||
Requests per poll: 1. Plus one detail per new role if a date or ISO country is wanted.
|
||||
|
||||
Bot protection: none met.
|
||||
|
||||
Worth a parser: yes, cheapest of all. Pinpoint is popular among UK and Nordic scale-ups and mid-size firms (inferred); two companies in this list.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://xeneta.pinpointhq.com/postings.json
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://inmusicbrands.pinpointhq.com/postings.json
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://xeneta.pinpointhq.com/en/postings/a85d5d45-b334-4f5c-aaea-d0b399b08a2e
|
||||
```
|
||||
|
||||
### iCIMS (classic portal), Jibe, and TalentBrew
|
||||
|
||||
"iCIMS" in the app covers three different front ends, so there is no single method.
|
||||
|
||||
**Classic portal** (Mambu, verified):
|
||||
|
||||
1. `GET https://careers-{tenant}.icims.com/sitemap.xml`: one `<loc>` per job (`/jobs/{id}/{slug}/job`) with `lastmod`, plus `/jobs/intro`. Matched the 8 cards on the list page (verified). robots.txt allows it and lists the sitemap.
|
||||
2. Alternative: `GET https://careers-{tenant}.icims.com/jobs/search?ss=1&in_iframe=1&pr={page}` with `pr` starting at 0. Cards are `li.iCIMS_JobCardItem` with location ("NL-Amsterdam"), title anchor and a snippet; the header says "Page 1 of N". `pr=1` on Mambu returned no cards. Page size and behaviour on multi-page tenants not verified (Mambu has one page).
|
||||
3. Detail: `GET .../jobs/{id}/x/job?in_iframe=1` (slug not needed, verified) carries a JSON-LD JobPosting with `datePosted`, `description`, `jobLocation[].address.addressCountry`, `validThrough`, `employmentType`.
|
||||
|
||||
Identifier: tenant subdomain (`careers-{tenant}`); some tenants use other prefixes (`experienced-arm`, `careers-holdings-workingatbooking`; inferred that the app must store the whole host). Per role: numeric job id.
|
||||
|
||||
Requests per poll: 1 sitemap (or `ceil(total/pageSize)` list pages) plus one detail per new role.
|
||||
|
||||
Bot protection: none met. Some tenants return "gone" (`careers-workingatbooking.icims.com` gave 404) or redirect to the customer's own site (Arm: JS redirect to `careers.arm.com/search-jobs`).
|
||||
|
||||
**Jibe** (Booking.com, verified): JSON API on the company's own `jobs.` host.
|
||||
|
||||
1. `GET https://jobs.booking.com/api/jobs?page={p}&limit=100[&brand=Booking.com]`, `page` starting at 1. Response has `totalCount` (146) and `jobs[].data` with all fields including full description. `limit=200` is a 422, so 100 is the cap. Page 2 returned the remaining 46. Brand filter is `brand=` (98 roles for Booking.com, 48 for Booking Holdings); `brands=` is ignored.
|
||||
2. Response is large (1.7 MB per 100 roles) because it repeats filter facets and HTML descriptions.
|
||||
3. Identifier: host and the path prefix (`/booking/`), `slug` (equals `req_id`). Requests per poll: `ceil(total/100)` (2). Bot protection: none met.
|
||||
4. Other Jibe sites use the same `/api/jobs` path (inferred; only Booking tested).
|
||||
|
||||
**TalentBrew / Radancy** (Arm, verified): the list is HTML inside JSON.
|
||||
|
||||
1. `GET https://careers.arm.com/search-jobs/results?ActiveFacetID=0&CurrentPage={n}&RecordsPerPage=100&TotalContentResults=&Distance=50&RadiusUnitType=0&Keywords=&Location=&ShowRadius=False&IsPagination=False&CustomFacetName=&FacetTerm=&FacetType=0&SearchResultsModuleName=Search+Results&SearchFiltersModuleName=Search+Filters&SortCriteria=0&SortDirection=0&SearchType=5&PostalCode=&ResultsType=0&fc=&fl=&fcf=&afc=&afl=&afcf=` returns `{"filters":"<html>","results":"<html>","hasJobs":true,...}`. A shortened query returned 60 bytes and no jobs, so send the full parameter set.
|
||||
2. In `results`, `<section id="search-results" data-total-job-results="399" data-total-pages="4" data-current-page="1">`. Job cards are `li.job-card` with `a.job-card__title[href][data-job-id]`, `span.location`, `span.category`, optional `span.job-card__intro`. Cards with class `job-card--content` are non-job content pages; ignore them.
|
||||
3. Pages 1 to 3 hold 100 jobs each and page 4 holds 99 (verified). The `data-total-results="516"` includes 117 content items, so use `data-total-job-results`.
|
||||
4. `https://careers.arm.com/sitemap.xml` (169 KB) lists all 399 `/job/...` URLs in one request without titles (verified).
|
||||
5. Detail: JSON-LD `datePosted` ("2026-6-25", unpadded), `description`, `identifier` (requisition number "2025-16272"), `jobLocation[].address`.
|
||||
6. Identifier: site host and org id `33099` (the number in every job URL); per role the trailing numeric id.
|
||||
|
||||
Worth a parser:
|
||||
- Classic iCIMS: sitemap plus detail JSON-LD is easy and generic; iCIMS is very common among large enterprises and mid-caps (inferred). Only Mambu is on classic in this list.
|
||||
- Jibe: a clean API, but it is a different product (Booking only here). Cheap to build (about 30 lines) if Booking.com matters.
|
||||
- TalentBrew: HTML scraping of a Radancy front end; used by big employers (Arm). Sitemap plus detail JSON-LD is the generic route and shares code with classic iCIMS.
|
||||
- Recommendation: one "sitemap plus JSON-LD detail" reader covers classic iCIMS, TalentBrew and (partly) Phenom. That is a better investment than three host parsers.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://careers-mambu.icims.com/sitemap.xml
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://careers-mambu.icims.com/jobs/search?ss=1&in_iframe=1&pr=0'
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://careers-mambu.icims.com/jobs/3461/x/job?in_iframe=1'
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://jobs.booking.com/api/jobs?page=1&limit=100&brand=Booking.com'
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://careers.arm.com/sitemap.xml
|
||||
```
|
||||
|
||||
### Gem
|
||||
|
||||
Verified for sennder and Productboard.
|
||||
|
||||
One method: yes, the board is identified by a vanity path and one GraphQL endpoint serves all boards. The company's own site either embeds it (sennder: `gemJobBoardUrl` in the page) or links to `jobs.gem.com/{board}` (Productboard). The HTML at `jobs.gem.com/{board}` is an empty React shell with no embedded data, so the GraphQL call is the only way without a browser.
|
||||
|
||||
Endpoint (found by reading `static.gem.com/scripts/jobBoards.*.js` and `ExternalJobBoardList.*.js`): `POST https://jobs.gem.com/api/public/graphql` with `Content-Type: application/json`. The Apollo batch variant `/api/public/graphql/batch` also exists in the bundle (not used). No auth, no CSRF token needed (I sent an Origin and Referer of jobs.gem.com; whether they are required was not tested).
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. List: query `JobBoardList` with `variables {"boardId": "{board}"}` selecting `oatsExternalJobPostings(boardId){ jobPostings { id extId title locations{ id name city isoCountry isRemote extId } job{ id department{ id name extId } locationType employmentType } } }`. Returns all roles in one response (33 and 11), no paging, no total field (count the array).
|
||||
2. Detail: query `ExternalJobPosting($boardId, $extId)` selecting `oatsExternalJobPosting(boardId, extId){ id title descriptionHtml extId firstPublishedTsSec locations{...} job{ locationType employmentType requisitionId teamDisplayName department{name} } compensationHtml }`. The role URL is `https://jobs.gem.com/{board}/{extId}`.
|
||||
3. `extId` is either numeric (Greenhouse-synced, "8084509002") or an opaque string ("am9icG9zdDoG..."); store it as a string.
|
||||
4. Query text is kept in `gem-q.py`-style form below; the GraphQL schema is fixed by the bundle, so the queries may break when Gem ships a new bundle (risk).
|
||||
|
||||
Identifier: board vanity path (`senndertechnologies-gmbh`, `productboard`), the segment after `jobs.gem.com/`. Per role `extId`.
|
||||
|
||||
Requests per poll: 1, plus one detail per new role for the date and description.
|
||||
|
||||
Bot protection: none met (plain POST from curl worked). Rate limits not hit.
|
||||
|
||||
Worth a parser: yes. One POST for the whole board. Gem is used by US-founded and growth-stage tech companies and by some European scale-ups (inferred); two companies in this list.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." -X POST -H 'Content-Type: application/json' -H 'Origin: https://jobs.gem.com' --data '{"operationName":"JobBoardList","variables":{"boardId":"senndertechnologies-gmbh"},"query":"query JobBoardList($boardId: String!) { oatsExternalJobPostings(boardId: $boardId) { jobPostings { id extId title locations { id name city isoCountry isRemote extId } job { id department { id name extId } locationType employmentType } } } }"}' https://jobs.gem.com/api/public/graphql
|
||||
curl -m 20 -A "Mozilla/5.0 ..." -X POST -H 'Content-Type: application/json' -H 'Origin: https://jobs.gem.com' --data '{"operationName":"ExternalJobPosting","variables":{"boardId":"senndertechnologies-gmbh","extId":"8084509002"},"query":"query ExternalJobPosting($boardId: String!, $extId: String!) { oatsExternalJobPosting(boardId: $boardId, extId: $extId) { id title descriptionHtml extId firstPublishedTsSec locations { id extId name city isoCountry isRemote } job { id locationType employmentType requisitionId teamDisplayName department { id extId name } } compensationHtml } }"}' https://jobs.gem.com/api/public/graphql
|
||||
```
|
||||
|
||||
### Eightfold
|
||||
|
||||
Verified for Infineon only; the suggested `/api/apply/v2/jobs` endpoint does not work there.
|
||||
|
||||
One method: the PCSX search API is the same on any Eightfold career site (inferred; only Infineon tested). Infineon fronts Eightfold with its own domain `jobs.infineon.com`, so the identifier is the company host plus the `domain` parameter (`infineon.com`, visible in the HTML as `domain=infineon.com`). Many customers use `{name}.eightfold.ai` instead (inferred).
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `GET https://{host}/api/pcsx/search?domain={domain}&query=&location=&sort_by=relevance&start={n}` returns `{"status":200,"data":{"positions":[...],"count":1292,"filterDef":{...}}}`. Ten positions per page; `num=100` is ignored (verified: 10 returned). `start=10` gives the next ten. `start=1285` gave the last 7 (verified).
|
||||
2. `sort_by=timestamp` orders roughly by newest (first page timestamps descending, but the last page was not sorted), so early-stop is not reliable.
|
||||
3. Position fields: `id`, `displayJobId`, `name`, `locations`, `standardizedLocations`, `postedTs`, `department`, `workLocationOption`, `positionUrl`.
|
||||
4. Detail: `GET https://{host}/api/pcsx/position_details?position_id={id}&domain={domain}&hl=en` returns `data.jobDescription` (HTML), `publicUrl`, `efcustom*` fields.
|
||||
5. robots.txt of jobs.infineon.com allows `/api/pcsx` and `/careers`, and disallows everything else. No sitemap (404).
|
||||
6. `/api/apply/v2/jobs?domain=infineon.com&start=0&num=10` returns 403 `{"message":"Not authorized for PCSX"}`. Some other tenants may still allow it (inferred; not tested).
|
||||
|
||||
Identifier: host and `domain`; per role `id` (numeric, store as a string) or `displayJobId`.
|
||||
|
||||
Requests per poll: `ceil(total/10)`, so 130 for Infineon (about two minutes at one per second). Twice a day is fine, but this is the heaviest poll in the set. The `count` field lets you compare totals to skip work only when nothing changed (a changed total is not required for new roles, so this is only a hint).
|
||||
|
||||
Bot protection: none met. Unauthenticated JSON is served.
|
||||
|
||||
Worth a parser: moderate. Eightfold is used by large enterprises (Infineon and others); it is less common in European tech scale-ups and Nordic mid-caps (inferred). One company in this list, and paging cost is high, so build it only if you want large-enterprise coverage.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://jobs.infineon.com/api/pcsx/search?domain=infineon.com&query=&location=&sort_by=relevance&start=0'
|
||||
curl -m 20 -A "Mozilla/5.0 ..." 'https://jobs.infineon.com/api/pcsx/position_details?position_id=563808971955206&domain=infineon.com&hl=en&queried_location=&triggered_by='
|
||||
```
|
||||
|
||||
### Phenom
|
||||
|
||||
Verified for Just Eat Takeaway and Philips.
|
||||
|
||||
One method: yes. Both sites are Phenom "Career Connect" sites on the company's domain and answer the same `POST /widgets` call. Per-site parameters: `refNum` (in the HTML, `TAKEGLOBAL`, `PHILUS`), the locale path (`/global/en`, `/na/en`), `lang` (`en_global`, `en_na`), `country` (`global`, `na`) and `pageId` (`page11`/`page9-ds-migration` for JET, `page6` for Philips; the value in `phApp.pageId` on the page). The `pageId` I used for JET (`page11`) differs from the home page value (`page9-ds-migration`) and still worked, so it may not be critical (not tested with a wrong value).
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `POST https://{host}/widgets` with `Content-Type: application/json` and this body: `{"lang":"en_global","deviceType":"desktop","country":"global","pageName":"search-results","ddoKey":"refineSearch","sortBy":"","subsearch":"","from":0,"jobs":true,"counts":true,"all_fetch":false,"pageType":"landingPage","size":100,"clearAll":false,"jdsource":"facets","isSliderEnable":false,"pageId":"page11","siteType":"external","location":"","keywords":"","global":true,"selected_fields":{},"sort":{"order":"desc","field":"postedDate"},"locationData":{}}`.
|
||||
2. Response: `refineSearch.totalHits` (103), `refineSearch.hits` (roles returned), `refineSearch.data.jobs[]`. Page with `from += size`. `from=100,size=10` returned the last 3 (verified). `size=200` returned all 103; `size=500` returned 500 of 865 on Philips (1.7 MB). `size=100` is the safe value.
|
||||
3. Job fields: `title`, `jobSeqNo` (`TAKEGLOBALR053200ENGLOBAL`, used in the URL), `reqId`, `postedDate`, `city`, `country`, `cityStateCountry`, `descriptionTeaser`, `category`, `type`, `applyUrl`, `latitude`, `longitude`.
|
||||
4. Role URL: `https://{host}/{country}/{lang}/job/{jobSeqNo}/{Title-With-Dashes}` (from the sitemap and detail page; the slug part is decorative, not tested). Detail page has JSON-LD JobPosting with `description` (HTML-escaped entities, decode twice), `datePosted`, `addressCountry`, `employmentType`, `identifier.value` (req id).
|
||||
5. Cross-check: `sitemap.xml` under the locale path lists every job URL (JET: 103 `/job/` URLs of 191, equal to totalHits; Philips: `sitemap_index.xml` points at sitemap1 and sitemap2 with 500 and 376 job URLs, against 865 hits, so the sitemap includes some duplicates or other locales). A sitemap route (sitemap plus JSON-LD detail) works with no POST at all.
|
||||
6. The `na` Philips site scopes to North America (306). To read all of Philips use `/global/en`. Whether per-country sites (`/de/de` etc.) exist as subsets is inferred.
|
||||
|
||||
Identifier: host, locale path, `refNum` and `pageId`; per role `jobSeqNo` (the `reqId` is not unique across languages; inferred).
|
||||
|
||||
Requests per poll: `ceil(total/100)`: 2 for JET (or 1 with size 200), 9 for Philips global. Descriptions need one detail page per new role.
|
||||
|
||||
Bot protection: none met from curl. The site sets cookies but the POST worked without any.
|
||||
|
||||
Worth a parser: yes. Phenom is common among large employers (Philips, JET here; many others in retail, pharma, industrials; inferred). Not typical for Nordic small companies. Two companies in this list, both large, so the return is good.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." -X POST -H 'Content-Type: application/json' -H 'Origin: https://careers.justeattakeaway.com' --data @body.json https://careers.justeattakeaway.com/widgets
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://careers.justeattakeaway.com/global/en/sitemap.xml
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://www.careers.philips.com/global/en/sitemap_index.xml
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://careers.justeattakeaway.com/global/en/job/TAKEGLOBALR053200ENGLOBAL/Corporate-Affairs-Specialist
|
||||
```
|
||||
|
||||
(`body.json` is the body from step 1, with `pageId` `page6` for Philips.)
|
||||
|
||||
### Welcome to the Jungle
|
||||
|
||||
Verified for Yousign/Youtrust.
|
||||
|
||||
One method: yes. All companies live in a shared Algolia index, so the identifier is the organization slug. The app's slug is stale here (`yousign` yields 0; `youtrust` yields 12), so slugs need to be taken from the company's `welcometothejungle.com/en/companies/{slug}/jobs` URL, and that URL can change on rebrand.
|
||||
|
||||
Algorithm:
|
||||
|
||||
1. `GET https://www.welcometothejungle.com/en/companies-v1/{slug}/jobs` (the `/en/companies/{slug}/jobs` URL answers 200 but only contains a meta refresh to `companies-v1`). The HTML embeds a JSON config with `ALGOLIA_APPLICATION_ID` (`CSEKHVMS53`), `ALGOLIA_API_KEY_CLIENT` (`<redacted>`, a search-only key that the site itself ships to every visitor) and `ALGOLIA_JOBS_INDEX_PREFIX` (`wttj_jobs_production`). Re-read these from the page if the key rotates instead of hard-coding them.
|
||||
2. `POST https://CSEKHVMS53-dsn.algolia.net/1/indexes/wttj_jobs_production_en/query` with headers `X-Algolia-Application-Id`, `X-Algolia-API-Key`, `Content-Type: application/json`, `Origin: https://www.welcometothejungle.com` and body `{"query":"","filters":"organization.slug:{slug}","hitsPerPage":100,"page":0}`. Response has `nbHits` (12), `nbPages`, `hits[]`. Page with `page += 1` while `page < nbPages`.
|
||||
3. The index suffix is the display language (`_en`, `_fr` exist; `_de` does not). The `_en` and `_fr` indices return the same 12 roles with the same `reference` UUIDs, so read one index only.
|
||||
4. A free-text query for "youtrust" matched 16 hits, including other companies' roles, so always filter by `organization.slug`.
|
||||
5. Hit fields: `name`, `slug`, `reference` (UUID), `published_at`, `offices[]`, `remote`, `contract_type`, `salary_*`, `summary`, `key_missions`, `profile`, `organization`. Full description text is not in the hit; the job page carries JSON-LD JobPosting with `description`, `datePosted`, `baseSalary`, `jobLocation` (fetched, 200).
|
||||
6. Role URL: `https://www.welcometothejungle.com/en/companies/{orgSlug}/jobs/{slug}`.
|
||||
|
||||
Identifier: organization slug; per role `reference` (UUID) or `slug`.
|
||||
|
||||
Requests per poll: 1 (plus 1 page load to refresh the key if it is not cached), plus a detail page per new role if the full description is wanted.
|
||||
|
||||
Bot protection: none met on Algolia. The WTTJ pages answer normal GETs. Algolia's public key is search-only, and its use is limited to what the site does itself. Keep volume low.
|
||||
|
||||
Worth a parser: only if French or Southern European companies matter. WTTJ is widely used by French companies and increasingly by other European ones as an aggregator, and many of those also use another ATS underneath (inferred). One company in this list. Note that a company can appear on WTTJ without its real ATS being WTTJ, and some companies have no roles there.
|
||||
|
||||
```
|
||||
curl -m 20 -A "Mozilla/5.0 ..." https://www.welcometothejungle.com/en/companies-v1/yousign/jobs
|
||||
curl -m 20 -A "Mozilla/5.0 ..." -X POST -H 'X-Algolia-Application-Id: CSEKHVMS53' -H 'X-Algolia-API-Key: <redacted>' -H 'Content-Type: application/json' -H 'Origin: https://www.welcometothejungle.com' --data '{"query":"","filters":"organization.slug:youtrust","hitsPerPage":100,"page":0}' 'https://CSEKHVMS53-dsn.algolia.net/1/indexes/wttj_jobs_production_en/query'
|
||||
```
|
||||
|
||||
## Summary of recommendations
|
||||
|
||||
| Host | Readable without a browser | Effort | Companies here | Recommendation |
|
||||
|---|---|---|---|---|
|
||||
| Pinpoint | yes, 1 GET | trivial | 2 | build |
|
||||
| BambooHR | yes, 1 GET plus detail | trivial | 1 | build |
|
||||
| Gem | yes, 1 POST (GraphQL) | small | 2 | build |
|
||||
| Oracle Recruiting Cloud | yes, 1 GET | small | 1 | build (cleanest API; only 1 company) |
|
||||
| Phenom | yes, 1 POST or sitemap | small | 2 | build |
|
||||
| iCIMS classic / TalentBrew | yes, sitemap plus JSON-LD detail | small, shared | 1 (Mambu) plus Arm | build the generic "sitemap plus JSON-LD" reader |
|
||||
| Jibe | yes, 1 to 2 GETs | trivial | 1 (Booking) | optional; cheap once JSON-LD reader exists |
|
||||
| Eightfold PCSX | yes, but 10 per page | medium (130 requests for Infineon) | 1 | build only if large enterprises matter |
|
||||
| Welcome to the Jungle | yes, Algolia | small | 1 | optional; slug staleness is the main risk |
|
||||
|
||||
What I verified by a live request: every listing endpoint named in the table, every paging claim marked verified above, the detail endpoints for Oracle, BambooHR, Pinpoint (page), iCIMS (page), Gem, Eightfold, Phenom (page) and the WTTJ job page. What I only infer: multi-page behaviour of classic iCIMS and BambooHR/Pinpoint on large boards, the Oracle job URL pattern, other Eightfold tenants allowing `/api/apply/v2/jobs`, whether Gem requires Origin/Referer headers, how widespread each host is, and how long the WTTJ Algolia key stays valid.
|
||||
@@ -1,97 +0,0 @@
|
||||
# Ingest review, 2026-09-29
|
||||
|
||||
Review of the path from source to ranking: the source plugins, the per-role text read, embedding and scoring of roles without text, locations, dates, field coverage, and filtering at the source. Five read-only agents worked on a snapshot of the live database (33,881 open roles, schema v17, before the duplicates migration) and on one live board per host. The planner checked the main claims against the code. This review is the input for the plan that follows; it is folded into `decisions.md` and deleted once that plan lands.
|
||||
|
||||
## State in one paragraph
|
||||
|
||||
The source layer is plugin-style: one module per host in `plugins/atshosts/`, a contract checked at import, and 8 parallel workers over batches of 10 boards. Agents work as intended: each role is scored against every prompt of an agent, the best one wins, and votes, favourites, gates and limits belong to each agent. The gaps are in what the modules keep from what the hosts send, in the order the text is fetched, and in the fields the filters rely on. 11,952 open roles (35%) have no text. Extract has never run, so seniority comes from the title alone and pay mostly from three hosts. The live server predates the duplicates merge and has not applied migration 18.
|
||||
|
||||
## Findings
|
||||
|
||||
Ordered by impact on getting the right roles ranked. Effort: S is under a day for an agent, M is a work package.
|
||||
|
||||
### Text
|
||||
|
||||
1. **SmartRecruiters and Workable have no per-role read.** 7,391 roles (Bosch, Delivery Hero, Wise, Ubisoft) are ranked on the title alone. Both hosts have a per-posting endpoint. Fix: a `detail` in `smartrecruiters.py` and `workable.py`. M.
|
||||
2. **The per-role read goes in list order, not by score.** `plugins/ats.py:667` takes the first 200 unread roles per board. Agent 2's ranks 2 to 11 were SAP roles without text, waiting up to four pulls. Detail reads run one after another inside the pull batch, so batches with such hosts take 300 to 440 s against 10 s for the rest, and the cascade waits 12 minutes. Fix: order the queue by best score (or gate pass for roles not yet scored), then move detail reads into a task of their own after `pull`, followed by embed and score, on its own schedule. M.
|
||||
3. **List teasers are dropped.** `ats.py:703` blanks the description of every role not read in that pull, so the Phenom and Oracle teasers (1,066 roles) are lost, and `feed_text` is cut from the blanked value. Fix: keep the teaser in `feed_text` and embed from it until the full text arrives. S.
|
||||
4. **Only an empty-to-filled description triggers a new embedding.** `embed.document` reads `title` and `description` only. A hand-fetched posting (`article_text`), a teaser replaced by full text, or an edited posting is never embedded again, and extract ignores them too. Jobindex roles keep their 237-character teaser for good. Fix: one text rule, `article_text or description or feed_text`, in embed and extract; a hash of the embedded text on `vectors`, re-embedding when it changes; the fetch endpoint queues embed and score. S to M.
|
||||
5. **Lever keeps only the intro paragraph.** `lever.py:102` drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way. S.
|
||||
6. **Extract would mark a title-only role done for good.** When text arrives, `upsert_role` drops the vector but not `extracted_at`. Harmless today since extract is off; must be fixed before switching it on. S.
|
||||
|
||||
### Roles without text in the ranking
|
||||
|
||||
Title-only roles are not shut out. They hold 15 of agent 2's top 30, 47 of the top 200 and 96 of the top 500. Their fit is noisy rather than inflated: on 400 roles with text, the title alone correlates at 0.72 (Spearman) with the full text, and 18 of the top 40 stay in the top 40. Findings 1 to 3 fix the cause. Until then, a "title only" mark in the feed shows which scores are guesses.
|
||||
|
||||
### Locations
|
||||
|
||||
7. **Seven hosts keep one place of several.** Ashby `secondaryLocations` (147 of 831 at OpenAI), Lever `allLocations` (37 of 79 at Spotify), Personio `additionalOffices` (116 of 320), Greenhouse `offices` (120 of 401 at Cloudflare), Workable `locations[]`, Recruitee `locations[]`, Platsbanken `workplace_addresses[]`. SuccessFactors' `detail` keeps the last address of several (`successfactors.py:402`) and then withholds the list's "+2 more". Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold and Jibe keep them all, joined with "; ", which the place matching reads word by word. Fix: a shared `join_places()` in `_shared.py` and every module uses it. S.
|
||||
8. **Placeholders stored as places.** Greenhouse "Hybrid" or "In-Office" (340 Cloudflare roles, real office in `offices`), Workday "N Locations" until read, empty `locationsText` at LEGO (192 roles, city in the URL path), Recruitee "Remote job", Workday "Country - City" which `country_of` does not read (63 roles). S each.
|
||||
9. **A role with a place but no country is gated wrongly.** The `place` gate has evidence whenever `place` is filled, the `country` gate has none when `country` is empty, so a German role at Zeiss in "Oberkochen; Jena" fails agent 2's `any` group. 1,262 open roles. A joined line over several countries becomes `EU`, which the country gate does not expand: 714 roles, 265 of them naming a Danish, Swedish or German place. Fix: fill the country from the place per town, and match `EU` by its places. S to M.
|
||||
10. **Postings copied per location are not one role.** 1,741 company-and-title groups (5,021 roles) are one posting per location; `dupes.py` leaves them apart on purpose. Nothing is lost, the feed is only noisier. Owner decision whether to fold them.
|
||||
|
||||
### Remote
|
||||
|
||||
11. **Ashby marks hybrid roles remote.** `ashby.py:116` maps `isRemote`, which is true for hybrid roles; `workplaceType` is the right field. 2,537 Ashby roles are stored remote, about 23 of 550 at OpenAI really are. Every remote gate passes them. S.
|
||||
|
||||
### Dates
|
||||
|
||||
12. **Eightfold and Phenom map a refresh date.** Eightfold `postedTs` is refreshed in bulk (a whole page says today, `creationTs` is up to eight weeks older). Phenom `postedDate` is the same; `dateCreated` is the posting date. Fix both mappings, then trust Eightfold. S.
|
||||
13. **Untrusted hosts that can be trusted.** Jibe `posted_date` equals the page's `datePosted`; HR Manager's `pubDate` is monotone in the id; Deel's dates predate first sight by the same margin as Ashby's; Gem's detail date is plausible. Add them to `TRUSTED_POSTED_HOSTS`, better as a `TRUST_POSTED` attribute on each module so a new host states it. Pinpoint states no date. Platsbanken's local time is read as UTC. S.
|
||||
|
||||
### Coverage and filters
|
||||
|
||||
| Field | Filled on open roles | Source |
|
||||
|---|---|---|
|
||||
| Text | 65% | hosts; 7,391 never, 4,549 waiting for the per-role read |
|
||||
| Seniority | 43% | title words only; extract never ran |
|
||||
| Pay figures | 12% | Greenhouse, Ashby, Lever almost alone |
|
||||
| Remote | 25% | hosts that state it, Ashby wrong |
|
||||
| Country | 94% | place lookup |
|
||||
| Permit, languages | 0% | extract only |
|
||||
|
||||
14. **Seniority stays near half from titles.** A hand sample of 59 roles without a level was 85% individual contributors with no level word, one manager. Better title rules add about one point. The text adds about nine: 2,937 of them state "N years of experience". Three hosts state a level and none is mapped: Personio `seniority` (all 320 roles), SmartRecruiters `experienceLevel` (about half), Recruitee `experience_code`; Jobylon `experience`. Fix: map the coarse levels (executive, director, entry, student) from those hosts, and let extract read the rest. S, plus extract on.
|
||||
15. **Pay in the text but not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and mostly US. Extract, or a regex with the period, would fill them. S to M.
|
||||
16. **The seniority filter is only useful with "no level" on.** It passes an empty level by default (`store.py:912`). In agent 2's top 200, "manager and up" keeps 37 known and 70 unknown and drops 93; with "no level" off it keeps 37. 14 of the top 200 are interns or working students, since the agent has no rule on level.
|
||||
|
||||
### Losing roles
|
||||
|
||||
17. **Caps leave boards that never close.** At `MAX_PAGES` (Eightfold 1,000, Workday 2,000, Workable 1,000) a read is incomplete and closes nothing, so dead roles stay open and roles beyond the cap never arrive. No attached board is near a cap today; Infineon and Microsoft are not attached because of it. Fix: filtering at the source (below), and closing an incomplete board's roles after N incomplete pulls by age.
|
||||
18. **A read that returns nothing is final.** `ats.py:696` marks a role done when `detail` returns without text (Oracle closed requisitions, empty SuccessFactors blocks). Phenom raises instead and retries every pull. Fix: retry an empty read after some days. S.
|
||||
19. **An empty answer closes a board.** SmartRecruiters, Ashby and Gem answer a complete empty list for an unknown id; the shrink guard below covers it. S.
|
||||
20. **A disabled agent keeps stale scores**, shown if it is picked in the feed. Extract and summarise pick by date, not score. S each.
|
||||
|
||||
### Jobindex and robots.txt
|
||||
|
||||
21. **Our Jobindex requests match its robots.txt `Disallow`** (`/jobsoegning*sort=`, `*page=`, `*geoareaid=`), and the search providers read no robots.txt. This contradicts the decision that robots.txt is honoured. Owner decision: drop Jobindex, read only what robots.txt allows (no paging, no area), or accept it for this source.
|
||||
|
||||
## Filtering at the source
|
||||
|
||||
Of 33,881 open roles, 25,480 are gated for agent 2. 13,558 of those sit on hosts that filter by place server-side. Greenhouse and Ashby, which hold most of the rest, ignore every place parameter.
|
||||
|
||||
| Host | Parameter | Behaviour, verified live | Roles saved |
|
||||
|---|---|---|---|
|
||||
| SmartRecruiters | `?country=de` | one country per request; DE+SE+DK sum exactly to what we hold; `remote` ignored | 6,210 of 7,238 |
|
||||
| SuccessFactors | `optionsFacetsDD_country=DK` | works on 4 of 4 tenants; no remote | 2,880 of 3,738 |
|
||||
| Workday | `appliedFacets` with the tenant's location facet | ids read from the first answer; multi-location roles kept; remote is a separate facet on some tenants | 1,765 of 2,377 |
|
||||
| Eightfold | `location=Sweden` | one country per request; multi-location roles kept; makes Infineon (1,288 to 163) and Microsoft (2,320 to 60) readable | 467 at Ericsson, plus the two new boards |
|
||||
| Phenom | `selected_fields.country` list | list unions; multi-location roles kept; no remote | 903 of 1,007 |
|
||||
| Oracle | `selectedLocationsFacet` ids, `ORA_REMOTE` | ids per pod, small countries not always listed; unknown id returns the whole board | 761 of 826 |
|
||||
| Workable, Jibe, Lever, Teamtailor | country name, or exact city | small boards | under 1,000 together |
|
||||
| Greenhouse, Ashby, Recruitee, the rest | none | | 0 |
|
||||
|
||||
Design: the reach is derived at pull time from the enabled agents' `any` gates (countries, towns mapped to their country, remote), the union over agents, and `all` when any agent has a non-place member in that group. A starred company uses the `require` reach (Europe). A board can pin `all` or its own reach in `subscriptions.params`. Only boards above about 300 roles are filtered. A host module gets an optional `queries(get, slug, reach)` that returns one request series per country; `_paged` unions the series by id, and the read is complete only when every series is. A complete filtered read closes the roles outside the filter on purpose; a wider filter reopens them. Guards: roles with a vote, application or note are not closed; a filtered read that returns nothing or under 10% of the board counts as failed; a host that ignores the parameter returns a superset, which is safe. Remote roles outside the wanted countries are lost on the four hosts that cannot ask for remote, about 120 today. Rows from a single-country series carry that country, which also fixes finding 9 for those hosts. This is package 11 of `docs/plan-hosts-wave-2.md`, with the reach derived from the agents instead of stored by hand. Order: SmartRecruiters, Eightfold, Workday and SuccessFactors, Phenom and Oracle.
|
||||
|
||||
## Plugin contract
|
||||
|
||||
Adding a host is one file, but the contract does not cover: the date trust (a set in `app/models.py`), fields beyond a fixed list (`ats.py` builds `raw` from fixed keys, so a host cannot state country, seniority or employment type), and `Row` as a bare dict. `search.py` is outside the contract: its own getter, no robots.txt check, no size cap, three providers in one file. Repeated in many modules and worth moving to `_shared.py`: HTML to text (16 modules), the remote word map (11, spelled differently, the cause of finding 11), pay lines (5), place joining (done by hand in 2).
|
||||
|
||||
## Order proposed
|
||||
|
||||
1. Restart the server so migration 18 applies.
|
||||
2. Text: teaser kept (3), one text rule and re-embed on change (4), extract reset (6), Lever and Recruitee bodies (5). S each, disjoint files.
|
||||
3. Locations and remote: `join_places` in every module (7), placeholders (8), Ashby `workplaceType` (11), country from place and `EU` in the gate (9).
|
||||
4. Dates (12, 13) and host-stated seniority (14): small per-module changes, can ride with step 3.
|
||||
5. Detail read by score in its own task (2), then SmartRecruiters and Workable `detail` (1).
|
||||
6. Filtering at the source.
|
||||
7. Extract on for each agent's top 500, then re-measure seniority and pay coverage before relying on those filters.
|
||||
+19
-19
@@ -59,14 +59,14 @@ Numbers are from the live database after the pull of 2026-09-29. For the "list,
|
||||
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
|
||||
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
|
||||
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request; 15 s per request for the search providers | A slow board fails for this pull and is asked again next pull. |
|
||||
| Pages per board | 100 | A board above it (1,000 roles at Workable or Eightfold, 2,000 at Workday, 5,000 at Teamtailor, 10,000 at SmartRecruiters) is read incomplete: roles on the first 100 pages are stored and new ones found, but no role of that board is ever closed by list. |
|
||||
| Pages per board | 100 | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
|
||||
| Wait | 0.5 s before every page after the first and before every per-role read; none between search pages | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
|
||||
| Per-role reads | 200 per board per pull | The rest wait for the next pull. A 404 or 410 counts as read; another failure is retried on a later pull. |
|
||||
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing. A role listed again reopens. A board switched off closes its roles at once. |
|
||||
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, robots.txt. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize. |
|
||||
| A board that fails for 21 days | all its open roles close at the next pull | The next answer reopens what it lists. |
|
||||
| robots.txt | read before the list for the 14 hosts that name a page: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Jibe, Gem, BambooHR, Pinpoint, Deel, Jobylon, Rippling, HiBob, HR Manager; kept for one day per board | A disallow fails the board with "robots.txt disallows it" and counts toward the 21 days. An unreachable file (5xx, 429, timeout) fails the board for this pull and is asked again next pull. A 4xx means no rules, which allows. `Crawl-delay` is not honoured. The eight API hosts (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor) and the search providers read no robots.txt. |
|
||||
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a per-role read: the per-role reads of that board stop until the next pull | Nothing is closed on a list cut short by a 429. A search that fails is skipped for the pull. |
|
||||
|
||||
How far a board is read: a pull asks each board over 300 roles for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of 30 times, unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application.
|
||||
@@ -171,7 +171,7 @@ What no source does:
|
||||
| Target | the subdomain, e.g. `pinpoint:xeneta` |
|
||||
| Companies | 1 board: Xeneta |
|
||||
| List | `https://<sub>.pinpointhq.com/postings.json`, the whole board in one answer; no total, no paging keys |
|
||||
| Requests per pull | 1, plus robots.txt once a day |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, skills and benefits sections; live: 2 of 2 |
|
||||
| Date | none: the list states none and the role page is not read for it; the role counts as posted when first seen |
|
||||
| Place and country | `location.name` (else `city`) plus `province`, one place; country from that line |
|
||||
@@ -187,7 +187,7 @@ What no source does:
|
||||
| Target | the board slug on `jobs.deel.com`, with case, e.g. `deel:klarna` |
|
||||
| Companies | 1 board: Klarna |
|
||||
| List | the HTML page `https://jobs.deel.com/<slug>`, from which the Next.js server payload is read; the whole board in one answer; no total |
|
||||
| Requests per pull | 1, plus robots.txt once a day |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text (`richtextDescription`); live: 96 of 96 |
|
||||
| Date | the publication's `createdAt`, else the posting's; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every `jobLocations` name, joined with "; "; country from that line (91 of 96) |
|
||||
@@ -203,7 +203,7 @@ What no source does:
|
||||
| Target | the subdomain of `careers.hibob.com`, e.g. `hibob:dixa` |
|
||||
| Companies | 1 board: Dixa |
|
||||
| List | `https://<sub>.careers.hibob.com/api/job-ad` with the header `companyIdentifier: <sub>`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1, plus robots.txt once a day |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, requirements, benefits; live: 2 of 2 |
|
||||
| Date | `publishedAt`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `site` plus `country`, one place |
|
||||
@@ -272,7 +272,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the career site's host, e.g. `jibe:jobs.booking.com` |
|
||||
| Companies | 1 board: Booking.com |
|
||||
| List | `https://<host>/api/jobs?limit=100&page=<n>`, 100 per request, the most the host serves; states `totalCount` |
|
||||
| Requests per pull | roles / 100, rounded up, plus robots.txt once a day; 2 for Booking.com |
|
||||
| Requests per pull | roles / 100, rounded up; 2 for Booking.com |
|
||||
| Description | in the list, full text: description and qualifications; live: 140 of 140 |
|
||||
| Date | `posted_date`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `full_location`, several places joined by the host; country 140 of 140 |
|
||||
@@ -288,14 +288,14 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the portal's customer name, e.g. `hrmanager:stsgroup_tr` |
|
||||
| Companies | none attached; Seaborg, the one seed company on it, is on the Watchlist as `robots` |
|
||||
| List | the RSS feed `api.hr-manager.net/JobPortal.svc/<customer>/PositionList/rss/?incads=true`, 100 per request by `skip`; no total, so the read goes on to an empty page |
|
||||
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page, plus robots.txt once a day |
|
||||
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page |
|
||||
| Description | in the list, the whole advertisement |
|
||||
| Date | `pubDate`; not trusted |
|
||||
| Place and country | none: the feed names no place |
|
||||
| Remote | none |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from a portal or API link; not probed |
|
||||
| Limitations | The portal's robots.txt on `candidate.hr-manager.net` disallows everything but the application form; the feed's API host has no robots.txt, so the module's check allows it. The owner decided to leave the board unattached. Without `take` the host answers 25 roles. |
|
||||
| Limitations | The portal's robots.txt on `candidate.hr-manager.net` disallows everything but the application form; the feed's API host has no robots.txt. The owner decided to leave Seaborg unattached. Without `take` the host answers 25 roles. |
|
||||
|
||||
## Hosts read as list, then each role
|
||||
|
||||
@@ -306,7 +306,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the career site's host, or host plus a brand path, e.g. `successfactors:careers.vestas.com`, `successfactors:jobs.teradyne.com/MiR` |
|
||||
| Companies | 14 boards: Novo Nordisk, Vestas, Carlsberg |
|
||||
| List | the site's HTML search page `/search/?startrow=<n>`, parsed in its two known layouts; page size is the site's own (10, 15, 25 or 100) and cannot be set; a total where the pagination label states one |
|
||||
| Requests per pull | roles / the site's page size, rounded up, plus up to 200 role pages, plus robots.txt once a day; per-board sizes are not in these notes |
|
||||
| Requests per pull | roles / the site's page size, rounded up, plus up to 200 role pages; per-board sizes are not in these notes |
|
||||
| Description | per role, the `itemprop="description"` block of the role page; live: 1,651 of 3,738 after one pull |
|
||||
| Date | the list's date column where it parses; the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's location, with a two-letter country code turned into the name; the role page's microdata replaces it; country 3,675 of 3,738 |
|
||||
@@ -323,7 +323,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | `<tenant>.<wdN>/<Site>`, site with case, e.g. `workday:lego.wd103/LEGO_External` |
|
||||
| Companies | 13 boards: LEGO Group, Zendesk, Equinor |
|
||||
| List | a POST to `/wday/cxs/<tenant>/<Site>/jobs`, 20 per request by offset (21 or more answers 400); the first page states the total |
|
||||
| Requests per pull | roles / 20, rounded up, plus up to 200 role reads, plus robots.txt once a day; for a board of 1,000 roles 50 + 200 on the first pull |
|
||||
| Requests per pull | roles / 20, rounded up, plus up to 200 role reads; for a board of 1,000 roles 50 + 200 on the first pull |
|
||||
| Description | per role, `jobDescription` from the role's JSON; live: 1,411 of 2,377 after one pull |
|
||||
| Date | the list's "Posted N Days Ago" as a day, possibly one day off; "Posted 30+ Days Ago" and other languages give none; the role's `startDate` replaces it; trusted |
|
||||
| Place and country | the list has `locationsText` ("Copenhagen" or "2 Locations"); the per-role read adds every location and the country. The country therefore arrives with the per-role read: 1,724 of 2,377 after one pull |
|
||||
@@ -340,7 +340,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | `<pod host>/<siteNumber>`, e.g. `oracle:fa-evmr-saasfaprod1.fa.ocs.oraclecloud.com/CX_1` |
|
||||
| Companies | 4 boards: Danske Bank, Nokia, Milestone Systems, Sinch |
|
||||
| List | the REST resource `recruitingCEJobRequisitions`, 200 per request by offset, with secondary places expanded; states `TotalJobsCount` |
|
||||
| Requests per pull | roles / 200, rounded up, plus up to 200 role reads, plus robots.txt once a day; 3 + 200 for Nokia's roughly 600 on the first pull |
|
||||
| Requests per pull | roles / 200, rounded up, plus up to 200 role reads; 3 + 200 for Nokia's roughly 600 on the first pull |
|
||||
| Description | per role: external description, responsibilities and qualifications joined; the list's one-paragraph `ShortDescriptionStr` is not stored; live: 432 of 826 after one pull |
|
||||
| Date | `PostedDate`; the role's `ExternalPostedStartDate` replaces it when earlier; trusted |
|
||||
| Place and country | the primary place plus every secondary one, joined with "; "; country 769 of 826 |
|
||||
@@ -357,7 +357,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the career site's host plus its locale root, e.g. `phenom:www.careers.philips.com/global/en` |
|
||||
| Companies | 3 boards: King, Just Eat Takeaway.com, Philips |
|
||||
| List | a POST to `https://<host>/widgets`, 100 per request by offset; states `totalHits` |
|
||||
| Requests per pull | roles / 100, rounded up, plus up to 200 role pages, plus robots.txt once a day; 9 + 200 for Philips' 865 on the first pull |
|
||||
| Requests per pull | roles / 100, rounded up, plus up to 200 role pages; 9 + 200 for Philips' 865 on the first pull |
|
||||
| Description | per role, the JSON-LD `JobPosting` description of the role page; the list's `descriptionTeaser` is not stored; live: 335 of 1,007 after one pull |
|
||||
| Date | the list's `postedDate`; the role page's `datePosted` replaces it when earlier (Philips' list gave the day of the read for a role posted 2026-09-08); trusted |
|
||||
| Place and country | every place of `multi_location`, joined with "; "; country 975 of 1,007 |
|
||||
@@ -374,7 +374,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | `<tenant host>/<company domain>`, e.g. `eightfold:jobs.ericsson.com/ericsson.com` |
|
||||
| Companies | 1 board: Ericsson |
|
||||
| List | `https://<host>/api/pcsx/search?domain=<domain>&sort_by=timestamp`, newest first, 10 per request whatever `num` asks; states `count` |
|
||||
| Requests per pull | roles / 10, rounded up, at most 100, plus up to 200 `position_details` reads, plus robots.txt once a day; 63 + 200 for Ericsson's 630 on the first pull |
|
||||
| Requests per pull | roles / 10, rounded up, at most 100, plus up to 200 `position_details` reads; 63 + 200 for Ericsson's 630 on the first pull |
|
||||
| Description | per role, `jobDescription`; live: 200 of 630 after one pull |
|
||||
| Date | `postedTs`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location, repeated names dropped, joined with "; "; country 622 of 630 |
|
||||
@@ -391,7 +391,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the board name on `jobs.gem.com`, with case, e.g. `gem:productboard` |
|
||||
| Companies | 2 boards: sennder, Productboard |
|
||||
| List | a GraphQL POST to `jobs.gem.com/api/public/graphql`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1, plus one GraphQL POST per role not yet read (at most 200), plus robots.txt once a day |
|
||||
| Requests per pull | 1, plus one GraphQL POST per role not yet read (at most 200) |
|
||||
| Description | per role, `descriptionHtml`; live: 44 of 44 |
|
||||
| Date | per role only, `firstPublishedTsSec`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location's city, joined with "; "; `isoCountry` is fetched and not used; country from the line (41 of 44) |
|
||||
@@ -407,7 +407,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the numeric company id, e.g. `jobylon:3143` |
|
||||
| Companies | 2 boards: Einride, Vipps MobilePay |
|
||||
| List | the careers widget's embed page `cdn.jobylon.com/jobs/companies/<id>/embed/v2/`, whose script literal holds the whole board; no total |
|
||||
| Requests per pull | 1, plus one role page on `emp.jobylon.com` per role not yet read (at most 200), plus robots.txt once a day; 1 + 33 for Einride on the first pull |
|
||||
| Requests per pull | 1, plus one role page on `emp.jobylon.com` per role not yet read (at most 200); 1 + 33 for Einride on the first pull |
|
||||
| Description | per role, the JSON-LD description of the role page; live: 33 of 33 |
|
||||
| Date | the list's day ("Sept. 25, 2026"); the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's place names, which can be cut short ("Houston +3 more"); the per-role read fills every place; country 32 of 33 |
|
||||
@@ -423,7 +423,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the board slug on `ats.rippling.com`, with case, e.g. `rippling:prisma-careers` |
|
||||
| Companies | 1 board: Prisma |
|
||||
| List | `api.rippling.com/platform/api/ats/v1/board/<slug>/jobs`, the whole board in one array, one entry per role and place, merged by role; no total |
|
||||
| Requests per pull | 1, plus one role record per role not yet read (at most 200), plus robots.txt once a day; 1 + 2 for Prisma |
|
||||
| Requests per pull | 1, plus one role record per role not yet read (at most 200); 1 + 2 for Prisma |
|
||||
| Description | per role, `description.role`; live: 2 of 2 |
|
||||
| Date | per role only, `createdOn`; trusted |
|
||||
| Place and country | the list's place labels, joined with "; "; the per-role read fills every work location |
|
||||
@@ -439,7 +439,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the subdomain, e.g. `bamboohr:configit` |
|
||||
| Companies | 1 board: Configit |
|
||||
| List | `https://<sub>.bamboohr.com/careers/list`, the whole board in one answer; a list shorter than its `totalCount` fails the board rather than being read as whole |
|
||||
| Requests per pull | 1, plus one `/careers/<id>/detail` per role not yet read (at most 200), plus robots.txt once a day; 1 + 2 for Configit |
|
||||
| Requests per pull | 1, plus one `/careers/<id>/detail` per role not yet read (at most 200); 1 + 2 for Configit |
|
||||
| Description | per role; live: 2 of 2 |
|
||||
| Date | per role only, `datePosted`; trusted |
|
||||
| Place and country | the list's city and state; the country arrives with the per-role read |
|
||||
@@ -513,4 +513,4 @@ Jobnet and Jooble were left out: Jobnet's search answers 401 outside its own sit
|
||||
| `key` | 3 | the only listing answers with a key the page's script carries | Maersk, New Work SE, Yousign (Welcome to the Jungle) |
|
||||
| no reason | 9 | on Teamtailor, but the careers link is a marketing site and the career host is not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial |
|
||||
|
||||
What is planned for the `unread` companies and for the gaps in hosts already read (Eightfold's page cap, SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, a country filter in a target) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
|
||||
What is planned for the `unread` companies and for the gaps in hosts already read (SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, Avature for Siemens) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
|
||||
@@ -10,7 +10,7 @@ as they are found. The site is case-insensitive for the API (Red Hat's
|
||||
`jobs` and `Jobs` both work) but its robots.txt paths are not, so the slug
|
||||
keeps the case the URL wrote (`KEEP_CASE`).
|
||||
|
||||
Written from live career sites on 2026-09-29 (`docs/plan-hosts/workday.md`).
|
||||
Written from live career sites on 2026-09-29.
|
||||
The listing carries no description, so `detail` reads each role's own JSON.
|
||||
|
||||
Countries and remote are filtered on the server by the tenant's own facets
|
||||
|
||||
@@ -458,7 +458,7 @@ def _found(text: str) -> list[tuple[str, str]]:
|
||||
return [(host, slug) for host, slug, _url in discover.boards_in(text)]
|
||||
|
||||
|
||||
#: The twelve career sites of `docs/plan-hosts/workday.md`, as a person opens
|
||||
#: The twelve career sites surveyed on 2026-09-29, as a person opens
|
||||
#: them, and the slug discovery names for each.
|
||||
TWELVE = [
|
||||
("https://zendesk.wd1.myworkdayjobs.com/zendesk", "zendesk.wd1/zendesk"),
|
||||
|
||||
Reference in new issue
Block a user