Merge the wiring of the new search providers (wp-src-wiring)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-10-03 12:24:56 +02:00
commit a9c387c130
16 files changed
+422 -39

No files matched your search

+50 -2
View File
@@ -85,6 +85,9 @@ Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open
| HiBob | 1 | 2 | whole board | all | in the list | figures | pay never seen filled |
| BambooHR | 1 | 2 | list, then each role | all | per role | none | paging of a large board unverified |
| Adzuna (search) | 0 | 0 | paged | 50 | teaser | figures | never run live; needs a key |
| freehire (search) | 0 | 0 | paged | 100 | in the list | figures | 10,000 results per search; the query is noisy |
| Jobnet (search) | 0 | 0 | paged | 100 | in the list | none | 10,000 results per search; needs the header `x-csrf: 1` |
| Akademikernes Jobbank (search) | 0 | 0 | paged | 100 | per role | none | not sorted by date, and no total, so never complete |
"Figures" means yearly `salary_min`, `salary_max` and `currency`, plus a pay line; "line only" means a pay line (`salary_text`) and no figures. Figures are yearly: a monthly figure is multiplied by 12 where the host states the period, and an hourly, daily or weekly one is kept as a line only.
@@ -505,7 +508,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
## The search providers
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
### Jobindex
@@ -552,7 +555,52 @@ A saved search is a query with its parameters, stored as a `search` subscription
| Key | a free key from developer.adzuna.com, as `sources.search.adzuna` in the config or `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`; without one the search fails |
| Limitations | Built against Adzuna's documented response and never run live; no Adzuna role is in the database. |
Jobnet and Jooble were left out: Jobnet's search answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
### freehire
| | |
|---|---|
| Saved search | `q`, optionally `country` (lower-case ISO codes, comma-separated; empty for all) and `remote` (a work mode of freehire's: `MODES` in the module) |
| List | `freehire.me/api/v1/agent/jobs/search`, newest first, 100 per page; states `meta.total`. A public API with no key, 300 requests a minute; robots.txt allows `/` and invites API use with a descriptive user agent |
| Pages read | 5 by default (500 results), 10 at most (1,000); the API ends at `offset + limit = 10000` |
| Text | the whole text, as HTML in the answer |
| Date | `posted_at`; trusted |
| Place | `cities` and `location`; the country only when the row names exactly one |
| Remote | `work_mode` |
| Pay | figures from `enrichment`, only where `salary_period` is a year |
| Key | none |
| Limitations | freehire aggregates about 50 hosts, and keeps the one a row came from in `raw` (`source`: `greenhouse`, `jobnet`, ...). A row whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it by the dupes rule. Unquoted words in `q` are OR over title, company, text and place, so a query is noisy: narrow it by country. |
### Jobnet
| | |
|---|---|
| Saved search | `q` and optionally `region`, one of Jobnet's five region ids (`REGIONS` in the module); empty for all of Denmark |
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login; this supersedes the earlier note that its search answers 401 outside its own session |
| Pages read | 5 by default (500 results), 10 at most (1,000); page 101 at 100 a page answers empty |
| Text | the whole text, as HTML in the answer |
| Date | `publicationDate`; trusted |
| Place | `postalDistrictName`, else `municipality`; country Denmark unless the ad names another |
| Remote | none |
| Pay | none |
| Key | none |
| Limitations | The state's portal: mostly the public sector and small companies, and about a fifth of its ads link to an employer's own ATS. The match is loose, so a phrase does not narrow it. There is no robots.txt (`/robots.txt` redirects to a login). The ad's deadline is kept in `raw` only. |
### Akademikernes Jobbank
| | |
|---|---|
| Saved search | `q` and optionally `region` (an `amt` id from the site's own search form: `REGIONS` in the module, 2 is Storkøbenhavn) and `remote` (`helt` fully remote, `delvist` partly) |
| List | the RSS feed `jobbank.dk/job/rss/`, 100 items a page; an empty page ends it. It states no total and is not sorted by date |
| Pages read | at most the module's `ceiling`, which keeps a run under about 20 pages |
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by `enrich` for the roles an agent ranks high. The listing carries none |
| Date | `datePosted` from the role's page; trusted |
| Place | the region in the item's description line, and `jobLocation.addressLocality` from the page; country Denmark |
| Remote | not stated by a row; the search's `remote` filter narrows the list |
| Pay | none |
| Key | none |
| Limitations | Academic and research roles, plus some corporate ads. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Cloudflare may answer a challenge instead of the page, which is an error for that role and not an empty text. |
Jooble was left out: it gives a key only on a manual application and answered a Cloudflare challenge. Jobdanmark was left out too: about a quarter of its newest ads are on Jobnet, freehire carries the rest as source `jobdanmark`, and its list has no text. LinkedIn's guest pages are out because its robots.txt disallows `/` and its terms forbid automated access; LinkedIn enters only as alert mail. Indeed has no public search API.
## Sources that cannot be read, and why