Wire freehire, Jobnet and Akademikernes Jobbank into the CLI, the Sources form, dupes and the docs

Provider names, the CLI's choices and the dupes rule's search hosts now come from readers.PROVIDERS, and the regions, work modes and remote words from the modules' REGIONS, MODES and REMOTE, read with getattr so main works before and after the readers merge.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Sonnet 5.5 committed 2026-10-03 12:24:34 +02:00
1 parent 018a1ad895
commit 6e6e655203
16 files changed
+422 -39

No files matched your search

+50 -2
View File
@@ -85,6 +85,9 @@ Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open
| HiBob | 1 | 2 | whole board | all | in the list | figures | pay never seen filled |
| BambooHR | 1 | 2 | list, then each role | all | per role | none | paging of a large board unverified |
| Adzuna (search) | 0 | 0 | paged | 50 | teaser | figures | never run live; needs a key |
| freehire (search) | 0 | 0 | paged | 100 | in the list | figures | 10,000 results per search; the query is noisy |
| Jobnet (search) | 0 | 0 | paged | 100 | in the list | none | 10,000 results per search; needs the header `x-csrf: 1` |
| Akademikernes Jobbank (search) | 0 | 0 | paged | 100 | per role | none | not sorted by date, and no total, so never complete |
"Figures" means yearly `salary_min`, `salary_max` and `currency`, plus a pay line; "line only" means a pay line (`salary_text`) and no figures. Figures are yearly: a monthly figure is multiplied by 12 where the host states the period, and an hourly, daily or weekly one is kept as a line only.
@@ -505,7 +508,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
## The search providers
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
### Jobindex
@@ -552,7 +555,52 @@ A saved search is a query with its parameters, stored as a `search` subscription
| Key | a free key from developer.adzuna.com, as `sources.search.adzuna` in the config or `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`; without one the search fails |
| Limitations | Built against Adzuna's documented response and never run live; no Adzuna role is in the database. |
Jobnet and Jooble were left out: Jobnet's search answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
### freehire
| | |
|---|---|
| Saved search | `q`, optionally `country` (lower-case ISO codes, comma-separated; empty for all) and `remote` (a work mode of freehire's: `MODES` in the module) |
| List | `freehire.me/api/v1/agent/jobs/search`, newest first, 100 per page; states `meta.total`. A public API with no key, 300 requests a minute; robots.txt allows `/` and invites API use with a descriptive user agent |
| Pages read | 5 by default (500 results), 10 at most (1,000); the API ends at `offset + limit = 10000` |
| Text | the whole text, as HTML in the answer |
| Date | `posted_at`; trusted |
| Place | `cities` and `location`; the country only when the row names exactly one |
| Remote | `work_mode` |
| Pay | figures from `enrichment`, only where `salary_period` is a year |
| Key | none |
| Limitations | freehire aggregates about 50 hosts, and keeps the one a row came from in `raw` (`source`: `greenhouse`, `jobnet`, ...). A row whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it by the dupes rule. Unquoted words in `q` are OR over title, company, text and place, so a query is noisy: narrow it by country. |
### Jobnet
| | |
|---|---|
| Saved search | `q` and optionally `region`, one of Jobnet's five region ids (`REGIONS` in the module); empty for all of Denmark |
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login; this supersedes the earlier note that its search answers 401 outside its own session |
| Pages read | 5 by default (500 results), 10 at most (1,000); page 101 at 100 a page answers empty |
| Text | the whole text, as HTML in the answer |
| Date | `publicationDate`; trusted |
| Place | `postalDistrictName`, else `municipality`; country Denmark unless the ad names another |
| Remote | none |
| Pay | none |
| Key | none |
| Limitations | The state's portal: mostly the public sector and small companies, and about a fifth of its ads link to an employer's own ATS. The match is loose, so a phrase does not narrow it. There is no robots.txt (`/robots.txt` redirects to a login). The ad's deadline is kept in `raw` only. |
### Akademikernes Jobbank
| | |
|---|---|
| Saved search | `q` and optionally `region` (an `amt` id from the site's own search form: `REGIONS` in the module, 2 is Storkøbenhavn) and `remote` (`helt` fully remote, `delvist` partly) |
| List | the RSS feed `jobbank.dk/job/rss/`, 100 items a page; an empty page ends it. It states no total and is not sorted by date |
| Pages read | at most the module's `ceiling`, which keeps a run under about 20 pages |
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by `enrich` for the roles an agent ranks high. The listing carries none |
| Date | `datePosted` from the role's page; trusted |
| Place | the region in the item's description line, and `jobLocation.addressLocality` from the page; country Denmark |
| Remote | not stated by a row; the search's `remote` filter narrows the list |
| Pay | none |
| Key | none |
| Limitations | Academic and research roles, plus some corporate ads. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Cloudflare may answer a challenge instead of the page, which is an error for that role and not an empty text. |
Jooble was left out: it gives a key only on a manual application and answered a Cloudflare challenge. Jobdanmark was left out too: about a quarter of its newest ads are on Jobnet, freehire carries the rest as source `jobdanmark`, and its list has no text. LinkedIn's guest pages are out because its robots.txt disallows `/` and its terms forbid automated access; LinkedIn enters only as alert mail. Indeed has no public search API.
## Sources that cannot be read, and why