From 018a1ad895f3ee3c7fe2e094f9f48dd5921b907a Mon Sep 17 00:00:00 2001 From: Wolf Date: Sat, 3 Oct 2026 12:12:49 +0200 Subject: [PATCH] Plan three more search providers: freehire, Jobnet and Akademikernes Jobbank Co-Authored-By: Claude Opus 5.5 --- docs/plan-search-sources.md | 70 +++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 docs/plan-search-sources.md diff --git a/docs/plan-search-sources.md b/docs/plan-search-sources.md new file mode 100644 index 0000000..525a38b --- /dev/null +++ b/docs/plan-search-sources.md @@ -0,0 +1,70 @@ +# Plan: three more search providers (freehire, Jobnet, Akademikernes Jobbank) + +Written 2026-10-03 after comparing roledesk with github.com/MadsLorentzen/ai-job-search, which reads five portals roledesk does not. Each was checked live on 2026-10-03; the raw answers are summarised below. Folded into `decisions.md` and deleted when the work lands. + +## What is added and what is not + +- **freehire.me**: a public JSON API (OpenAPI at `/openapi.yaml`, no key, 300 requests a minute on the search) over about 50 hosts. Denmark alone has 6,215 open rows from 66 sources, among them employers whose boards roledesk does not follow. The search answer carries the full text. robots.txt allows `/` for `*` and its comments invite API use with a descriptive user agent. +- **Jobnet** (jobnet.dk, the state's portal): the SPA's backend `/bff/FindJob/Search` answers 401 without the header `x-csrf: 1` and 200 with it; no login. This supersedes the decision "Jobnet's search endpoint answers 401 outside its own site's session". No robots.txt (`/robots.txt` redirects to a login). The search answer carries the full HTML text. Mostly Danish public sector and SMEs; about a fifth of its ads link to an employer's ATS. +- **Akademikernes Jobbank** (jobbank.dk): an RSS feed per search and a detail page with JSON-LD `JobPosting`. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Academic and research roles, plus corporate ads (LEO Pharma, Ørsted); no overlap with Jobindex in the sample. +- **Not Jobdanmark.** About a quarter of its newest ads are on Jobnet too, freehire carries the rest as source `jobdanmark`, and it has no text in the listing (a 50 KB HTML page per role). +- **Not LinkedIn's guest pages.** robots.txt is `User-agent: *` `Disallow: /` and the terms forbid automated access; the decision "LinkedIn enters only as alert mail" stands. + +## The three readers + +Each is a module in `plugins/readers/` with `scope="search"`, written like `platsbanken.py` (JSON) or `jobindex.py` (RSS), and a test file with fixtures under `tests/fixtures/search/` cut from a live answer. Raw answers from 2026-10-03 are in `/tmp/claude-1000/-home-claude-cboxcode-jobboard/44c787fe-f1eb-4e02-bfb2-010327cbd736/scratchpad/` (`fh*.json`, `jn*.json`, `jb*.xml`); a reader may make a few more polite requests to cut a fixture. Tests never touch the network. + +The names below are the contract between the readers and WP4: WP4 imports them, so a reader must export them as written. + +### freehire (`plugins/readers/freehire.py`) + +- `NAME = "freehire"`, `LABEL = "freehire (tech, many countries)"`. +- Search: `GET https://freehire.me/api/v1/agent/jobs/search` with `q`, `limit=100`, `offset`, `sort=posted_at`, `order=desc`, `description_format=html`, repeated `countries` (lower-case ISO), `work_mode`. +- `READS`: `listing="paged"`, `page_size=100`, `ceiling=10000` (`offset + limit <= 10000`), `text="listing"`, `complete="stated"` (`meta.total`), `filters=Filters(query="one", country="many", remote="one")`, `places="many"`, `fields=Fields(pay="figures", posted="trusted", remote="stated", country="stated")` plus whatever `Fields` declares for seniority and employment if the row carries them, and a dated `robots` string. +- `PARAMS = {"q": "query", "country": "country", "remote": "remote"}`. `clean_params` takes `q`, `country` as a comma-separated list of ISO codes (empty for all) and `remote` as `"remote"`, `"hybrid"`, `"onsite"` or `""`. +- `MODES: dict[str, str]`, `work_mode` value -> label, for the form. +- Rows: `source_id` the `public_slug`; `url` the record's `url` with `utm_source` removed; `places` from `cities`/`location`; `country` the first of `countries` when there is exactly one, else empty; `remote` from `work_mode`; pay from `enrichment` only when `salary_period` is a year; `description` through `html_text`. Keep the freehire `source` (e.g. `greenhouse`, `jobnet`) in `raw` so dupes and debugging can see where a row came from. +- `q` is noisy: unquoted words are OR over title, company, text and place. Say so in the module docstring and the form hint. + +### Jobnet (`plugins/readers/jobnet.py`) + +- `NAME = "jobnet"`, `LABEL = "Jobnet (Denmark)"`. +- Search: `GET https://jobnet.dk/bff/FindJob/Search` with header `x-csrf: 1`, `searchString`, `regions`, `orderType=PublicationDate`, `resultsPerPage=100`, `pageNumber` (1-indexed). Total in `totalJobAdCount`; page 101 at 100 a page answers empty. +- `READS`: `listing="paged"`, `page_size=100`, `ceiling=10000`, `text="listing"`, `complete="stated"`, `filters=Filters(query="one", place="one")`, `places="one"`, `fields=Fields(posted="trusted", country="stated")`, a dated `robots` string naming the redirect and the header. +- `PARAMS = {"q": "query", "region": "place"}`, `REGIONS: dict[str, str]` the five region ids (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`) -> Danish label; `""` is all of Denmark. +- Rows: `source_id` the `jobAdId`; `url` the `jobAdUrl` when it is a web URL (the employer's own page) else `https://jobnet.dk/find-job/{jobAdId}`; place from `postalDistrictName`/`municipality`; `country` `DK` unless `country` names another (it is Danish: "Tyskland"); `posted` from `publicationDate`; `description` the HTML through `html_text`; `applicationDeadline` (with `1900-01-01` meaning none) only in `raw`, since `Row` has no deadline. +- `x-csrf` goes through `Get(..., headers=...)`; if the runner's `Get` does not pass headers for a search reader, say so in the hand-back rather than working around it. + +### Akademikernes Jobbank (`plugins/readers/jobbank.py`) + +- `NAME = "jobbank"`, `LABEL = "Akademikernes Jobbank (Denmark)"`. +- Search: `GET https://jobbank.dk/job/rss/?key=…&amt=…&fjernarbejde=…&page=N` (the trailing slash matters, without it 301). 100 items a page, an empty page ends it; the feed states no total and is not sorted by date. +- `READS`: `listing="paged"`, `page_size=100`, a `ceiling` that keeps a run under about 20 pages, `text="detail"`, `complete="never"`, `filters=Filters(query="one", place="one")` and remote if the contract lets a search declare it, `places="one"`, `fields=Fields(posted="trusted", country="stated")`, a dated `robots` string quoting the `/job/` allow and the `/fwd/webjob/` disallow. +- `PARAMS = {"q": "query", "region": "place", "remote": "remote"}` (drop `remote` if the contract cannot take it for a search), `REGIONS: dict[str, str]` the `amt` ids -> label as the site's own search form lists them (2 is Storkøbenhavn), `REMOTE: dict[str, str]` `{"helt": "Fully remote", "delvist": "Partly remote"}`. +- `detail()` reads `/job/{id}/…` and takes the JSON-LD `JobPosting`: `description` through `html_text`, `datePosted`, `validThrough` into `raw`, `jobLocation.addressLocality`. It never follows `/fwd/webjob/`. Cloudflare may answer a challenge; a challenge is an error for that role, not an empty text. +- Rows from the RSS item: `source_id` the numeric id in the link, `url` the item link, company and region parsed from the description line ("Fuldtidsjob hos LEO PHARMA A/S, Storkøbenhavn (Ansøgningsfrist: 18.10.2026)"). + +## Work packages + +The first three own only their new files and can run in parallel. WP4 owns every shared file and builds against the names above, with stubs where a module is not merged yet; the planner merges WP1 to WP3 first, then WP4. + +| WP | Files | Model | +|---|---|---| +| WP1 freehire | `plugins/readers/freehire.py`, `tests/test_readers_freehire.py`, `tests/fixtures/search/freehire-*` | Sonnet | +| WP2 Jobnet | `plugins/readers/jobnet.py`, `tests/test_readers_jobnet.py`, `tests/fixtures/search/jobnet-*` | Sonnet | +| WP3 Jobbank | `plugins/readers/jobbank.py`, `tests/test_readers_jobbank.py`, `tests/fixtures/search/jobbank-*` | Sonnet | +| WP4 wiring | `app/cli.py`, `app/dupes.py`, `app/config.py`, `app/jobs/sources.py`, `app/web/pages.py`, `app/web/api.py`, `app/templates/settings/sources.html`, `app/static/sources.js`, `config.example.yaml`, `README.md`, `plugins/README.md`, tests of those | Sonnet | + +WP4 in detail: + +- Provider names come from `readers.PROVIDERS` instead of the lists in `app/cli.py` (`choices`) and `app/dupes.py` (`SEARCH_HOSTS`), which closes that item of `docs/open.md`. +- `app searches add` gets `--country` reused for freehire (comma-separated), `--region` for Jobnet and Jobbank (help text naming the ids), `--remote` for freehire and Jobbank. +- The Sources form gets the three providers in the select, with a hint per provider (freehire: words are OR, noisy, narrow by country; Jobnet: loose match, no phrases; Jobbank: words are OR) and the region and remote selects filled from the modules' `REGIONS`, `MODES` and `REMOTE` through `pages.py`, as Platsbanken's regions are today. +- Dupes: a freehire row whose `url` is a board roledesk reads is a copy of that board's role; the existing rule keeps the board copy over a search copy, so `SEARCH_HOSTS` must contain the three new names. Check that the rule matches on the stripped URL or on company, title and town, and add a test with a freehire copy of a Greenhouse role. +- README's Sources paragraph and `plugins/README.md` name the three providers; `config.example.yaml` shows one search each. + +## Done when + +- `uv run pytest` passes on main after the merges. +- On a copy of the live database, not the live one, `app searches add` for one search per provider (freehire `q="head of data" country=dk`, Jobnet `q="data science" region=HovedstadenOgBornholm`, Jobbank `q="data director" region=2`) saves and a pull reads it. +- `decisions.md` records the three providers, the Jobnet header, and why Jobdanmark and LinkedIn's guest pages are out; this plan is deleted.