Files
jobboard/docs/plan-search-sources.md
T

9.5 KiB

Plan: three more search providers (freehire, Jobnet, Akademikernes Jobbank)

Written 2026-10-03 after comparing roledesk with github.com/MadsLorentzen/ai-job-search, which reads five portals roledesk does not. Each was checked live on 2026-10-03; the raw answers are summarised below. Folded into decisions.md and deleted when the work lands.

What is added and what is not

  • freehire.me: a public JSON API (OpenAPI at /openapi.yaml, no key, 300 requests a minute on the search) over about 50 hosts. Denmark alone has 6,215 open rows from 66 sources, among them employers whose boards roledesk does not follow. The search answer carries the full text. robots.txt allows / for * and its comments invite API use with a descriptive user agent.
  • Jobnet (jobnet.dk, the state's portal): the SPA's backend /bff/FindJob/Search answers 401 without the header x-csrf: 1 and 200 with it; no login. This supersedes the decision "Jobnet's search endpoint answers 401 outside its own site's session". No robots.txt (/robots.txt redirects to a login). The search answer carries the full HTML text. Mostly Danish public sector and SMEs; about a fifth of its ads link to an employer's ATS.
  • Akademikernes Jobbank (jobbank.dk): an RSS feed per search and a detail page with JSON-LD JobPosting. robots.txt allows /job/ and disallows /fwd/webjob/, which is never followed. Academic and research roles, plus corporate ads (LEO Pharma, Ørsted); no overlap with Jobindex in the sample.
  • Not Jobdanmark. About a quarter of its newest ads are on Jobnet too, freehire carries the rest as source jobdanmark, and it has no text in the listing (a 50 KB HTML page per role).
  • Not LinkedIn's guest pages. robots.txt is User-agent: * Disallow: / and the terms forbid automated access; the decision "LinkedIn enters only as alert mail" stands.

The three readers

Each is a module in plugins/readers/ with scope="search", written like platsbanken.py (JSON) or jobindex.py (RSS), and a test file with fixtures under tests/fixtures/search/ cut from a live answer. Raw answers from 2026-10-03 are in /tmp/claude-1000/-home-claude-cboxcode-jobboard/44c787fe-f1eb-4e02-bfb2-010327cbd736/scratchpad/ (fh*.json, jn*.json, jb*.xml); a reader may make a few more polite requests to cut a fixture. Tests never touch the network.

The names below are the contract between the readers and WP4: WP4 imports them, so a reader must export them as written.

freehire (plugins/readers/freehire.py)

  • NAME = "freehire", LABEL = "freehire (tech, many countries)".
  • Search: GET https://freehire.me/api/v1/agent/jobs/search with q, limit=100, offset, sort=posted_at, order=desc, description_format=html, repeated countries (lower-case ISO), work_mode.
  • READS: listing="paged", page_size=100, ceiling=10000 (offset + limit <= 10000), text="listing", complete="stated" (meta.total), filters=Filters(query="one", country="many", remote="one"), places="many", fields=Fields(pay="figures", posted="trusted", remote="stated", country="stated") plus whatever Fields declares for seniority and employment if the row carries them, and a dated robots string.
  • PARAMS = {"q": "query", "country": "country", "remote": "remote"}. clean_params takes q, country as a comma-separated list of ISO codes (empty for all) and remote as "remote", "hybrid", "onsite" or "".
  • MODES: dict[str, str], work_mode value -> label, for the form.
  • Rows: source_id the public_slug; url the record's url with utm_source removed; places from cities/location; country the first of countries when there is exactly one, else empty; remote from work_mode; pay from enrichment only when salary_period is a year; description through html_text. Keep the freehire source (e.g. greenhouse, jobnet) in raw so dupes and debugging can see where a row came from.
  • q is noisy: unquoted words are OR over title, company, text and place. Say so in the module docstring and the form hint.

Jobnet (plugins/readers/jobnet.py)

  • NAME = "jobnet", LABEL = "Jobnet (Denmark)".
  • Search: GET https://jobnet.dk/bff/FindJob/Search with header x-csrf: 1, searchString, regions, orderType=PublicationDate, resultsPerPage=100, pageNumber (1-indexed). Total in totalJobAdCount; page 101 at 100 a page answers empty.
  • READS: listing="paged", page_size=100, ceiling=10000, text="listing", complete="stated", filters=Filters(query="one", place="one"), places="one", fields=Fields(posted="trusted", country="stated"), a dated robots string naming the redirect and the header.
  • PARAMS = {"q": "query", "region": "place"}, REGIONS: dict[str, str] the five region ids (HovedstadenOgBornholm, Midtjylland, Syddanmark, OevrigeSjaelland, Nordjylland) -> Danish label; "" is all of Denmark.
  • Rows: source_id the jobAdId; url the jobAdUrl when it is a web URL (the employer's own page) else https://jobnet.dk/find-job/{jobAdId}; place from postalDistrictName/municipality; country DK unless country names another (it is Danish: "Tyskland"); posted from publicationDate; description the HTML through html_text; applicationDeadline (with 1900-01-01 meaning none) only in raw, since Row has no deadline.
  • x-csrf goes through Get(..., headers=...); if the runner's Get does not pass headers for a search reader, say so in the hand-back rather than working around it.

Akademikernes Jobbank (plugins/readers/jobbank.py)

  • NAME = "jobbank", LABEL = "Akademikernes Jobbank (Denmark)".
  • Search: GET https://jobbank.dk/job/rss/?key=…&amt=…&fjernarbejde=…&page=N (the trailing slash matters, without it 301). 100 items a page, an empty page ends it; the feed states no total and is not sorted by date.
  • READS: listing="paged", page_size=100, a ceiling that keeps a run under about 20 pages, text="detail", complete="never", filters=Filters(query="one", place="one") and remote if the contract lets a search declare it, places="one", fields=Fields(posted="trusted", country="stated"), a dated robots string quoting the /job/ allow and the /fwd/webjob/ disallow.
  • PARAMS = {"q": "query", "region": "place", "remote": "remote"} (drop remote if the contract cannot take it for a search), REGIONS: dict[str, str] the amt ids -> label as the site's own search form lists them (2 is Storkøbenhavn), REMOTE: dict[str, str] {"helt": "Fully remote", "delvist": "Partly remote"}.
  • detail() reads /job/{id}/… and takes the JSON-LD JobPosting: description through html_text, datePosted, validThrough into raw, jobLocation.addressLocality. It never follows /fwd/webjob/. Cloudflare may answer a challenge; a challenge is an error for that role, not an empty text.
  • Rows from the RSS item: source_id the numeric id in the link, url the item link, company and region parsed from the description line ("Fuldtidsjob hos LEO PHARMA A/S, Storkøbenhavn (Ansøgningsfrist: 18.10.2026)").

Work packages

The first three own only their new files and can run in parallel. WP4 owns every shared file and builds against the names above, with stubs where a module is not merged yet; the planner merges WP1 to WP3 first, then WP4.

WP Files Model
WP1 freehire plugins/readers/freehire.py, tests/test_readers_freehire.py, tests/fixtures/search/freehire-* Sonnet
WP2 Jobnet plugins/readers/jobnet.py, tests/test_readers_jobnet.py, tests/fixtures/search/jobnet-* Sonnet
WP3 Jobbank plugins/readers/jobbank.py, tests/test_readers_jobbank.py, tests/fixtures/search/jobbank-* Sonnet
WP4 wiring app/cli.py, app/dupes.py, app/config.py, app/jobs/sources.py, app/web/pages.py, app/web/api.py, app/templates/settings/sources.html, app/static/sources.js, config.example.yaml, README.md, plugins/README.md, tests of those Sonnet

WP4 in detail:

  • Provider names come from readers.PROVIDERS instead of the lists in app/cli.py (choices) and app/dupes.py (SEARCH_HOSTS), which closes that item of docs/open.md.
  • app searches add gets --country reused for freehire (comma-separated), --region for Jobnet and Jobbank (help text naming the ids), --remote for freehire and Jobbank.
  • The Sources form gets the three providers in the select, with a hint per provider (freehire: words are OR, noisy, narrow by country; Jobnet: loose match, no phrases; Jobbank: words are OR) and the region and remote selects filled from the modules' REGIONS, MODES and REMOTE through pages.py, as Platsbanken's regions are today.
  • Dupes: a freehire row whose url is a board roledesk reads is a copy of that board's role; the existing rule keeps the board copy over a search copy, so SEARCH_HOSTS must contain the three new names. Check that the rule matches on the stripped URL or on company, title and town, and add a test with a freehire copy of a Greenhouse role.
  • README's Sources paragraph and plugins/README.md name the three providers; config.example.yaml shows one search each.

Done when

  • uv run pytest passes on main after the merges.
  • On a copy of the live database, not the live one, app searches add for one search per provider (freehire q="head of data" country=dk, Jobnet q="data science" region=HovedstadenOgBornholm, Jobbank q="data director" region=2) saves and a pull reads it.
  • decisions.md records the three providers, the Jobnet header, and why Jobdanmark and LinkedIn's guest pages are out; this plan is deleted.