Fold the search providers plan into the decisions and delete it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8a033e028d
commit
8c139f7f5a
3 files changed
+7
-76
No files matched your search
+3
-1
@@ -66,7 +66,9 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**An employer known only from a search gets a company row when it is first judged, and the Companies list derives that it is one.** Jobindex and Adzuna roles carry a `company_key` with no row in `companies` (67 keys with open roles on the live data of 2026-09-29), so they could not be hidden or liked. `Store.set_affinity` now creates a row for a key that has roles, named by the name its roles give most often, with nothing else set, and the reader draws the control for any role with a `company_key`. Such a row has no board, `ats` or `careers_url`, which the Companies list read as "board not found" and put under "Not readable yet". The rule: a company with no `ats` subscription (not even a detached one) and no `ats`, `careers_url` or `domain` is from job boards (`from_search` on `/api/companies`), labelled so and left out of that filter. Derived rather than a column, because every way of following a company writes one of those (the seed and discover a domain, Follow and `set-board` a board, which detaching hides but keeps), and a search-only company that later gets a domain or a board is followed from then on without a flag to flip.
|
||||
|
||||
**Jobindex and Adzuna, not Jobnet or Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jobnet's search endpoint answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
|
||||
**Jobindex and Adzuna first, not Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API. Jobnet was left out at first because its search answered 401; see the next paragraph.
|
||||
|
||||
**freehire, Jobnet and Akademikernes Jobbank are the fourth to sixth search providers (2026-10-03).** They came from comparing roledesk with github.com/MadsLorentzen/ai-job-search, which reads them. freehire.me is a public JSON API over about 50 hosts (no key, 300 requests a minute, robots.txt invites API use) whose search carries the full text; most of its Danish rows are employers roledesk does not follow, and a copy of a board role folds into the board's by the dupes rule, which now treats every `readers.PROVIDERS` name as a search host. Its query is an OR of words over title, company, text and place, so a search is narrowed by country. Jobnet's `/bff/FindJob/Search` answers 401 without the header `x-csrf: 1` and 200 with it, no login; it has no robots.txt and carries the full text. Jobbank is an RSS feed per search with the text on each ad's page as JSON-LD, so it is the first search reader with `detail`: enrich and the Fetch button read a search role's page through the board path (`plugins.ats.reader_of`), with the same bar, budgets and per-role state, and `Store.enrich_waiting` hands over search roles too. Its robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. On a copy of the live data, a first page of each gave 266 roles, of which 20 were copies of roles already read. Jobdanmark is left out: about a quarter of its ads are on Jobnet, freehire carries the rest, and its listing has no text. LinkedIn's guest endpoints are left out: robots.txt disallows `/` for every agent and the terms forbid automated access.
|
||||
|
||||
**Platsbanken is the third search provider, for Sweden.** The JobTech API (`jobsearch.api.jobtechdev.se/search`) needs no key, states a total, so completeness follows Adzuna's rule, and filters by the taxonomy id of a län (`platsbanken.REGIONS`; Skåne is `CaRE_1nn_cSU`). It refuses an offset above 2000, which the page limit never reaches. The ads carry no usable remote field, so `remote` is read off the headline. Free text ranks by all words rather than requiring them; a quoted phrase narrows it.
|
||||
|
||||
|
||||
+4
-1
@@ -40,7 +40,10 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la
|
||||
|
||||
- **Remote roles are not read from hosts that cannot filter on remote.** On SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" is asked as "DK, SE"; since 2026-09-30 the filtered read keeps the remote roles it has and the line says "remote not asked", but new remote roles on those 21 boards are never fetched, and the 72 closed on 2026-09-29 and 30 (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3) reopen only if a read lists them. Reading such boards whole is the owner's call; Bosch is about 70 requests a pull.
|
||||
- **SmartRecruiters' API host disallows `/` for every agent but LinkedInBot** (read 2026-09-30), which covers the listing and the detail requests; `READS.robots` says "owner to decide" and the eleven boards are read meanwhile. Jobylon allows `*` but disallows named job crawlers (JobrapidoBot, JoobleBot, AdzunaBot); ours is not one of them. HR Manager's record says the owner decided on 2026-09-29 to read the API host; confirm it.
|
||||
- **Provider names are hard-coded in nine places** (`cli.py`, `dupes.py`, `sources.py`, `pages.py`, `sources.html`, `sources.js`, `api.py`, `posting.py`) instead of `readers.PROVIDERS`, and `has_key` re-implements `resolve_key`.
|
||||
- **`has_key` re-implements `resolve_key`**, and the form hard-codes the labels of Jobindex, Adzuna and Platsbanken; since 2026-10-03 the CLI, dupes, `search_params` and the other providers in the form come from `readers.PROVIDERS`.
|
||||
- **A search role's detail places are not kept across pulls.** `plugins/search.py` has no `_KEPT_FROM_DETAIL`, so the next pull may overwrite the town Jobbank's `detail` added; the text is kept. Per-role detail state of a search is never pruned, since a search never lists everything.
|
||||
- **Jobbank's JSON-LD `addressLocality` is sometimes not a town** ("Ørsted" on an Ørsted ad), and the feed's regions (`Øresundsregionen`, `Udlandet (øvrige)`) will show under Unmapped.
|
||||
- **freehire states seniority and employment type, but the reader declares neither**, since a stated word needs a table in `app/prep/hosts/`. Jobnet keeps `utm_*` on employer links; dupes match on company, title and town, so it does not matter there.
|
||||
- **`ats.py` keeps compatibility names that only tests reach for:** `_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused` and `DETAIL_FIELDS`.
|
||||
- **`title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`.** Wave 2's WP9 must own `_runner.py` (`plan-hosts-wave-2.md`).
|
||||
- **Some `posted="trusted"` declarations rest on the detail read** (Workday, BambooHR, Rippling) **or on a fallback the host moves** (Greenhouse `updated_at`, Phenom `postedDate`).
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
# Plan: three more search providers (freehire, Jobnet, Akademikernes Jobbank)
|
||||
|
||||
Written 2026-10-03 after comparing roledesk with github.com/MadsLorentzen/ai-job-search, which reads five portals roledesk does not. Each was checked live on 2026-10-03; the raw answers are summarised below. Folded into `decisions.md` and deleted when the work lands.
|
||||
|
||||
## What is added and what is not
|
||||
|
||||
- **freehire.me**: a public JSON API (OpenAPI at `/openapi.yaml`, no key, 300 requests a minute on the search) over about 50 hosts. Denmark alone has 6,215 open rows from 66 sources, among them employers whose boards roledesk does not follow. The search answer carries the full text. robots.txt allows `/` for `*` and its comments invite API use with a descriptive user agent.
|
||||
- **Jobnet** (jobnet.dk, the state's portal): the SPA's backend `/bff/FindJob/Search` answers 401 without the header `x-csrf: 1` and 200 with it; no login. This supersedes the decision "Jobnet's search endpoint answers 401 outside its own site's session". No robots.txt (`/robots.txt` redirects to a login). The search answer carries the full HTML text. Mostly Danish public sector and SMEs; about a fifth of its ads link to an employer's ATS.
|
||||
- **Akademikernes Jobbank** (jobbank.dk): an RSS feed per search and a detail page with JSON-LD `JobPosting`. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Academic and research roles, plus corporate ads (LEO Pharma, Ørsted); no overlap with Jobindex in the sample.
|
||||
- **Not Jobdanmark.** About a quarter of its newest ads are on Jobnet too, freehire carries the rest as source `jobdanmark`, and it has no text in the listing (a 50 KB HTML page per role).
|
||||
- **Not LinkedIn's guest pages.** robots.txt is `User-agent: *` `Disallow: /` and the terms forbid automated access; the decision "LinkedIn enters only as alert mail" stands.
|
||||
|
||||
## The three readers
|
||||
|
||||
Each is a module in `plugins/readers/` with `scope="search"`, written like `platsbanken.py` (JSON) or `jobindex.py` (RSS), and a test file with fixtures under `tests/fixtures/search/` cut from a live answer. Raw answers from 2026-10-03 are in `/tmp/claude-1000/-home-claude-cboxcode-jobboard/44c787fe-f1eb-4e02-bfb2-010327cbd736/scratchpad/` (`fh*.json`, `jn*.json`, `jb*.xml`); a reader may make a few more polite requests to cut a fixture. Tests never touch the network.
|
||||
|
||||
The names below are the contract between the readers and WP4: WP4 imports them, so a reader must export them as written.
|
||||
|
||||
### freehire (`plugins/readers/freehire.py`)
|
||||
|
||||
- `NAME = "freehire"`, `LABEL = "freehire (tech, many countries)"`.
|
||||
- Search: `GET https://freehire.me/api/v1/agent/jobs/search` with `q`, `limit=100`, `offset`, `sort=posted_at`, `order=desc`, `description_format=html`, repeated `countries` (lower-case ISO), `work_mode`.
|
||||
- `READS`: `listing="paged"`, `page_size=100`, `ceiling=10000` (`offset + limit <= 10000`), `text="listing"`, `complete="stated"` (`meta.total`), `filters=Filters(query="one", country="many", remote="one")`, `places="many"`, `fields=Fields(pay="figures", posted="trusted", remote="stated", country="stated")` plus whatever `Fields` declares for seniority and employment if the row carries them, and a dated `robots` string.
|
||||
- `PARAMS = {"q": "query", "country": "country", "remote": "remote"}`. `clean_params` takes `q`, `country` as a comma-separated list of ISO codes (empty for all) and `remote` as `"remote"`, `"hybrid"`, `"onsite"` or `""`.
|
||||
- `MODES: dict[str, str]`, `work_mode` value -> label, for the form.
|
||||
- Rows: `source_id` the `public_slug`; `url` the record's `url` with `utm_source` removed; `places` from `cities`/`location`; `country` the first of `countries` when there is exactly one, else empty; `remote` from `work_mode`; pay from `enrichment` only when `salary_period` is a year; `description` through `html_text`. Keep the freehire `source` (e.g. `greenhouse`, `jobnet`) in `raw` so dupes and debugging can see where a row came from.
|
||||
- `q` is noisy: unquoted words are OR over title, company, text and place. Say so in the module docstring and the form hint.
|
||||
|
||||
### Jobnet (`plugins/readers/jobnet.py`)
|
||||
|
||||
- `NAME = "jobnet"`, `LABEL = "Jobnet (Denmark)"`.
|
||||
- Search: `GET https://jobnet.dk/bff/FindJob/Search` with header `x-csrf: 1`, `searchString`, `regions`, `orderType=PublicationDate`, `resultsPerPage=100`, `pageNumber` (1-indexed). Total in `totalJobAdCount`; page 101 at 100 a page answers empty.
|
||||
- `READS`: `listing="paged"`, `page_size=100`, `ceiling=10000`, `text="listing"`, `complete="stated"`, `filters=Filters(query="one", place="one")`, `places="one"`, `fields=Fields(posted="trusted", country="stated")`, a dated `robots` string naming the redirect and the header.
|
||||
- `PARAMS = {"q": "query", "region": "place"}`, `REGIONS: dict[str, str]` the five region ids (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`) -> Danish label; `""` is all of Denmark.
|
||||
- Rows: `source_id` the `jobAdId`; `url` the `jobAdUrl` when it is a web URL (the employer's own page) else `https://jobnet.dk/find-job/{jobAdId}`; place from `postalDistrictName`/`municipality`; `country` `DK` unless `country` names another (it is Danish: "Tyskland"); `posted` from `publicationDate`; `description` the HTML through `html_text`; `applicationDeadline` (with `1900-01-01` meaning none) only in `raw`, since `Row` has no deadline.
|
||||
- `x-csrf` goes through `Get(..., headers=...)`; if the runner's `Get` does not pass headers for a search reader, say so in the hand-back rather than working around it.
|
||||
|
||||
### Akademikernes Jobbank (`plugins/readers/jobbank.py`)
|
||||
|
||||
- `NAME = "jobbank"`, `LABEL = "Akademikernes Jobbank (Denmark)"`.
|
||||
- Search: `GET https://jobbank.dk/job/rss/?key=…&amt=…&fjernarbejde=…&page=N` (the trailing slash matters, without it 301). 100 items a page, an empty page ends it; the feed states no total and is not sorted by date.
|
||||
- `READS`: `listing="paged"`, `page_size=100`, a `ceiling` that keeps a run under about 20 pages, `text="detail"`, `complete="never"`, `filters=Filters(query="one", place="one")` and remote if the contract lets a search declare it, `places="one"`, `fields=Fields(posted="trusted", country="stated")`, a dated `robots` string quoting the `/job/` allow and the `/fwd/webjob/` disallow.
|
||||
- `PARAMS = {"q": "query", "region": "place", "remote": "remote"}` (drop `remote` if the contract cannot take it for a search), `REGIONS: dict[str, str]` the `amt` ids -> label as the site's own search form lists them (2 is Storkøbenhavn), `REMOTE: dict[str, str]` `{"helt": "Fully remote", "delvist": "Partly remote"}`.
|
||||
- `detail()` reads `/job/{id}/…` and takes the JSON-LD `JobPosting`: `description` through `html_text`, `datePosted`, `validThrough` into `raw`, `jobLocation.addressLocality`. It never follows `/fwd/webjob/`. Cloudflare may answer a challenge; a challenge is an error for that role, not an empty text.
|
||||
- Rows from the RSS item: `source_id` the numeric id in the link, `url` the item link, company and region parsed from the description line ("Fuldtidsjob hos LEO PHARMA A/S, Storkøbenhavn (Ansøgningsfrist: 18.10.2026)").
|
||||
|
||||
## Work packages
|
||||
|
||||
The first three own only their new files and can run in parallel. WP4 owns every shared file and builds against the names above, with stubs where a module is not merged yet; the planner merges WP1 to WP3 first, then WP4.
|
||||
|
||||
| WP | Files | Model |
|
||||
|---|---|---|
|
||||
| WP1 freehire | `plugins/readers/freehire.py`, `tests/test_readers_freehire.py`, `tests/fixtures/search/freehire-*` | Sonnet |
|
||||
| WP2 Jobnet | `plugins/readers/jobnet.py`, `tests/test_readers_jobnet.py`, `tests/fixtures/search/jobnet-*` | Sonnet |
|
||||
| WP3 Jobbank | `plugins/readers/jobbank.py`, `tests/test_readers_jobbank.py`, `tests/fixtures/search/jobbank-*` | Sonnet |
|
||||
| WP4 wiring | `app/cli.py`, `app/dupes.py`, `app/config.py`, `app/jobs/sources.py`, `app/web/pages.py`, `app/web/api.py`, `app/templates/settings/sources.html`, `app/static/sources.js`, `config.example.yaml`, `README.md`, `plugins/README.md`, tests of those | Sonnet |
|
||||
|
||||
WP4 in detail:
|
||||
|
||||
- Provider names come from `readers.PROVIDERS` instead of the lists in `app/cli.py` (`choices`) and `app/dupes.py` (`SEARCH_HOSTS`), which closes that item of `docs/open.md`.
|
||||
- `app searches add` gets `--country` reused for freehire (comma-separated), `--region` for Jobnet and Jobbank (help text naming the ids), `--remote` for freehire and Jobbank.
|
||||
- The Sources form gets the three providers in the select, with a hint per provider (freehire: words are OR, noisy, narrow by country; Jobnet: loose match, no phrases; Jobbank: words are OR) and the region and remote selects filled from the modules' `REGIONS`, `MODES` and `REMOTE` through `pages.py`, as Platsbanken's regions are today.
|
||||
- Dupes: a freehire row whose `url` is a board roledesk reads is a copy of that board's role; the existing rule keeps the board copy over a search copy, so `SEARCH_HOSTS` must contain the three new names. Check that the rule matches on the stripped URL or on company, title and town, and add a test with a freehire copy of a Greenhouse role.
|
||||
- README's Sources paragraph and `plugins/README.md` name the three providers; `config.example.yaml` shows one search each.
|
||||
|
||||
## Done when
|
||||
|
||||
- `uv run pytest` passes on main after the merges.
|
||||
- On a copy of the live database, not the live one, `app searches add` for one search per provider (freehire `q="head of data" country=dk`, Jobnet `q="data science" region=HovedstadenOgBornholm`, Jobbank `q="data director" region=2`) saves and a pull reads it.
|
||||
- `decisions.md` records the three providers, the Jobnet header, and why Jobdanmark and LinkedIn's guest pages are out; this plan is deleted.
|
||||
|
||||
## WP5: text for a search reader with `detail`
|
||||
|
||||
Added after WP3: Jobbank's feed has no text and nothing calls a search reader's `detail()`. `enrich.plan` reads only `source="ats"` subscriptions and `readers.HOSTS`, and `plugins.ats.read_board_details` looks the module up in `HOSTS`. WP5 lets enrich read the best roles of a search subscription whose reader has `detail` (the same bar, budgets and per-role state as a board), and makes the reader's Fetch button use `detail()` for such a role. Owns `app/jobs/enrich.py`, `plugins/ats.py`, `app/jobs/posting.py` and their tests; Sonnet.
|
||||
Reference in new issue
Block a user