Merge the wiring of the new search providers (wp-src-wiring)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-10-03 12:24:56 +02:00
commit a9c387c130
16 files changed
+422 -39

No files matched your search

+6 -3
View File
@@ -1,6 +1,6 @@
# roledesk
A job-search engine for one person. It reads the public job boards of the companies you follow (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor, Workday, SuccessFactors, Oracle, Phenom, Eightfold and nine smaller hosts, one module each in `plugins/readers/`) and runs saved searches on boards that span companies (Jobindex, Adzuna, Platsbanken), keeps every posting with when it was first seen and when it closed, scores the open roles for each of your search agents, and tracks the applications you make.
A job-search engine for one person. It reads the public job boards of the companies you follow (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor, Workday, SuccessFactors, Oracle, Phenom, Eightfold and nine smaller hosts, one module each in `plugins/readers/`) and runs saved searches on boards that span companies (Jobindex, Adzuna, Platsbanken, freehire, Jobnet, Akademikernes Jobbank), keeps every posting with when it was first seen and when it closed, scores the open roles for each of your search agents, and tracks the applications you make.
Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeline/`), password (`auth/`) and deploy kit (`deploy/`), plus the plugin contract and tools from news (`desk/`). FastAPI, Jinja2, SQLite, `uv`, no build step.
@@ -22,6 +22,9 @@ The Sources sub-tab does all of this; the same actions exist on the command line
```bash
uv run app searches add jobindex "Copenhagen golang" --q "golang go-udvikler" --area 1
uv run app searches add adzuna "Berlin backend" --q backend --country de --where Berlin
uv run app searches add freehire "Head of data" --q "head of data" --country dk
uv run app searches add jobnet "Data science" --q "data science" --region HovedstadenOgBornholm
uv run app searches add jobbank "Data director" --q "data director" --region 2
uv run app searches list
uv run app companies discover "Pleo" pleo.io --add # find its board on its own site, then follow it
uv run app companies set-board lunar greenhouse:lunar # attach a board to a company, or replace the one it has
@@ -41,7 +44,7 @@ The seed is a hand-picked list of European tech employers, not a list of boards:
`app stages` shows or switches extract, summarise and enrich, `app seniority` shows per host how often the stated level equals the model's, `app dupes` recomputes which open roles are copies of one posting, and `app prep` maps the host words and folds the copies without a pull.
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. A freehire query is free text over about 50 hosts whose words are OR over title, company, text and place, so it is noisy: `--country` takes comma-separated ISO codes (`dk,se`) and `--remote` a work mode. A Jobnet query is a loose match without phrases and `--region` one of its five regions (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`). An Akademikernes Jobbank query is OR words, `--region` its `amt` id (2 is Storkøbenhavn) and `--remote` `helt` or `delvist`. A freehire role whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
## The tabs
@@ -58,7 +61,7 @@ The cog opens `/settings`: a rail of pages on the left, the page beside it, stic
| | |
|---|---|
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/prefer rules as rows, fetch terms, weights and limits); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna or Platsbanken: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise and enrich, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise and enrich, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
| **Account** | the password, sessions and passkeys. Open on loopback, locked on anything else |
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its `taste` weight scores a role by how close it is to the roles the agent liked minus the ones it passed.
+24 -4
View File
@@ -46,8 +46,19 @@ if str(_ROOT) not in sys.path:
sys.path.insert(0, str(_ROOT))
def _ids(provider: str, table: str, lead: str) -> str:
"""`lead` and the `id label` pairs of a search reader's `REGIONS`, `MODES` or `REMOTE`
for a help text, or nothing when the reader is not loaded or has no such table."""
from app import jobs
pairs = ", ".join(f"{key or 'empty'} {label}".replace("%", "%%")
for key, label in jobs.sources.provider_table(provider, table).items())
return f"{lead}{pairs}" if pairs else ""
def build_parser() -> argparse.ArgumentParser:
from app.models import HAND_REASONS
from plugins import readers
parser = argparse.ArgumentParser(
prog="app", description="roledesk: a job-search engine on desk and the pipeline queue.")
@@ -128,17 +139,25 @@ def build_parser() -> argparse.ArgumentParser:
searches_sub = searches.add_subparsers(dest="action", required=True)
search_add = searches_sub.add_parser(
"add", help="run a search once, then save it for every pull (or update it)")
search_add.add_argument("provider", choices=["jobindex", "adzuna", "platsbanken"])
search_add.add_argument("provider", choices=list(readers.PROVIDERS))
search_add.add_argument("name", help='what to call it, e.g. "Copenhagen backend"')
search_add.add_argument("--q", required=True,
help="the query, in the provider's own syntax")
search_add.add_argument("--area", type=int, default=0,
help="jobindex geoareaid: 1 Region Hovedstaden, 8 Aarhus")
search_add.add_argument("--country", default="",
help="adzuna country: gb, de, nl, fr, at, be, es, it, pl, ch")
help="adzuna country: gb, de, nl, fr, at, be, es, it, pl, ch; "
"freehire countries, comma-separated ISO codes: dk,se, "
"empty all")
search_add.add_argument("--where", default="", help="adzuna place, e.g. Berlin")
search_add.add_argument("--region", default="",
help="platsbanken län id: CaRE_1nn_cSU is Skåne, empty all of Sweden")
help="platsbanken län id: CaRE_1nn_cSU is Skåne, empty all of "
"Sweden" + _ids("jobnet", "REGIONS", "; jobnet region")
+ _ids("jobbank", "REGIONS", "; jobbank amt"))
search_add.add_argument("--remote", default="",
help="freehire work mode" + _ids("freehire", "MODES", ": ")
+ _ids("jobbank", "REMOTE", "; jobbank remote work: ")
+ ", empty any")
search_add.add_argument("--pages", type=int, default=0,
help="result pages per pull (default 5, at most 10)")
searches_sub.add_parser("list", help="every saved search and its open roles")
@@ -608,7 +627,8 @@ def add_search(config: config_mod.Config, store: Store, args: argparse.Namespace
try:
params = jobs.sources.search_params(args.provider, args.q, area=args.area,
country=args.country, where=args.where,
region=args.region, pages=args.pages)
region=args.region, remote=args.remote,
pages=args.pages)
sub, found, complete = jobs.sources.save_search(config, store, args.provider, args.name,
params)
except ValueError as exc:
+3 -1
View File
@@ -166,7 +166,9 @@ class SearchSeed(Section):
"""A saved search as `config.yaml` lists it: one `search` subscription,
imported once like the agents. `params` are the provider's own, see
plugins/search.py: `q`, `area` and `pages` for jobindex; `q`, `country`,
`where` and `pages` for adzuna; `q` and `region` for platsbanken. The provider is any reader of
`where` and `pages` for adzuna; `q` and `region` for platsbanken; `q`,
`country` and `remote` for freehire; `q` and `region` for jobnet; `q`,
`region` and `remote` for jobbank. The provider is any reader of
`scope="search"`, so a new one needs no edit here."""
name: str
+18 -5
View File
@@ -37,18 +37,31 @@ the strings boards really write and the storage layer only feeds it rows.
from __future__ import annotations
import re
import sys
from collections.abc import Iterable
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
from typing import NamedTuple
from app import places
from app.models import country_of
#: Hosts that search across companies rather than run a company's own board.
#: A board closes a role reliably and carries the full text, so it is kept
#: over a copy found here.
SEARCH_HOSTS: frozenset[str] = frozenset({"jobindex", "adzuna", "platsbanken"})
def is_search_host(host: str) -> bool:
"""Whether `host` is a provider that searches across companies rather than
a company's own board: a reader of `scope="search"`. A board closes a role
reliably and carries the full text, so it is kept over a copy found here.
`plugins` is a top-level folder beside `app`, reached lazily, with the
repo root put on the path when it is not there yet, as
`app.models._declared_posted` does."""
try:
from plugins import readers
except ImportError:
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from plugins import readers
return host in readers.PROVIDERS
@dataclass(frozen=True)
@@ -101,7 +114,7 @@ def keep_order(c: Copy) -> tuple[bool, bool, bool, str, str]:
board over a search provider, then one with a description, then the
earliest `posted_at` (none last), then the smallest `uid`, so the choice
is the same on every run."""
return (c.host in SEARCH_HOSTS, not c.has_description, not c.posted_at,
return (is_search_host(c.host), not c.has_description, not c.posted_at,
c.posted_at or "", c.uid)
+13 -8
View File
@@ -26,9 +26,6 @@ from plugins import readers
if TYPE_CHECKING:
from app.jobs.discover import Discovery
#: The providers, from the reader modules that declare `scope="search"`.
PROVIDERS = tuple(readers.PROVIDERS)
def set_fetch(store: Store, sub_id: int, fetch: bool) -> tuple[Subscription, int]:
"""Switch one board or search on or off. Off closes its open roles and
@@ -356,20 +353,28 @@ def seed_companies(store: Store, path: str | Path) -> dict[str, Any]:
"problems": problems}
def provider_table(provider: str, name: str) -> dict[str, str]:
"""The table `name` (`REGIONS`, `MODES` or `REMOTE`) of a search reader, as
the form and the CLI list its values; empty when the provider is not
loaded or declares none, so a page works before a reader is merged."""
return dict(getattr(readers.PROVIDERS.get(provider), name, None) or {})
def search_params(provider: str, q: str, *, area: int = 0, country: str = "",
where: str = "", region: str = "", pages: int = 0) -> dict[str, Any]:
where: str = "", region: str = "", remote: str = "",
pages: int = 0) -> dict[str, Any]:
"""The `params` a search subscription stores, with only the keys its
provider reads: those of its module's `PARAMS` whose filter dimension
`READS.filters` declares, cleaned by the module's `clean_params`, which
refuses a value the provider would not accept."""
if provider not in PROVIDERS:
raise ValueError(f"provider is one of {', '.join(PROVIDERS)}")
if provider not in readers.PROVIDERS:
raise ValueError(f"provider is one of {', '.join(readers.PROVIDERS)}")
if not q.strip():
raise ValueError("a search needs a query")
module = readers.PROVIDERS[provider]
declared = module.READS.filters.declared()
given: dict[str, Any] = {"q": q.strip(), "area": area, "country": country,
"where": where, "region": region}
"where": where, "region": region, "remote": remote}
wanted = {name: given[name] for name, dimension in module.PARAMS.items()
if dimension in declared}
params = module.clean_params(wanted)
@@ -379,7 +384,7 @@ def search_params(provider: str, q: str, *, area: int = 0, country: str = "",
def has_key(config: Config, provider: str) -> bool:
"""Whether `provider` can run: jobindex and platsbanken always, adzuna with a key in the
"""Whether `provider` can run: every provider but adzuna always, adzuna with a key in the
config or the environment."""
if provider != "adzuna":
return True
+20 -7
View File
@@ -1,7 +1,8 @@
/* The Job boards section of Sources: one query on one board per row, on
* Jobindex, Adzuna or Platsbanken. The UI says "job board" for what the API and the CLI
* call a search (`/api/searches`, `app searches`); company boards are
* companies.js's section, since a board is a company's own, not a query's.
* Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank.
* The UI says "job board" for what the API and the CLI call a search
* (`/api/searches`, `app searches`); company boards are companies.js's
* section, since a board is a company's own, not a query's.
*
* Edit turns a row into the form from the page's `<template>`, in place, and
* New job board puts a blank form on top of the list, as agents.js does for
@@ -172,7 +173,13 @@
$("#s-area", f).value = String(p.area || (source ? 0 : 1));
$("#s-country", f).value = p.country || "de";
$("#s-where", f).value = p.where || "";
$("#s-region", f).value = p.region || "";
const of = (provider) => source && source.provider === provider;
$("#s-region", f).value = of("platsbanken") ? p.region || "" : "";
$("#s-countries", f).value = of("freehire") ? p.country || "" : "";
$("#s-mode", f).value = of("freehire") ? p.remote || "" : "";
$("#s-region-jobnet", f).value = of("jobnet") ? String(p.region || "") : "";
$("#s-region-jobbank", f).value = of("jobbank") ? String(p.region || "") : "";
$("#s-remote-jobbank", f).value = of("jobbank") ? p.remote || "" : "";
$("#s-pages", f).value = String(p.pages || 10);
showProvider(f);
f.addEventListener("input", () => { f.dataset.dirty = "true"; });
@@ -209,15 +216,21 @@
$("#s-q", editor).focus({ preventScroll: true });
}
// Region, country and remote word each have one control per provider that
// reads them; the body takes the shown provider's, and "" where it has none.
function searchBody(f) {
const provider = $("#s-provider", f).value;
const own = (controls) => (controls[provider] ? $(controls[provider], f).value.trim() : "");
return {
provider: $("#s-provider", f).value,
provider,
name: $("#s-name", f).value.trim(),
q: $("#s-q", f).value.trim(),
area: Number($("#s-area", f).value) || 0,
country: $("#s-country", f).value,
country: own({ adzuna: "#s-country", freehire: "#s-countries" }),
where: $("#s-where", f).value.trim(),
region: $("#s-region", f).value,
region: own({ platsbanken: "#s-region", jobnet: "#s-region-jobnet",
jobbank: "#s-region-jobbank" }),
remote: own({ freehire: "#s-mode", jobbank: "#s-remote-jobbank" }),
pages: Number($("#s-pages", f).value) || 0,
};
}
+28 -2
View File
@@ -19,7 +19,7 @@
top of the list on New, as agents.js does for the Search agents cards. #}
<section class="card" id="job-boards" data-section>
<h2>Job boards</h2>
<p class="lede">A query on Jobindex, Adzuna or Platsbanken finds roles at any company. Switching one off closes its roles until you switch it back on.</p>
<p class="lede">A query on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank finds roles at any company. Switching one off closes its roles until you switch it back on.</p>
<div class="agents-head">
<p id="searches-count" class="count" role="status"></p>
@@ -38,6 +38,7 @@
<option value="jobindex">Jobindex (Denmark)</option>
<option value="adzuna">Adzuna (UK and EU)</option>
<option value="platsbanken">Platsbanken (Sweden)</option>
{% for value, label in providers %}<option value="{{ value }}">{{ label }}</option>{% endfor %}
</select></label>
<label>Name <input type="text" id="s-name" required placeholder="Copenhagen backend"></label>
</div>
@@ -45,7 +46,10 @@
<input type="text" id="s-q" required spellcheck="false">
<span class="hint" data-provider="jobindex">Jobindex syntax: words are OR, <code>"a b"</code> is a phrase, <code>+word</code> is required.</span>
<span class="hint" data-provider="adzuna" hidden>Every word must appear.</span>
<span class="hint" data-provider="platsbanken" hidden>Free text; every word counts towards the match.</span></label>
<span class="hint" data-provider="platsbanken" hidden>Free text; every word counts towards the match.</span>
<span class="hint" data-provider="freehire" hidden>Words are OR over title, company, text and place, so a query is noisy; narrow it by country.</span>
<span class="hint" data-provider="jobnet" hidden>A loose match: words, no phrases.</span>
<span class="hint" data-provider="jobbank" hidden>Words are OR.</span></label>
<div class="form-row">
<label data-provider="jobindex">Area
<select id="s-area">
@@ -61,6 +65,28 @@
<option value="">All of Sweden</option>
{% for value, label in regions %}<option value="{{ value }}">{{ label }}</option>{% endfor %}
</select></label>
<label data-provider="freehire" hidden>Countries <input type="text" id="s-countries" placeholder="dk,se" spellcheck="false">
<span class="hint">ISO codes, comma-separated; empty for all</span></label>
<label data-provider="freehire" hidden>Work mode
<select id="s-mode">
<option value="">Any</option>
{% for value, label in modes %}<option value="{{ value }}">{{ label }}</option>{% endfor %}
</select></label>
<label data-provider="jobnet" hidden>Region
<select id="s-region-jobnet">
<option value="">All of Denmark</option>
{% for value, label in jobnet_regions %}<option value="{{ value }}">{{ label }}</option>{% endfor %}
</select></label>
<label data-provider="jobbank" hidden>Region
<select id="s-region-jobbank">
<option value="">All of Denmark</option>
{% for value, label in jobbank_regions %}<option value="{{ value }}">{{ label }}</option>{% endfor %}
</select></label>
<label data-provider="jobbank" hidden>Remote work
<select id="s-remote-jobbank">
<option value="">Any</option>
{% for value, label in jobbank_remote %}<option value="{{ value }}">{{ label }}</option>{% endfor %}
</select></label>
<label>Pages
<input type="number" id="s-pages" min="1" max="10" value="10">
<span class="hint">per pull, up to 10</span></label>
+3 -1
View File
@@ -1043,6 +1043,7 @@ class SearchBody(BaseModel):
country: str = ""
where: str = ""
region: str = ""
remote: str = ""
pages: Annotated[int, Field(ge=0, le=10)] = 0
@@ -1050,7 +1051,8 @@ def _params(body: SearchBody) -> dict[str, Any]:
try:
return jobs.sources.search_params(body.provider, body.q, area=body.area,
country=body.country, where=body.where,
region=body.region, pages=body.pages)
region=body.region, remote=body.remote,
pages=body.pages)
except ValueError as exc:
raise HTTPException(422, str(exc)) from exc
+28 -3
View File
@@ -27,8 +27,11 @@ keeps working.
functions are the classifier's vocabulary and the hosts are what it parses,
and this reaches `plugins/` the same way `app.cli` does, since a plugin is
app-agnostic and never reaches back into `app`. The
Jobindex areas, Adzuna countries and Platsbanken regions on the Sources page
come from the search readers in `plugins/readers/` for the same reason.
Jobindex areas, Adzuna countries, the regions of Platsbanken, Jobnet and
Jobbank, freehire's work modes and Jobbank's remote words on the Sources page
come from the search readers in `plugins/readers/` for the same reason. The
last four and the select's other providers are read from `readers.PROVIDERS`,
so the page works whether or not a provider's module is loaded.
"""
from __future__ import annotations
@@ -41,6 +44,7 @@ from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates
from app import jobs
from app.models import LADDER, REMOTE, RUNGS, STATUSES
from app.web.deps import StoreDep
@@ -48,6 +52,7 @@ _ROOT = Path(__file__).resolve().parents[2]
if str(_ROOT) not in sys.path:
sys.path.insert(0, str(_ROOT))
from plugins import readers # noqa: E402
from plugins.ats import FUNCTIONS, HOSTS # noqa: E402
from plugins.readers.adzuna import COUNTRIES as ADZUNA_COUNTRIES # noqa: E402
from plugins.readers.jobindex import AREAS as JOBINDEX_AREAS # noqa: E402
@@ -110,6 +115,25 @@ HOST_OPTIONS = list(HOSTS.items())
LADDER_JSON = {"rungs": list(RUNGS), "levels": {k: list(v) for k, v in LADDER.items()}}
def search_options() -> dict[str, list[tuple[str, str]]]:
"""The `value, label` options the Sources form's providers add to its
selects, read from `readers.PROVIDERS` on each request: the providers the
template does not name itself, and the regions of Jobnet and Jobbank,
freehire's work modes and Jobbank's remote words, each in the reader's
order and without its empty value, which the form puts first as "all". All
empty for a provider whose module is not loaded."""
def options(provider: str, table: str) -> list[tuple[str, str]]:
return [(value, label) for value, label in jobs.sources.provider_table(provider, table).items()
if value]
return {"providers": [(n, m.LABEL) for n, m in readers.PROVIDERS.items()
if n not in ("jobindex", "adzuna", "platsbanken")],
"jobnet_regions": options("jobnet", "REGIONS"),
"jobbank_regions": options("jobbank", "REGIONS"),
"modes": options("freehire", "MODES"),
"jobbank_remote": options("jobbank", "REMOTE")}
def asset(path: str) -> str:
"""A static URL stamped with the file's modification time, so a changed
file is a different URL and a cached stale copy is never the answer."""
@@ -180,7 +204,8 @@ def agents(request: Request) -> HTMLResponse:
@router.get("/settings/sources", response_class=HTMLResponse)
def sources(request: Request) -> HTMLResponse:
return page(request, "settings/sources.html", areas=AREA_OPTIONS,
countries=COUNTRY_OPTIONS, regions=REGION_OPTIONS, hosts=HOST_OPTIONS)
countries=COUNTRY_OPTIONS, regions=REGION_OPTIONS, hosts=HOST_OPTIONS,
**search_options())
@router.get("/settings/account", response_class=HTMLResponse)
+8 -2
View File
@@ -187,6 +187,12 @@ agents: []
# Starter searches, imported once while there are no `search` subscriptions.
# provider is a search reader: jobindex (Denmark, no key: q, area, pages),
# adzuna (the UK and most of the EU, a free key: q, country, where, pages) or
# platsbanken (Sweden, no key: q, region: the id of a län, or empty). See
# plugins/readers/.
# platsbanken (Sweden, no key: q, region: the id of a län, or empty),
# freehire (many countries, no key: q, country: lower-case ISO codes, comma
# separated, remote: a work mode), jobnet (Denmark, no key: q, region: one of
# its five region ids) or jobbank (Akademikernes Jobbank, Denmark, no key: q,
# region: an amt id, remote: helt or delvist). See plugins/readers/.
searches: []
# - {name: Head of data, provider: freehire, params: {q: head of data, country: dk}}
# - {name: Data science, provider: jobnet, params: {q: data science, region: HovedstadenOgBornholm}}
# - {name: Data director, provider: jobbank, params: {q: data director, region: "2"}}
+50 -2
View File
@@ -85,6 +85,9 @@ Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open
| HiBob | 1 | 2 | whole board | all | in the list | figures | pay never seen filled |
| BambooHR | 1 | 2 | list, then each role | all | per role | none | paging of a large board unverified |
| Adzuna (search) | 0 | 0 | paged | 50 | teaser | figures | never run live; needs a key |
| freehire (search) | 0 | 0 | paged | 100 | in the list | figures | 10,000 results per search; the query is noisy |
| Jobnet (search) | 0 | 0 | paged | 100 | in the list | none | 10,000 results per search; needs the header `x-csrf: 1` |
| Akademikernes Jobbank (search) | 0 | 0 | paged | 100 | per role | none | not sorted by date, and no total, so never complete |
"Figures" means yearly `salary_min`, `salary_max` and `currency`, plus a pay line; "line only" means a pay line (`salary_text`) and no figures. Figures are yearly: a monthly figure is multiplied by 12 where the host states the period, and an hourly, daily or weekly one is kept as a line only.
@@ -505,7 +508,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
## The search providers
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
### Jobindex
@@ -552,7 +555,52 @@ A saved search is a query with its parameters, stored as a `search` subscription
| Key | a free key from developer.adzuna.com, as `sources.search.adzuna` in the config or `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`; without one the search fails |
| Limitations | Built against Adzuna's documented response and never run live; no Adzuna role is in the database. |
Jobnet and Jooble were left out: Jobnet's search answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
### freehire
| | |
|---|---|
| Saved search | `q`, optionally `country` (lower-case ISO codes, comma-separated; empty for all) and `remote` (a work mode of freehire's: `MODES` in the module) |
| List | `freehire.me/api/v1/agent/jobs/search`, newest first, 100 per page; states `meta.total`. A public API with no key, 300 requests a minute; robots.txt allows `/` and invites API use with a descriptive user agent |
| Pages read | 5 by default (500 results), 10 at most (1,000); the API ends at `offset + limit = 10000` |
| Text | the whole text, as HTML in the answer |
| Date | `posted_at`; trusted |
| Place | `cities` and `location`; the country only when the row names exactly one |
| Remote | `work_mode` |
| Pay | figures from `enrichment`, only where `salary_period` is a year |
| Key | none |
| Limitations | freehire aggregates about 50 hosts, and keeps the one a row came from in `raw` (`source`: `greenhouse`, `jobnet`, ...). A row whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it by the dupes rule. Unquoted words in `q` are OR over title, company, text and place, so a query is noisy: narrow it by country. |
### Jobnet
| | |
|---|---|
| Saved search | `q` and optionally `region`, one of Jobnet's five region ids (`REGIONS` in the module); empty for all of Denmark |
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login; this supersedes the earlier note that its search answers 401 outside its own session |
| Pages read | 5 by default (500 results), 10 at most (1,000); page 101 at 100 a page answers empty |
| Text | the whole text, as HTML in the answer |
| Date | `publicationDate`; trusted |
| Place | `postalDistrictName`, else `municipality`; country Denmark unless the ad names another |
| Remote | none |
| Pay | none |
| Key | none |
| Limitations | The state's portal: mostly the public sector and small companies, and about a fifth of its ads link to an employer's own ATS. The match is loose, so a phrase does not narrow it. There is no robots.txt (`/robots.txt` redirects to a login). The ad's deadline is kept in `raw` only. |
### Akademikernes Jobbank
| | |
|---|---|
| Saved search | `q` and optionally `region` (an `amt` id from the site's own search form: `REGIONS` in the module, 2 is Storkøbenhavn) and `remote` (`helt` fully remote, `delvist` partly) |
| List | the RSS feed `jobbank.dk/job/rss/`, 100 items a page; an empty page ends it. It states no total and is not sorted by date |
| Pages read | at most the module's `ceiling`, which keeps a run under about 20 pages |
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by `enrich` for the roles an agent ranks high. The listing carries none |
| Date | `datePosted` from the role's page; trusted |
| Place | the region in the item's description line, and `jobLocation.addressLocality` from the page; country Denmark |
| Remote | not stated by a row; the search's `remote` filter narrows the list |
| Pay | none |
| Key | none |
| Limitations | Academic and research roles, plus some corporate ads. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Cloudflare may answer a challenge instead of the page, which is an error for that role and not an empty text. |
Jooble was left out: it gives a key only on a manual application and answered a Cloudflare challenge. Jobdanmark was left out too: about a quarter of its newest ads are on Jobnet, freehire carries the rest as source `jobdanmark`, and its list has no text. LinkedIn's guest pages are out because its robots.txt disallows `/` and its terms forbid automated access; LinkedIn enters only as alert mail. Indeed has no public search API.
## Sources that cannot be read, and why
+72
View File
@@ -0,0 +1,72 @@
"""Fixtures shared by the app's tests."""
from __future__ import annotations
import sys
from pathlib import Path
from types import ModuleType
from typing import Any
import pytest
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, ROOT.as_posix())
from plugins import readers # noqa: E402
from plugins.readers import Filters, Reads # noqa: E402
def _stub(name: str, label: str, params: dict[str, str], filters: Filters,
**tables: dict[str, str]) -> ModuleType:
"""A search provider as `plugins/readers/<name>.py` is planned to declare it:
`NAME`, `LABEL`, `PARAMS`, `clean_params` and the tables that fill the
form. Only what `app/` reads, and no listing, since a test that needs one
fakes `plugins.search.run_search`. `clean_params` strips each value, lowers a
country and refuses a region or remote word the tables do not hold."""
module = ModuleType(f"plugins.readers.{name}")
module.NAME, module.LABEL, module.PARAMS = name, label, params
module.READS = Reads(scope="search", listing="paged", page_size=100, ceiling=10000,
text="listing", complete="stated", filters=filters, places="one",
robots="read: stub, checked 2026-10-03")
for table, values in tables.items():
setattr(module, table, values)
allowed = {"region": tables.get("REGIONS"), "remote": tables.get("MODES")
or tables.get("REMOTE")}
def clean_params(given: dict[str, Any]) -> dict[str, Any]:
out = {key: str(value).strip() for key, value in given.items()}
for key, known in allowed.items():
if known is not None and out.get(key) and out[key] not in known:
raise ValueError(f"{key} is not one of {', '.join(known)}")
if "country" in out:
out["country"] = out["country"].lower()
return out
module.clean_params = clean_params
return module
@pytest.fixture
def search_stubs(monkeypatch: pytest.MonkeyPatch) -> dict[str, ModuleType]:
"""freehire, jobnet and jobbank as stub modules in `readers.PROVIDERS`, so
the wiring is tested with or without the real readers merged; a real
module of the name is replaced for the test."""
stubs = {
"freehire": _stub("freehire", "freehire (tech, many countries)",
{"q": "query", "country": "country", "remote": "remote"},
Filters(query="one", country="many", remote="one"),
MODES={"remote": "Remote", "hybrid": "Hybrid", "onsite": "On-site"}),
"jobnet": _stub("jobnet", "Jobnet (Denmark)", {"q": "query", "region": "place"},
Filters(query="one", place="one"),
REGIONS={"HovedstadenOgBornholm": "Hovedstaden og Bornholm",
"Midtjylland": "Midtjylland"}),
"jobbank": _stub("jobbank", "Akademikernes Jobbank (Denmark)",
{"q": "query", "region": "place", "remote": "remote"},
Filters(query="one", place="one", remote="one"),
REGIONS={"2": "Storkøbenhavn", "8": "Østjylland"},
REMOTE={"helt": "Fully remote", "delvist": "Partly remote"}),
}
for name, module in stubs.items():
monkeypatch.setitem(readers.PROVIDERS, name, module)
return stubs
+12
View File
@@ -181,6 +181,18 @@ def test_keep_order_ranks():
key=keep_order).uid == "a"
def test_every_search_provider_loses_to_a_board(search_stubs):
from app.dupes import is_search_host
assert all(is_search_host(h) for h in
("jobindex", "adzuna", "platsbanken", "freehire", "jobnet", "jobbank"))
assert not any(is_search_host(h) for h in ("greenhouse", "workday", "teamtailor", ""))
board = cp("z", host="greenhouse", desc=False, posted="2026-09-09")
for host in ("freehire", "jobnet", "jobbank"):
search = cp("a", host=host, desc=True, posted="2026-01-01")
assert min([search, board], key=keep_order) is board
def test_a_row_without_a_town_joins_only_one_group():
# "Germany" matches Berlin and Munich; joining both would make them one role.
rows = [cp("a", place="Berlin"), cp("b", place="Munich"), cp("c", place="Germany")]
+15
View File
@@ -61,6 +61,21 @@ def test_a_search_copy_of_a_board_role_is_hidden_behind_the_board(store: Store)
assert dup_of(store) == {board.uid: None, search.uid: board.uid}
def test_a_freehire_copy_of_a_greenhouse_role_is_hidden_behind_the_board(
store: Store, search_stubs: dict[str, object]) -> None:
"""freehire aggregates other hosts' postings; the same role it lists from
a board roledesk reads is one role, and the board's copy is the one kept
even though freehire's is the one with a description."""
board = role("board", company="Lunar", title="Head of Data", place="Copenhagen",
board="greenhouse", body="", day=22)
copy = role("copy", company="Lunar", title="Head of Data", place="Copenhagen, Denmark",
board=None, provider="freehire", source="greenhouse", day=20)
for item in (copy, board):
store.upsert_role(item)
store.rebuild_dupes()
assert dup_of(store) == {board.uid: None, copy.uid: board.uid}
def kept_of(store: Store, *items: Item) -> str:
"""The uid of the one of these roles that is not a copy."""
found = dup_of(store)
+71 -1
View File
@@ -576,7 +576,7 @@ def test_search_params_keeps_only_what_the_provider_reads() -> None:
def test_the_providers_and_their_params_follow_the_reader_declarations() -> None:
from plugins import readers
assert sources.PROVIDERS == tuple(readers.PROVIDERS) == ("jobindex", "adzuna", "platsbanken")
assert tuple(readers.PROVIDERS)[:3] == ("jobindex", "adzuna", "platsbanken")
assert sources.search_params("adzuna", " backend ", country=" DE ", where=" Berlin ",
area=1, region="x", pages=2) == {
"q": "backend", "country": "de", "where": "Berlin", "pages": 2}
@@ -589,3 +589,73 @@ def test_search_params_refuses_a_region_that_is_no_swedish_lan() -> None:
sources.search_params("platsbanken", "data", region="zYpL_aGk_QY8") # Midland
with pytest.raises(ValueError, match="provider"):
sources.search_params("indeed", "data")
def test_search_params_of_the_three_newer_providers_follow_their_declarations(
search_stubs: dict[str, Any]) -> None:
assert sources.search_params("freehire", " head of data ", country=" DK,se ",
remote="remote", area=1, region="x", pages=2) == {
"q": "head of data", "country": "dk,se", "remote": "remote", "pages": 2}
assert sources.search_params("jobnet", "data", region="Midtjylland", country="dk",
remote="remote") == {"q": "data", "region": "Midtjylland"}
assert sources.search_params("jobbank", "data", region="2", remote="helt") == {
"q": "data", "region": "2", "remote": "helt"}
with pytest.raises(ValueError, match="region"):
sources.search_params("jobnet", "data", region="Fyn")
with pytest.raises(ValueError, match="remote"):
sources.search_params("freehire", "data", remote="helt")
def test_the_provider_error_lists_the_loaded_readers(search_stubs: dict[str, Any]) -> None:
with pytest.raises(ValueError, match=r"freehire.*jobnet.*jobbank"):
sources.search_params("indeed", "data")
def test_a_provider_table_is_empty_for_a_reader_that_is_not_loaded(
search_stubs: dict[str, Any], monkeypatch: pytest.MonkeyPatch) -> None:
from plugins import readers
assert sources.provider_table("jobbank", "REMOTE") == {
"helt": "Fully remote", "delvist": "Partly remote"}
assert sources.provider_table("freehire", "REGIONS") == {} # it declares none
monkeypatch.delitem(readers.PROVIDERS, "jobbank")
assert sources.provider_table("jobbank", "REGIONS") == {}
assert sources.provider_table("nobody", "MODES") == {}
def test_cli_searches_add_takes_the_newer_providers_and_their_flags(
search_stubs: dict[str, Any], store: Store, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None:
monkeypatch.setattr(sources, "try_search", lambda config, provider, name, params: (
[{"title": "Data lead", "company": "Acme", "place": "København",
"url": "https://x.example/1"}], True))
config = tmp_path / "config.yaml"
config.write_text(f"database: {store.path}\n")
def add(*argv: str) -> int:
return main(["-c", str(config), "searches", "add", *argv])
assert add("freehire", "Head of data", "--q", "head of data", "--country", "DK,SE",
"--remote", "hybrid") == 0
assert add("jobnet", "Data science", "--q", "data science", "--region",
"HovedstadenOgBornholm") == 0
assert add("jobbank", "Data director", "--q", "data director", "--region", "2",
"--remote", "delvist") == 0
capsys.readouterr()
saved = {s.board: s.params for s in store.subscriptions(source="search")}
assert saved == {
"freehire": {"q": "head of data", "country": "dk,se", "remote": "hybrid"},
"jobnet": {"q": "data science", "region": "HovedstadenOgBornholm"},
"jobbank": {"q": "data director", "region": "2", "remote": "delvist"}}
assert add("jobnet", "Bad", "--q", "x", "--region", "Fyn") == 1
assert "region" in capsys.readouterr().err
def test_cli_searches_add_help_names_the_regions_and_work_modes(
search_stubs: dict[str, Any], capsys: pytest.CaptureFixture[str]) -> None:
with pytest.raises(SystemExit):
main(["searches", "add", "-h"])
text = " ".join(capsys.readouterr().out.split())
assert "freehire" in text and "jobnet" in text and "jobbank" in text
assert "HovedstadenOgBornholm" in text and "2 Storkøbenhavn" in text
assert "hybrid Hybrid" in text and "helt Fully remote" in text
+51
View File
@@ -1357,6 +1357,57 @@ def test_a_search_is_tried_saved_updated_and_removed(
assert again["source"]["id"] == sub_id and again["source"]["on"] is True
def test_the_newer_providers_save_with_their_own_params(
client: TestClient, store: Store, search_stubs: dict[str, Any],
monkeypatch: pytest.MonkeyPatch) -> None:
_fake_search(monkeypatch, 2)
freehire = client.post("/api/searches", json={
"provider": "freehire", "name": "Head of data", "q": "head of data",
"country": "DK,se", "remote": "hybrid", "area": 1, "region": "x"}).json()
assert freehire["source"]["target"] == "freehire:head-of-data"
assert freehire["source"]["params"] == {"q": "head of data", "country": "dk,se",
"remote": "hybrid"}
jobnet = client.post("/api/searches/try", json={
"provider": "jobnet", "q": "data", "region": "Midtjylland", "remote": "helt"})
assert jobnet.status_code == 200 and jobnet.json()["found"] == 2
jobbank = client.post("/api/searches", json={
"provider": "jobbank", "name": "Data director", "q": "data director", "region": "2",
"remote": "helt"}).json()
assert jobbank["source"]["params"] == {"q": "data director", "region": "2", "remote": "helt"}
refused = client.post("/api/searches", json={
"provider": "jobnet", "name": "Bad", "q": "data", "region": "Fyn"})
assert refused.status_code == 422 and "region" in refused.json()["detail"]
assert [s.board for s in store.subscriptions(source="search")] == ["freehire", "jobbank"]
def test_the_sources_form_offers_the_providers_that_are_loaded(
client: TestClient, search_stubs: dict[str, Any]) -> None:
page = client.get("/settings/sources").text
for option in ('<option value="freehire">freehire (tech, many countries)</option>',
'<option value="jobnet">Jobnet (Denmark)</option>',
'<option value="jobbank">Akademikernes Jobbank (Denmark)</option>',
'<option value="HovedstadenOgBornholm">Hovedstaden og Bornholm</option>',
'<option value="2">Storkøbenhavn</option>',
'<option value="hybrid">Hybrid</option>',
'<option value="delvist">Partly remote</option>'):
assert option in page
for control in ('id="s-countries"', 'id="s-mode"', 'id="s-region-jobnet"',
'id="s-region-jobbank"', 'id="s-remote-jobbank"', 'id="s-region"'):
assert page.count(control) == 1
assert 'value="jobindex"' in page and 'value="platsbanken"' in page
def test_the_sources_form_renders_without_the_newer_providers(
client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:
from plugins import readers
for name in ("freehire", "jobnet", "jobbank"):
monkeypatch.delitem(readers.PROVIDERS, name, raising=False)
page = client.get("/settings/sources").text
assert 'value="adzuna"' in page and 'value="freehire"' not in page
assert 'id="s-region-jobnet"' in page and 'value="HovedstadenOgBornholm"' not in page
def test_a_search_that_finds_nothing_or_is_malformed_is_refused(
client: TestClient, store: Store, monkeypatch: pytest.MonkeyPatch) -> None:
_fake_search(monkeypatch, 0)