Merge main into wp-review-contract

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-30 15:57:46 +02:00
commit fe2a1b0347
36 files changed
+1741 -596

No files matched your search

+14 -9
View File
@@ -1,6 +1,6 @@
# roledesk
A job-search engine for one person. It reads the public job boards of the companies you follow (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor, Workday, SuccessFactors, Oracle, Phenom, Eightfold and a dozen smaller hosts, one module each in `plugins/readers/`) and runs saved searches on boards that span companies (Jobindex, Adzuna), keeps every posting with when it was first seen and when it closed, scores the open roles for each of your search agents, and tracks the applications you make.
A job-search engine for one person. It reads the public job boards of the companies you follow (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor, Workday, SuccessFactors, Oracle, Phenom, Eightfold and nine smaller hosts, one module each in `plugins/readers/`) and runs saved searches on boards that span companies (Jobindex, Adzuna, Platsbanken), keeps every posting with when it was first seen and when it closed, scores the open roles for each of your search agents, and tracks the applications you make.
Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeline/`), password (`auth/`) and deploy kit (`deploy/`), plus the plugin contract and tools from news (`desk/`). FastAPI, Jinja2, SQLite, `uv`, no build step.
@@ -8,12 +8,12 @@ Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeli
```bash
uv sync --all-extras
uv run app companies seed # 362 European employers from seed/companies.json, 175 with a readable board; a re-run adds new rows and boards
uv run app job pull # read every board and search, embed and score them, extract the best, rescore, summarise
uv run app companies seed # 362 European employers from seed/companies.json, 286 with a board; a re-run adds new rows and boards
uv run app job pull # read every board and search, map and fold, embed and score, read the best roles' pages, extract, rescore, summarise
uv run app serve # http://127.0.0.1:8790
```
`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the chip input's. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents every key, including the `extract:` and `summarise:` blocks, which set the default for the two model stages; the switch under Settings, Sources, Runs overrides it without a restart, and `app stages` does the same from the command line.
`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the chip input's. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents the keys, among them the `extract:` and `summarise:` blocks, which set the default for the two model stages; the switch under Settings, Sources, Runs overrides it without a restart, and `app stages` does the same from the command line.
## Sources: job boards and company boards
@@ -32,10 +32,15 @@ uv run app companies by-hand # the companies read by h
uv run app companies edit revolut --note "needs a login" # name, link, note or reason
uv run app companies checked revolut # its careers page was looked at just now; the Watchlist shows the date
uv run app companies remove revolut # refused while a board is on or it has open roles
uv run app companies size infineon # measure its board again; --missing does every board with no size
uv run app agents list # the agents and their prompts; agents prompt adds, changes or --remove a prompt
uv run app prep unmapped # the host words no table maps; prep unlinked lists the place texts no place resolves
```
The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read (its own pages, a host with no module, a site whose robots.txt disallows reading) is still a row, with its careers page and the reason, shown on the Watchlist tab for checking by hand and under Companies as "not readable yet", so the gap is visible and one module at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles, and asks first when there are open roles to lose; a board is read once before it is kept, and one that lists no roles yet is kept too. Setting a board queues a pull right after, so its roles are read by the one already running or the very next one, rather than waiting for the schedule. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
`app stages` shows or switches extract, summarise and enrich, `app seniority` shows per host how often the stated level equals the model's, `app dupes` recomputes which open roles are copies of one posting, and `app prep` maps the host words and folds the copies without a pull.
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
## The tabs
@@ -44,7 +49,7 @@ Three tabs, and a cog for settings.
| | |
|---|---|
| **Roles** | newsdesk's feed: a filter rail (Show read, Best first or Newest first, first seen, tags, more filters), the open roles with company, place, age, tags and a summary, votes (▲ like ● neither way ▼ pass) and a Save button. A title opens the reader beside the list; on a phone it takes the screen. `j`/`k` move, `Enter` opens, `/` searches |
| **Roles** | newsdesk's feed: a filter rail (Viewing as, Show read, Best first or Newest first, first seen, tags, and under more filters company and place chips, seniority from, to and track, function, remote, pay and votes), the open roles with company, place, age, tags and a summary, votes (▲ like ● neither way ▼ pass) and a Save button. A title opens the reader beside the list; on a phone it takes the screen. `j`/`k` move, `Enter` opens, `/` searches |
| **Applications** | the saved roles and their status, each change recorded as an event |
| **Watchlist** | the companies whose board the app cannot read, for going through their careers pages by hand: Yes or No marks whether one is worth checking (a filter shows the unrated, the yes, the no or all), the careers page opens in a new tab and notes when, and a missing or wrong link is set in the row. A company whose board is failing is listed too. Separate from the affinity a search agent has |
@@ -52,19 +57,19 @@ The cog opens `/settings`: a rail of pages on the left, the page beside it, stic
| | |
|---|---|
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/prefer rules as rows, weights); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna or Platsbanken: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, following a company, setting or changing a company's board) and runs (Run the pull now, what is running, the schedule, and the Problems log) |
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/prefer rules as rows, fetch terms, weights and limits); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna or Platsbanken: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise and enrich, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
| **Account** | the password, sessions and passkeys. Open on loopback, locked on anything else |
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its `taste` weight scores a role by how close it is to the roles the agent liked minus the ones it passed.
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A search role's posting is a teaser; the reader offers a Fetch button that scrapes the original listing once and stores the text. A Summarise button asks for a short summary of that one role; The switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, bounded to the top roles of each agent by that agent's own `limits` so the cost stays flat.
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role that `enrich` has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role; the switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, bounded to the top roles of each agent by that agent's own `limits` so the cost stays flat.
## Docs
- [`docs/decisions.md`](docs/decisions.md): what was decided for roledesk and why.
- [`docs/open.md`](docs/open.md): what is left to do, for whoever picks this up next.
- [`plugins/README.md`](plugins/README.md): every reader, its limits and the Watchlist; [`docs/filter-survey.md`](docs/filter-survey.md): which source-side filters each host takes.
- `docs/plan-*.md`: plans in progress, deleted when their work lands.
- `docs/plan-*.md`: plans in progress, folded into `decisions.md` and deleted when their work lands.
- [`desk/README.md`](desk/README.md), [`pipeline/README.md`](pipeline/README.md), [`auth/README.md`](auth/README.md), [`deploy/README.md`](deploy/README.md): the copied modules, where each comes from and how to refresh it.
- [`docs/hosting.md`](docs/hosting.md), [`docs/release.md`](docs/release.md), [`docs/server.md`](docs/server.md), [`docs/password-protection.md`](docs/password-protection.md), [`docs/llm-backends.md`](docs/llm-backends.md): the boilerplate's deployment docs.
+1 -1
View File
@@ -6,7 +6,7 @@
app companies seed|add|edit|checked|remove|by-hand|set-board|detach|discover|check|size
app searches add|list|remove
app agents list|prompt
app stages [extract|summarise on|off]
app stages [extract|summarise|enrich on|off]
app dupes
app prep [unmapped|unlinked] [--remap]
app db init
+21 -3
View File
@@ -16,8 +16,9 @@ the starter agents; `config.example.yaml` documents every key.
from __future__ import annotations
import os
import sys
from pathlib import Path
from typing import Any, Literal
from typing import Any
import yaml
from pydantic import Field, ValidationError, field_validator, model_validator
@@ -165,12 +166,29 @@ class SearchSeed(Section):
"""A saved search as `config.yaml` lists it: one `search` subscription,
imported once like the agents. `params` are the provider's own, see
plugins/search.py: `q`, `area` and `pages` for jobindex; `q`, `country`,
`where` and `pages` for adzuna."""
`where` and `pages` for adzuna; `q` and `region` for platsbanken. The provider is any reader of
`scope="search"`, so a new one needs no edit here."""
name: str
provider: Literal["jobindex", "adzuna"]
provider: str
params: dict[str, Any] = Field(default_factory=dict)
@field_validator("provider")
@classmethod
def _a_search_provider(cls, provider: str) -> str:
# `plugins` is a top-level folder beside `app` and is not on the path
# when this module is imported first, so it is reached the way
# `app.models` reaches it (`_declared_posted`), and only when a seed
# is validated.
root = str(Path(__file__).resolve().parents[1])
if root not in sys.path:
sys.path.insert(0, root)
from plugins.readers import PROVIDERS
if provider not in PROVIDERS:
raise ValueError(f"provider is one of {', '.join(PROVIDERS)}, not {provider!r}")
return provider
class Config(Section):
"""The whole file."""
+10 -2
View File
@@ -180,14 +180,22 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
Under `NULL` there is no queue to take the chain, so this raises rather
than report a pull that never happened; `app job pull` claims the run and
starts the chain on a queue of its own instead of calling this.
starts the chain on a queue of its own instead of calling this. Anything
that raises once the run is claimed, building the chain or handing it
over, releases the claim first, or the next pull would wait
`pull.ABANDONED` for a run that never started.
"""
run = pull.claim_run(store)
if not run:
since = pull.running(store)
context.notice(already_running(since))
return {"skipped": pull.ALREADY_RUNNING, "running": since}
if not context.task(pull_chain(run)):
try:
started = context.task(pull_chain(run))
except Exception:
pull.release_run(store, run)
raise
if not started:
pull.release_run(store, run)
raise RuntimeError("no queue to start the pull cascade on")
return {"started": " > ".join(CASCADES["pull"]), "run": run}
+282 -107
View File
@@ -18,11 +18,14 @@ is swept across pulls and closes once the sweep reached its end (see
`plugins/readers/_runner.py`).
A board read with a filter (`ats:filtered:<target>`, from the agents' reach,
`app.jobs.reach`, or its subscription's `params`) lists only the roles in reach, so its close keeps open the roles a
person acted on: those with a vote or an application (`_close_unlisted`). A
filtered read, or one of a reader that declares `empty="suspect"`, that lists
under a tenth of what the same read listed last time is recorded by the
runner as a failure, not as a list to close against.
`app.jobs.reach`, or its subscription's `params`) lists only the roles in
reach, so its close keeps open the roles a person acted on: those with a vote
or an application (`_close_unlisted`). When the reach wanted remote roles and
the reader could not ask for them (`ats:remote_unasked:<target>`), the read
says nothing of the remote roles either, so they stay open too and close only
once the board has not listed them for `STALE_DAYS` (`_close_aged`). Any read
that lists under a tenth of what the same read listed last time is recorded
by the runner as a failure, not as a list to close against (`_runner.read`).
A board swept across pulls (`ats:swept:<target>`, a complete read that finished
a sweep an earlier pull began) closes a role only when two finished sweeps in a
@@ -33,14 +36,21 @@ not swept clears them.
A read narrowed by the agents' fetch terms (`ats:narrowed:<target>`) lists
the roles matching a word, which says nothing of the roles it did not list,
so it closes nothing; only an unfiltered or a place-filtered complete read
closes.
so it closes by list nothing: a role of such a board closes once the board
has not listed it for `STALE_DAYS`, the rule searches have.
A search closes the same way when it listed everything that matches (see
`plugins/search.py` for how each provider knows). One that did not closes its
roles only once no search has seen them for `STALE_DAYS`. A board
that has failed on every fetch for `STALE_DAYS` (`ats:failing:<target>`)
closes all of its roles, and its next answer reopens what it lists.
roles only once no search has seen them for `STALE_DAYS`. A board or a
search that has failed on every fetch for `STALE_DAYS` (`ats:failing:<target>`,
`search:failing:<target>`) closes all of its roles, and its next answer
reopens what it lists. A target that raises in its batch after its read, a
row that does not store, is recorded as failing the same way and the batch
goes on with the next one (`run_batch`).
The settle says what needs a look in one notice (`settled`): the targets
that did not answer or were read incomplete, and the boards that closed more
than `CLOSE_NOTICE` roles or more than half of theirs.
One pull at a time. A run is claimed in the database (`claim_run`: `pull:run`
set only while the previous run is settled, in one statement), so the
@@ -60,7 +70,7 @@ from datetime import UTC, datetime, timedelta
from functools import partial
from typing import Any
from pipeline import NULL, RunContext, Tally, Task, job
from pipeline import NULL, Cancelled, Notice, RunContext, Tally, Task, job
from app.config import Config
from app.jobs.reach import reaches
@@ -91,6 +101,19 @@ STALE_DAYS = 21
#: `ats:failing:<target>` -> `{"since", "reason"}`, removed on its next answer.
FAILING = "ats:failing:"
#: The same for a search, kept by the `search` plugin.
SEARCH_FAILING = "search:failing:"
#: A board that closes more roles than this in one settle is named in the
#: settle's notice, and so is one that closes more than half of its open
#: roles, once it had at least `CLOSE_SHARE_MIN` open: a board of two closing
#: one is an ordinary day.
CLOSE_NOTICE = 50
CLOSE_SHARE_MIN = 10
#: How many targets a notice names in its text; its detail names them all.
NOTICE_NAMES = 10
#: The `state` key that records the starter searches were imported.
SEARCHES_SEEDED = "searches:seeded"
@@ -142,27 +165,38 @@ def _boards(config: Config, store: Store, run: str = "") -> list[dict[str, Any]]
`close_boards` compares against. `run` is a marker the caller claimed
already (the `pull` job, `app job pull`); without one the finder claims
its own, and queues nothing when another run is in flight. A run with
nothing to read is settled here, since no settle will see a batch of it."""
seed_searches(config, store)
subs = {source: store.subscriptions(source=source, fetchable=True) for source in SOURCES}
targets = [(source, [s.target for s in subs[source]]) for source in SOURCES]
if not any(found for _, found in targets):
if run:
release_run(store, run)
return []
run = run or claim_run(store)
if not run:
log.warning("pull: %s since %s; this one reads nothing", ALREADY_RUNNING,
running(store))
return []
# The agents' reach, once per run, each board's in its batch (`app.jobs.reach`).
reach_of = reaches(store, subs["ats"])
batches = [{"targets": found[i:i + BATCH_SIZE], "source": source, "run": run}
for source, found in targets for i in range(0, len(found), BATCH_SIZE)]
for batch in batches:
if batch["source"] == "ats":
batch["reach"] = {t: reach_of[t] for t in batch["targets"] if t in reach_of}
return batches
nothing to read is settled here, since no settle will see a batch of it,
and so is a run whose finder raises: no batch and no settle would follow,
and the next pull would wait `ABANDONED` for a claim nobody holds."""
held = run
try:
seed_searches(config, store)
subs = {source: store.subscriptions(source=source, fetchable=True)
for source in SOURCES}
targets = [(source, [s.target for s in subs[source]]) for source in SOURCES]
if not any(found for _, found in targets):
if run:
release_run(store, run)
return []
run = run or claim_run(store)
if not run:
log.warning("pull: %s since %s; this one reads nothing", ALREADY_RUNNING,
running(store))
return []
held = run
# The agents' reach, once per run, each board's in its batch (`app.jobs.reach`).
reach_of = reaches(store, subs["ats"])
batches: list[dict[str, Any]] = [
{"targets": found[i:i + BATCH_SIZE], "source": source, "run": run}
for source, found in targets for i in range(0, len(found), BATCH_SIZE)]
for batch in batches:
if batch["source"] == "ats":
batch["reach"] = {t: reach_of[t] for t in batch["targets"] if t in reach_of}
return batches
except Exception:
if held:
release_run(store, held)
raise
@job("pull_batch", "Reading boards", resource="http")
@@ -185,9 +219,19 @@ def run_batch(config: Config, store: Store, *, targets: list[str], source: str =
plugin as its `run` option, so its stamps belong to that run whatever the
stored marker says by the time the batch runs. `reach` is `target ->` the
reach the run computed for a board (`app.jobs.reach`), handed to the
plugin as its `reach` option.
plugin as its `reach` option. `STALE_DAYS` goes as `stale_days`, after
which the runner's 10% guard accepts the answer it kept refusing.
Each target is guarded on its own. A read that fails is the plugin's to
record; anything raised after it, a row `upsert_role` refuses or a bug
past the read, is recorded here under `<source>:failing:<target>` with
its reason, and the batch goes on with the next target. Otherwise one
bad row would cost the targets after it in the batch their read on every
pull, since batches are stable, and leave no trace on any of them. A
Kill (`Cancelled`) still ends the batch.
"""
from plugins.ats import ATSSource
from plugins.readers import _runner
from plugins.search import SearchSource
# Fetchable now, not when the pull was queued: a board switched off on the
@@ -200,24 +244,39 @@ def run_batch(config: Config, store: Store, *, targets: list[str], source: str =
if sub is None:
continue
context.check()
options = dict(config.sources.get(source) or {}) | {"run": run}
plugin: ATSSource | SearchSource
if source == "search":
options["searches"] = [{"url": target, "label": sub.label, "params": sub.params}]
plugin = SearchSource(options)
else:
# A board's params are its filter; a reader's key is kept with
# the searches' (`sources.search.<name>`).
options["boards"] = [{"url": target, "label": sub.company or sub.label,
"params": sub.params}]
options["keys"] = dict(config.sources.get("search") or {})
options["reach"] = (reach or {}).get(target)
plugin = ATSSource(options)
plugin.state = store.state
for item in plugin.fetch():
item.source = source
store.upsert_role(item, sub)
stored += 1
options = dict(config.sources.get(source) or {}) | {"run": run,
"stale_days": STALE_DAYS}
items: Any = None
try:
plugin: ATSSource | SearchSource
if source == "search":
options["searches"] = [{"url": target, "label": sub.label,
"params": sub.params}]
plugin = SearchSource(options)
else:
# A board's params are its filter; a reader's key is kept with
# the searches' (`sources.search.<name>`).
options["boards"] = [{"url": target, "label": sub.company or sub.label,
"params": sub.params}]
options["keys"] = dict(config.sources.get("search") or {})
options["reach"] = (reach or {}).get(target)
plugin = ATSSource(options)
plugin.state = store.state
items = plugin.fetch()
for item in items:
item.source = source
store.upsert_role(item, sub)
stored += 1
except Cancelled:
raise
except Exception as exc:
log.warning("pull: %s failed after its read (%s); on to the next target",
target, exc, exc_info=True)
_runner.record_failure(store.state, f"{source}:failing:{target}", exc)
finally:
# The plugin's generator holds its HTTP client until it is closed.
if items is not None and hasattr(items, "close"):
items.close()
return {"roles": stored, "run": run}
@@ -235,7 +294,8 @@ def _run_of(tally: Tally, context: RunContext) -> str:
def close_boards(config: Config, store: Store, tally: Tally, *,
context: RunContext = NULL) -> dict[str, Any]:
"""The run's settle: close every board this run answered in full and
stopped listing a role from, and tally what happened.
stopped listing a role from, tally what happened, and say what needs a
look.
Reads `self.state` rather than the batch jobs' own results, because a
`pull_batch` job that raised partway through still leaves the boards it
@@ -243,27 +303,41 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
`ats:ok:<target>` was not written during *this* run — it failed, or its
whole batch job never got to it — is never closed by its list, and a board
that answered incompletely is not either. A search closes by its list only
when that list was complete, and otherwise by `STALE_DAYS`. A board that
has failed on every fetch since `STALE_DAYS` ago closes all its roles.
when that list was complete, and otherwise by `STALE_DAYS`. A board or a
search that has failed on every fetch since `STALE_DAYS` ago closes all
its roles (`failing_boards`). How a read that answered closes is
`_close_read`'s.
The stamps are compared with this run's own marker, the one its batches
carried, never with the stored `pull:run`, which a later run may have
replaced. A settle with no batch behind it (a refused pull, nothing to
read) closes nothing by list or by age. Last, the run is marked settled,
which lets the next pull start.
read) closes nothing by list or by age and says nothing. Last, the run is
marked settled, which lets the next pull start.
A board or search switched off while its batch was in flight had its
roles reopened by that batch, after the switch closed them, so every
switched-off one is closed again here.
switched-off one is closed again here; those closes are asked for and not
named in the notice.
The counts go into the result (`unanswered` counts boards and searches
whose stamp is not this run's), and one notice (`settled`) names every
target unanswered, every board read incomplete other than one mid-sweep,
and every board that closed more than `CLOSE_NOTICE` roles or more than
half the open roles it had, when there is any. The result alone was
seen by nobody: a settle closed 12,588 roles with nothing on screen.
The `prep` stage that follows maps what this run stored and folds the
copies (`Store.rebuild_dupes`), so a closed kept role hands its group on
and the roles this run added join theirs.
"""
answered = searched = failed = incomplete = closed = 0
answered = searched = incomplete = closed = 0
unanswered: list[str] = []
partial: list[str] = []
many: list[tuple[str, int, int]] = []
stale = (datetime.now(UTC) - timedelta(days=STALE_DAYS)).isoformat(timespec="seconds")
failing = failing_boards(store)
run_marker = _run_of(tally, context)
open_before = _open_counts(store) if run_marker else {}
for source in SOURCES:
for sub in store.subscriptions(source=source):
if not (sub.subscribed and sub.fetch):
@@ -273,60 +347,158 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
for sub in store.subscriptions(source=source, fetchable=True):
stamp = store.state.get(f"{source}:ok:{sub.target}", "")
if not (stamp and stamp == run_marker):
failed += 1
since = failing.get(sub.target, ("", ""))[0] if source == "ats" else ""
if since and since < stale:
closed += store.close_subscription(sub.id)
continue
if source == "search":
searched += 1
unanswered.append(sub.target)
since = failing.get(sub.target, ("", ""))[0]
shut = store.close_subscription(sub.id) if since and since < stale else 0
else:
answered += 1
if not store.state.get(f"{source}:complete:{sub.target}", ""):
if source == "ats":
incomplete += 1
continue
if store.state.get(f"{source}:narrowed:{sub.target}", ""):
continue
seen_ids = json.loads(store.state.get(f"{source}:ids:{sub.target}", "[]"))
is_filtered = bool(store.state.get(f"{source}:filtered:{sub.target}", ""))
if store.state.get(f"{source}:swept:{sub.target}", ""):
closed += _close_swept(store, source, sub, seen_ids, is_filtered)
continue
store.state.pop(f"{source}:missing:{sub.target}", None)
if is_filtered:
closed += _close_unlisted(store, source, seen_ids, sub.id)
else:
closed += store.mark_closed(source, seen_ids, sub.id)
if source == "search":
searched += 1
else:
answered += 1
read = _close_read(store, source, sub, stale)
if read is None:
if source == "ats":
incomplete += 1
if not store.state.get(f"ats:sweep:{sub.target}"):
partial.append(sub.target)
continue
shut = read
closed += shut
before = open_before.get(sub.id, 0)
if shut > CLOSE_NOTICE or (before >= CLOSE_SHARE_MIN and 2 * shut > before):
many.append((sub.target, shut, before))
if searched:
closed += store.close_unseen("search", stale)
if run_marker:
if unanswered or partial or many:
context.notice(settled(unanswered, partial, many, failing,
boards=answered, searches=searched, closed=closed))
release_run(store, run_marker)
roles = sum(int(result.get("roles", 0)) for result in tally.results)
return {"boards": answered, "searches": searched, "failed": failed,
return {"boards": answered, "searches": searched, "unanswered": len(unanswered),
"incomplete": incomplete, "roles": roles, "closed": closed}
def _close_unlisted(store: Store, source: str, seen_ids: list[str], subscription_id: int) -> int:
def _close_read(store: Store, source: str, sub: Any, stale: str) -> int | None:
"""Close what one target's read of this run says has closed, and return
how many; None when the read was incomplete, which closes nothing.
A read narrowed by terms closes only the roles it has not listed for
`STALE_DAYS` (`_close_aged`). A read that finished a sweep closes what
two sweeps in a row missed (`_close_swept`); any other complete read
clears that record and closes what it did not list, all of it unfiltered
(`Store.mark_closed`), filtered sparing the roles acted on
(`_close_unlisted`). A filtered read that could not ask for remote roles
(`remote_unasked`) spares the remote ones as well, and closes those by
age instead."""
state, target = store.state, sub.target
if not state.get(f"{source}:complete:{target}", ""):
return None
seen_ids = json.loads(state.get(f"{source}:ids:{target}", "[]"))
if state.get(f"{source}:narrowed:{target}", ""):
return _close_aged(store, source, sub.id, seen_ids, stale)
is_filtered = bool(state.get(f"{source}:filtered:{target}", ""))
spare_remote = is_filtered and bool(state.get(f"{source}:remote_unasked:{target}", ""))
aged = (_close_aged(store, source, sub.id, seen_ids, stale, remote_only=True)
if spare_remote else 0)
if state.get(f"{source}:swept:{target}", ""):
return aged + _close_swept(store, source, sub, seen_ids, is_filtered,
spare_remote=spare_remote)
state.pop(f"{source}:missing:{target}", None)
if is_filtered:
return aged + _close_unlisted(store, source, seen_ids, sub.id,
spare_remote=spare_remote)
return store.mark_closed(source, seen_ids, sub.id)
def _open_counts(store: Store) -> dict[int, int]:
"""`subscription id -> open roles`, before the settle closes any, for the
share a board closed."""
with store.transaction() as conn:
return {int(row[0]): int(row[1]) for row in conn.execute(
"""SELECT subscription_id, count(*) FROM roles
WHERE closed_at IS NULL AND subscription_id IS NOT NULL
GROUP BY subscription_id""")}
def settled(unanswered: list[str], incomplete: list[str], many: list[tuple[str, int, int]],
failing: dict[str, tuple[str, str]], *, boards: int, searches: int,
closed: int) -> Notice:
"""The notice of a settle that has something to look at: the targets
that did not answer (with the reason their failing record gives), the
boards read incomplete, and the boards that closed many roles, each
as `closed of open`, with the totals. The text names the first
`NOTICE_NAMES` of each, the detail all of them, one a line."""
def named(target: str) -> str:
reason = failing.get(target, ("", ""))[1]
return f"{target} ({reason})" if reason else target
groups = [("Not answering", [named(t) for t in unanswered]),
("Read incomplete", list(incomplete)),
("Closed many", [f"{t} {shut} of {before}" for t, shut, before in many])]
text = [f"The pull read {boards} boards and {searches} searches and closed "
f"{closed:,} roles."]
detail: list[str] = []
for title, names in groups:
if not names:
continue
shown = ", ".join(names[:NOTICE_NAMES])
more = len(names) - NOTICE_NAMES
text.append(f"{title} ({len(names)}): {shown}" + (f" and {more} more." if more > 0
else "."))
detail += [f"{title}: {name}" for name in names]
return Notice(code="pull.settled", level="warning", kind="expected", key="pull",
text=" ".join(text), detail="\n".join(detail), where="app.jobs.pull")
#: The roles a partial read never closes by its list: one with a vote (any,
#: neutral included) or an application (which holds the notes).
_ACTED_ON = """ AND uid NOT IN (SELECT uid FROM role_votes)
AND uid NOT IN (SELECT role_uid FROM applications
WHERE role_uid IS NOT NULL)"""
#: The remote roles a filtered read that could not ask for them leaves alone.
_NOT_REMOTE = " AND remote != 'remote'"
def _close_unlisted(store: Store, source: str, seen_ids: list[str], subscription_id: int,
*, spare_remote: bool = False) -> int:
"""`Store.mark_closed` for a filtered read: close every open role of the
board it did not list, except a role someone acted on, one with a vote
(any, neutral included) or an application (which holds the notes). A
board it did not list, except a role someone acted on (`_ACTED_ON`). A
filtered read cannot tell a role gone from a role outside its filter, and
a role acted on stays until the board is read without the filter or
fails for `STALE_DAYS`."""
fails for `STALE_DAYS`. With `spare_remote` the remote roles stay as
well, since the read could not ask for them."""
with store.transaction() as conn:
return conn.execute(
"""UPDATE roles SET closed_at = ?
WHERE source = ? AND subscription_id = ? AND closed_at IS NULL
AND source_id NOT IN (SELECT value FROM json_each(?))
AND uid NOT IN (SELECT uid FROM role_votes)
AND uid NOT IN (SELECT role_uid FROM applications
WHERE role_uid IS NOT NULL)""",
f"""UPDATE roles SET closed_at = ?
WHERE source = ? AND subscription_id = ? AND closed_at IS NULL
AND source_id NOT IN (SELECT value FROM json_each(?)){_ACTED_ON}"""
+ (_NOT_REMOTE if spare_remote else ""),
(store.now(), source, subscription_id, json.dumps(list(seen_ids)))).rowcount
def _close_aged(store: Store, source: str, subscription_id: int, seen_ids: list[str],
stale: str, *, remote_only: bool = False) -> int:
"""Close the open roles of one board that its read did not list and that
no read has listed since `stale` (`roles.last_seen`, which `upsert_role`
writes on every listing): the rule for what a read cannot speak for. A
read narrowed by terms closes all such roles, as a search does; with
`remote_only`, a filtered read that could not ask for remote closes only
its remote roles so, sparing those acted on as its list does."""
where = " AND remote = 'remote'" + _ACTED_ON if remote_only else ""
with store.transaction() as conn:
return conn.execute(
f"""UPDATE roles SET closed_at = ?
WHERE source = ? AND subscription_id = ? AND closed_at IS NULL
AND last_seen < ?
AND source_id NOT IN (SELECT value FROM json_each(?)){where}""",
(store.now(), source, subscription_id, stale,
json.dumps(list(seen_ids)))).rowcount
def _close_swept(store: Store, source: str, sub: Any, seen_ids: list[str],
filtered: bool) -> int:
filtered: bool, *, spare_remote: bool = False) -> int:
"""The close of a board whose complete read finished a sweep across
pulls: its open roles missing from `seen_ids` that also are in
`<source>:missing:<target>`, the ones the previous finished sweep missed.
@@ -334,15 +506,14 @@ def _close_swept(store: Store, source: str, sub: Any, seen_ids: list[str],
missed may only have shifted back past the cursor when roles closed
between pulls, so one sweep is not evidence that it is gone. A filtered
read keeps its exceptions as `_close_unlisted` does: a role acted on
stays open, and stays in `missing`."""
stays open, and stays in `missing`, and so does a remote role with
`spare_remote`."""
key = f"{source}:missing:{sub.target}"
try:
earlier = {str(i) for i in json.loads(store.state.get(key) or "[]")}
except (TypeError, ValueError):
earlier = set()
acted_on = """ AND uid NOT IN (SELECT uid FROM role_votes)
AND uid NOT IN (SELECT role_uid FROM applications
WHERE role_uid IS NOT NULL)""" if filtered else ""
acted_on = (_ACTED_ON if filtered else "") + (_NOT_REMOTE if spare_remote else "")
with store.transaction() as conn:
missed = [row[0] for row in conn.execute(
"""SELECT source_id FROM roles
@@ -365,16 +536,20 @@ def _close_swept(store: Store, source: str, sub: Any, seen_ids: list[str],
def failing_boards(store: Store) -> dict[str, tuple[str, str]]:
"""`target -> (since, reason)` for every `ats` board whose last fetch
failed: when the failures began, and what the latest one said."""
"""`target -> (since, reason)` for every `ats` board and every search
whose last fetch failed: when the failures began, and what the latest
one said. No board host is also a search provider, so a board's target
(`host:slug`) and a search's (`provider:name`) never collide in one
dict."""
out: dict[str, tuple[str, str]] = {}
for key, value in store.state_items(FAILING).items():
try:
record = json.loads(value)
except ValueError:
continue
out[key[len(FAILING):]] = (str(record.get("since") or ""),
str(record.get("reason") or ""))
for prefix in (FAILING, SEARCH_FAILING):
for key, value in store.state_items(prefix).items():
try:
record = json.loads(value)
except ValueError:
continue
out[key[len(prefix):]] = (str(record.get("since") or ""),
str(record.get("reason") or ""))
return out
+13 -7
View File
@@ -9,6 +9,8 @@ good for a given search. For each agent:
column filled for at least one, is gated ("none of
country DK / remote remote")
prefer gates `weight` when the role matches, 0 when it does not
exclude gates a role that matches one of the values keeps its row, with
score 0 and the value ("title account executive")
A gate rejects on evidence, never on absence: a role whose column is empty
is not gated by it. `extract` is what fills an empty column, and it reads
@@ -29,7 +31,8 @@ roles out of extract's reach for good.
Every term is 0..1. The score is the weighted sum over the sum of the weights
(the agent's four and its prefer gates'), so it stays 0..1. A favourite
company (`affinity = 2`) skips the `any` group but not the `require` gates, and
company (`affinity = 2`) skips the `any` group but not the `require` or
`exclude` gates, and
is scored like any other company with its affinity term at 1: nothing puts
its roles above a better fit. A hidden one (`-1`) is not scored at all. A
role with no vector is scored without `fit` and `taste`, unless `fit` is all
@@ -156,14 +159,17 @@ def _reason(gate: Gate) -> str:
def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
"""Why this role is gated, as reasons: every require gate it fails, and
the `any` group when it matches none of them. A gate with an empty column
(state -1) neither passes nor fails, so the group fails only when some
member had evidence. A favourite company's roles skip the `any` group but
not the require gates, so a star widens a search to a company's roles
within e.g. Europe, not to the whole world."""
"""Why this role is gated, as reasons: every require gate it fails, every
exclude gate it matches, and the `any` group when it matches none of them.
A gate with an empty column (state -1) neither passes nor fails, so the
group fails only when some member had evidence. A favourite company's
roles skip the `any` group but not the require and exclude gates, so a
star widens a search to a company's roles within e.g. Europe, not to the
whole world, and not to the titles the agent leaves out."""
states = list(zip(gates, candidate.gates, strict=True))
reasons = [_reason(g) for g, state in states if g.mode == "require" and state == 0]
reasons += [f"{g.field} {g.values[state - 1]}" for g, state in states
if g.mode == "exclude" and state > 0]
group = [(g, state) for g, state in states if g.mode == "any"]
if candidate.affinity != 2 and group and not any(state == 1 for _, state in group) \
and any(state == 0 for _, state in group):
+6 -3
View File
@@ -11,11 +11,14 @@ finders ask it (`extract.extract_task`, `summarise.summarise_task`).
`enrich` (reading the best roles on their own pages, `app/jobs/enrich.py`)
asks no model and is on by default (`enrich.enabled`); its switch is
`stage:enrich`, set the same way through `PUT /api/stages/enrich`.
`stage:enrich`, set the same way as the others: by the Runs page
(`PUT /api/stages/enrich`) and by `app stages enrich on|off`.
`describe` is what `GET /api/stages` and `app stages` show: for each stage
with a switch, the model stages and `enrich`, whether it is on, where that comes from, the model it uses, and who
would answer.
would answer. The model is the one a call is made with: `agent.model` when
set, since it overrides the per-task model everywhere (`desk/modeltask.py`),
else the stage's own.
"""
from __future__ import annotations
@@ -98,7 +101,7 @@ def describe_one(config: Config, store: Store, name: str) -> Stage:
backend, said = answer(config)
return Stage(name=name, on=stage_on(config, store, name),
source="config" if stored(store, name) is None else "stored",
model=getattr(getattr(config, name), "model", ""), backend=backend,
model=config.agent.model or getattr(config, name).model, backend=backend,
answer=said)
+12 -5
View File
@@ -1167,21 +1167,26 @@ class Application(Model):
#: The role columns a gate can name. `company` matches on the company key;
#: `place` takes the Place filter's tokens (`app.places`: `country:DK`,
#: `region:skane`, `place:malmo`, bare text a `place:`) on whole words; the rest are
#: `region:skane`, `place:malmo`, bare text a `place:`) on whole words; `title`
#: matches a value as whole words anywhere in the title, folded as `title_key`
#: folds it, so `account executive` finds "Enterprise Account Executive, UK"
#: and `data engineer` does not find "Data Engineering Manager"; the rest are
#: case-insensitive equality, except `salary_min` (at least the value) and
#: `languages` (every language the role requires is one of the values).
GateField = Literal["place", "country", "remote", "seniority", "function", "salary_min",
"currency", "permit", "languages", "company"]
"currency", "permit", "languages", "company", "title"]
GATE_FIELDS: tuple[str, ...] = get_args(GateField)
GateMode = Literal["require", "any", "prefer"]
GateMode = Literal["require", "any", "prefer", "exclude"]
class Gate(Model):
"""One condition of an agent. `require` gates a role that fails it (the
row is kept with the reason); the `any` gates together are one group a
role must match at least one of ("in Denmark, or remote"); `prefer` adds
`weight` to its score when it matches. A gate rejects on evidence only: a
`weight` to its score when it matches; `exclude` gates a role that matches
any of its values ("title account executive"), a favourite's too. A gate
rejects on evidence only: a
role whose column is empty is not rejected, since `extract` fills the
column later for the roles that rank, and only for those."""
@@ -1297,7 +1302,9 @@ class Agent(Model):
class Candidate(Model):
"""What scoring reads about one open role for one agent: its posted
date (for `recency`), its company's affinity for that agent, and per
gate `1` match, `0` no match, `-1` the column is empty."""
gate `1` match, `0` no match, `-1` the column is empty. An `exclude`
gate's match is the position of the first value it matched, from 1, so
the reason can name it."""
uid: str
posted_at: str
+13 -4
View File
@@ -39,8 +39,8 @@
"HU", "IE", "IT", "LV", "LT", "LU", "MT", "NL", "PL", "PT", "RO", "SK", "SI", "ES", "SE",
"IS", "LI", "NO", "GB", "CH", "EU"];
const FIELDS = ["place", "country", "remote", "seniority", "function", "salary_min",
"currency", "permit", "languages", "company"];
const MODES = [["require", "must"], ["any", "one of"], ["prefer", "prefer"]];
"currency", "permit", "languages", "company", "title"];
const MODES = [["require", "must"], ["any", "one of"], ["prefer", "prefer"], ["exclude", "not"]];
const HYBRID_WEIGHT = 0.05;
/** @type {object[]} every agent, as the server lists them */
@@ -227,11 +227,14 @@
const must = of("require").map(phrase);
const prefer = of("prefer").map((g) => phrase(g).replace(/^with /, ""));
if (where.hybrid) prefer.unshift("hybrid");
const not = of("exclude").map(phrase);
const sentences = [];
const any = whereSentence(where, of("any"));
if (any) sentences.push(any);
if (must.length) sentences.push(`Only roles ${must.join(" and ")}.`);
if (prefer.length) sentences.push(`Prefers ${prefer.join("; ")}.`);
if (where.titles) sentences.push(`Leaves out titles with ${orList(where.titles.values.map((t) => `“${t}”`))}.`);
if (not.length) sentences.push(`Leaves out roles ${not.join(" or ")}.`);
return sentences.join(" ");
}
@@ -497,7 +500,7 @@
// that is Europe and the countries line exactly, and only with Remote
// ticked; any other must on country is a row, and Remote reads anywhere.
function distribute(gates) {
const where = { countries: null, places: null, remote: null, europe: null, hybrid: null };
const where = { countries: null, places: null, remote: null, europe: null, hybrid: null, titles: null };
const rest = [];
const musts = [];
const plain = (g) => g.weight === undefined || g.weight === 1;
@@ -512,6 +515,7 @@
else if (g.field === "remote" && g.mode === "any" && plain(g) && only(g, "remote")) slot = "remote";
else if (g.field === "place" && g.mode === "any" && plain(g)) slot = "places";
else if (g.field === "remote" && g.mode === "prefer" && only(g, "hybrid")) slot = "hybrid";
else if (g.field === "title" && g.mode === "exclude" && plain(g)) slot = "titles";
if (slot && !where[slot]) where[slot] = g;
else rest.push(g);
}
@@ -561,7 +565,9 @@
const { where, rest } = distribute(row ? row.gates : []);
const countries = countryInput(where.countries ? where.countries.values : []);
const places = wordInput(where.places ? where.places.values : [], "Add a town");
const titles = wordInput(where.titles ? where.titles.values : [], "account executive");
const terms = wordInput(row ? row.fetch_terms || [] : [], "machine learning, data science");
$("#a-titles", f).replaceWith(titles.node);
$("#a-terms", f).replaceWith(terms.node);
$("#a-countries", f).replaceWith(countries.node);
$("#a-towns", f).replaceWith(places.node);
@@ -601,7 +607,8 @@
});
// The gates the Where block says, in the order the starter agents have
// them: the must on country, the lines, then hybrid.
// them: the must on country, the lines, then hybrid; and the titles left
// out, the block under it.
function whereGates() {
const out = [];
const within = countries.values();
@@ -615,6 +622,8 @@
if ($("#a-hybrid", f).checked) {
out.push({ field: "remote", values: ["hybrid"], mode: "prefer", weight: Number($("#a-hybrid-weight", f).value) });
}
const left = titles.values();
if (left.length) out.push({ field: "title", values: left, mode: "exclude" });
return out;
}
// The gates Save sends: the Where block's, then the rows. The sentence
+2 -1
View File
@@ -1223,7 +1223,8 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
border-radius: .3rem; padding: .35rem .5rem;
}
/* The fieldsets that hold rows: a column, the Add button under the rows. */
.agent-form fieldset.prompts, .agent-form fieldset.where, .agent-form fieldset.terms,
.agent-form fieldset.prompts, .agent-form fieldset.where, .agent-form fieldset.titles,
.agent-form fieldset.terms,
.agent-form fieldset.rules {
display: flex; flex-direction: column; align-items: stretch; gap: .5rem; min-width: 0;
border: 1px solid var(--line); border-radius: .3rem;
+3 -2
View File
@@ -59,7 +59,7 @@
const company = companies.get(application.company_key);
cell(tr, "Company", company ? company.name : application.company_key);
const select = statusSelect(application.status);
select.addEventListener("change", () => setStatus(application.id, select));
select.addEventListener("change", () => setStatus(application.id, select, application.status));
cell(tr, "Status", select);
const saved = cell(tr, "Saved", savedOn(application.created_at));
saved.title = application.created_at;
@@ -69,7 +69,7 @@
count(applications.length, "tracked application");
}
async function setStatus(id, select) {
async function setStatus(id, select, was) {
select.disabled = true;
try {
await api(`/api/applications/${encodeURIComponent(id)}/status`, {
@@ -77,6 +77,7 @@
});
render(await api("/api/applications"));
} catch (err) {
select.value = was; // the change was refused, so the row still holds `was`
select.disabled = false;
}
}
+4 -4
View File
@@ -217,8 +217,8 @@
// Replacing a different board closes its roles (see set_board in
// app/jobs/sources.py); ask first, naming both boards and the count,
// the same as Detach and a removed search already do.
if (company.board && company.board !== board
&& !window.confirm(`Replace ${company.name}'s board (${company.board}) `
if (company.target && company.target !== board
&& !window.confirm(`Replace ${company.name}'s board (${company.target}) `
+ `with ${board}? Its ${count(company.open_roles, "open role")} close.`)) {
return;
}
@@ -429,8 +429,8 @@
// editor does, so it asks first too.
const existing = [...companies.values()]
.find((c) => c.name.toLowerCase() === name.toLowerCase());
if (existing && existing.board && existing.board !== board
&& !window.confirm(`Replace ${existing.name}'s board (${existing.board}) `
if (existing && existing.target && existing.target !== board
&& !window.confirm(`Replace ${existing.name}'s board (${existing.target}) `
+ `with ${board}? Its ${count(existing.open_roles, "open role")} close.`)) {
return;
}
+11 -4
View File
@@ -109,10 +109,17 @@
...(stage.model ? [el("span", "badge", stage.model)] : []));
row.append(name, stageSwitch(stage, () => renderStages(stages)));
const sub = el("span", "sub");
sub.append(`${DOES[stage.name]}. How deep it reads is set per search agent, under `);
const agents = el("a", "", "Search agents");
agents.href = "/settings/agents";
sub.append(agents, stage.source === "config"
if (stage.name === "enrich") {
/* Global, unlike the model stages: `enrich.top` and `enrich.min_score`. */
sub.append(`${DOES[stage.name]}. How many it reads is set for every agent alike by `
+ "enrich.top and enrich.min_score in config.yaml");
} else {
sub.append(`${DOES[stage.name]}. How deep it reads is set per search agent, under `);
const agents = el("a", "", "Search agents");
agents.href = "/settings/agents";
sub.append(agents);
}
sub.append(stage.source === "config"
? ". Following config.yaml until this switch is used."
: ".");
row.appendChild(sub);
+3 -2
View File
@@ -864,7 +864,8 @@
is the only time this page ever sees a role's `summary` or
`description` directly. */
function excerptOf(role) {
const text = (role.summary || role.description || "").replace(/\s+/g, " ").trim();
/* The server's order (`_EXCERPT_SOURCE`): summary, description, teaser. */
const text = (role.summary || role.description || role.teaser || "").replace(/\s+/g, " ").trim();
if (text.length <= 200) return text;
const cut = text.slice(0, 200);
const space = cut.lastIndexOf(" ");
@@ -1458,7 +1459,7 @@
async function fetchText(button) {
const uid = state.readingUid;
const role = await busy(button, "Fetching…",
() => api(`/api/roles/${encodeURIComponent(uid)}/text`, { method: "POST" }));
() => api(roleUrl(uid).replace(/(\?|$)/, "/text$1"), { method: "POST" }));
if (role) merge(role);
}
+1 -1
View File
@@ -176,7 +176,7 @@
text: n.text,
fix: n.fix,
detail: n.detail,
link: { label: "all messages", href: "/messages" },
link: { label: "all messages", href: "/settings/sources#messages" },
};
if (PROBLEM.has(n.level)) t.count = n.count;
if (n.level === "good") t.action = { label: "reload", run: () => location.reload() };
+16 -4
View File
@@ -219,6 +219,13 @@ def _gate_sql(gate: Gate, gaz: places.Gazetteer) -> tuple[str, str, list[Any]]:
if gate.field == "place":
match, params = _places_sql(values, gaz, table="r")
return "r.place = ''", match, params
if gate.field == "title":
# Whole words of the folded title: " account executive " inside
# " enterprise account executive uk ". A fold drops `%` and `_`, so a
# value cannot smuggle a LIKE wildcard in.
words = [f"% {title_key(v)} %" for v in values if title_key(v)]
match = " OR ".join("(' ' || r.title_key || ' ') LIKE ?" for _ in words) or "0"
return "r.title_key = ''", f"({match})", words
if gate.field == "company":
keys = [company_key(v) for v in values]
return ("r.company_key = ''",
@@ -1830,10 +1837,15 @@ class Store:
params: list[Any] = []
gaz = self.gazetteer()
for n, gate in enumerate(gates):
empty, match, values = _gate_sql(gate, gaz)
columns.append(
f"CASE WHEN {empty} THEN -1 WHEN {match} THEN 1 ELSE 0 END AS g{n}")
params += values
# An exclude gate says which of its values matched: one WHEN each.
parts = ([gate.model_copy(update={"values": [v]}) for v in gate.values]
if gate.mode == "exclude" else [gate])
whens = []
for k, part in enumerate(parts, 1):
empty, match, values = _gate_sql(part, gaz)
whens.append(f"WHEN {match} THEN {k}")
params += values
columns.append(f"CASE WHEN {empty} THEN -1 {' '.join(whens)} ELSE 0 END AS g{n}")
select = ", ".join(["r.uid", "r.posted_at",
"COALESCE(ac.affinity, 0) AS affinity", *columns])
rows = self._rows(
+8
View File
@@ -49,6 +49,14 @@
<span class="hint">This one ranks rather than filters: a hybrid role gains a little.</span>
<p id="a-where-sum" class="where-sum" role="status"></p>
</fieldset>
<fieldset class="titles">
<legend>Which titles to leave out?</legend>
<div class="where-line">
<span>Leave out titles with</span>
<div id="a-titles"></div>
</div>
<span class="hint">Separate phrases with a comma or Enter. A phrase matches whole words anywhere in the title, whatever the case: <code>account executive</code> takes "Enterprise Account Executive, UK", and <code>data engineer</code> leaves "Data Engineering Manager" alone. A shorter phrase takes more: <code>sales</code> takes Sales Engineer too. It applies to liked companies as well. A role left out keeps its row, and Show gated on Roles lists it with the phrase that took it.</span>
</fieldset>
<fieldset class="terms">
<legend>What big boards send</legend>
<div class="where-line">
+30 -8
View File
@@ -98,7 +98,13 @@ def _feed_filters(
raise HTTPException(422, f"{name} is one of {', '.join(RUNGS)}, not {rung!r}.")
if track and track not in TRACKS:
raise HTTPException(422, f"track is one of {', '.join(TRACKS)}, not {track!r}.")
levels = levels_between(level_from, level_to, track) if level_from or level_to or track else []
levels: tuple[str, ...] = ()
if level_from or level_to or track:
# The store reads an empty list as "no filter", so a range that holds no
# level must be refused here or it would list every role.
levels = levels_between(level_from, level_to, track)
if not levels:
raise HTTPException(422, "That seniority range holds no level.")
return {
"q": q or None, "company": company or None, "place": place or None,
"function": function or None, "remote": remote or None, "days": days,
@@ -359,14 +365,21 @@ def _known_role(store: Store, uid: str) -> Role:
@router.post("/roles/{uid}/text")
def fetch_text(uid: str, store: StoreDep) -> Role:
def fetch_text(uid: str, store: StoreDep, agent: int | None = None) -> ScoredRole:
"""Fetch the posting page once and keep it as plain text, for a role whose
description is a teaser (`jobs.posting.fetch_posting`); a Jobindex role
is read from the page its "Se jobbet" link leads to."""
is read from the page its "Se jobbet" link leads to. The role comes back as
`agent` sees it, as `GET /api/roles/{uid}` does: the page merges the answer
into the open role, and one without the vote would blank the reader's."""
agent_id = _agent_id(store, agent)
try:
return jobs.posting.fetch_posting(store, uid)
jobs.posting.fetch_posting(store, uid)
except jobs.posting.PostingError as exc:
raise HTTPException(exc.status, str(exc)) from exc
role = store.role(uid, agent_id)
if role is None:
raise HTTPException(404, f"no such role: {uid}")
return role
@router.post("/roles/{uid}/summarise", status_code=202)
@@ -436,12 +449,17 @@ class CompanyOut(BaseModel):
with no board shown that is not `from_search`, and `hand_reason` says why
it is read by hand (`app.models.HAND_REASONS`). `all_countries` and
`all_roles` are the company's two fetch overrides: its big boards read
with the agents' wide reach, and with no role-type narrowing."""
with the agents' wide reach, and with no role-type narrowing. `target` is
the attached board in full, `host:slug`, where `board` is its host alone;
the page compares it with the board typed to know whether a replacement
closes anything. `ats_slug` cannot stand in for it, since the careers site
may run on another host than the board."""
key: str
name: str
notes: str = ""
board: str
target: str = ""
subscription_id: int | None = None
on: bool | None = None
affinity: int
@@ -500,6 +518,7 @@ def _company_outs(store: Store, agent: int | None) -> list[CompanyOut]:
sub = subs.get(sub_id) if sub_id is not None else None
from_search = jobs.sources.known_from_search_only(c, ever_boarded)
out.append(CompanyOut(key=c.key, name=c.name, notes=c.notes, board=host,
target=target,
subscription_id=sub_id, on=on, affinity=c.affinity,
open_roles=counts.get(c.key, 0), ats=c.ats,
ats_slug=c.ats_slug, careers_url=c.careers_url,
@@ -865,9 +884,12 @@ def set_reach(sub_id: int, body: ReachBody, store: StoreDep) -> dict[str, Any]:
sub = store.subscription(sub_id)
if sub is None or sub.source != "ats":
raise HTTPException(404, f"no such board: {sub_id}")
codes = sorted({c.strip().upper() for c in body.countries if c.strip()})
if unknown := [c for c in codes if not places.known(f"country:{c}") or c == "EU"]:
raise HTTPException(422, f"Not a country code: {', '.join(unknown)}.")
named = sorted({c.strip() for c in body.countries if c.strip()})
tokens = {c: places.parse(f"country:{c}") for c in named}
if unknown := [c for c, t in tokens.items() if not places.known(str(t)) or t.value == "EU"]:
raise HTTPException(422, f"Not a country: {', '.join(unknown)}.")
# The ISO code is what the reach compares against, not the name or alias typed.
codes = sorted({t.value for t in tokens.values()})
if body.mode == "own" and not (codes or body.remote):
raise HTTPException(422, "Name a country or remote, or take the agents' reach.")
params = {k: v for k, v in sub.params.items() if k not in jobs.reach.OVERRIDE_KEYS}
+38 -6
View File
@@ -46,6 +46,10 @@ workers:
# `agent` extra.
agent:
backend: auto # auto | api | claude_code | off
model: "" # overrides extract.model and summarise.model when set
timeout: 300.0 # seconds for one model call, either backend
max_tokens: 8192 # the answer's cap per call
cli_path: "" # where the `claude` binary is, if not on PATH
# Title and description in, gate columns out (app/jobs/extract.py). Off by
# default: on the API backend every batch of 20 roles is a paid call. On, a
@@ -56,7 +60,10 @@ agent:
# `app stages extract on|off`) overrides it and needs no restart.
extract:
enabled: false
model: claude-haiku-4-5-20251001
model: claude-haiku-4-5-20251001 # agent.model wins when set
batch_size: 20 # roles per model call
on_ingest: auto # auto: in a pull when a model is reachable | always | never
thinking: off # off | adaptive
# About 60 words on a role: what it is, the stack, where, the pay if stated
# (app/jobs/summarise.py). Off by default. On, each pull summarises the best
@@ -67,7 +74,26 @@ extract:
# `app stages summarise on|off`) overrides it and needs no restart.
summarise:
enabled: false
model: claude-haiku-4-5-20251001
model: claude-haiku-4-5-20251001 # agent.model wins when set
batch_size: 10 # roles per model call
thinking: off # off | adaptive
# on_ingest (auto | always | never) is accepted here and ignored: a pull
# runs summarise when it is on and a model is reachable (app/jobs/cascade.py).
# Only extract reads it.
# Reading the best roles on their own pages, for the hosts whose list carries
# no text (app/jobs/enrich.py); asks no model. A role is read when it has no
# text and an enabled agent with a prompt ranks it ungated among its best
# `top` and at `min_score` or above. `enabled` is only the default for a fresh
# database: the switch under Settings, Sources, Runs (or `app stages enrich
# on|off`) overrides it and needs no restart.
enrich:
enabled: true
top: 500 # rank depth per agent
min_score: 0.5 # 0..1, the score bar
per_board: 200 # reads per board and run
per_run: 1000 # reads per run
retry_days: 7 # days before a page that gave no text is read again
# A role and each agent's prompts as vectors, for the fit term of a score
# (app/jobs/embed.py). Needs the `embeddings` extra; without it `embed` posts
@@ -94,6 +120,9 @@ embed:
# none, except the key an adzuna search needs (or ADZUNA_APP_ID and
# ADZUNA_APP_KEY in the environment, which keeps it out of this file).
sources: {}
# `enabled` is a key every plugin's block takes, but the pull builds its ats and
# search plugins from the subscriptions and does not read it: switch a board
# or a search off on the Sources page instead.
# search:
# adzuna: {app_id: "...", app_key: "..."}
@@ -124,8 +153,9 @@ sources: {}
# without blending them into one paragraph. Votes, gates and limits stay the
# agent's, shared by its prompts. intent, as in config.yaml, is one prompt
# named after the agent.
# The Search agents sub-tab lists and edits them; the order menu on Roles
# picks one by name. Two more, beside the starter in config.yaml:
# enabled (default true) leaves an agent out of the scores and the pull when
# false. The Search agents sub-tab lists and edits them; the order menu on
# Roles picks one by name. Two more, beside the starter in config.yaml:
agents: []
# - name: Remote from Europe
# prompts:
@@ -155,6 +185,8 @@ agents: []
# - {field: remote, values: [hybrid, remote], mode: prefer, weight: 0.05}
# Starter searches, imported once while there are no `search` subscriptions.
# provider is jobindex (Denmark, no key: q, area, pages) or adzuna (the UK and
# most of the EU, a free key: q, country, where, pages). See plugins/search.py.
# provider is a search reader: jobindex (Denmark, no key: q, area, pages),
# adzuna (the UK and most of the EU, a free key: q, country, where, pages) or
# platsbanken (Sweden, no key: q, region: the id of a län, or empty). See
# plugins/readers/.
searches: []
+21 -19
View File
@@ -16,7 +16,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
## Data
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one job under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one role under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, a row with no country matches Japan and Malaysia, and joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member; a vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own. A known miss: `Remote - USA NH` and `Remote - USA WA` merge, since a state code is not a town.
@@ -34,7 +34,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Structured pay is unpacked the same as any other board field, except that Greenhouse and Ashby need an extra query parameter to get it.** Ashby sends it with `?includeCompensation=true`, Lever's `salaryRange` and Recruitee's `salary` are already on the plain list response, and Greenhouse needs `pay_transparency=true` on the same request it already makes (no per-role fetch); Personio's XML rarely carries a `salaryInformation` block, but reads the same way when it does. Workable and SmartRecruiters give neither, checked live against their real boards (2026-09-29). Every host converts to yearly `salary_min`/`salary_max`/`currency`: a monthly figure is multiplied by 12, an hourly or a daily one is left out rather than annualised into a number nobody stated, and a range with no currency is ignored. Greenhouse can give several ranges for one role (one per office or pay zone); with more than one there is no way to tell which office the role in hand is at, so all of them are skipped rather than one being picked at random.
**Pay that does not normalise is not dropped, only kept out of the gate columns.** `roles.salary_text` (migration 12) is one human-readable line per parser, written whether or not the board's pay conformed — an hourly or a daily rate, several Greenhouse pay zones, a range with no currency — so a person can read it and do their own sums; it is filled through `raw` like the other columns, but replaced whole on every pull, including with `''` when a board stops stating pay, since the usual keep-filled rule (`_RAW_TEXT`) would otherwise leave a stale line behind. `salary_state` (`app.models.Role`) is derived, not stored: 'normalised' when `salary_min` or `salary_max` is set, 'stated' when only `salary_text` is, `''` otherwise; it is what the feed's Pay filter and badge read, and gates and scoring keep reading only `salary_min`/`salary_max`/`currency`, unchanged.
**Pay that does not normalise is not dropped, only kept out of the gate columns.** `roles.salary_text` (migration 12) is one human-readable line per parser, written whether or not the board's pay conformed (an hourly or a daily rate, several Greenhouse pay zones, a range with no currency), so a person can read it and do their own sums; it is filled through `raw` like the other columns, but replaced whole on every pull, including with `''` when a board stops stating pay, since the usual keep-filled rule (`_RAW_TEXT`) would otherwise leave a stale line behind. `salary_state` (`app.models.Role`) is derived, not stored: 'normalised' when `salary_min` or `salary_max` is set, 'stated' when only `salary_text` is, `''` otherwise; it is what the feed's Pay filter and badge read, and gates and scoring keep reading only `salary_min`/`salary_max`/`currency`, unchanged.
**`extract` reads the board's location too.** The model gets title, the board's `place` text, and the first 3000 characters of the description. Greenhouse's location line ("Berlin, Germany") is often the only place a country is stated, and it is part of the posting, so the "only what the posting states" contract still holds.
@@ -44,7 +44,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Postings are persisted.** Storing every posting is what gives `first_seen`, when the app itself first stored the role, and `closed_at`, set by `mark_closed` when a complete fetch of a board no longer lists a role. A role listed again is open again.
**`posted_at` (migration 10) is the earlier of `first_seen` and a trusted host's `published_at`; `first_seen` is left alone, as when the app found the role.** `first_seen` being the only honest posted date was true of Greenhouse's `updated_at`, which resets on every edit; `plugins.ats` already reads `first_published` instead, and that plus the other hosts' dates — Lever's `createdAt`, Ashby's `publishedAt`, SmartRecruiters' `releasedDate`, Recruitee's `published_at`/`created_at`, Personio's `createdAt`, Workable's `published`, and the two `search` providers' (Jobindex's RSS `pubDate`, Adzuna's `created`) — are creation dates, checked against live boards and the live database (`app.models.TRUSTED_POSTED_HOSTS`, with the evidence in its comment). A date in the future or before 2015 is not trusted either, host or no host. Without this, a newly added board's whole backlog scored as freshly posted for two weeks: on the live database, 10,223 of 15,415 board roles were posted more than 30 days before they were first seen (average gap in days: Ashby 78, Greenhouse 143, Lever 226, Personio 333), and open roles with `recency` (`app.jobs.score.recency`) above 0.5 fell from all 15,558 of them to 3,089 (19.9%) once it read `posted_at` instead. `upsert_role` keeps the earliest `posted_at` a role has ever had: an edit on the board can only move its own date later, never earlier, so it can only move `posted_at` earlier too, when the board finally reports the true one.
**`posted_at` (migration 10) is the earlier of `first_seen` and a trusted host's `published_at`; `first_seen` is left alone, as when the app found the role.** `first_seen` being the only honest posted date was true of Greenhouse's `updated_at`, which resets on every edit; `plugins.ats` already reads `first_published` instead, and that plus the other hosts' dates (Lever's `createdAt`, Ashby's `publishedAt`, SmartRecruiters' `releasedDate`, Recruitee's `published_at`/`created_at`, Personio's `createdAt`, Workable's `published`, and the two `search` providers' (Jobindex's RSS `pubDate`, Adzuna's `created`)) are creation dates, checked against live boards and the live database (`app.models.TRUSTED_POSTED_HOSTS`, with the evidence in its comment). A date in the future or before 2015 is not trusted either, host or no host. Without this, a newly added board's whole backlog scored as freshly posted for two weeks: on the live database, 10,223 of 15,415 board roles were posted more than 30 days before they were first seen (average gap in days: Ashby 78, Greenhouse 143, Lever 226, Personio 333), and open roles with `recency` (`app.jobs.score.recency`) above 0.5 fell from all 15,558 of them to 3,089 (19.9%) once it read `posted_at` instead. `upsert_role` keeps the earliest `posted_at` a role has ever had: an edit on the board can only move its own date later, never earlier, so it can only move `posted_at` earlier too, when the board finally reports the true one. `TRUSTED_POSTED_HOSTS` is gone: each reader declares `Fields(posted="trusted")`, read through `app/models.py`, and there are three search providers, Platsbanken's `publication_date` trusted beside the two named.
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with status in `saved, applied, screen, technical, onsite, offer, rejected, withdrawn`, `next_action`, `next_date`, and an append-only `application_events` written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.
@@ -58,7 +58,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Boards follow companies; searches find them.** `ats` only reads companies someone listed, so it cannot surface a role at a company nobody knew of, and the seed list came from a US-centred project: of 29,000 open roles, 31 were in Denmark. The `search` plugin runs saved queries on boards that span companies, as `search` subscriptions whose `params` are the query, and pulls them in the same task as the boards.
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list. A re-run attaches the board an entry names to a row that never had one, so a seed update that gains boards reaches an existing database; it never replaces a board, including a detached one, and never touches affinity, notes or status. Its boards go through `set_board` without the fetch, since they were read when the seed was written and `app companies check` finds the ones that died since.
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list. A re-run attaches the board an entry names to a row that never had one, so a seed update that gains boards reaches an existing database; it never replaces a board, including a detached one, and never touches affinity, notes or status. Its boards go through `set_board` without the fetch, since they were read when the seed was written and `app companies check` finds the ones that died since. The count of 175 is out of date: the seed has 286 boards, as the paragraph on the seed's gap below says.
**`discover` reads the site before it guesses.** A careers page that links its board is the authority, so the home page, its own careers links, then the usual paths are read for a link into a known host, and a page's own "open positions" link one level down. Only when no page names a host are Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio probed with the company's slug, because a board drawn by a script leaves no link. A probe hit on a common name can be another tenant (`recruitee:google` exists, and is not Google), which is why the seed's boards were fetched and read before they landed, and why `discover --add` reports the roles it found. For the same reason a page that names only an unreadable host ends discovery with that host: probing anyway, as the first version did, could replace a known Workday site with a stranger's Greenhouse board.
@@ -70,9 +70,9 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Platsbanken is the third search provider, for Sweden.** The JobTech API (`jobsearch.api.jobtechdev.se/search`) needs no key, states a total, so completeness follows Adzuna's rule, and filters by the taxonomy id of a län (`platsbanken.REGIONS`; Skåne is `CaRE_1nn_cSU`). It refuses an offset above 2000, which the page limit never reaches. The ads carry no usable remote field, so `remote` is read off the headline. Free text ranks by all words rather than requiring them; a quoted phrase narrows it.
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. An Adzuna search listed everything when it read as many distinct results as the `count` it states; a Jobindex search, whose feed states no count, when a page within its page limit came back with fewer than 20 items, counted before the parser drops any, so one malformed item does not pass for the end. Such a search closes like a board; the rest close once no search has listed them for 21 days. A Jobindex answer with no RSS channel raises instead of reading as an empty, complete search. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists.
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. An Adzuna search listed everything when it read as many distinct results as the `count` it states; a Jobindex search, whose feed states no count, when a page within its page limit came back with fewer than 20 items, counted before the parser drops any, so one malformed item does not pass for the end. Such a search closes like a board; the rest close once no search has listed them for 21 days. A Jobindex answer with no RSS channel raises instead of reading as an empty, complete search. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists. Jobindex has since been declared `complete="never"` (its feed states no total), so a Jobindex search closes its roles only by the 21 days and the short-page rule above no longer applies to it.
**Jobindex ads are read from the feed, not followed to the employer.** Each item links to Jobindex's own page, and its `/c?t=` redirect leads to the employer's site or ATS. Following it per role would count as a click in Jobindex's statistics and costs a request per role, so discovering a company's ATS board from a search result is left for later.
**Jobindex ads are read from the feed, not followed to the employer.** Each item links to Jobindex's own page, and its `/c?t=` redirect leads to the employer's site or ATS. Following it per role would count as a click in Jobindex's statistics and costs a request per role, so discovering a company's ATS board from a search result is left for later. The on-demand fetch (A search role's posting text is fetched on demand, below) does follow the "Se jobbet" link to the posting, for a role somebody opened; the pull still never does.
**Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch already in flight when the switch happens reopens them again, so the pull's settle closes every switched-off target once more.
@@ -88,13 +88,13 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**SmartRecruiters' board list has no description, so its text is read per role by `enrich`.** Its `/postings` endpoint gives title, location and department; the text is only on `/postings/{id}`, whose `jobAd.sections` hold `jobDescription`, `qualifications` and `additionalInformation` (joined) and `companyDescription` (left out, the same blurb on every role). Reading it for every role would multiply a board's requests by its roles, so the reader declares `text="detail"` and `enrich` reads only the ranked roles (2026-09-30; 209 open roles had no text before).
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing. The seven hosts became 22 modules, each with its own allow-list of host names; Greenhouse is still the only reader with a fallback feed.
**A board request is streamed, with a 30 s read timeout and a 60 s deadline.** `_get` counts the body as it arrives and refuses it at `MAX_BYTES`, where reading `response.content` first held the whole body in memory before the check. httpx's timeouts are per phase, so a board that keeps sending a byte now and then never timed out; the deadline covers the whole request, redirects and body included, and is checked between chunks. openings.go's 10 s read timeout failed `greenhouse:mongodb` and `greenhouse:feedzai` in a pull of 2026-09-28. Measured on 2026-09-29, three runs each of those two and the five Greenhouse boards with the most open roles, with and without `pay_transparency=true` (84 requests, every one a CloudFront miss): a body of up to 10.4 MB arrives in under a second once it starts, and the first byte waited up to 16.6 s, with the three slowest at 16.6, 14.4 and 10.9 s. The flag made no measurable difference; the slow first byte is the origin's, with or without it. A read may wait 30 s, about twice the worst seen, and the deadline is 60 s, which leaves a large board room on a slower link.
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by `enrich` for the ranked roles as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id.
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published role object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The role page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by `enrich` for the ranked roles as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id.
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse.
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse. Since then there are 22 host modules, not seven; a board over 100 pages is swept across pulls and closes by two finished sweeps (below), and `_paged` is `_runner.read_listing`.
**Teamtailor is read from the career site's `jobs.rss`, and a slug with a dot is the site's own host.** The official API needs a key per company; the feed needs none and carries title, link, guid, `pubDate`, the description, `remoteStatus` and locations, department and role in the `tt:` namespace. `teamtailor:acme` reads `acme.teamtailor.com` and `teamtailor:careers.acme.com` that host, since most career sites run on the company's domain. `_allowed_host` allows exactly that host, redirects included, and a slug must be a DNS name whose last label is not a number; beyond that a target may name any host that resolves, the same trust as any board the owner adds. The feed states no total, so it is read by `per_page=50` and `offset` to an empty page, and a page with nothing new ends the read as incomplete. 23 of the 32 seed companies answered on their `careers_url` host on 2026-09-29; the rest list a marketing site there.
@@ -104,9 +104,9 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
**Per-role text is read by the enrich run, ranked roles first.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. The pull reads listings only. After scoring, the `enrich` stage reads the role pages of roles without text that an enabled agent ranks in its top `enrich.top` (500) at `enrich.min_score` (0.5) or more, best score first, 200 per board and 1,000 per run, and only the roles it filled are embedded and scored again. It follows every pull, a saved agent and a star change, not a vote. A read that brought no text is tried again after `retry_days` (7). `detail` may fill `DETAIL_FIELDS` only, the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing. Completeness is decided by the listing alone. The pull used to read 200 pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered.
**Per-role text is read by the enrich run, ranked roles first.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. The pull reads listings only. After scoring, the `enrich` stage reads the role pages of roles without text that an enabled agent ranks in its top `enrich.top` (500) at `enrich.min_score` (0.5) or more, best score first, 200 per board and 1,000 per run, and only the roles it filled are embedded and scored again. It follows every pull, a saved agent and a star change, not a vote. A read that brought no text is tried again after `retry_days` (7). `detail` may fill `DETAIL_FIELDS` only, the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing. Completeness is decided by the listing alone. The pull used to read 200 pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered. Eightfold, SmartRecruiters and Workable list their roles without text too and are read the same way.
**robots.txt is decided when a reader is built, not checked at run time.** The builder reads the host's robots.txt, the owner decides per reader whether to read it anyway, and `READS.robots` records the decision and the date. The runtime matcher went with the readers rework. Jobindex is read in full although its robots.txt disallows `page=` and `geoareaid=`; Adzuna's API host disallows `/` for every agent; the owner decided to read its documented API with the account's key anyway, since the rule is aimed at crawlers. It needs a key in the config and has not run live yet. HR Manager's portal disallows everything but the application form; no attached company uses it. The companies once kept off the automatic route by robots.txt (Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy, Umbraco) stay on the Watchlist with reason `robots` until each is decided. The SuccessFactors reader uses `/search/` and `/job/` only.
**robots.txt is decided when a reader is built, not checked at run time.** The builder reads the host's robots.txt, the owner decides per reader whether to read it anyway, and `READS.robots` records the decision and the date. The runtime matcher went with the readers rework. Jobindex is read in full although its robots.txt disallows `sort=`, `page=` and `geoareaid=`; Adzuna's API host disallows `/` for every agent; the owner decided to read its documented API with the account's key anyway, since the rule is aimed at crawlers. It needs a key in the config and has not run live yet. HR Manager's portal disallows everything but the application form; no attached company uses it. The companies once kept off the automatic route by robots.txt (Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy, Umbraco) stay on the Watchlist with reason `robots` until each is decided. The SuccessFactors reader uses `/search/` and `/job/` only. Since then `hrmanager:teknologisk` (the Danish Technological Institute) has been attached and has 7 open roles, so an attached company does use HR Manager; and Novonesis, Darktrace and Criteo are Workday boards, no longer on the Watchlist.
**Every source is a reader module that declares what it can do.** The 22 hosts and the 3 search providers are modules in `plugins/readers/`, each with `READS = Reads(...)`: board or search, whole or paged listing, where the text is (listing, excerpt, detail, none), how completeness is known (stated, end, whole, never), page size and the host's ceiling, the filters it takes per dimension (none, one value a request, many, host ids through `facets`), whether it states one place, several, or one posting per place, which fields it states and whether its date is trusted, a key, a provisional title, whether an unknown slug answers an empty board, and the robots decision. Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `problems()` checks the declaration against the functions at import, and `tests/test_readers_contract.py` runs every module against fixtures, so a new reader that drops a place or a teaser fails there. The contract is the docstring of `plugins/readers/__init__.py`; the types are `_contract.py`.
@@ -154,7 +154,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**The embedding backend is effectively required.** The intent is a CV or a paragraph per agent, and TF-IDF cannot relate "Kubernetes" to "container orchestration". The model is `intfloat/multilingual-e5-small`, not the English-only `all-MiniLM-L6-v2` roledesk started with: the corpus turned European, and on a sample of the live roles (all open Danish-country roles plus a random sample of the rest, language labelled by a stop-word count) against the "Copenhagen engineering" agent's intent, MiniLM separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles, where a Danish-country engineering role beats the rest's non-engineering median only 68% of the time; multilingual-e5-small reaches an AUC of 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. A second multilingual candidate, `paraphrase-multilingual-MiniLM-L12-v2`, measured *worse* than plain MiniLM (AUC 0.60) and was dropped. e5 needs its own "query: " / "passage: " prefixes, which `app/jobs/embed.py` adds by matching the model name, not a separate setting.
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty.
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty. Summaries and tags came back as the `summarise` stage and the Summarise button (below); what stays off is desk's own `SummaryPolicy` and `TaggingPolicy`.
**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 whatever the weights are.
@@ -164,6 +164,8 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Gates match loosely where the column is free text.** `place` takes the Place filter's tokens on whole folded words, so `Berlin` matches "Berlin, Germany" and `Lund` no longer matches "Billund"; `company` compares company keys, `languages` passes when every required language is a value, `salary_min` is at least the smallest value, and the rest are case-insensitive equality. Each gate is one SQL `CASE` column, so a require gate filters while the gated row keeps every failed gate as its reason.
**`exclude` gates titles out by phrase, not by `function`.** Sales titles (Account Executive, Account Director, Client Partner) and individual data engineering ranked high for the owner's agent, since their postings read like the sales engineering and data leadership the prompts describe. `function = sales` could not remove them: the keyword classifier files "Solutions Engineer" and "pre-sales" under sales, so on the data of 2026-09-30 it would also have gated Field CTO, Director Field Engineering, Staff Solutions Engineer and every AI Deployment Strategist in the agent's top 400. The fourth mode, `exclude`, gates a role that matches any of its values, and the gate's `CASE` column returns the position of the first value that matched so the reason names it ("title account executive"). It applies to favourites, which skip only the `any` group. The new `title` field matches a value as whole words of `title_key`, `LIKE '% phrase %'` over the padded key, so `data engineer` gates "Senior MLOps/Data Engineer" and not "Data Engineering Manager", and Forward Deployed Engineer is untouched. The form has a Leave out titles line under Where. Bare `sales` and `vice president` stay off the owner's list, because they would take Sales Engineer and VP roles, and `account manager` stays off because it would take Technical Account Manager. Measured on the agent's 5,089 ranked roles, the list the owner took gated 283, 13 of them in the top 100.
**A role with no vector is scored without `fit`.** The fit weight leaves the denominator, so a role `embed` has not reached yet ranks on recency, affinity and prefer gates instead of sinking to the bottom. It is gated as "no vector" only when fit is the only weighted term, and as "no intent vector" when the intent could not be embedded.
**Each run replaces an agent's scores.** `replace_scores` deletes and inserts in one transaction, so a role closed or muted since the last run drops out rather than keeping a stale score, and a reader sees one list or the other.
@@ -172,15 +174,15 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**`embed` and `score` run in the server process.** newsdesk isolates `embed` in a child process with a timeout. roledesk's run is one bounded pass over open roles, and in-process keeps the embedder replaceable in tests. The cost is torch's memory staying in the server after the first run; isolate it if that becomes the problem.
**The pull cascade is `pull → embed → score → extract → score → summarise`.** Every stage is always in the chain and a stage that is off (the switch under Runs, else config.yaml) finds nothing, as in newsdesk. The free stages run first: embed and score cost no tokens, and their score is what bounds the two model stages. Extract reads only what ranks, then score runs again so the gates read what it filled. A stage is addressed by its position in the tuple, since `score` is in it twice. Extract reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly.
**The pull cascade is `pull → embed → score → extract → score → summarise`.** Every stage is always in the chain and a stage that is off (the switch under Runs, else config.yaml) finds nothing, as in newsdesk. The free stages run first: embed and score cost no tokens, and their score is what bounds the two model stages. Extract reads only what ranks, then score runs again so the gates read what it filled. A stage is addressed by its position in the tuple, since `score` is in it twice. Extract reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly. Later decisions added `prep` and `enrich`: the cascade is `pull → prep → embed → score → enrich → prep → embed → score → extract → score → summarise`, and `CASCADES` in `app/jobs/cascade.py` is the source.
**Extract is bounded like summarise: the top roles of each agent, by that agent's own `limits`.** Extract has short answers but would otherwise touch every open role, and the seed boards alone produced 29,000; at Haiku's rates that is tens of dollars for a first pass over roles nobody will look at. Bounded to the ranking, a pull reads a few hundred. `extract_top` is the backstop (default 500) and `extract_min` the real stop (default 0.55, about 300 roles for the starter agent): a fixed count stays flat whatever the boards produce, where a percentage would not. The bounds are per agent rather than config, because one agent may be a wide monitoring search that should read a thousand and another a narrow hunt that wants fifty; one global number fits neither. A batch dealt named uids finishes them whatever their rank, so a running batch is not cut short by a rescore.
**`extract_batch` and `summarise_batch` are `isolate=True`, with a timeout.** `context.check()` only runs between the calls `modeltask.one` makes, not inside one, so a call that hangs on either backend was otherwise unkillable short of restarting the server; a process boundary is what lets a Kill, or the timeout itself, actually end it. One call is already bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for summarise): a batch still running after that is stuck, not slow. The cost: an isolated job's context has no working `stop`, `task` or `fan_out` (see `pipeline/child.py`'s `context_for`), so a refusal or an account-level failure inside a batch can no longer stop the rest of that task's queued siblings — they run on to the same failure instead of being spared it.
**`extract_batch` and `summarise_batch` are `isolate=True`, with a timeout.** `context.check()` only runs between the calls `modeltask.one` makes, not inside one, so a call that hangs on either backend was otherwise unkillable short of restarting the server; a process boundary is what lets a Kill, or the timeout itself, actually end it. One call is already bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for summarise): a batch still running after that is stuck, not slow. The cost: an isolated job's context has no working `stop`, `task` or `fan_out` (see `pipeline/child.py`'s `context_for`), so a refusal or an account-level failure inside a batch can no longer stop the rest of that task's queued siblings, which run on to the same failure instead of being spared it.
## The pages
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models`, `plugins.ats.FUNCTIONS` and `plugins.ats.HOSTS`, so they cannot drift from what the store and the plugin accept). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change.
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models`, `plugins.ats.FUNCTIONS` and `plugins.ats.HOSTS`, so they cannot drift from what the store and the plugin accept). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change. Only Roles, Applications and Watchlist are tabs now: Agents and Companies are settings pages, and Companies is a section of Sources (Two tabs and a cog, Watchlist is a third tab and Settings are three sub-tabs, below).
**The feed loads whether or not its enhancements do.** `/api/agents` and `/api/companies` only dim hidden companies, draw the affinity control and pick the viewing agent's score order; the init block used to await both before ever calling `loadFeed`, so a slow or failed one left the feed blank for good behind a silent `catch (err) { return; }`. Now the feed's own request goes out alongside them rather than after, agents and companies fill in and redraw what they touch (the rail, the feed's dimming, the reader's affinity slot) whenever they land, and a feed already fetched with no agent known reloads once the real one is. A failed feed fetch of its own gets an empty state with a retry (`state.feedError`) instead of the same silent blank.
@@ -190,7 +192,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**The regions are a hand-written list.** Skåne, the Capital Region of Denmark, Zealand, Øresund (the two capital regions), Jutland and Funen, each with its names and towns. On the data of 2026-09-29 no Danish or Skåne town with more than one role was missing. A region elsewhere is added to `REGIONS`.
**`seniority` is read from the title until extract has read the role.** Extract had never run, so the column was empty on all 33,881 open roles and a seniority filter or gate had nothing to read. `seniority.from_title` matches whole words in English, Danish, Swedish, German and French and gives 43% of open roles a level. "X Manager" is a level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor; intern wins over a boss named in the title. Known mistakes: "VP" is always exec, "Field CTO" and "Chief of Staff" get none, "Account Director" is director. Extract overwrites the title's level, a level a board states wins over both.
**`seniority` is read from the title until extract has read the role.** Extract had never run, so the column was empty on all 33,881 open roles and a seniority filter or gate had nothing to read. `seniority.from_title` matches whole words in English, Danish, Swedish, German and French and gives 43% of open roles a level. "X Manager" is a level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor; intern wins over a boss named in the title. Known mistakes: "VP" is always exec, "Field CTO" and "Chief of Staff" get none, "Account Director" is director. Extract overwrites the title's level, a level a board states wins over both. Since the ladder, `roles.seniority` is the first of the model's level, the host's stated level and the title's, in that order (Seniority has three sources, above), so a stated level no longer wins over the model's.
**Seniority is a ladder of rungs with two tracks.** The first vocabulary was one line (intern to exec) and put a Principal below a Team Lead, and a Staff engineer below any manager, while the owner counts a Principal Solutions Engineer as very senior. From `staff` up, each rung pairs an individual contributor's level with a people lead's, as big tech firms grade them: `staff` with `lead` (a team lead) and `manager`, `principal` (also Senior Staff) with `senior_manager` (also Associate Director, AVP), `distinguished` (Senior Principal, Distinguished, Technical Fellow) with `director` (also Head of), then `exec`. `LADDER` in `app/models.py` gives each word its rung and track, and `roles.seniority` stays one column of words, so gates, `extract` and the host tables work on words. `lead` changed meaning: "Tech Lead", "Lead Engineer" and "Marketing Lead" are `staff`, and only a team lead, supervisor or shift lead is `lead`. In a title, a people level wins over a grade on its own rung, and a grade above it lifts it one rung ("Principal Engineering Manager" is `senior_manager`); a slash is "or" ("Senior/Staff Engineer" is `staff`). Principal sits one rung above Staff, although at some firms it is the top individual grade and at others a senior manager's peer. Migration 25 cleared the model levels the split made ambiguous (staff, lead, manager, director), and prep `VERSION` 2 mapped every open role again. On 2026-09-30 that gave, over 19,230 open roles without copies: staff 1,756, principal 535, distinguished 23, lead 222, manager 736, senior_manager 312, director 883, exec 101, none 9,565.
@@ -206,11 +208,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**A muted company is excluded in SQL, before `LIMIT`.** Filtering after `LIMIT` shipped first and showed nothing once Databricks' 884 freshly pulled roles filled the newest 200 rows and Databricks was muted.
**`GET /api/roles` serves a row, not a role.** It used to answer `roles.*`, the whole `Role` — `description`, `feed_text`, `raw` and `article_text` included, about 5.5 KB a role on average — for a feed that only ever shows a title, a few badges and an excerpt; a 1000-row restore was several megabytes of text a phone never renders. `Store.roles_page` selects only the fields a row draws (`RoleRow`), with `excerpt` built by `app.models.excerpt` off a `SUBSTR` of `summary` or `description`, never the column whole. Measured on a copy of the live database: a 60-row page went from 392 KiB to 39 KiB and a 1000-row restore from 6.6 MB to 631 KiB, both about a tenth. The reader is unaffected: it already fetched the whole role by uid when it opened one (`GET /api/roles/{uid}`, still the full `Role`); what changed is that the list row it painted from first, before that fetch lands, no longer carries a body to paint with, so the reader now waits on the fetch for the posting text and the original link the same way it always waited for the summary.
**`GET /api/roles` serves a row, not a role.** It used to answer `roles.*`, the whole `Role` (`description`, `feed_text`, `raw` and `article_text` included, about 5.5 KB a role on average) for a feed that only ever shows a title, a few badges and an excerpt; a 1000-row restore was several megabytes of text a phone never renders. `Store.roles_page` selects only the fields a row draws (`RoleRow`), with `excerpt` built by `app.models.excerpt` off a `SUBSTR` of `summary` or `description`, never the column whole. Measured on a copy of the live database: a 60-row page went from 392 KiB to 39 KiB and a 1000-row restore from 6.6 MB to 631 KiB, both about a tenth. The reader is unaffected: it already fetched the whole role by uid when it opened one (`GET /api/roles/{uid}`, still the full `Role`); what changed is that the list row it painted from first, before that fetch lands, no longer carries a body to paint with, so the reader now waits on the fetch for the posting text and the original link the same way it always waited for the summary.
## The phone
**Two tabs and a cog.** `TABS` in `app/web/pages.py` becomes Roles and Applications; the other six pages (Agents, Companies, Sources, Jobs, Schedules, Messages, Account) move behind a settings cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and six of them were setup or system pages.
**Two tabs and a cog.** `TABS` in `app/web/pages.py` becomes Roles and Applications; the other seven pages (Agents, Companies, Sources, Jobs, Schedules, Messages, Account) move behind a settings cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and six of them were setup or system pages. The Watchlist became a third tab and the settings pages became three sub-tabs, both below.
**Watchlist is a third tab: the companies the app cannot read, rated yes or no.** The owner goes through the careers pages of such companies by hand and wanted that list one click from Roles rather than behind the cog, so `TABS` has three entries and the phone bar's tabs are tighter. A company is on it when it has no board subscription and is not known from a search only, or when its board is failing: the rule of the Company boards filter "Not readable yet", read from `GET /api/companies`. The rating is `companies.watch` (migration 15: NULL unrated, 1 on the list, 0 not), global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. A row rated under Unrated leaves the list at once. On a phone the search field of Roles is left a few letters wide by the third tab, so while it has the focus the tabs and the queue chip step aside.
@@ -240,7 +242,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**`taste` is the liked-minus-passed centroid, as in newsdesk.** An agent's votes become one vector, the mean of the liked roles' embeddings minus the mean of the passed ones', and a role's taste term is its cosine to that vector, calibrated like fit, weighted beside fit, recency and affinity. It is the one term an agent learns rather than is told, and it costs no model call.
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest.
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest. The figure is of 2026-09-28: hosts that list a role without its text now leave 1,974 of 24,003 open roles without a description on 2026-09-30, until `enrich` reads them.
**Every link built from someone else's data goes through `safeUrl`.** A board, a search result, a discovered careers page and a saved application's `url` are all free text from outside the app; an `http(s)` check before it lands in an `href` is what keeps a stray `javascript:` link or a bare string typed into a field from being clickable. The reader needed this first, for `original_url`, so `safeUrl` moved to `app.js` beside the app's other shared helpers rather than staying roles.js's own; companies.js (`careers_url`), sources.js (a search sample's `url`) and applications.js (`application.url`) now use the same one.
+26 -24
View File
@@ -1,6 +1,6 @@
# Survey: which filters the hosts take at the source
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted.
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. It covers the nine readers that declare a filter (SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken, Adzuna) and Workable, Teamtailor, Jibe and Lever, which declare none. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed. Sections 2 and 3 began as the plan for the fetch terms, which it called F2; F2 has landed (`decisions.md`, Fetch terms), and those sections now say what was recommended and what became of it.
Marks used in the table:
@@ -72,7 +72,7 @@ Country and remote are left out; they are in the readers already. "Combine" is h
| category | `appliedFacets.jobFamily=[<id>, ...]` (30 values on LEGO); the parameter name is the tenant's (`jobFamilyGroup` elsewhere) | ids in one facet union: Mechanical Engineering (4) plus IT Engineering (8) with Denmark and `engineer` answered 12 | AND across facets and with `searchText` | untested | live |
| level | no level facet on LEGO; `jobFamily` has "Trainee / Intern / Apprentice" | | | | live |
| employment | `workerSubType` ("Job Type": Regular, Education Fixed Term, Student Worker, Temporary) and `timeType` (Full time, Part time) | as `jobFamily` | AND | untested | live |
| place | LEGO's `Location` facet lists countries only (13 values); other tenants nest a city facet under `locationMainGroup` | as `jobFamily` | | | code |
| place | LEGO's `Location` facet lists countries only (13 values); on some tenants `locationMainGroup` holds `locationCountry`, another country facet, and no city facet was seen | as `jobFamily` | | | code |
| notes | The reader's `facets` call already reads facet ids off one page; the facet list responds to `searchText` (counts change) | | | | live |
### Oracle Recruiting Cloud (`oracle:fa-evmr-saasfaprod1.fa.ocs.oraclecloud.com/CX_1`, Nokia, 554 roles)
@@ -86,7 +86,7 @@ Requests go through `finder=findReqs;siteNumber=CX_1,<name>=<value>,...`.
| category | `selectedCategoriesFacet=<id>` from `categoriesFacet` (10 values: Applied R&D 259, Direct Production 55 under the keyword) | joined with `;` (`%3B`): two ids answered 314 = 259 + 55 with the keyword: union | AND with the keyword | an id the pod does not know answers the whole board (superset) | live; unknown from code |
| level | none: `titlesFacet` holds function codes (`SWA` Software, `TRA` Trainee), `flexFieldsFacet` holds unnamed attributes | | | | live |
| employment | none seen; `selectedWorkplaceTypesFacet` is the work-location type (`ORA_HYBRID`, `ORA_ON_SITE`, `ORA_REMOTE`) | | | | live |
| place | `selectedLocationsFacet` lists 10 countries only; `workLocationsFacet` lists sites; the answer also echoes `Location`, `LocationId` and `Radius`, which suggests a free place search | | | | live, untested |
| place | `locationsFacet`, the response facet, lists 10 countries only, and `selectedLocationsFacet` is the request parameter that takes its ids; `workLocationsFacet` lists sites; the answer also echoes `Location`, `LocationId` and `Radius`, which suggests a free place search | | | | live, untested |
### Workable (`workable:starling-bank`, 48 roles)
@@ -94,10 +94,10 @@ Requests go through `finder=findReqs;siteNumber=CX_1,<name>=<value>,...`.
|---|---|---|---|---|---|
| keyword | `query` in the POST body of `/api/v3/accounts/<slug>/jobs`; `{"query": "engineer"}` answered 42 of 48 | | | | live |
| keyword scope | title and text, judged from the 6 of 10 hits that have it in the title: not determined | | | | untested |
| category | `department`: the site sends a list of names taken from each department's `filter` array; my body `["Engineering"]` was answered 400 (the request with `location` too), which of the two keys was refused is not known | | | | live (rejected), front end |
| category | `department`: the site sends a list of names taken from each department's `filter` array; the survey's body `["Engineering"]` was answered 400 (the request with `location` too), which of the two keys was refused is not known | | | | live (rejected), front end |
| level | none: no filter key | | | | front end |
| employment | `worktype`, and `workplace` (`hybrid`, `on_site`, `remote`) | | | | front end |
| place | `location`: a list of `{country, region, city, countryCode}` objects, the shape of a role's `locations`; my body left out `countryCode`, which may have caused the 400 | | | | front end |
| place | `location`: a list of `{country, region, city, countryCode}` objects, the shape of a role's `locations`; the survey's body left out `countryCode`, which may have caused the 400 | | | | front end |
### Teamtailor (`teamtailor:careers.shine.co`, 73 roles)
@@ -159,7 +159,7 @@ For agent 2 this costs nothing: its `any` group has DK and SE beside the towns,
**Which hosts could take a town instead.**
- Could, by what was seen: Eightfold (`location=`, its facet values are town strings; untested), Workable (`location` objects with `city`; untested body), Lever (`location=Stockholm`, live: exact text, matches any of a role's places), Adzuna (`where`, `location0..7`), Jobindex (`geoareaid`), Platsbanken (`municipality`).
- Could, with facets: Workday on tenants that have a city facet (LEGO's is countries only), SuccessFactors (`locationsearch=`, untested), Phenom (`location` and `locationData` in the body, untested), Jibe (a city facet is listed; the parameter is not known).
- Could, with facets: Workday if a tenant has a city facet (none seen: LEGO's is countries only, and `locationMainGroup` holds a country facet), SuccessFactors (`locationsearch=`, untested), Phenom (`location` and `locationData` in the body, untested), Jibe (a city facet is listed; the parameter is not known).
- Cannot: SmartRecruiters has `city=` and `region=` in its documentation but it is untested; Oracle lists ten countries and sites only; Teamtailor takes ids.
- Only the search providers do so today (`Filters(place="one")` on Adzuna, Jobindex and Platsbanken).
@@ -169,9 +169,11 @@ For agent 2 this costs nothing: its `any` group has DK and SE beside the towns,
2. Town strings are the host's own, unlike ISO codes: each host needs a name-to-string table or a facet lookup, and a town the host spells differently returns nothing and, on the closing guard, wrongly closes roles.
3. Hosts that state their places per role (all the big ones) leave the exact match to the gates and the gazetteer, which is already right.
So: F2 changes `reach.py` to carry `places` beside `countries` for a group that has towns and no country, and passes them only to readers that declare `place` (the three providers now; Eightfold and Lever if a later run shows the strings work). Everything else keeps the country.
So: the plan was to carry `places` beside `countries` in the reach for a group that has towns and no country, and to pass them only to readers that declare `place`. That was not built. `Reach` (`app/jobs/reach.py`) carries countries and remote, a town in a gate is fetched as its country (`decisions.md`, Fetch terms), and the three search providers take their `place` from the saved params (`PARAMS`), never from a reach. Eightfold and Lever wait for a run that shows the town strings work.
## 3. Recommendation
## 3. What was recommended, and what landed
Landed: `query` on SmartRecruiters, SuccessFactors, Workday, Phenom, Eightfold and Oracle (`Filters(query="one")`). Not built: the rows below marked "no" or "later", and the rows for Workable, Teamtailor and Jibe, which declare no `query` yet.
### Declarations per host
@@ -183,17 +185,17 @@ So: F2 changes `reach.py` to carry `places` beside `countries` for a group that
| Phenom | yes: `keywords` | later: `selected_fields.category` needs the value list | no: `positionLevel` is empty | no |
| Workday | yes: `searchText` | later: `jobFamily` ids from the existing `facets` call | no | no |
| Oracle | yes: `keyword` | later: `selectedCategoriesFacet` ids from `categoriesFacet` | no | no |
| Workable | yes: `query` | no: the body is not settled (400) | no | no |
| Teamtailor | yes, after one baseline request confirms the feed filters | no | no | no |
| Jibe | yes: `keywords` | later: `categories` | no | no |
| Workable | not built: `query` answered in the survey | no: the body is not settled (400) | no | no |
| Teamtailor | not built: needs a baseline request that confirms the feed filters | no | no | no |
| Jibe | not built: `keywords` | later: `categories` | no | no |
| Lever | no: no keyword parameter; boards are small and read whole | no | no | later, as above |
| Jobindex, Platsbanken, Adzuna | already `query` and `place`; `subid`, `occupation-field` and `category` are available for `category` | | | |
`level` is not worth declaring on any host: only SuccessFactors and Jibe have a facet for it, and on the tenants seen it is a tenant label or empty. Seniority stays a prep matter (title rules, host word).
`level` is not worth declaring on any host: only SuccessFactors and Jibe have a facet for it, and on the tenants seen it is a tenant label or empty (Lever documents `level=` and SmartRecruiters states `experienceLevel` on each role; neither is filtered on). Seniority stays a prep matter (title rules, host word).
### What F2 should build first
### What F2 built, and why in this order
The plan assumes terms narrow big boards. The live DB says how much:
The table shows how much terms can narrow big boards, from the live database of 2026-09-30:
| Board | Host | Open | In Europe | In DK or SE (today's reach) | Roles with a term in the title, Europe |
|---|---|---|---|---|---|
@@ -211,29 +213,29 @@ The plan assumes terms narrow big boards. The live DB says how much:
| Ericsson | Eightfold | 618 | 210 | 149 | 64 |
| Nokia | Oracle | 554 | 87 (54 with none) | 0 | 24 |
"Term in the title" is a count of the words head, director, VP, chief, CTO, CDO, data, AI, machine learning, forward deployed, solutions engineer or architect, professional services, research, innovation, principal, lead and manager in the title. It is a proxy for `fetch_terms`, and a lower bound on what a text filter returns: every host's keyword matched text, not only titles (Ericsson: `software` matched a thesis title without it; Nokia: `engineer` matched 468 of 554; Bosch: 1,713 of 4,830). Expect a keyword read to return several times these counts, and a broad term like `engineer` to cut little.
"Term in the title" is a count of the words head, director, VP, chief, CTO, CDO, data, AI, machine learning, forward deployed, solutions engineer or architect, professional services, research, innovation, principal, lead and manager in the title. It is a proxy for `fetch_terms`, and a lower bound on what a text filter returns: where the scope could be told, the keyword matched text and not only titles (SmartRecruiters, Eightfold and Oracle by the examples here; on SuccessFactors, Workday, Workable and Jibe it is not determined) (Ericsson: `software` matched a thesis title without it; Nokia: `engineer` matched 468 of 554; Bosch: 1,713 of 4,830). Expect a keyword read to return several times these counts, and a broad term like `engineer` to cut little.
Findings:
1. **Narrow reach needs no terms.** At DK and SE every big board is already at most 149 roles (Ericsson), and Bosch is two requests. Terms save requests only on the wide reach: the All roles tag, and the favourite reads the plan drops.
1. **Narrow reach needs no terms.** At DK and SE every big board is already at most 149 roles (Ericsson), and Bosch is two requests. Terms save requests only on the wide reach, which is the All countries tag; a company set to All roles takes no terms, and a favourite no longer widens the fetch (`decisions.md`).
2. **The wide reach is where terms pay.** SmartRecruiters reads Europe (33 countries) as the whole board, because `MAX_SERIES` is 8 for a `one` host: Bosch is 49 requests of 100 for 1,564 European roles. One `q` request per term (about 35% of the board for `engineer`, far less for `director` or `data`) replaces it.
3. **Order by the size of the whole read that terms replace**, then by how sure the parameter is:
3. **The order followed the size of the whole read that terms replace**, then how sure the parameter was:
1. SmartRecruiters `q`: Bosch, Delivery Hero, Wise and Coolblue, 6,700 roles in the live DB, one live-verified parameter, AND with country.
2. SuccessFactors `q`: SAP, Scania, Vestas and Novo Nordisk, 2,650 roles; verified. The reader already reads a country a request, so a term multiplies the series.
3. Workday `searchText` (Zeiss 848, LEGO 455), Phenom `keywords` (Philips 868), Eightfold `query` (Ericsson 618) and Oracle `keyword` (Nokia 554): each verified once, each a small change to the request body or URL.
4. **Category next, and only for Workday, Oracle, Jibe and Eightfold.** They state their own category ids in a response the reader already reads or can read (`facets`, `categoriesFacet`, `filter.categories`, `filterDef`), and the union across values is verified on Workday and Oracle. But an agent's term (`engineer`) is not a category name, so it needs a mapping from term to category per tenant that nobody has: not before terms have run on real boards.
5. **Level: never at the source.** **Place: the search providers now, Eightfold and Lever if a test proves a town string; the rest stay on country.**
4. **Category comes next, and only for Workday, Oracle, Jibe and Eightfold** (Phenom's needs its value list first). They state their own category ids in a response the reader already reads or can read (`facets`, `categoriesFacet`, `filter.categories`, `filterDef`), and the union across values is verified on Workday and Oracle. But an agent's term (`engineer`) is not a category name, so it needs a mapping from term to category per tenant that nobody has: not before terms have run on real boards.
5. **Level is not filtered at the source.** Place is filtered by the search providers; Eightfold and Lever wait for a test that proves a town string, and the rest stay on country.
### Guards F2 must keep
### Guards the fetch terms keep
- A term filter needs the closing guards to hold: a keyword read is not a whole board, so `ats:size` must not be overwritten by it, and roles outside it must not be closed as gone.
- Unknown-value behaviour differs: Lever returns nothing, Oracle returns everything, SuccessFactors returned something unrelated, Workable answers 400. The runner must treat a facet value as known only when the host's own facet list stated it, as `series` does for ids.
- A term filter needs the closing guards to hold: a keyword read is not a whole board, so it does not write `ats:size` (a pull no longer rewrites a size at all) and roles outside it are not closed as gone (`narrowed` in `plugins/readers/_runner.py`, `close_boards` in `app/jobs/pull.py`).
- Unknown-value behaviour differs: Lever returns nothing, Oracle returns everything, SuccessFactors returned something unrelated, Workable answered 400 to a body whose refused key is not known. The runner treats a facet value as known only when the host's own facet list stated it, as `series` does for ids.
- Keywords are text matches everywhere except Adzuna's `title_only`; the agent's own scoring still decides relevance.
### Open, and cheap to settle in F2's first hour
### Open, and cheap to settle with one request each
- SmartRecruiters: does `q="head of data"` mean all words or any, and are `department`, `function`, `experienceLevel` accepted (one request each)?
- Teamtailor: a baseline request to confirm the feed's `query` filter.
- Workable: which key of my body was refused (`department` or `location`).
- Workable: which key of the survey's body was refused (`department` or `location`).
- Eightfold: a `location` with a town string, and repeated `filter_function` values.
- Adzuna: everything, once the key exists.
+24 -7
View File
@@ -5,11 +5,11 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la
## Text
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only `summarise` reads. `embed` and `extract` read `description or teaser`, so a Jobindex role keeps being ranked on its teaser. The fix is one text rule, `article_text or description or teaser`, in embed and extract, and queuing embed and score after a fetch.
- **An edited posting is never embedded again.** The vector is dropped only when a teaser is replaced by a description. A hash of the embedded text on `vectors` would re-embed on any change.
- **An edited posting is never embedded again.** The vector is dropped only when the text ranks better than before (a title-only role gaining a teaser, or a teaser replaced by a description). A hash of the embedded text on `vectors` would re-embed on any change.
- **Extract marks a title-only role done for good.** `upsert_role` does not reset `extracted_at` when text arrives. Extract is on, so this bites. `_EXTRACT_WAITING` re-reads roles whose `seniority_model` is empty, which covers level and not pay.
- **Lever keeps only the intro paragraph.** `lever.py` reads `description` and drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way.
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 16.5% of open roles. Extract can return `salary_min` and `salary_max` but has read about 500 roles so far.
- **A title-only role has no mark in the feed.** 1,304 open roles have no text, 869 of them Eightfold. A mark would show which fits are guesses.
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 17.9% of open roles. Extract can return `salary_min` and `salary_max` but has read 554 roles so far.
- **A title-only role has no mark in the feed.** 1,180 open roles have no description, teaser or fetched posting: 677 of them Workday, 256 SmartRecruiters, 150 Workable, 57 SuccessFactors and 26 Eightfold. A mark would show which fits are guesses.
## Ranking
@@ -19,17 +19,34 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la
## Sources
- **The boilerplate moved on.** `deploy/` is at boilerplate `1679e88` and lacks `wsl-setup.sh` and `wsl-setup.ps1`; `pipeline/` and `auth/` differ from boilerplate HEAD (three languages, installable phone). Refresh them with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
- **The boilerplate moved on.** `pipeline/` equals boilerplate `a1c6f0d` once line endings are ignored: 87 files of the working tree have CRLF although the index has LF, so a plain `diff -r` shows every file as different (use `--strip-trailing-cr`). `auth/` lacks `c6a90ff`, the phone commit (`auth/README.md`, `auth/static/account.js`, `auth/templates/login.html`). `deploy/` is at `1679e88` and lacks the WSL scripts of `e1b3746` (`wsl-setup.sh`, `wsl-setup.ps1`), and `docs/release.md` lacks its `wsl.md` sentence. `desk/` is news `a275c38` less the prune and lags news by four commits (`b0e1f6f`, `f01a6bd`, `3bb6775`, `7b9c8a2`). The three languages of boilerplate `7bb7da2` are starter code in `app/`, not part of the copies. Refresh with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
- **Six hosts' dates are not trusted.** Eightfold, Jibe, Gem, Deel, HiBob and HR Manager declare `posted="listed"`, so their backlog counts as posted when first seen. Eightfold now maps `creationTs`, Jibe's `posted_date` equals the page's `datePosted`, and HR Manager's `pubDate` rises with the id; each needs one check on a live board before it declares `posted="trusted"`. Platsbanken's local time is read as UTC.
- **A mistyped SmartRecruiters or Gem slug attaches as an empty board.** Both answer an empty listing for an unknown id. `empty="suspect"` and the 10% guard keep a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board.** All three answer an empty listing for an unknown id and declare `empty="suspect"`. The 10% guard keeps a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the runner waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
- **A SuccessFactors site on a company's own domain is not discovered.** Its pages link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
- **HR Manager: decisions and data disagree.** `decisions.md` and `plugins/README.md` say no attached company uses it, but the live database has `hrmanager:teknologisk` attached with 0 open roles. Seaborg stays unattached until the owner decides whether the feed counts as allowed.
- **Stripe's places carry a "Locations" suffix.** 282 open roles have place text like "Singapore Locations" or "Dublin; Ireland Locations".
- **HR Manager is attached although its robots.txt disallows reading.** `hrmanager:teknologisk` (the Danish Technological Institute) has 7 open roles, while `READS.robots` says "disallowed by robots.txt; owner to decide". Seaborg stays unattached until the owner decides whether the feed counts as allowed.
- **Whether SmartRecruiters' identifier is case-sensitive is unsettled.** The module docstring says it is (`BoschGroup`), `decisions.md` says it is not, checked live; the live boards hold the identifiers with case. A live request with a lowercased identifier settles it.
- **The seed lags the live database.** The seed has 286 boards and the database 310: it lacks Microsoft, Infineon, Novonesis, Darktrace, Criteo, NVIDIA, HR Manager and seventeen more, and names five of them as `unread` or `robots`. `app companies seed` never replaces a board, so the seed is brought in line by hand.
- **Stripe's places carry a "Locations" suffix.** 316 open roles (285 without copies) have place text like "Singapore Locations" or "Dublin; Ireland Locations".
## The pages
- **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite.
- **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it.
- **`company_key` keeps `ø`.** Aliases make "Orsted" and "Ørsted" one company; folding the key itself would change the primary key of `companies`.
## From the review of 2026-09-30
- **Remote roles are not read from hosts that cannot filter on remote.** On SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" is asked as "DK, SE", and the filtered read closes remote roles outside them: 72 on 2026-09-29 and 30 (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3). Reading such boards whole is the owner's call; Bosch is about 70 requests a pull.
- **Provider names are hard-coded in nine places** (`cli.py`, `dupes.py`, `sources.py`, `pages.py`, `sources.html`, `sources.js`, `api.py`, `posting.py`) instead of `readers.PROVIDERS`, and `has_key` re-implements `resolve_key`.
- **`ats.py` keeps compatibility names that only tests reach for:** `_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused` and `DETAIL_FIELDS`.
- **`title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`.** Wave 2's WP9 must own `_runner.py` (`plan-hosts-wave-2.md`).
- **Some `posted="trusted"` declarations rest on the detail read** (Workday, BambooHR, Rippling) **or on a fallback the host moves** (Greenhouse `updated_at`, Phenom `postedDate`).
- **Dupes fold same-host roles with the same title and town without comparing text.** 99 open copies share under 30% of their words with the kept role.
- **Enrich never reads a role below the bar.** 349 roles without text are ranked on their title for good.
- **A pull never re-measures a board's size.** Recheck is the only way, so a board that grows past 300 roles stays read whole until someone rechecks it.
- **Fetch terms are configured on no agent**, so no board is read with terms today.
- **A starred company on a big filtered board is scored on roles the fetch closes.** All countries is the workaround.
- **`SMALL` (300), `STALE_DAYS` (21), the guard's 10% and `DETAIL_CAP` (200) are code constants**, and the pages hard-code 300 in tooltips.
- **`The todo list.md` holds items the code has done** (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.
+8 -8
View File
@@ -6,7 +6,7 @@ The board filter this plan had as WP11 landed as the agents' reach (`decisions.m
## Where it stands
76 companies are on the Watchlist, each with the careers page opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
In the seed (`seed/companies.json`, 362 companies, 286 with a board), 76 companies have no board, each with the careers page opened on 2026-09-29 and, but for nine, a reason; 35 carry `unread`: a program can read them and nothing does yet. The live Watchlist has moved on: 89 companies have no board (14 more were added by hand), Infineon and Microsoft are attached as Eightfold boards, and Novonesis, Darktrace and Criteo as Workday boards.
| Route | Companies | Roles seen |
|---|---|---|
@@ -16,11 +16,11 @@ The board filter this plan had as WP11 landed as the agents' reach (`decisions.m
| A host that is read, with a gap | Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures.
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures; `own-1.md` and `own-2.md` name `out-own-1.json` and `out-own-2.json`, which do not exist, and `survey-own.json` is the whole record.
## Decisions to take first
**A listing that knows no title.** A sitemap lists URLs, and the listing rewrites the title on every pull, so a sitemap reader would show a title made from the URL. The contract already has `Reads(title="provisional")` (`plugins/readers/_contract.py`), but nothing honours it: `upsert_role` still overwrites a title that `detail` filled. WP9 adds that rule.
**A listing that knows no title.** A sitemap lists URLs, and the listing rewrites the title on every pull, so a sitemap reader would show a title made from the URL. The contract already has `Reads(title="provisional")` (`plugins/readers/_contract.py`), but nothing honours it: `upsert_role` overwrites the title on every pull. And `detail` cannot fill a title yet, since `_runner.DETAIL_FIELDS` has no `title` and the contract says the listing rewrites it. WP9 adds both rules: `title` in `DETAIL_FIELDS`, so the runner takes it from `detail`, and `upsert_role` keeping a stored title over a provisional one.
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
@@ -30,26 +30,26 @@ Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tes
| WP | Model | Builds | Needs |
|---|---|---|---|
| 9 Provisional title | Opus | `upsert_role` keeps a stored title over a provisional one, and its tests; owns `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | |
| 10 Sitemap reader | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
| 9 Provisional title | Opus | `title` in `DETAIL_FIELDS`, `upsert_role` keeping a stored title over a provisional one, and their tests; owns `plugins/readers/_runner.py`, `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | `plan-remote.md` WP-A |
| 10 Sitemap reader | Sonnet | `sitemap`: role URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9.
WP9 owns files that `plan-remote.md` WP-A also owns (`app/storage/`, `plugins/`, `tests/`) and `plan-review.md` R1 edits (`_runner.py`, `ats.py`), so it waits until those have merged. WP12 to WP15 own new modules only and can start at once and in parallel. WP10 waits for WP9.
### WP10 notes: Mambu and Arm
Both were read live on 2026-09-29 without a browser or a block.
- **Mambu, classic iCIMS.** `https://careers-mambu.icims.com/sitemap.xml` has one `<loc>` per job (`/jobs/{id}/{slug}/job`) with `lastmod`, plus `/jobs/intro`; robots.txt allows it and lists the sitemap. The detail `.../jobs/{id}/x/job?in_iframe=1` carries a JSON-LD `JobPosting` with `datePosted`, `description`, `jobLocation[].address.addressCountry`, `validThrough` and `employmentType`. Tenants use other subdomain prefixes, so the target stores the whole host. 8 roles.
- **Mambu, classic iCIMS.** `https://careers-mambu.icims.com/sitemap.xml` has one `<loc>` per role (`/jobs/{id}/{slug}/job`) with `lastmod`, plus `/jobs/intro`; robots.txt allows it and lists the sitemap. The detail `.../jobs/{id}/x/job?in_iframe=1` carries a JSON-LD `JobPosting` with `datePosted`, `description`, `jobLocation[].address.addressCountry`, `validThrough` and `employmentType`. Tenants use other subdomain prefixes, so the target stores the whole host. 8 roles.
- **Arm, TalentBrew (Radancy).** The `*-arm.icims.com` hosts only redirect to `careers.arm.com`. `https://careers.arm.com/sitemap.xml` lists all 399 `/job/...` URLs in one request, without titles. The detail's JSON-LD has `datePosted` (unpadded, "2026-6-25"), `description`, `identifier` and `jobLocation[].address`. The paged list at `/search-jobs/results` is HTML inside JSON and only answers the full parameter set; its `data-total-results` counts content pages, `data-total-job-results` counts roles.
## Planner, after the merges
- Attach each board with `app companies set-board`, which reads it first and clears the reason.
- Bring `seed/companies.json` in line with the live database: the board of every company that has one, the reason of every company that has none.
- Bring `seed/companies.json` in line with the live database: the board of every company that has one (24 more than the seed's 286, among them Microsoft, Infineon, Novonesis, Darktrace, Criteo and NVIDIA), the reason of every company that has none (five differ; `app companies seed` never replaces a board, so this is done by hand).
- Declare `Fields(posted="trusted")` on a reader whose date is a posting date, with the evidence in the docstring.
- Restart the server by its PID after a backup in `../jobboard-backups/`.
- Find the career host of the nine Teamtailor companies by opening each careers page.
+3 -1
View File
@@ -42,7 +42,9 @@ Open roles without copies: 20,578. `remote` is `remote` on 3,177, `hybrid` on 3,
4. **A role whose area is not stated matches no remote chip in the feed.** In an agent's gate it is empty, which by the gate rule neither passes nor fails, so it ranks, `extract` reads it and the rescore gates it.
5. **For remote, the model wins, as it does for seniority.** `remote_model` is new, and `remote` is the model's value, else the host's word, else the place line. The prompt says remote when the hire works from home most of the time, even with a regular office week, and hybrid when there are set office days each week. Black Forest Labs then reads as remote.
6. **Extract reads the area once more for the roles it has read.** A role waits while `remote_where_model` is NULL (never asked). `[]` means asked and not stated. The existing per-agent bounds apply, so no new cost bound is needed.
7. **Agents get a gate field `remote_from`, whose values are place tokens.** It matches as the chip does. The live agent's `remote: [remote]` gate keeps working until the owner edits it, since the Where block then writes `remote_from`.
7. **An agent's scope is explicit** (owner, 2026-09-30). The `require` gate on `country`, Europe for the owner's agent, is the first line of the Where block: "Only roles in: Europe / these countries / everywhere". A company set to All countries is read within it, as `reach` already does. A remote role is in scope when its area overlaps the scope or is Anywhere, so the scope gate reads a remote role through its area, and "Home Based - Worldwide" stays in Europe's scope.
8. **Plain Remote is every remote role within the scope.** The token `remote:any` (label "Remote") and the agent's existing `remote: [remote]` any gate both mean it. "Remote · Denmark" narrows it to the roles open to someone living in Denmark, which includes Europe-wide and Anywhere roles. There is no "Remote · anywhere" suggestion. Anywhere is part of every remote chip, and the badge still says "remote · anywhere".
9. **Agents get a gate field `remote_from`, whose values are place tokens.** It matches as the chip does and is the Where block's "only when open to someone in" line. The `remote: [remote]` gate stays the plain "remote within the scope" line, so the live agent loads and saves unchanged.
## Contract between the packages
+256 -201
View File
@@ -1,52 +1,89 @@
# Where roles come from
This file lists every source roledesk reads roles from, how each is read and what it cannot deliver, as of the code and the live database of 2026-09-29. The contract for writing a reader (a host module or a search provider) is the docstring of `plugins/readers/__init__.py`, with its types and the declaration check in `plugins/readers/_contract.py` and the conformance test in `tests/test_readers_contract.py`, not this file.
This file lists every source roledesk reads roles from, how each is read and what it cannot deliver, as of the code on main and the live database of 2026-09-30. The contract for writing a reader (a host module or a search provider) is the docstring of `plugins/readers/__init__.py`, with its types and the declaration check in `plugins/readers/_contract.py` and the conformance test in `tests/test_readers_contract.py`, not this file.
The code has two plugins: `ats` (`plugins/ats.py`) reads the board of a company someone listed, with one module per host in `plugins/readers/`; `search` (`plugins/search.py`) runs saved queries on sites that list roles across companies. Below, a module of `ats` is a "host" and a module of `search` is a "provider". A company's list of roles on a host is its "board"; a board is attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `enrich:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
## Flow
The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section says what each step leaves behind.
1. A reader declares what it does (`READS`) and how to ask and parse (`feed_url` and `parse`, or `page` and `rows`, and optionally `detail` and `facets`).
2. `pull` queues batches of 10 targets (`app/jobs/pull.py`). In each batch `ats` or `search` calls `_runner.read`: it asks the reader for the facets a filter needs, builds the series of requests from the reach, reads the listing whole or in pages, continues or starts a sweep, and applies the 10% guard.
3. The plugin stores every row with `upsert_role` and then stamps the target's state (`self.state`, the `state` table). A search stamps only `ok`, `complete` and `ids`. Keys are `ats:<key>:<target>`:
| Key | Written by | Read by |
|---|---|---|
| `ok` | `_runner.stamp`, the run marker, after every row was stored | `close_boards`: a target whose marker is not this run's did not answer |
| `complete` | `stamp` | `close_boards`: only a complete read closes |
| `ids` | `stamp`, the compound ids the read listed | `close_boards`: closes open roles not in it |
| `filtered` | `stamp`, a read by country, place or remote | `close_boards`: keeps roles with a vote or an application |
| `narrowed` | `stamp`, a read by fetch terms | `close_boards`: closes nothing |
| `swept` | `stamp`, a complete read that finished a sweep begun on an earlier pull | `close_boards`: applies the two-strikes rule |
| `sweep` | `stamp`, the cursor of a read cut at `MAX_PAGES` | `_runner.read` on the next pull |
| `count` | `stamp`, the roles a whole read listed | `_runner.read`, the 10% guard |
| `missing` | `close_boards` (`_close_swept`), the ids the last finished sweep missed | the next finished sweep |
| `failing` | `ats._board` through `record_failure`, `{since, reason}`; removed on the next answer | `close_boards` (21 days), the Companies list, `enrich` |
| `facets` | `_runner.facets_for`, kept one day | `reach.site_facets`, the Companies line |
| `size`, `sizechecked` | `ats.keep_size`: `set_board`, Recheck, `app companies size`, or the first read of an unsized board | `reach.sizes`, which decides whether the board is filtered |
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`enrich`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `enrich.plan` through `detail_due` |
4. `close_boards`, the settle of the pull, closes what a complete read did not list, closes a board that failed for `STALE_DAYS`, and releases the run.
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score`, `enrich` (reads the role pages of the best-ranked roles without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `summarise`. Extract and summarise are switches under Runs; enrich is on by default.
## The three ways a source is read
**Whole board.** One request returns every role on the board with its text. The read is complete whenever the answer parses, so the pull can close every role the answer no longer lists.
**Whole board.** One request returns every role, with its text for most hosts. The read is complete whenever the answer parses, so the pull can close every role the answer no longer lists.
**Paged.** The list arrives in pages of a fixed size, read one after another 0.5 s apart, at most 100 pages per board. The read is complete when the number of distinct roles read equals the total the host states, or, for a host that states none, when the read reaches an empty page. A board larger than 100 pages is read incomplete on every pull (see the shared rules).
**Paged.** The list arrives in pages of a fixed size (`READS.page_size`), read one after another `PAGE_DELAY` (0.5 s) apart, at most `MAX_PAGES` (100) pages a pull. The read is complete when the number of distinct roles read equals the total the host states (`complete="stated"`), or, for a host that states none, when the read reaches an empty page (`complete="end"`); a Jobindex search is never complete (`complete="never"`). A board of more than 100 pages is swept: each pull stores what it read and the next continues from the saved cursor, and the read is complete when a sweep reaches the end. Roles closing between pulls shift an offset-paged host's later roles back past the cursor, so a role of a swept board closes only when two finished sweeps in a row missed it. A host with a `ceiling` (Workday 2,000) answers nothing past it: a read that reaches it is incomplete, is not swept and closes nothing, so a filter must bring the board under it.
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text. The text is on each role's own page, which is read once per role: at most 200 roles per board per pull, roles never tried before roles whose read failed. A new board of 800 roles has all its text after four pulls. Until its own page is read, a role is in the feed with title, place and the list's date, and it is ranked without its text. Three consequences follow from the code:
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The `enrich` stage reads those pages after scoring (`app/jobs/enrich.py`):
- `embed` encodes a role once. A role listed before its page was read is encoded from its title; when the text arrives on a later pull its vector is dropped, and the next `embed` encodes it again with the text. Until then its fit is the title's.
- The per-role read may fill only description, place, remote, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
- It takes open roles without text that an enabled agent with a prompt ranks ungated within its best `enrich.top` (500) at `enrich.min_score` (0.5) or above, best score first.
- It reads at most `enrich.per_board` (200) roles per board and `enrich.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
- A read that gave no text is tried again after `enrich.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
- It runs after every pull, after a saved agent and after a star change, not after a vote, and it can be switched off under Runs.
A role below the bar for every agent is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles that rank do. Four consequences follow from the code:
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow `enrich` encode it again.
- A role page may fill only `_runner.DETAIL_FIELDS`: description, location and places, teaser, remote, country, employment, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
- Completeness, and so closing, is decided by the list alone. A board of 800 roles closes its roles from the first pull.
- SmartRecruiters and Workable list their roles in pages and belong here too, since their list has no text.
## Overview
Numbers are from the live database after the pull of 2026-09-29. For the "list, then each role" hosts the description share is after one pull and rises by up to 200 roles per board per pull.
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles `enrich` has read.
| Source | Boards | Open roles | How it is read | Roles per request | Description | Pay | Limit that matters most |
|---|---|---|---|---|---|---|---|
| Greenhouse | 85 | 10,216 | whole board | all | in the list | figures | a board over 96 MiB loses its text and pay |
| SmartRecruiters | 10 | 7,238 | paged | 100 | none | none | no text: ranked on the title alone |
| Ashby | 69 | 4,261 | whole board | all | in the list | figures | an unknown board name answers an empty board |
| SuccessFactors | 14 | 3,738 | list, then each role | the site's own | per role | none | default locale only |
| Workday | 13 | 2,377 | list, then each role | 20 | per role | none | 2,000 roles per board; country arrives per role |
| Lever | 24 | 1,033 | whole board | all | in the list | figures | one place per role |
| Phenom | 3 | 1,007 | list, then each role | 100 | per role | none | three sites checked |
| Oracle Recruiting Cloud | 4 | 826 | list, then each role | 200 | per role | none | no pay field |
| Eightfold | 1 | 630 | list, then each role | 10 | per role | none | 1,000 roles per board |
| Teamtailor | 35 | 598 | paged | 50 | in the list | none | no pay; nine companies without a found site |
| Jobindex (search) | – | 538 | paged | 20 | excerpt in the list | none | excerpt only; at most 200 results per search |
| Platsbanken (search) | – | 446 | paged | 100 | in the list | none | at most 1,000 results per search |
| Personio | 3 | 322 | whole board | all | in the list | figures | one place, no country |
| Recruitee | 8 | 177 | whole board | all | in the list | figures | an empty board looks like no board to discovery |
| Workable | 7 | 153 | paged | 10 | none | none | no text: ranked on the title alone |
| Jibe | 1 | 140 | paged | 100 | in the list | none | Booking Holdings' roles read with Booking.com's |
| Greenhouse | 87 | 10,457 | whole board | all | in the list | figures | a board over 96 MiB loses its text and pay |
| Ashby | 74 | 4,964 | whole board | all | in the list | figures | an unknown board name answers an empty board |
| Workday | 17 | 2,041 | list, then each role | 20 | per role (1,280 read) | none | 2,000 roles per board; country arrives per role |
| Lever | 26 (25 read) | 1,347 | whole board | all | in the list, the intro paragraph only | figures | text is the opening paragraph |
| Oracle Recruiting Cloud | 5 | 937 | list, then each role | 200 | per role (603 read), teaser in the list | none | no pay field |
| SuccessFactors | 14 | 749 | list, then each role | the site's own | per role (681 read) | none | default locale only |
| Teamtailor | 38 | 611 | paged | 50 | in the list | none | no pay; nine companies without a found site |
| Jobindex (search) | 12 saved (8 on) | 562 | paged | 20 | teaser | none | teaser only; at most 200 results per search; never complete |
| Platsbanken (search) | 6 saved | 439 | paged | 100 | in the list | none | at most 1,000 results per search |
| Phenom | 5 | 389 | list, then each role | 100 | per role (142 read), teaser in the list | none | locale spelling known from three sites |
| SmartRecruiters | 11 | 343 | list, then each role | 100 | per role (1 read) | none | one country a request; remote is not filtered |
| Personio | 3 | 320 | whole board | all | in the list | figures | no country field |
| Eightfold | 3 | 185 | list, then each role | 10 | per role (158 read) | none | 10 per request, so 1,000 roles a pull |
| Recruitee | 8 | 176 | whole board | all | in the list | figures | an empty board looks like no board to discovery |
| Workable | 8 | 157 | list, then each role | 10 | per role (1 read) | none | answers 429 to bursts |
| Jibe | 1 | 137 | paged | 100 | in the list | none | Booking Holdings' roles read with Booking.com's |
| Deel | 1 | 96 | whole board | all | in the list | line only | pay has no period |
| Gem | 2 | 44 | list, then each role | all | per role | none | undocumented query, can break |
| Jobylon | 2 | 33 | list, then each role | all | per role | none | scraped from the careers widget |
| Gem | 2 | 45 | list, then each role | all | per role (44 read) | none | undocumented query, can break |
| Jobylon | 2 | 33 | list, then each role | all | per role (33 read), teaser in the list | none | scraped from the careers widget |
| HR Manager | 1 | 7 | paged | 100 | in the list | none | the portal's robots.txt disallows reading; the owner has not decided |
| Rippling | 1 | 2 | list, then each role | all | per role | figures | text, date and pay cost one request per role |
| Pinpoint | 1 | 2 | whole board | all | in the list | figures | no date |
| HiBob | 1 | 2 | whole board | all | in the list | figures | pay never seen filled |
| BambooHR | 1 | 2 | list, then each role | all | per role | none | paging of a large board unverified |
| Adzuna (search) | – | 0 | paged | 50 | excerpt in the list | figures | never run live; needs a key |
| HR Manager | 0 | 0 | paged | 100 | in the list | none | not attached: the portal's robots.txt disallows reading |
| Adzuna (search) | 0 | 0 | paged | 50 | teaser | figures | never run live; needs a key |
"Figures" means yearly `salary_min`, `salary_max` and `currency`, plus a pay line; "line only" means a pay line (`salary_text`) and no figures. Figures are yearly: a monthly figure is multiplied by 12 where the host states the period, and an hourly, daily or weekly one is kept as a line only.
@@ -54,28 +91,28 @@ Numbers are from the live database after the pull of 2026-09-29. For the "list,
| Limit | Value | What it means |
|---|---|---|
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade `pull`, `embed`, `score`, `extract`, `score`, `summarise`; `extract` and `summarise` are off by default. |
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. One pull runs at a time. |
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); enrich is on by default. |
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. Enrich reads one batch per board on the same pool. One pull runs at a time. |
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request; 15 s per request for the search providers | A slow board fails for this pull and is asked again next pull. |
| Pages per board | 100 | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
| Wait | 0.5 s before every page after the first and before every per-role read; none between search pages | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
| Per-role reads | 200 per board per pull | The rest wait for the next pull. A 404 or 410 counts as read; another failure is retried on a later pull. |
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing. A role listed again reopens. A board switched off closes its roles at once. |
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. |
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize. |
| A board that fails for 21 days | all its open roles close at the next pull | The next answer reopens what it lists. |
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request | The same for boards, role pages and searches. A slow board fails for this pull and is asked again next pull. |
| Pages per board | 100 a pull | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
| Wait | 0.5 s before every page after the first, for boards and searches alike, and before every role page | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
| Per-role reads | `enrich`, after scoring: 200 per board and 1,000 per run, ranked roles first | See "List, then each role". Roles the agents do not rank are not read. |
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing. A role listed again reopens. A board switched off closes its roles at once. A filtered read keeps roles with a vote or an application; a read by fetch terms closes nothing. |
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. A Jobindex search is never complete, so its roles close by the 21 days only. |
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, and a filtered, terms or `empty="suspect"` read that lists under 10% of what the same read listed last time (`SUSPECT_SHARE`). A search that fails is skipped for the pull and leaves no record. |
| A board that fails for 21 days | all its open roles close at the next pull | The next answer reopens what it lists. `STALE_DAYS` is the one 21. |
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a per-role read: the per-role reads of that board stop until the next pull | Nothing is closed on a list cut short by a 429. A search that fails is skipped for the pull. |
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next enrich run | Nothing is closed on a list cut short by a 429. |
How far a board is read: a pull asks each board over 300 roles for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of 30 times, unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application.
How far a board is read: a pull asks each board of 300 roles or more for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of once per country (33 codes today), unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application. A board's size is set once, when it is added, on Recheck or by `app companies size`, and a pull does not rewrite it; a board with no size yet is sized first in its batch and then read as its size says, and a board whose reader cannot narrow the reach is read whole and sized by that read. A host that cannot filter a dimension ignores it: SmartRecruiters, Phenom, SuccessFactors and Eightfold have no remote filter, so their reach asks for countries only.
Fetch terms narrow a wide read by words. Each agent has a list of them (`fetch_terms`, set in the agent editor); the terms of a pull are the union over the agents that are on, and none when one of those agents has none. They go only to a board read with the wide reach: a company set to All countries, or a board whose reach is more countries than a one-country host is asked for (`MAX_SERIES`, 8). The narrow reach (Denmark, Sweden) leaves at most about 150 roles on any big board, so it takes no terms, and nor does a board under 300 roles, one never read whole, or one with its own reach. A company set to All roles takes none on any of its boards. A host that takes a keyword gets one series per term, unioned by id, each combined with the country series (an AND on every host the survey tested). When terms times the country series exceed 8, the place part of the reach goes first, whole, and the terms stay; more than 8 terms read the board unfiltered. Every host matches the whole text, not only the title, so a term like `engineer` cuts less than it looks. A read with terms is never the whole board: it records no size, closes no role it did not list, and the 10% guard holds it only to a read with the same terms and places. The search providers take no terms: each saved search has its own query.
Fetch terms narrow a wide read by words. Each agent has a list of them (`fetch_terms`, set in the agent editor); the terms of a pull are the union over the agents that are on, and none when one of those agents has none. They go only to a board read with the wide reach: a company set to All countries, or a board whose reach is more countries than a one-country host is asked for (`MAX_SERIES`, 8). The narrow reach (Denmark, Sweden) leaves at most about 150 roles on any big board, so it takes no terms, and nor does a board under 300 roles or one with its own reach. A company set to All roles takes none on any of its boards. A host that takes a keyword gets one series per term, unioned by id, each combined with the country series; the survey shows that as an AND on SmartRecruiters, Eightfold, SuccessFactors, Phenom and Workday. When terms times the country series exceed 8, the place part of the reach goes first, whole, and the terms stay; more than 8 terms read the board unfiltered. The survey found the keyword matching the ad text and not only the title where it could tell, and could not tell on SuccessFactors, Workday, Workable and Jibe (`docs/filter-survey.md`), so a term like `engineer` cuts less than it looks. A read with terms is never the whole board: it records no size, closes no role it did not list, and the 10% guard holds it only to a read with the same terms and places. The search providers take no terms: each saved search has its own query.
What no source does:
- Filter on the host's side for a host other than SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom and Eightfold. Greenhouse, Ashby, Recruitee and the rest are read whole.
- Filter on the host's side for a host other than SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom and Eightfold. Greenhouse, Ashby, Recruitee and the rest are read whole. The search providers filter by their own query, place and country.
- Filter by category or level at the source. Workday, Oracle, Jibe and Eightfold state category ids, but nothing maps an agent's terms to them yet (`docs/filter-survey.md`).
- Read more than one board per company. DFDS is read from its Danish Teamtailor board only.
- Use a key taken from a site's script. Maersk, New Work and Welcome to the Jungle are on the Watchlist for that reason.
@@ -89,14 +126,14 @@ What no source does:
| | |
|---|---|
| Target | the board token, e.g. `greenhouse:trustpilot` |
| Companies | 85 boards: Trustpilot, Cognite, Wolt |
| Companies | 87 boards: Trustpilot, Cognite, Wolt |
| List | `boards-api.greenhouse.io/v1/boards/<token>/jobs` with `content=true` and `pay_transparency=true`, the whole board in one answer; no total |
| Requests per pull | 1; 2 when the first answer is over 96 MiB |
| Description | in the list, full text (`content`); live: 10,216 of 10,216 |
| Date | `first_published`, else `updated_at`; trusted as the posting date |
| Place and country | `location.name`, one place; the app reads the country from that line (9,307 of 10,216) |
| Remote | none from the host; the app reads remote or hybrid from the place line (2,131 roles) |
| Pay | figures from `pay_input_ranges` when there is exactly one range; a range whose title says hour or day is a line only; any other range is taken as yearly. Several ranges give one line per range and no figures. Live: 2,170 with figures, 2,503 with a line |
| Description | in the list, full text (`content`) |
| Date | `first_published`, else `updated_at`, which the host moves on an edit; trusted as the posting date |
| Place and country | every `offices` entry; `location.name` when the role has no office or the name says more, with placeholder names such as "Hybrid" dropped. The host states no country; it is resolved from the places |
| Remote | none from the host; prep reads remote, hybrid or onsite from the place words |
| Pay | figures from `pay_input_ranges` when there is exactly one range; a range whose title says hour or day is a line only; any other range is taken as yearly. Several ranges give one line per range and no figures |
| Found by discovery | yes, from a link to `boards.greenhouse.io`, `job-boards.greenhouse.io` or the embed script; a guessed name is probed first of all hosts |
| Limitations | Over 96 MiB the board is read again without `content=true`: titles land, text and pay of the whole board are lost. An `eu.greenhouse.io` link is read from the main API. The pay flag must be sent, or no range comes back. |
@@ -105,32 +142,34 @@ What no source does:
| | |
|---|---|
| Target | the site name, or `eu/<site>` on Lever's EU instance, e.g. `lever:spotify`, `lever:eu/tomtom` |
| Companies | 24 boards: Spotify, TomTom, Pipedrive |
| Companies | 26 boards, one switched off: Spotify, TomTom, Pipedrive |
| List | `api.lever.co/v0/postings/<site>?mode=json` (`api.eu.lever.co` for `eu/`), the whole board in one answer; no total, no paging parameter sent |
| Requests per pull | 1 |
| Description | in the list, `description` (else `descriptionPlain`); the module reads no `lists` or `additional` sections, so it is the opening text; live: 1,023 of 1,033 |
| Description | in the list, `description` (else `descriptionPlain`); the module reads no `lists` or `additional` sections, so it is the opening text |
| Date | `createdAt`; trusted |
| Place and country | `categories.location`, one place; `allLocations` is not read; country from that line (1,012 of 1,033) |
| Remote | `workplaceType`, the host's own word (1,032 of 1,033) |
| Pay | figures from `salaryRange` for `per-year-salary` and `per-month-salary` (×12); other intervals a line only; live: 207 with figures, 210 with a line |
| Place and country | `categories.location` and every `allLocations` entry; the host's `country` is handed on when the role has one place, otherwise the country is resolved from the places |
| Remote | `workplaceType`, the host's own word |
| Employment | `categories.commitment`, the host's own word |
| Pay | figures from `salaryRange` for `per-year-salary` and `per-month-salary` (x12); other intervals a line only |
| Found by discovery | yes, from a `jobs.lever.co` or `api.lever.co` link on either instance; a guessed name is probed on the US instance, then on the EU one |
| Limitations | The same site name on the two instances is two companies. Only one place per role is read. |
| Limitations | The same site name on the two instances is two companies. Only the opening text of an ad is read (`docs/open.md`). |
### Ashby
| | |
|---|---|
| Target | the board name, which may carry a dot or be a uuid, e.g. `ashby:pleo`, `ashby:jimdo.com` |
| Companies | 69 boards: Pleo, Too Good To Go, Lovable |
| Companies | 74 boards: Pleo, Too Good To Go, Lovable |
| List | `api.ashbyhq.com/posting-api/job-board/<name>?includeCompensation=true`, the whole board in one answer; no total |
| Requests per pull | 1 |
| Description | in the list, full text (`descriptionHtml`); live: 4,261 of 4,261 |
| Description | in the list, full text (`descriptionHtml`) |
| Date | `publishedAt`; trusted |
| Place and country | `location`, one place; secondary places are not read; country from that line (4,003 of 4,261) |
| Remote | `isRemote`, true or false (2,625 with a value) |
| Pay | figures from the first `Salary` component with interval `1 YEAR` or `1 MONTH` (×12); an hourly one a line only; bonus and equity appear in the line only. Live: 1,482 with figures, 1,499 with a line |
| Place and country | the primary `location` and every `secondaryLocations` entry; country resolved from the places |
| Remote | `workplaceType` (Remote, Hybrid, OnSite) as the host words, else `isRemote`, which is true for hybrid roles too |
| Employment | `employmentType`, the host's own word |
| Pay | figures from the first `Salary` component with interval `1 YEAR` or `1 MONTH` (x12); an hourly one a line only; bonus and equity appear in the line only; when no component states a bound the line falls back to `compensationTierSummary` or `scrapeableCompensationSalarySummary` |
| Found by discovery | yes, from a `jobs.ashbyhq.com` link; a guessed name is probed |
| Limitations | The pay flag must be sent, or no compensation comes back. An unknown board name answers an empty board, so a board with no open roles looks like no board to discovery, and a mistyped name reads as an empty, complete board. The name is read without case, so a lowercased slug works. |
| Limitations | The pay flag must be sent, or no compensation comes back. An unknown board name answers an empty board, so a board with no open roles looks like no board to discovery. `READS.empty="suspect"` keeps a known board from closing its roles on such an answer; a mistyped name attached as a new board still reads as an empty, complete board (`docs/open.md`). |
### Recruitee
@@ -140,11 +179,12 @@ What no source does:
| Companies | 8 boards: bunq, idealo, Climeworks |
| List | `https://<sub>.recruitee.com/api/offers/`, the whole board in one answer; no total |
| Requests per pull | 1 |
| Description | in the list, `description` only (`requirements` is not read); live: 177 of 177 |
| Description | in the list, `description` only (`requirements` is not read) |
| Date | `published_at`, else `created_at`; trusted |
| Place and country | `location`, else `city`, one place; country from that line (149 of 177) |
| Remote | `remote`, true or false (29 with a value) |
| Pay | figures from `salary` for period `year` and `month` (×12); other periods a line only; live: 12 with figures, 41 with a line |
| Place and country | every `locations` entry as "city, state, country"; `location` or `city` when there are none; a remote offer with no place gets the place "Remote" |
| Remote | `remote`, true or false |
| Seniority, employment | `experience_code` and `employment_type_code`, the host's own words |
| Pay | figures from `salary` for period `year` and `month` (x12); other periods a line only |
| Found by discovery | yes, from a `<sub>.recruitee.com` link; a guessed name is probed |
| Limitations | A board with no offers answers an empty list and looks like no board to discovery. A board on the company's own domain is not read (planned in wave 2, Quantum Systems). |
@@ -156,11 +196,12 @@ What no source does:
| Companies | 3 boards: Penneo, tado, 1KOMMA5° |
| List | the XML feed `https://<sub>.jobs.personio.de/xml`, the whole board in one answer; no total |
| Requests per pull | 1 |
| Description | in the list, every `jobDescription` value joined; live: 321 of 322 |
| Description | in the list, every `jobDescription` value joined |
| Date | `createdAt`; trusted |
| Place and country | `office`, one place, no country field; country from that line (232 of 322) |
| Remote | true when the office text contains "remote", else unknown (37 roles) |
| Pay | figures from `salaryInformation` for `yearly` and `monthly` (×12); hourly, daily, weekly a line only; live: 1 role |
| Place and country | `office` and every `additionalOffices` office; the feed has no country field, so it is resolved from the places |
| Remote | none for the role; a place whose text contains "remote" is a remote place, and prep makes the role remote if any place is |
| Seniority, employment | the host's `seniority` and its contract (or `schedule`) words |
| Pay | figures from `salaryInformation` for `yearly` and `monthly` (x12); hourly, daily, weekly a line only; rare |
| Found by discovery | yes, from a `jobs.personio.de` or `jobs.personio.com` link; a guessed name is probed |
| Limitations | The feed has no role link; it is built as `/job/<id>`. Only the feed's default language is read. The company Personio itself is on the Watchlist as `blocked`. |
@@ -172,11 +213,12 @@ What no source does:
| Companies | 1 board: Xeneta |
| List | `https://<sub>.pinpointhq.com/postings.json`, the whole board in one answer; no total, no paging keys |
| Requests per pull | 1 |
| Description | in the list, full text: description, responsibilities, skills and benefits sections; live: 2 of 2 |
| Description | in the list, full text: description, responsibilities, skills and benefits sections |
| Date | none: the list states none and the role page is not read for it; the role counts as posted when first seen |
| Place and country | `location.name` (else `city`) plus `province`, one place; country from that line |
| Place and country | `location.name` (else `city`) plus `province`, one place; country resolved from that line |
| Remote | `workplace_type`: onsite, hybrid, remote |
| Pay | figures when `compensation_visible`, for annual and monthly (×12) frequencies; others a line only. The frequency spellings are guessed, since no board read so far shows pay |
| Employment | `employment_type`, the host's own word |
| Pay | figures when `compensation_visible`, for annual and monthly (x12) frequencies; others a line only. The frequency spellings are guessed, since no board read so far shows pay |
| Found by discovery | yes, from a `<sub>.pinpointhq.com` link; not probed |
| Limitations | Written from Xeneta's board, two roles. Whether a large board pages `/postings.json` is unverified. A board shared by several brands (inMusic with Native Instruments) is read whole, with no filter on division. |
@@ -188,11 +230,12 @@ What no source does:
| Companies | 1 board: Klarna |
| List | the HTML page `https://jobs.deel.com/<slug>`, from which the Next.js server payload is read; the whole board in one answer; no total |
| Requests per pull | 1 |
| Description | in the list, full text (`richtextDescription`); live: 96 of 96 |
| Description | in the list, full text (`richtextDescription`) |
| Date | the publication's `createdAt`, else the posting's; not trusted, so the role counts as posted when first seen |
| Place and country | every `jobLocations` name, joined with "; "; country from that line (91 of 96) |
| Remote | `workArrangementEnum`: onsite, hybrid, remote (95 of 96) |
| Pay | line only, when visible: currency and two amounts with no period, so no yearly figure can be told from a monthly one (90 roles) |
| Place and country | every `jobLocations` name as a place, repeated names dropped, each carrying the role's remote flag; country resolved from the places |
| Remote | `workArrangementEnum`: ON_SITE, HYBRID, REMOTE |
| Employment | the first `jobEmploymentTypes` name |
| Pay | line only, when visible: currency and two amounts with no period, so no yearly figure can be told from a monthly one |
| Found by discovery | yes, from a `jobs.deel.com` link, case kept; not probed |
| Limitations | Deel's `/api/` is disallowed by robots.txt, so the board is read out of the page's React payload, which a Next.js release can change. Written from Klarna's board only. A wrongly cased slug answers `Job Board Not Found` and fails the board. |
@@ -202,68 +245,34 @@ What no source does:
|---|---|
| Target | the subdomain of `careers.hibob.com`, e.g. `hibob:dixa` |
| Companies | 1 board: Dixa |
| List | `https://<sub>.careers.hibob.com/api/job-ad` with the header `companyIdentifier: <sub>`, the whole board in one answer; no total |
| List | `https://<sub>.careers.hibob.com/api/job-ad` with the header `companyIdentifier: <sub>`, the whole board in one answer; no total. Declared as paged with `complete="end"` because `page` may send headers; it reads one page |
| Requests per pull | 1 |
| Description | in the list, full text: description, responsibilities, requirements, benefits; live: 2 of 2 |
| Description | in the list, full text: description, responsibilities, requirements, benefits |
| Date | `publishedAt`; not trusted, so the role counts as posted when first seen |
| Place and country | `site` plus `country`, one place |
| Place and country | `site` plus `country`. A company that hires at two sites posts one ad per site with its own id (`places="copies"`); `app/dupes.py` folds them into one role |
| Remote | `workspaceTypeId`: remote, hybrid, onsite |
| Pay | figures from the `payTransparency*` fields for year and month (×12); hour, day and week a line only. No board seen had them filled, so the period spellings are guessed |
| Employment | `employmentType`, the host's own word |
| Pay | figures from the `payTransparency*` fields for year and month (x12); hour, day and week a line only. No board seen had them filled, so the period spellings are guessed |
| Found by discovery | yes, from a `<sub>.careers.hibob.com` link; not probed |
| Limitations | Written from two boards (Dixa, Liquidia). The header is the public subdomain, not a key; without it the API answers 401. |
## Hosts read in pages
### SmartRecruiters
| | |
|---|---|
| Target | the company identifier, e.g. `smartrecruiters:BoschGroup` |
| Companies | 10 boards: Bosch (about 4,800 roles), Delivery Hero, Wise |
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
| Requests per pull | roles / 100, rounded up; 49 for Bosch |
| Description | none. The list carries no text, and the module does not read the per-posting endpoint, which would cost one request per role. These 7,238 roles are ranked on the title alone, and `extract` and `summarise` have nothing to read. Live: 0 of 7,238 |
| Date | `releasedDate`; trusted |
| Place and country | `fullLocation`, else city and country; country 7,230 of 7,238 |
| Remote | `location.remote`, true or false (122 remote) |
| Pay | none: the list states no pay |
| Found by discovery | yes, from a `jobs.`, `careers.` or `api.smartrecruiters.com` link; a guessed name is probed |
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see below. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
| Limitations | 10,000 roles per board (100 pages). The identifier is read without case, so a lowercased slug works. An unknown identifier answers an empty list, not an error: a mistyped slug reads as a complete board with no roles and would close a company's roles. |
The reader's "Fetch the posting" button shows for these roles, since their description is empty; it reads one posting page on demand and does not feed `embed` or `extract`. Whether it reads a SmartRecruiters page cleanly was not checked.
### Workable
| | |
|---|---|
| Target | the account name, e.g. `workable:huggingface` |
| Companies | 7 boards: Hugging Face, Kahoot!, Starling Bank |
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
| Requests per pull | roles / 10, rounded up, 0.5 s apart; at most 16 for any board attached (153 roles in all) |
| Description | none. The text is only on a per-role endpoint, which the module does not call. These roles are ranked on the title alone, and `extract` and `summarise` have nothing to read. Live: 0 of 153 |
| Date | `published`; trusted |
| Place and country | city, region and country joined; country 153 of 153 |
| Remote | the `remote` flag or `workplace` remote gives remote; the rest is stored as unknown (22 remote); hybrid is not read |
| Pay | none: the list states no pay |
| Found by discovery | yes, from an `apply.workable.com` or `<account>.workable.com` link; not probed, since a run of guesses trips Workable's rate limit |
| Limitations | 1,000 roles per board (100 pages of 10). Workable answers bursts with 429, which is why pages are 0.5 s apart; a 429 after the first page ends the read incomplete. The role link is built from the shortcode. |
### Teamtailor
| | |
|---|---|
| Target | the subdomain of `teamtailor.com`, or the whole host of a career site on the company's domain, e.g. `teamtailor:lunar`, `teamtailor:careers.trackunit.com` |
| Companies | 35 boards: Lunar, Trackunit, DFDS |
| Companies | 38 boards: Lunar, Trackunit, DFDS |
| List | the RSS feed `https://<site>/jobs.rss`, 50 per request by offset; no total, so the read goes on to an empty page |
| Requests per pull | roles / 50, rounded up, plus 1 for the empty page; at most 13 for any board attached (598 roles in all) |
| Description | in the list, full text; live: 598 of 598 |
| Requests per pull | roles / 50, rounded up, plus 1 for the empty page; the largest board attached has 75 roles, so 3 |
| Description | in the list, full text |
| Date | `pubDate`; trusted |
| Place and country | every `tt:location` as "city, country", joined with "; "; country 582 of 598 |
| Remote | `remoteStatus`: fully is remote, hybrid, onsite; `none` and `temporary` stay unknown (290 with a value) |
| Place and country | every `tt:location` as a place "city, country", each remote when `remoteStatus` says so; country resolved from the places |
| Remote | `remoteStatus`, handed on as the host's word: fully is remote, hybrid, onsite; `none` and `temporary` stay unknown |
| Pay | none: the feed states no pay |
| Found by discovery | yes, from a `<sub>.teamtailor.com` link; a page that loads Teamtailor's CDN names the page's own host as the board; a guessed name is probed |
| Limitations | 5,000 roles per board; a board of exactly 5,000 is also incomplete, since no empty page is reached. A page with nothing new ends the read incomplete. Nine seed companies have no board because their careers link is a marketing site, not the career host. |
| Limitations | A board of more than 100 pages is swept across pulls and completes on the empty page. A page with nothing new ends the read incomplete. Nine seed companies have no board because their careers link is a marketing site, not the career host. |
### Jibe
@@ -273,10 +282,11 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
| Companies | 1 board: Booking.com |
| List | `https://<host>/api/jobs?limit=100&page=<n>`, 100 per request, the most the host serves; states `totalCount` |
| Requests per pull | roles / 100, rounded up; 2 for Booking.com |
| Description | in the list, full text: description and qualifications; live: 140 of 140 |
| Description | in the list, full text: description and qualifications |
| Date | `posted_date`; not trusted, so the role counts as posted when first seen |
| Place and country | `full_location`, several places joined by the host; country 140 of 140 |
| Place and country | `full_location` split at ";" into places; the host's `country_code` is handed on when the role has one place |
| Remote | none (`location_type` is a kind of coordinates) |
| Employment | `employment_type`, the host's own word |
| Pay | none: the `salary_*` fields were 0 on every role and carry no currency |
| Found by discovery | no slug: a link to Jibe's CDN names the host only; the target is set by hand |
| Limitations | Only Booking.com's site was read. No brand filter: the Booking Holdings roles are read with the Booking.com ones. The site's robots.txt asks for `Crawl-delay: 5`, which is not honoured. |
@@ -285,8 +295,8 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
| | |
|---|---|
| Target | the portal's customer name, e.g. `hrmanager:stsgroup_tr` |
| Companies | none attached; Seaborg, the one seed company on it, is on the Watchlist as `robots` |
| Target | the portal's customer name, e.g. `hrmanager:teknologisk` |
| Companies | 1 board: Danish Technological Institute (`hrmanager:teknologisk`, 7 open roles). Seaborg, the one seed company on the portal, is unattached and on the Watchlist as `robots` |
| List | the RSS feed `api.hr-manager.net/JobPortal.svc/<customer>/PositionList/rss/?incads=true`, 100 per request by `skip`; no total, so the read goes on to an empty page |
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page |
| Description | in the list, the whole advertisement |
@@ -295,10 +305,49 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
| Remote | none |
| Pay | none |
| Found by discovery | yes, from a portal or API link; not probed |
| Limitations | The portal's robots.txt on `candidate.hr-manager.net` disallows everything but the application form; the feed's API host has no robots.txt. The owner decided to leave Seaborg unattached. Without `take` the host answers 25 roles. |
| Limitations | The portal's robots.txt on `candidate.hr-manager.net` disallows everything but the application form; the feed's API host has no robots.txt. `READS.robots` says "owner to decide", and the board of the Danish Technological Institute is attached all the same (`docs/open.md`). Without `take` the host answers 25 roles. |
## Hosts read as list, then each role
For every host below the pull reads the list only; the description arrives when `enrich` reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
### SmartRecruiters
| | |
|---|---|
| Target | the company identifier, e.g. `smartrecruiters:BoschGroup` |
| Companies | 11 boards: Bosch (about 4,800 roles on the site, 13 open in reach), Delivery Hero, Wise |
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
| Requests per pull | one series per country of the reach (two for Bosch: DK and SE), each roles / 100, rounded up; a board read whole is roles / 100 |
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by `enrich` for ranked roles only: 1 of 343 open roles has text |
| Date | `releasedDate`; trusted |
| Place and country | `fullLocation`, else city and country, with empty parts removed; the host's `country` is handed on |
| Remote | `location.remote`, true or false, as the host's word |
| Seniority, employment | `experienceLevel` and `typeOfEmployment`, the host's ids |
| Pay | none: the list states no pay |
| Found by discovery | yes, from a `jobs.`, `careers.` or `api.smartrecruiters.com` link; a guessed name is probed |
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
| Limitations | 100 pages a pull; a larger board is swept across pulls. Whether the identifier is case-sensitive is unsettled: the module docstring says it is, `docs/decisions.md` says it is not (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until `enrich` has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
### Workable
| | |
|---|---|
| Target | the account name, e.g. `workable:huggingface` |
| Companies | 8 boards: Hugging Face, Kahoot!, Starling Bank |
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
| Requests per pull | roles / 10, rounded up, 0.5 s apart; the largest board has 50 roles, so 5 |
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by `enrich` for ranked roles only: 1 of 157 open roles has text |
| Date | `published`; trusted |
| Place and country | every `locations` entry as "city, region, country", each remote when the role is; `location` when there are none |
| Remote | `workplace` (remote, hybrid, on_site) as the host's word, else the `remote` flag |
| Employment | `type`, the host's own word |
| Pay | none: the list states no pay |
| Found by discovery | yes, from an `apply.workable.com` or `<account>.workable.com` link; not probed, since a run of guesses trips Workable's rate limit |
| Limitations | 100 pages of 10 a pull; a larger board is swept across pulls. Workable answers bursts with 429, which is why pages and role reads are 0.5 s apart; a 429 after the first page ends the read incomplete. The role link is built from the shortcode. |
### SuccessFactors
| | |
@@ -306,83 +355,84 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
| Target | the career site's host, or host plus a brand path, e.g. `successfactors:careers.vestas.com`, `successfactors:jobs.teradyne.com/MiR` |
| Companies | 14 boards: Novo Nordisk, Vestas, Carlsberg |
| List | the site's HTML search page `/search/?startrow=<n>`, parsed in its two known layouts; page size is the site's own (10, 15, 25 or 100) and cannot be set; a total where the pagination label states one |
| Requests per pull | roles / the site's page size, rounded up, plus up to 200 role pages; per-board sizes are not in these notes |
| Description | per role, the `itemprop="description"` block of the role page; live: 1,651 of 3,738 after one pull |
| Requests per pull | roles / the site's page size, rounded up |
| Description | per role, the `itemprop="description"` block of the role page |
| Date | the list's date column where it parses; the role page's `datePosted` replaces it when earlier; trusted |
| Place and country | the list's location, with a two-letter country code turned into the name; the role page's microdata replaces it; country 3,675 of 3,738 |
| Remote | none |
| Place and country | the list's location, with a two-letter country code turned into the name; the role page's microdata replaces it |
| Remote | none from the host |
| Pay | none |
| Found by discovery | yes, from a link to a role, `/go/` page or search on the site's own host; a link to the SAP back end names the host only; not probed. A site on the company's own domain whose pages link only the back end is set by hand |
| Filter at the source | `optionsFacetsDD_country=DK`, one country a request; no remote facet. Vestas DK: 64 of 668 (2026-09-30). Which countries is the agents' reach, see below. Fetch terms as the form's `q=engineer`, one a request, AND with the country (Vestas DK: 40 of 64) |
| Limitations | Only the default locale's roles are listed: a role that exists only in another locale (Carlsberg's French brand, 37 of 105) is not read. Only `/search/` and `/job/` are read, never `/services/` or the `successfactors.eu` back end, so a company with only the back end (Bang & Olufsen) cannot be read. 100 pages per board, so 1,000 to 10,000 roles depending on the page size. A third layout or a bot wall that answers 200 fails the board with "not a SuccessFactors search page". Read from this machine only; not yet from the deployed server. |
| Filter at the source | `optionsFacetsDD_country=DK`, one country a request; no remote facet. Vestas DK: 64 of 668 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as the form's `q=engineer`, one a request, AND with the country (Vestas DK: 40 of 64) |
| Limitations | Only the default locale's roles are listed: a role that exists only in another locale (Carlsberg's French brand, 37 of 105) is not read. Only `/search/` and `/job/` are read, never `/services/` or the `successfactors.eu` back end, so a company with only the back end (Bang & Olufsen) cannot be read. 100 pages a pull; a larger site is swept across pulls. A third layout or a bot wall that answers 200 fails the board with "not a SuccessFactors search page". Read from this machine only; not yet from the deployed server. |
### Workday
| | |
|---|---|
| Target | `<tenant>.<wdN>/<Site>`, site with case, e.g. `workday:lego.wd103/LEGO_External` |
| Companies | 13 boards: LEGO Group, Zendesk, Equinor |
| Companies | 17 boards: LEGO Group, Zendesk, Equinor, and since 2026-09-30 Criteo, Darktrace, Novonesis and NVIDIA |
| List | a POST to `/wday/cxs/<tenant>/<Site>/jobs`, 20 per request by offset (21 or more answers 400); the first page states the total |
| Requests per pull | roles / 20, rounded up, plus up to 200 role reads; for a board of 1,000 roles 50 + 200 on the first pull |
| Description | per role, `jobDescription` from the role's JSON; live: 1,411 of 2,377 after one pull |
| Requests per pull | roles / 20, rounded up; 50 for a board of 1,000 roles |
| Description | per role, `jobDescription` from the role's JSON |
| Date | the list's "Posted N Days Ago" as a day, possibly one day off; "Posted 30+ Days Ago" and other languages give none; the role's `startDate` replaces it; trusted |
| Place and country | the list has `locationsText` ("Copenhagen" or "2 Locations"); the per-role read adds every location and the country. The country therefore arrives with the per-role read: 1,724 of 2,377 after one pull |
| Remote | `remoteType`, where the tenant sends it: remote, hybrid, onsite (585 with a value) |
| Place and country | the list has `locationsText` ("Copenhagen" or "2 Locations"); the per-role read adds every location and the country, so the country arrives with the role page |
| Remote | `remoteType`, where the tenant sends it, as the tenant's own word ("Remote", "Hybrid", "Full Remote", "No"); prep maps it |
| Employment | `timeType`, the host's own word |
| Pay | none: no pay field; pay appears only inside the text |
| Found by discovery | yes, from a `myworkdayjobs.com` link with a site; not probed |
| Filter at the source | the tenant's own facets in `appliedFacets`, ids read by `facets` once a day: the facet whose values are countries, and any value that reads as remote. Two facets intersect, so remote is a series of its own. LEGO DK: 97 of 455 (2026-09-30). A tenant with no country facet is read whole. Which countries is the agents' reach, see below. Fetch terms as `searchText` in the body, one a request, AND with the facets (LEGO: 106 for `engineer`) |
| Limitations | 2,000 roles per board (100 pages of 20). A tenant has several sites and each is its own board. robots.txt paths are matched without case, as Workday serves any case. Every tenant is behind Cloudflare and was read from this machine only. |
| Filter at the source | the tenant's own facets in `appliedFacets`, ids read by `facets` once a day: the facet whose values are countries, and any value that reads as remote. Two facets intersect, so remote is a series of its own. LEGO DK: 97 of 455 (2026-09-30). A tenant with no country facet is read whole. Which countries is the agents' reach, see above. Fetch terms as `searchText` in the body, one a request, AND with the facets (LEGO: 106 for `engineer`) |
| Limitations | 2,000 roles per board (100 pages of 20), the `ceiling`. A tenant has several sites and each is its own board. Every tenant is behind Cloudflare and was read from this machine only. The slug's case is kept (`KEEP_CASE`). |
### Oracle Recruiting Cloud
| | |
|---|---|
| Target | `<pod host>/<siteNumber>`, e.g. `oracle:fa-evmr-saasfaprod1.fa.ocs.oraclecloud.com/CX_1` |
| Companies | 4 boards: Danske Bank, Nokia, Milestone Systems, Sinch |
| Companies | 5 boards: Danske Bank, Nokia, Milestone Systems, Sinch, Pioneer Centre for AI |
| List | the REST resource `recruitingCEJobRequisitions`, 200 per request by offset, with secondary places expanded; states `TotalJobsCount` |
| Requests per pull | roles / 200, rounded up, plus up to 200 role reads; 3 + 200 for Nokia's roughly 600 on the first pull |
| Description | per role: external description, responsibilities and qualifications joined; the list's one-paragraph `ShortDescriptionStr` is not stored; live: 432 of 826 after one pull |
| Requests per pull | roles / 200, rounded up; 3 for Nokia's roughly 600 |
| Description | per role: external description, responsibilities and qualifications joined. The list's one-paragraph `ShortDescriptionStr` is stored as the teaser (`READS.teaser=True`), which is shown and embedded until the description arrives |
| Date | `PostedDate`; the role's `ExternalPostedStartDate` replaces it when earlier; trusted |
| Place and country | the primary place plus every secondary one, joined with "; "; country 769 of 826 |
| Remote | `WorkplaceTypeCode`: remote, hybrid, onsite (741 with a value) |
| Place and country | the primary place plus every secondary one, each a place of the role |
| Remote | `WorkplaceTypeCode` as the host's word (e.g. `ORA_HYBRID`); prep maps it |
| Pay | none: the resource states no pay |
| Found by discovery | yes, from a candidate site or role link on the pod, or a REST call naming `siteNumber`; not probed |
| Filter at the source | `selectedLocationsFacet` with the pod's geography ids, and `selectedWorkplaceTypesFacet=ORA_REMOTE` as a series of its own. `locationsFacet` lists the ten biggest countries only, so a country missing from a full list is read unfiltered: fa-evmr (Nokia) has neither DK nor SE listed and is read whole. Which countries is the agents' reach, see below. Fetch terms as `keyword=` in the finder, one a request; it matches the text (Nokia: 468 of 554 for `engineer`) |
| Limitations | The host is the tenant's Fusion pod, not the company's domain (`jobs.nokia.com` redirects to it). Links are built for locale `en`. The department fields were empty on every role of the four boards. |
| Filter at the source | `selectedLocationsFacet` with the pod's geography ids, and `selectedWorkplaceTypesFacet=ORA_REMOTE` as a series of its own. `locationsFacet` lists the ten biggest countries only, so a country missing from a full list is read unfiltered: fa-evmr (Nokia) has neither DK nor SE listed and is read whole. Which countries is the agents' reach, see above. Fetch terms as `keyword=` in the finder, one a request; it matches the text (Nokia: 468 of 554 for `engineer`) |
| Limitations | The host is the tenant's Fusion pod, not the company's domain (`jobs.nokia.com` redirects to it). Links are built for locale `en`. The department fields were empty on every role of the first four boards; Pioneer's were not checked. |
### Phenom
| | |
|---|---|
| Target | the career site's host plus its locale root, e.g. `phenom:www.careers.philips.com/global/en` |
| Companies | 3 boards: King, Just Eat Takeaway.com, Philips |
| Companies | 5 boards: King, Just Eat Takeaway.com, Philips, BCG, Verisure |
| List | a POST to `https://<host>/widgets`, 100 per request by offset; states `totalHits` |
| Requests per pull | roles / 100, rounded up, plus up to 200 role pages; 9 + 200 for Philips' 865 on the first pull |
| Description | per role, the JSON-LD `JobPosting` description of the role page; the list's `descriptionTeaser` is not stored; live: 335 of 1,007 after one pull |
| Date | the list's `postedDate`; the role page's `datePosted` replaces it when earlier (Philips' list gave the day of the read for a role posted 2026-09-08); trusted |
| Place and country | every place of `multi_location`, joined with "; "; country 975 of 1,007 |
| Remote | per role only, from the page's `checkRemote` (120 with a value) |
| Requests per pull | roles / 100, rounded up; 9 for Philips' roughly 880 |
| Description | per role, the JSON-LD `JobPosting` description of the role page. The list's `descriptionTeaser` is stored as the teaser (`READS.teaser=True`) |
| Date | the list's `dateCreated`, else `postedDate`, which the site moves on a refresh; the role page's `datePosted` replaces it when earlier; trusted |
| Place and country | every place of `multi_location`, each a place of the role |
| Remote | per role only, from the page's `checkRemote` |
| Pay | none |
| Found by discovery | yes, from a `/<country>/<lang>/search-results` link; a link to Phenom itself names the host only; not probed |
| Filter at the source | `selected_fields.country`, a list of English names in one request; no remote. Philips NL and DE: 111 of 873 (2026-09-30). Which countries is the agents' reach, see below. Fetch terms as `keywords` in the body, one a request, AND with the countries; it matches the text, not the title |
| Limitations | The locale is built as `<lang>_<country>`, the spelling of the three sites read; another site may spell it differently. 10,000 roles per board. A role page without a description fails the read and is retried on a later pull. |
| Filter at the source | `selected_fields.country`, a list of English names in one request; no remote. Philips NL and DE: 111 of 873 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `keywords` in the body, one a request, AND with the countries; it matches the text, not the title |
| Limitations | The locale is built as `<lang>_<country>`, the spelling of the three sites first read; another site may spell it differently. 100 pages a pull; a larger board is swept across pulls. A role page without a description fails the read and is retried on a later run. |
### Eightfold
| | |
|---|---|
| Target | `<tenant host>/<company domain>`, e.g. `eightfold:jobs.ericsson.com/ericsson.com` |
| Companies | 1 board: Ericsson |
| Companies | 3 boards: Ericsson, Microsoft, Infineon |
| List | `https://<host>/api/pcsx/search?domain=<domain>&sort_by=timestamp`, newest first, 10 per request whatever `num` asks; states `count` |
| Requests per pull | roles / 10, rounded up, at most 100, plus up to 200 `position_details` reads; 63 + 200 for Ericsson's 630 on the first pull |
| Description | per role, `jobDescription`; live: 200 of 630 after one pull |
| Date | `postedTs`; not trusted, so the role counts as posted when first seen |
| Place and country | every location, repeated names dropped, joined with "; "; country 622 of 630 |
| Remote | `workLocationOption`: onsite, hybrid, remote (628 with a value) |
| Requests per pull | roles / 10, rounded up, at most 100 |
| Description | per role, `jobDescription` from `position_details` |
| Date | `creationTs`, else `postedTs`, which the host refreshes in bulk; not trusted, so the role counts as posted when first seen |
| Place and country | every location, repeated names dropped, each a place of the role |
| Remote | `workLocationOption`: onsite, hybrid, remote |
| Pay | none |
| Found by discovery | yes, from an `eightfold.ai` link that carries `domain=`; a bare `eightfold.ai` link names the host only; not probed |
| Filter at the source | `location=<country name>`, one country a request; no remote. Ericsson SE: 154 of 618 (2026-09-30). Which countries is the agents' reach, see below. Fetch terms as `query=`, one a request, AND with the location; it matches title, text and skills (Ericsson SE: 102 of 154 for `software`) |
| Limitations | With 10 per request and 100 pages, a board over 1,000 roles is never read to the end; unless the country filter brings it under the cap: Infineon (1,289, 163 in DK and SE) and Microsoft (about 2,300, 60) become readable that way. Verified on three tenants only. |
| Filter at the source | `location=<country name>`, one country a request; no remote. Ericsson SE: 154 of 618 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `query=`, one a request, AND with the location; it matches title, text and skills (Ericsson SE: 102 of 154 for `software`) |
| Limitations | With 10 per request and 100 pages a pull, a board over 1,000 roles is swept across pulls; what the host answers past offset 1,000 was not checked. The country filter brings big boards under the cap: Microsoft (size 2,345) has 29 open roles in reach, and Infineon (999 stored) has none in DK and SE. Verified on three tenants only. |
### Gem
@@ -390,15 +440,16 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|---|---|
| Target | the board name on `jobs.gem.com`, with case, e.g. `gem:productboard` |
| Companies | 2 boards: sennder, Productboard |
| List | a GraphQL POST to `jobs.gem.com/api/public/graphql`, the whole board in one answer; no total |
| Requests per pull | 1, plus one GraphQL POST per role not yet read (at most 200) |
| Description | per role, `descriptionHtml`; live: 44 of 44 |
| List | a GraphQL POST to `jobs.gem.com/api/public/graphql`, the whole board in one answer; no total. Declared as paged with `complete="end"`; it reads one page |
| Requests per pull | 1 |
| Description | per role, `descriptionHtml`, one GraphQL POST per role |
| Date | per role only, `firstPublishedTsSec`; not trusted, so the role counts as posted when first seen |
| Place and country | every location's city, joined with "; "; `isoCountry` is fetched and not used; country from the line (41 of 44) |
| Remote | `locationType`: onsite, hybrid, remote (44 of 44) |
| Place and country | every location as a place (the city, else the location name); `isoCountry` is fetched and not used |
| Remote | `locationType`: onsite, hybrid, remote |
| Employment | `employmentType`, the host's own word |
| Pay | none; `compensationHtml` is fetched but was null on every role read |
| Found by discovery | yes, from a `jobs.gem.com` link, case kept; not probed |
| Limitations | The GraphQL queries were read out of the page's script bundle and are undocumented; a new bundle from Gem can rename a field and fail the board. An unknown board answers an empty list, not an error, so a wrongly cased or mistyped name reads as an empty, complete board. |
| Limitations | The GraphQL queries were read out of the page's script bundle and are undocumented; a new bundle from Gem can rename a field and fail the board. An unknown board answers an empty list, not an error: `READS.empty="suspect"` protects a known board, a mistyped name attached as a new board reads as an empty, complete board. |
### Jobylon
@@ -406,12 +457,13 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|---|---|
| Target | the numeric company id, e.g. `jobylon:3143` |
| Companies | 2 boards: Einride, Vipps MobilePay |
| List | the careers widget's embed page `cdn.jobylon.com/jobs/companies/<id>/embed/v2/`, whose script literal holds the whole board; no total |
| Requests per pull | 1, plus one role page on `emp.jobylon.com` per role not yet read (at most 200); 1 + 33 for Einride on the first pull |
| Description | per role, the JSON-LD description of the role page; live: 33 of 33 |
| List | the careers widget's embed page `cdn.jobylon.com/jobs/companies/<id>/embed/v2/`, whose script literal holds the whole board; no total. The list carries a `summary`, stored as the teaser |
| Requests per pull | 1 |
| Description | per role, the JSON-LD description of the role page on `emp.jobylon.com` |
| Date | the list's day ("Sept. 25, 2026"); the role page's `datePosted` replaces it when earlier; trusted |
| Place and country | the list's place names, which can be cut short ("Houston +3 more"); the per-role read fills every place; country 32 of 33 |
| Remote | `workspace`: remote, hybrid, onsite (32 of 33) |
| Place and country | the list's place names, which can be cut short ("Houston +3 more"); the per-role read fills every place |
| Remote | `workspace`: remote, hybrid, onsite |
| Seniority, employment | `experience` and `employment_type`, the host's own words |
| Pay | none; JSON-LD pay is not read |
| Found by discovery | yes, from an `emp.` or `cdn.jobylon.com` link or a `jbl_company_id` on the company's own site; not probed |
| Limitations | Jobylon's own feeds are addressed by a uuid the company holds, so the widget is scraped; a change to its script literal breaks the board. An unknown id answers 404 and fails the board. |
@@ -423,14 +475,15 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
| Target | the board slug on `ats.rippling.com`, with case, e.g. `rippling:prisma-careers` |
| Companies | 1 board: Prisma |
| List | `api.rippling.com/platform/api/ats/v1/board/<slug>/jobs`, the whole board in one array, one entry per role and place, merged by role; no total |
| Requests per pull | 1, plus one role record per role not yet read (at most 200); 1 + 2 for Prisma |
| Description | per role, `description.role`; live: 2 of 2 |
| Requests per pull | 1 |
| Description | per role, `description.role`, one role record per role |
| Date | per role only, `createdOn`; trusted |
| Place and country | the list's place labels, joined with "; "; the per-role read fills every work location |
| Remote | remote only when every place label starts with "remote"; otherwise unknown |
| Pay | figures, per role only, when the record has exactly one pay range with YEAR or MONTH (×12) frequency; no pay line |
| Place and country | the list's place labels as places; the per-role read replaces them with every work location |
| Remote | the reader sets no role-level remote; a place whose label starts with "remote" is a remote place, and prep makes the role remote if any place is |
| Employment | `employmentType.label`, the host's own word |
| Pay | figures, per role only, when the record has exactly one pay range with YEAR or MONTH (x12) frequency; no pay line |
| Found by discovery | yes, from an `ats.rippling.com` or API link, case kept; not probed |
| Limitations | Text, date and pay each wait for the per-role read. Several pay zones give no figures. A wrongly cased slug answers `Job Board not found` and fails the board. |
| Limitations | Text, date and pay each wait for the role page. Several pay zones give no figures. A wrongly cased slug answers `Job Board not found` and fails the board. |
### BambooHR
@@ -439,33 +492,34 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
| Target | the subdomain, e.g. `bamboohr:configit` |
| Companies | 1 board: Configit |
| List | `https://<sub>.bamboohr.com/careers/list`, the whole board in one answer; a list shorter than its `totalCount` fails the board rather than being read as whole |
| Requests per pull | 1, plus one `/careers/<id>/detail` per role not yet read (at most 200); 1 + 2 for Configit |
| Description | per role; live: 2 of 2 |
| Requests per pull | 1 |
| Description | per role, `/careers/<id>/detail` |
| Date | per role only, `datePosted`; trusted |
| Place and country | the list's city and state; the country arrives with the per-role read |
| Place and country | the list's city and state as one place; the country arrives with the role page |
| Remote | `isRemote`, else `locationType` 0 onsite, 1 remote, 2 hybrid; the codes are assumed from Configit's board |
| Employment | `employmentStatusLabel`, the host's own word |
| Pay | none; the per-role `compensation` was null |
| Found by discovery | yes, from a `<sub>.bamboohr.com` link; not probed |
| Limitations | Written from one board of two roles. Whether a large board pages `/careers/list` is unverified. |
## The search providers
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. The text a provider sends is stored as the description; the reader shows "Fetch the posting" for any role whose description is under 500 characters, which fetches the posting page once, on demand, and stores it beside the description. The fetched text is what the reader and `summarise` use; `embed` and `extract` read the description only. A search that fails is skipped for the pull. No provider reads robots.txt.
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
### Jobindex
| | |
|---|---|
| Saved search | `q` in Jobindex's syntax (words are OR, `"a b"` a phrase, `+word` required) and optionally `area`, Jobindex's region id (1 Region Hovedstaden, 8 Aarhus, ...) |
| List | the RSS twin of the search page, `jobindex.dk/jobsoegning.rss`, newest first, 20 per page; no total. Complete when a page within the limit comes back short |
| List | the RSS twin of the search page, `jobindex.dk/jobsoegning.rss`, newest first, 20 per page; no total. `READS.complete="never"`: a search is never complete, so its roles close by the 21 days only |
| Pages read | 5 by default (100 results), 10 at most (200) |
| Text | an excerpt of a couple of hundred characters from the feed; live: 537 of 538 have one. The full text only through "Fetch the posting", on demand, since every fetch of Jobindex's link counts as a click in its statistics |
| Text | an excerpt of a couple of hundred characters from the feed, stored as the teaser; 561 of 562 open roles have some text. The full text only through "Fetch the posting", on demand, since every fetch of Jobindex's link counts as a click in its statistics |
| Date | `pubDate`; trusted |
| Place | the item's area, else the search's area; country always Denmark |
| Remote | from place and title words (41 roles) |
| Remote | the reader states none; prep reads place and title words |
| Pay | none |
| Key | none |
| Limitations | The company is the logo's alt text or what follows the title's last comma. Items are not followed to the employer's site. |
| Limitations | The company is the logo's alt text or what follows the title's last comma. Items are not followed to the employer's site during a search; the on-demand fetch follows the "Se jobbet" link. |
### Platsbanken
@@ -474,10 +528,10 @@ A saved search is a query with its parameters, stored as a `search` subscription
| Saved search | `q` and optionally `region`, a län id from JobTech's taxonomy (Skåne is `CaRE_1nn_cSU`); empty for all of Sweden |
| List | `jobsearch.api.jobtechdev.se/search`, newest first, 100 per page; states `total` |
| Pages read | 5 by default (500 results), 10 at most (1,000); the API refuses an offset above 2,000, which the page limit never reaches |
| Text | `description.text` of the ad; live: 446 of 446. The code does not say whether this is the whole ad or a shortened one |
| Text | `description.text` of the ad, the whole ad |
| Date | `publication_date`; trusted |
| Place | the municipality, else the region; country always Sweden |
| Remote | from headline words only, since `workplace_model` is mostly empty (0 roles) |
| Place | every entry of `workplace_addresses` (municipality, else region, else country) as a place of the role; country always Sweden |
| Remote | `workplace_model.label` where present, which is mostly empty; prep also reads headline words |
| Pay | none |
| Key | none |
| Limitations | A region is a län; there is no municipality or city filter. Free text ranks by all words rather than requiring them; a quoted phrase narrows it. |
@@ -489,10 +543,10 @@ A saved search is a query with its parameters, stored as a `search` subscription
| Saved search | `q`, `country` (gb, de, nl, fr, at, be, es, it, pl, ch; not Denmark) and optionally `where`, a free place text |
| List | `api.adzuna.com/v1/api/jobs/<country>/search/<page>`, newest first, 50 per page; states `count` |
| Pages read | 5 by default (250 results), 10 at most (500) |
| Text | Adzuna's `description` snippet; its length is not stated in the code |
| Text | Adzuna's `description` snippet, stored as the teaser; its length is not stated in the code |
| Date | `created`; trusted |
| Place | `location.display_name`; country from the search |
| Remote | from title words only |
| Remote | prep reads the place line, then title words |
| Pay | figures from `salary_min` and `salary_max` in the country's currency, dropped when Adzuna marks them as predicted |
| Key | a free key from developer.adzuna.com, as `sources.search.adzuna` in the config or `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`; without one the search fails |
| Limitations | Built against Adzuna's documented response and never run live; no Adzuna role is in the database. |
@@ -501,16 +555,17 @@ Jobnet and Jooble were left out: Jobnet's search answers 401 outside its own sit
## Sources that cannot be read, and why
76 of the 362 seed companies have no board and are on the Watchlist with a careers link opened on 2026-09-29.
The table is the seed's: 76 of the 362 companies in `seed/companies.json` have no board there and carry a careers link the survey opened on 2026-09-29. The live database has moved on, and its Watchlist tab and `app companies by-hand` are the current list. Since the seed was written, Infineon and Microsoft (Eightfold) and Novonesis, Darktrace and Criteo (Workday) have been attached, so they are struck from the company lists below; the seed still names them.
| Reason | Count | What it means | Companies |
|---|---|---|---|
| `unread` | 35 | readable by a program; no module reads it yet | Zalando, Ørsted, Siemens, Infineon, Microsoft, Google DeepMind, Amazon, Apple, ASML, Netcompany, Tryg, DNB |
| `robots` | 10 | the site's robots.txt disallows the page a board would be read from | Novonesis, Darktrace, Criteo (Workday), Bang & Olufsen (SuccessFactors), Pandora, Terma, Zenegy, Umbraco, Snyk, Seaborg (HR Manager) |
| `group` | 8 | the roles are inside another company's board, with no field that separates them | Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio, Native Instruments, Visma |
| `blocked` | 7 | the site answers a program with a challenge | Tradeshift, Personio, Adjust, Revolut, Silverfin, HashiCorp, Meta |
| `gone` | 4 | no careers page | wefox, WeTransfer, Featurespace, Unbabel |
| `key` | 3 | the only listing answers with a key the page's script carries | Maersk, New Work SE, Yousign (Welcome to the Jungle) |
| no reason | 9 | on Teamtailor, but the careers link is a marketing site and the career host is not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial |
| Reason | Seed | Live 2026-09-30 | What it means | Companies |
|---|---|---|---|---|
| `unread` | 35 | 35 | readable by a program; no module reads it yet | Zalando, Ørsted, Siemens, Google DeepMind, Amazon, Apple, ASML, Netcompany, Tryg, DNB |
| `robots` | 10 | 7 | the site's robots.txt disallows the page a board would be read from | Bang & Olufsen (SuccessFactors), Pandora, Terma, Zenegy, Umbraco, Snyk, Seaborg (HR Manager) |
| `group` | 8 | 8 | the roles are inside another company's board, with no field that separates them | Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio, Native Instruments, Visma |
| `blocked` | 7 | 8 | the site answers a program with a challenge | Tradeshift, Personio, Adjust, Revolut, Silverfin, HashiCorp, Meta |
| `gone` | 4 | 5 | no careers page | wefox, WeTransfer, Featurespace, Unbabel |
| `key` | 3 | 3 | the only listing answers with a key the page's script carries | Maersk, New Work SE, Yousign (Welcome to the Jungle) |
| `added` | – | 14 | added by hand or by an import and not looked at yet | Danish Centre for AI Innovation, 2021.AI, McKinsey, AMD, Ingka Group and nine more |
| no reason | 9 | 9 | on Teamtailor, but the careers link is a marketing site and the career host is not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial |
What is planned for the `unread` companies and for the gaps in hosts already read (SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, Avature for Siemens) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
The live counts are companies with no `ats` board subscription; the company lists are the seed's, less the five attached companies. What is planned for the `unread` companies and for the gaps in hosts already read (SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, Avature for Siemens) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
+41 -23
View File
@@ -16,7 +16,9 @@ The allow-list and the classifier are ported from `openings.go` and
A target is `host:slug`, e.g. `greenhouse:databricks`, `ashby:deepl`,
`lever:eu/tomtom`, `teamtailor:careers.acme.com`; each host module says what
its slug is. Every board is read independently; one that fails is logged,
recorded as failing under `ats:failing:<target>`, and skipped. A board's
recorded as failing under `ats:failing:<target>`, and skipped, and so is one
whose read lists under a tenth of what it listed last time (`_runner.read`'s
guard), until it has failed that way for the pull's `stale_days`. A board's
filter is the `reach` option the pull hands it (the agents', see
`app/jobs/reach.py`), over which its subscription `params` (`{"countries":
["DK"], "remote": true}`) win, narrowed to what its reader declares
@@ -26,7 +28,8 @@ the whole board: it prunes no per-role state.
A board's size (`board_size`) is set once, when the board is added
(`app.jobs.sources.set_board`) or rechecked, and kept under
`ats:size:<target>` with the day it was checked under
`ats:sizechecked:<target>` (`keep_size`); the reach reads it to leave small
`ats:sizechecked:<target>` (`keep_size`, which also resets the guard's count,
so a Recheck accepts a board that shrank); the reach reads it to leave small
boards whole. A pull does not rewrite it. A board with no size yet is sized
in its batch before it is read: its reach then says `unsized` and the size
under which a board is read whole (`small`, `app.jobs.reach.SMALL`). A board
@@ -106,8 +109,9 @@ _PARSERS: dict[str, Callable[[bytes, str], list[Row]]] = {
name: module.parse for name, module in readers.HOSTS.items() if hasattr(module, "parse")}
def _limits() -> _runner.Limits:
return _runner.Limits(max_pages=MAX_PAGES, page_delay=PAGE_DELAY, detail_cap=DETAIL_CAP)
def _limits(stale_days: int | None = None) -> _runner.Limits:
return _runner.Limits(max_pages=MAX_PAGES, page_delay=PAGE_DELAY, detail_cap=DETAIL_CAP,
stale_days=stale_days)
def _allowed_host(host: str, slug: str) -> Callable[[str], bool]:
@@ -339,8 +343,7 @@ def board_size(target: str, *, client: httpx.Client | None = None) -> int:
host, _, slug = target.partition(":")
module = readers.HOSTS[host]
get = _getter(client, host, slug)
if (_runner.declared(module) and module.READS.listing == "paged"
and module.READS.complete == "stated"):
if module.READS.listing == "paged" and module.READS.complete == "stated":
_data, _results, total, _cursor = module.page(get, slug, None, readers.Ask())
if isinstance(total, int):
return total
@@ -348,11 +351,18 @@ def board_size(target: str, *, client: httpx.Client | None = None) -> int:
return len(_runner.read_listing(module, slug, get, limits=limits).ids)
def keep_size(state: Any, target: str, size: int, *, today: str = "") -> None:
def keep_size(state: Any, target: str, size: int, *, today: str = "",
reset: bool = True) -> None:
"""Keep board `target`'s size and the day it was checked (ISO,
`today`, else today in UTC)."""
`today`, else today in UTC). With `reset`, the default and what a
Recheck does, the 10% guard's count goes too (`ats:count:<target>`), so
the next read of a board that really shrank is taken as it is instead of
failing as suspect; the plugin sizing a board as it reads it passes
False, since its count is the read it just stamped."""
state[f"ats:size:{target}"] = str(size)
state[f"ats:sizechecked:{target}"] = today or datetime.now(UTC).date().isoformat()
if reset:
state.pop(f"ats:count:{target}", None)
failure_reason = _runner.failure_reason
@@ -400,14 +410,15 @@ class ATSSource(Source):
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
module = readers.HOSTS[host]
get = _getter(client, host, slug)
ask, unsized = self._ask(client, target, module)
ask, unsized, remote = self._ask(client, target, module)
stale_days = self.options.get("stale_days")
listing = _runner.read(self.state, "ats", target, module, slug, get, ask,
limits=_limits())
limits=_limits(int(stale_days) if stale_days else None),
remote=remote)
except Exception as exc:
log.warning("ats: %s failed (%s)", target, exc)
_runner.record_failure(self.state, failing, exc)
return
self.state.pop(failing, None)
def compound(source_id: str) -> str:
# Matches `roles.source_id`, which `Store.mark_closed` compares.
@@ -435,11 +446,13 @@ class ATSSource(Source):
yield item_of(row, target, label)
# Stamped only once every row went out, so a board that blew up half
# way through is neither marked ok nor has a partial id list treated
# as the whole of it.
# as the whole of it, and it stays failing until then: a row that
# fails to store (`app.jobs.pull.run_batch`) leaves the record in place.
_runner.stamp(self.state, "ats", target, run_marker, listing,
[compound(i) for i in listing.ids])
self.state.pop(failing, None)
if unsized and listing.complete and not (listing.filtered or listing.narrowed):
keep_size(self.state, target, len(listing.ids))
keep_size(self.state, target, len(listing.ids), reset=False)
if detail_sets is not None:
# A complete listing drops the ids it no longer lists.
for key, ids in zip(detail_keys(target), detail_sets, strict=False):
@@ -447,9 +460,11 @@ class ATSSource(Source):
self.state[key] = json.dumps(sorted(ids))
def _ask(self, client: httpx.Client, target: str,
module: Any) -> tuple[readers.Ask, bool]:
module: Any) -> tuple[readers.Ask, bool, bool]:
"""What `target` is asked for, from the reach its batch handed it and
its params, and whether it still needs a size. A board with no size
its params, whether it still needs a size, and whether that reach
and params want remote roles (`_runner.remote_wanted`, which the
runner compares with what it could ask). A board with no size
yet (`unsized` in its reach) whose reach would narrow the read is
sized first (`board_size`, kept by `keep_size`), and read whole under
`small` roles. One whose read is whole anyway is sized by that read,
@@ -457,16 +472,18 @@ class ATSSource(Source):
given = self.options.get("reach")
params, keys = self.params_for(target), self.options.get("keys")
if not isinstance(given, dict) or not given.get("unsized"):
return _runner.ask_for(module, params, given, keys=keys), False
reach = {k: v for k, v in given.items() if k not in ("unsized", "small")}
ask = _runner.ask_for(module, params, reach or None, keys=keys)
return (_runner.ask_for(module, params, given, keys=keys), False,
_runner.remote_wanted(module, params, given))
reach = {k: v for k, v in given.items() if k not in ("unsized", "small")} or None
ask = _runner.ask_for(module, params, reach, keys=keys)
if not (_runner.filtered(ask) or _runner.narrowed(module, ask)):
return ask, True
return ask, True, _runner.remote_wanted(module, params, reach)
size = board_size(target, client=client)
keep_size(self.state, target, size)
keep_size(self.state, target, size, reset=False)
if size < int(given.get("small") or 0):
reach = None
ask = _runner.ask_for(module, params, None, keys=keys)
return ask, False
return ask, False, _runner.remote_wanted(module, params, reach)
# ------------------------------------------------------ the per-role reads
@@ -567,7 +584,8 @@ def read_board_details(state: Any, target: str, rows: list[Row], *, client: http
def item_of(row: Row, target: str, label: str) -> Item:
"""One row of board `target` as the `Item` the store takes: its id
compound, `host:slug:id`, as `roles.source_id` has it."""
compound, `host:slug:id`, as `roles.source_id` has it. A row without a
date is dated now, as `Item` itself would date it."""
host, _, slug = target.partition(":")
description = row.get("description") or ""
return Item(
@@ -576,7 +594,7 @@ def item_of(row: Row, target: str, label: str) -> Item:
title=row["title"],
excerpt=description[:300],
body=description,
published_at=row.get("posted"),
published_at=row.get("posted") or datetime.now(UTC),
subscription=label,
kind="role",
raw=_runner.raw_of(row) | {
+152 -74
View File
@@ -5,23 +5,25 @@ A reader says how its host answers; this module reads it: the guarded fetch
(`fetch`, bound to one target by `getter`), the `Ask` a subscription resolves
to (`ask_for`) and its series (`series`), the listing whole or paged, one
series per `one`-filter value unioned by id (`read_listing`), completeness,
the sweep of an oversized paged board across pulls, the 10% guard on a
filtered or `empty="suspect"` read (`read`), the per-role detail reads
the sweep of an oversized paged board across pulls, the 10% guard on every
read that is not a sweep (`read`), the per-role detail reads
(`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
bookkeeping (`record_failure`, `failure_reason`). `plugins/ats.py` and
`plugins/search.py` are thin `Source` classes over it: they pass their own
limits (`Limits`) and state prefix (`ats`, `search`), so their state keys stay
as they were.
A module without `READS` is read with the older signatures, `feed_url(slug)`
and `page(get, slug, cursor)`, and never gets an `Ask`.
as they were. Every reader declares `READS`; a module without it is refused
by the package before it gets here.
State keys, under the Source's prefix and the target (`ats:sweep:<target>`):
sweep a paged read cut at `max_pages`: `{"asks", "series", "cursor",
"seen", "rows", "whole"}`, continued by the next pull
count `{"asks", "count"}`, the roles the last whole read listed, for
the 10% guard
count `{"asks", "count"}`, the roles the last read that was not a
sweep listed, for the 10% guard
failing `{"since", "reason"}`, kept by the Source (`record_failure`);
the guard gives way once `since` is `Limits.stale_days` old
remote_unasked "1" when the reach wanted remote and the read could not
ask for it (`Listing.remote_unasked`), else ""
facets `{"checked", "facets"}`, a board's `facets`, kept `FACETS_TTL`
swept "1" when the last stamped listing finished a sweep carried over
from an earlier pull (`Listing.swept`), else "". Offset paging
@@ -87,10 +89,16 @@ DETAIL_FIELDS = ("description", "location", "places", "teaser", "remote", "count
#: How long a board's `facets` are kept before they are asked again.
FACETS_TTL = timedelta(days=1)
#: A filtered or `empty="suspect"` read listing under this share of what the
#: same read listed last time counts as a failure.
#: A read listing under this share of what the same read listed last time
#: counts as a failure (`read`).
SUSPECT_SHARE = 0.1
#: The last count from which the guard holds every read. A read that is
#: filtered, narrowed by terms or of an `empty="suspect"` reader is held from
#: a count of 1, as it was before every read was; a whole read of a board
#: that listed a handful of roles can drop to none by itself.
SUSPECT_MIN = 10
#: The keys of a row that an `Item` carries in its own fields, not in `raw`.
ITEM_KEYS = ("source_id", "title", "url", "description", "posted")
@@ -105,8 +113,8 @@ class TooLarge(ValueError):
class Suspect(Exception):
"""A filtered or `empty="suspect"` read that listed under `SUSPECT_SHARE`
of what it listed last time."""
"""A read that listed under `SUSPECT_SHARE` of what it listed last time
(see `read`)."""
class MissingKey(ValueError):
@@ -115,12 +123,15 @@ class MissingKey(ValueError):
@dataclass(frozen=True, slots=True)
class Limits:
"""What one Source allows a read: pages, the pause between requests, and
the detail reads a pull."""
"""What one Source allows a read: pages, the pause between requests, the
detail reads a pull, and the days after which the 10% guard accepts the
answer it keeps refusing (`stale_days`, the pull's `STALE_DAYS`; None,
for a Source run outside a pull, never)."""
max_pages: int = MAX_PAGES
page_delay: float = PAGE_DELAY
detail_cap: int = DETAIL_CAP
stale_days: int | None = None
@dataclass(slots=True)
@@ -134,7 +145,9 @@ class Listing:
neither are the whole board. `swept` says a complete listing finished a
sweep carried over from an earlier pull: `ids` were read across pulls,
and roles closing meanwhile may have shifted an offset-paged host's later
roles back past the cursor, so `ids` can miss a role that is still open."""
roles back past the cursor, so `ids` can miss a role that is still open.
`remote_unasked` says a filtered read was meant to ask for remote roles
as well and could not, so its `ids` say nothing of the remote roles."""
rows: list[Row]
complete: bool
@@ -144,12 +157,14 @@ class Listing:
asks: str = ""
narrowed: bool = False
swept: bool = False
remote_unasked: bool = False
# --------------------------------------------------------------- modules
def declared(module: ModuleType) -> bool:
"""Whether `module` declares `READS`, and so takes `ask`."""
"""Whether `module` declares `READS`. Every reader does, so the runner no
longer asks; `app.jobs.reach` still does."""
return isinstance(getattr(module, "READS", None), Reads)
@@ -334,6 +349,33 @@ def _cost(reads: Reads, wanted: Mapping[str, Any]) -> int:
return count
def _wanted(module: ModuleType, subscription_params: Mapping[str, Any] | None,
reach: Mapping[str, Any] | None) -> tuple[dict[str, Any], bool]:
"""The dimensions `reach` and then the params set, params winning, before
they are narrowed to what the reader declares, and whether the read is
wide. A reach of more than `MAX_SERIES` countries on a `one` country
filter is dropped whole, which makes the read wide; a `remote` of False
asks nothing."""
given = reach or {}
reached = _dimensions(module, given)
wide = bool(given.get("wide"))
if module.READS.filters.country == "one" and len(reached.get("country", ())) > MAX_SERIES:
reached = {}
wide = True
wanted = reached | _dimensions(module, dict(subscription_params or {}))
if wanted.get("remote") is False:
del wanted["remote"]
return wanted, wide
def remote_wanted(module: ModuleType, subscription_params: Mapping[str, Any] | None = None,
reach: Mapping[str, Any] | None = None) -> bool:
"""Whether the reach or the params want remote roles, whether or not the
reader can ask for them. `read_listing` compares it with what the series
asked (`Listing.remote_unasked`)."""
return _wanted(module, subscription_params, reach)[0].get("remote") is True
def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None = None,
reach: Mapping[str, Any] | None = None, *,
keys: Mapping[str, Any] | None = None) -> Ask:
@@ -346,7 +388,8 @@ def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None =
dropped. A `remote` of False asks nothing. A search's params go whole into
`Ask.params`; a reader with `READS.key` gets the resolved key under
`Ask.params["key"]` (see `resolve_key`, which raises when it is missing).
A module without `READS` gets `Ask()`.
A remote the reader cannot filter on is dropped here; `remote_wanted`
still says it was wanted.
Fetch terms (`reach["terms"]`) go only to a board reader that declares
`query="one"`, and only on a wide read: `reach["wide"]` (a company set
@@ -358,19 +401,10 @@ def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None =
whole, since remote without its countries would read only the remote
roles; over it with terms alone, the terms go too and the board is read
unfiltered."""
if not declared(module):
return Ask()
reads: Reads = module.READS
params = dict(subscription_params or {})
given = reach or {}
reached = _dimensions(module, given)
wide = bool(given.get("wide"))
if reads.filters.country == "one" and len(reached.get("country", ())) > MAX_SERIES:
reached = {}
wide = True
wanted = reached | _dimensions(module, params)
if wanted.get("remote") is False:
del wanted["remote"]
wanted, wide = _wanted(module, params, given)
declared_now = reads.filters.declared()
wanted = {name: value for name, value in wanted.items() if name in declared_now}
terms = _terms(given.get("terms"))
@@ -400,7 +434,7 @@ def narrowed(module: ModuleType, ask: Ask) -> bool:
query. Such a read is never the whole board, nor the board within its
places, so it closes nothing. A search's query is what
the search is, and does not count."""
board = not declared(module) or module.READS.scope == "board"
board = module.READS.scope == "board"
return bool(ask.terms) or (board and bool(ask.query))
@@ -417,10 +451,7 @@ def series(module: ModuleType, ask: Ask,
remote part beside places is then left out. Each of those is then asked
once per fetch term (`ask.terms`), with the term as `query`: the host
intersects a keyword with its facets, as the survey found on every
host, so the union by id is "one of the terms, in these places". `[ask]`
for a module without `READS`."""
if not declared(module):
return [ask]
host, so the union by id is "one of the terms, in these places"."""
if ask.remote and (ask.countries or ask.places):
parts = [*_series(module, ask.model_copy(update={"remote": None}), facets),
*_series(module, ask.model_copy(update={"countries": (), "places": ()}),
@@ -450,8 +481,8 @@ def _series(module: ModuleType, ask: Ask, facets: Mapping[str, Mapping[str, str]
known = known_facets.get(dimension) or {}
mapped = tuple(dict.fromkeys(known[v] for v in values if v in known))
if not mapped:
log.info("readers: %s knows none of %s %s", module.NAME, dimension,
", ".join(values))
log.warning("readers: %s knows none of %s %s; nothing asked", module.NAME,
dimension, ", ".join(values))
return []
if dimension == "country":
countries = mapped
@@ -497,7 +528,7 @@ def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
for index, ask in enumerate(asks):
if index:
time.sleep(limits.page_delay)
url = module.feed_url(slug, ask) if declared(module) else module.feed_url(slug)
url = module.feed_url(slug, ask)
try:
body = get(url)
except TooLarge:
@@ -514,7 +545,8 @@ def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None, *,
limits: Limits | None = None, sweep: Mapping[str, Any] | None = None,
can_sweep: bool = False,
facets: Mapping[str, Mapping[str, str]] | None = None) -> Listing:
facets: Mapping[str, Mapping[str, str]] | None = None,
remote: bool = False) -> Listing:
"""One target's listing, read and parsed. Raises on any failure of the
first request, and on a reader that does not understand an answer.
@@ -532,22 +564,35 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
its series and cursor; the sweep is complete when it reaches the end,
and `ids` then hold every id it listed, and the listing is `swept`. A
read at the reader's `ceiling` is not swept: the host answers nothing
past it."""
past it.
A filtered read that asks no series, because an `ids` dimension names
no value the host's facets know, asked nothing, which is no evidence of
anything: it is incomplete and closes nothing. `filtered` is a board's
only: a search's places are what the search is, as its query is.
`remote` says the reach or the params wanted remote roles; when no series
asked for them (a reader without a remote filter, or one whose facets
state none beside places) the listing is `remote_unasked`."""
limits = limits or Limits()
ask = ask or Ask()
reads: Reads = module.READS
asks = series(module, ask, facets)
is_filtered = filtered(ask)
is_filtered = filtered(ask) and reads.scope == "board"
by_words = narrowed(module, ask)
unasked = (is_filtered and (remote or bool(ask.remote))
and not any(a.remote or a.remote_ids for a in asks))
key = fingerprint(asks)
if not asks:
return Listing([], True, [], is_filtered, None, key, by_words)
log.warning("readers: %s:%s: the filter maps to nothing the host knows; "
"nothing asked, not closing", module.NAME, slug)
return Listing([], False, [], True, None, key, by_words, remote_unasked=unasked)
if not hasattr(module, "page"):
rows = _whole(module, slug, get, asks, limits)
return Listing(list(rows.values()), True, list(rows), is_filtered, None, key, by_words)
whole_rows = _whole(module, slug, get, asks, limits)
return Listing(list(whole_rows.values()), True, list(whole_rows), is_filtered, None,
key, by_words, remote_unasked=unasked)
reads: Reads | None = module.READS if declared(module) else None
never = reads is not None and reads.complete == "never"
ceiling = reads.ceiling if reads is not None else None
never = reads.complete == "never"
ceiling = reads.ceiling
start, cursor, seen, earlier, whole = 0, None, set(), [], True
resumed = bool(sweep and sweep.get("asks") == key)
if sweep and resumed:
@@ -568,10 +613,7 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
if pages:
time.sleep(limits.page_delay)
try:
if reads is not None:
data, results, total, cursor = module.page(get, slug, cursor, asks[index])
else:
data, results, total, cursor = module.page(get, slug, cursor)
data, results, total, cursor = module.page(get, slug, cursor, asks[index])
except httpx.HTTPStatusError as exc:
if not pages or exc.response.status_code != 429:
raise
@@ -603,15 +645,17 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
listed = list(dict.fromkeys(earlier + list(rows)))
if not stop:
return Listing(list(rows.values()), whole, listed, is_filtered, None, key, by_words,
resumed and whole)
resumed and whole, unasked)
log.warning("readers: %s:%s: stopped after %d pages (%s); not complete",
module.NAME, slug, pages, stop)
at_ceiling = ceiling is not None and len(seen) >= ceiling
if not can_sweep or never or at_ceiling or not _storable(cursor):
return Listing(list(rows.values()), False, list(rows), is_filtered, None, key, by_words)
return Listing(list(rows.values()), False, list(rows), is_filtered, None, key, by_words,
remote_unasked=unasked)
keep = {"asks": key, "series": index, "cursor": cursor, "seen": sorted(seen),
"rows": listed, "whole": whole}
return Listing(list(rows.values()), False, list(rows), is_filtered, keep, key, by_words)
return Listing(list(rows.values()), False, list(rows), is_filtered, keep, key, by_words,
remote_unasked=unasked)
def _storable(cursor: Any) -> bool:
@@ -637,7 +681,7 @@ def facets_for(module: ModuleType, slug: str, get: Get, ask: Ask,
"""The reader's `facets` when `ask` sets a dimension it declares `ids`
(a country, a place, or remote),
kept in `state[key]` for `FACETS_TTL`; `{}` otherwise."""
if not declared(module) or not hasattr(module, "facets"):
if not hasattr(module, "facets"):
return {}
ids = [name for name, reach in module.READS.filters.declared().items() if reach == "ids"]
if not any(getattr(ask, _ASK_FIELD[name]) for name in ids if name != "query"):
@@ -659,46 +703,79 @@ def facets_for(module: ModuleType, slug: str, get: Get, ask: Ask,
def read(state: MutableMapping[str, str], prefix: str, target: str, module: ModuleType,
slug: str, get: Get, ask: Ask | None = None, *,
limits: Limits | None = None, can_sweep: bool = True) -> Listing:
limits: Limits | None = None, can_sweep: bool = True, remote: bool = False) -> Listing:
"""`read_listing` for one target with its state under `<prefix>:...:<target>`:
the facets it needs, the sweep it continues, and the 10% guard. A read
that is filtered, narrowed by terms, or whose reader declares
`empty="suspect"`, and that listed under `SUSPECT_SHARE` of what the same
read listed last time (the same `fingerprint`, so a term read is only
held to a term read with the same terms and places), raises `Suspect`. Nothing is written here but the facets; `stamp` keeps
the rest once the rows are stored."""
the facets it needs, the sweep it continues, and the 10% guard. Nothing
is written here but the facets; `stamp` keeps the rest once the rows are
stored.
The guard holds every read that is not a sweep: one that listed under
`SUSPECT_SHARE` of what the same read listed last time (the same
`fingerprint`, so a term read is only held to a term read with the same
terms and places) raises `Suspect`, and the Source records the target
as failing, so its roles stay open. A host whose answer changed shape
parses to no rows with a 200, and closing a board on that would lose it
whole. It holds from a last count of `SUSPECT_MIN`, or of 1 for a read
that is filtered, narrowed or of an `empty="suspect"` reader. A board
that really shrank is accepted once the target has been failing for
`limits.stale_days` (its `<prefix>:failing:<target>` record), and its
stamp then resets the count; a Recheck resets it at once
(`plugins.ats.keep_size`)."""
limits = limits or Limits()
ask = ask or Ask()
facets = facets_for(module, slug, get, ask, state, f"{prefix}:facets:{target}")
listing = read_listing(module, slug, get, ask, limits=limits, can_sweep=can_sweep,
sweep=_load(state, f"{prefix}:sweep:{target}"), facets=facets)
suspect = declared(module) and module.READS.empty == "suspect"
sweep=_load(state, f"{prefix}:sweep:{target}"), facets=facets,
remote=remote)
held = listing.filtered or listing.narrowed or module.READS.empty == "suspect"
last = _load(state, f"{prefix}:count:{target}")
count = int(last.get("count") or 0) if last.get("asks") == listing.asks else 0
if ((listing.filtered or listing.narrowed or suspect) and listing.sweep is None and count
if (listing.sweep is None and count >= (1 if held else SUSPECT_MIN)
and len(listing.ids) < SUSPECT_SHARE * count):
raise Suspect(f"listed {len(listing.ids)} roles, under "
f"{SUSPECT_SHARE:.0%} of the {count} it listed last time")
why = (f"listed {len(listing.ids)} roles, under "
f"{SUSPECT_SHARE:.0%} of the {count} it listed last time")
if _failing_for(state, f"{prefix}:failing:{target}", limits.stale_days):
log.warning("readers: %s:%s: %s, accepted after %d days of failing",
module.NAME, slug, why, limits.stale_days)
return listing
raise Suspect(why)
return listing
def _failing_for(state: Mapping[str, str], key: str, days: int | None) -> bool:
"""Whether the failure kept under `key` began at least `days` ago; False
for `days` None or no record."""
since = str(_load(state, key).get("since") or "")
if days is None or not since:
return False
try:
began = datetime.fromisoformat(since)
except ValueError:
return False
return datetime.now(UTC) - began >= timedelta(days=days)
def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
listing: Listing, ids: list[str]) -> None:
"""Record a read whose rows are stored: `ok` (the run's marker),
`complete`, `ids` (the compound ids, what the pull closes against),
`filtered`, `narrowed` (a read by terms, which the pull never closes
against), whether it finished a sweep from earlier pulls (`swept`), the
against), whether it finished a sweep from earlier pulls (`swept`),
whether a filtered read left remote unasked (`remote_unasked`), the
sweep to continue or none, and the count for the 10% guard once a read
is whole."""
is not a sweep. A read that asked nothing keeps the count it had."""
state[f"{prefix}:ok:{target}"] = run
state[f"{prefix}:complete:{target}"] = "1" if listing.complete else ""
state[f"{prefix}:swept:{target}"] = "1" if listing.swept else ""
state[f"{prefix}:ids:{target}"] = json.dumps(ids)
state[f"{prefix}:filtered:{target}"] = "1" if listing.filtered else ""
state[f"{prefix}:narrowed:{target}"] = "1" if listing.narrowed else ""
state[f"{prefix}:remote_unasked:{target}"] = "1" if listing.remote_unasked else ""
if listing.sweep is not None:
state[f"{prefix}:sweep:{target}"] = json.dumps(listing.sweep)
else:
state.pop(f"{prefix}:sweep:{target}", None)
return
state.pop(f"{prefix}:sweep:{target}", None)
if listing.asks != fingerprint([]):
state[f"{prefix}:count:{target}"] = json.dumps({"asks": listing.asks,
"count": len(listing.ids)})
@@ -766,7 +843,7 @@ def read_details(module: ModuleType, name: str, slug: str, get: Get, rows: list[
if ignored := sorted(set(answer) - set(DETAIL_FIELDS)):
log.debug("readers: %s:%s: detail of %s: %s left out, not in DETAIL_FIELDS",
name, slug, row["source_id"], ", ".join(ignored))
taken: Row = {k: v for k, v in answer.items() if k in DETAIL_FIELDS} # type: ignore[assignment]
taken: Row = {k: v for k, v in answer.items() if k in DETAIL_FIELDS}
if taken.get("location") and "places" not in taken:
taken["places"] = [{"text": taken["location"], "remote": None}]
filled[key] = taken
@@ -776,12 +853,12 @@ def read_details(module: ModuleType, name: str, slug: str, get: Get, rows: list[
for row in rows:
key = compound(row)
if key in filled:
out.append(row | {"description": ""} | filled[key]) # type: ignore[operator]
out.append(row | {"description": ""} | filled[key])
elif key in detailed:
out.append(row | {"description": "", "location": "", "places": [], # type: ignore[operator]
out.append(row | {"description": "", "location": "", "places": [],
"remote": None})
else:
out.append(row | {"description": ""}) # type: ignore[operator]
out.append(row | {"description": ""})
if listed is not None:
done &= listed
failing &= listed
@@ -791,8 +868,9 @@ def read_details(module: ModuleType, name: str, slug: str, get: Get, rows: list[
# -------------------------------------------------------------------- raw
def places_of(row: Mapping[str, Any]) -> list[Place]:
"""A row's places: its `places`, or, for a row without (an undeclared
reader's), its `location` as the one place, split nothing."""
"""A row's places: its `places`, or, for a row without (a raw row stored
before readers returned places), its `location` as the one place, split
nothing."""
places = row.get("places")
if not isinstance(places, list):
location = str(row.get("location") or "").strip()
+59 -40
View File
@@ -17,14 +17,19 @@ one run is not closed here. A search that read everything that matches was
complete (`complete="stated"`, as many distinct results as the stated total),
and the pull closes what it stopped listing; the rest close once no search
has seen them for `STALE_DAYS` (see `app/jobs/pull.py`).
In a pull a search is read by `_runner.read` under the prefix `search`, as a
board is under `ats`: the 10% guard holds it, `_runner.stamp` records it,
and a search that fails or is suspect is kept as failing under
`search:failing:<target>` (`_runner.record_failure`) until it answers, which
the settle reads as it reads a failing board's.
"""
from __future__ import annotations
import json
import logging
import sys
from collections.abc import Iterator, Mapping
from collections.abc import Iterator, Mapping, MutableMapping
from datetime import UTC, datetime
from pathlib import Path
from types import ModuleType
@@ -65,8 +70,9 @@ def _pages(params: Mapping[str, Any]) -> int:
return DEFAULT_PAGES
def _limits(params: Mapping[str, Any]) -> _runner.Limits:
return _runner.Limits(max_pages=_pages(params), page_delay=PAGE_DELAY)
def _limits(params: Mapping[str, Any], stale_days: int | None = None) -> _runner.Limits:
return _runner.Limits(max_pages=_pages(params), page_delay=PAGE_DELAY,
stale_days=stale_days)
def provider_of(target: str) -> str:
@@ -76,10 +82,13 @@ def provider_of(target: str) -> str:
def _read(client: httpx.Client, target: str, params: Mapping[str, Any],
keys: Mapping[str, Any] | None) -> tuple[ModuleType, _runner.Listing]:
keys: Mapping[str, Any] | None, state: MutableMapping[str, str] | None = None,
stale_days: int | None = None) -> tuple[ModuleType, _runner.Listing]:
"""One search read through the runner: its provider and its listing.
Raises ValueError for an unknown provider or a missing key, before any
request."""
request. With `state` (a pull) it is `_runner.read` under `search`, the
10% guard included and never swept; without (a search tried before it is
saved) a bare `read_listing`, since there is nothing to compare with."""
provider = provider_of(target)
if not provider:
raise ValueError(f"{target!r} is not provider:name with provider one of "
@@ -88,7 +97,11 @@ def _read(client: httpx.Client, target: str, params: Mapping[str, Any],
slug = target.partition(":")[2]
ask = _runner.ask_for(module, params, keys=keys)
get = _runner.getter(client, module.allowed(slug), provider)
return module, _runner.read_listing(module, slug, get, ask, limits=_limits(params))
limits = _limits(params, stale_days)
if state is None:
return module, _runner.read_listing(module, slug, get, ask, limits=limits)
return module, _runner.read(state, "search", target, module, slug, get, ask,
limits=limits, can_sweep=False)
def run_search(target: str, params: dict[str, Any],
@@ -130,37 +143,43 @@ class SearchSource(Source):
run_marker = str(self.options.get("run") or "") or datetime.now(UTC).isoformat(
timespec="seconds")
for target, label in self.targets():
provider = provider_of(target)
if not provider:
log.warning("search: %r is not provider:name; skipped", target)
continue
try:
_module, listing = _read(client, target, self.params_for(target),
self.options)
except Exception as exc:
log.warning("search: %s failed (%s)", target, exc)
continue
yield from self._search(client, target, label, run_marker)
seen_ids = []
for row in listing.rows:
source_id = f"{provider}:{row['source_id']}"
seen_ids.append(source_id)
description = row.get("description") or ""
yield Item(
source_id=source_id,
location=row["url"],
title=row["title"],
excerpt=description[:300],
body=description,
published_at=row.get("posted"),
subscription=label,
kind="role",
raw=_runner.raw_of(row) | {
"company": row.get("company") or "",
"provider": provider,
"search": target,
},
)
self.state[f"search:ok:{target}"] = run_marker
self.state[f"search:complete:{target}"] = "1" if listing.complete else ""
self.state[f"search:ids:{target}"] = json.dumps(seen_ids)
def _search(self, client: httpx.Client, target: str, label: str,
run_marker: str) -> Iterator[Item]:
"""One search: read, its rows out as items, then stamped. A search
that fails, a suspect answer included, is recorded under
`search:failing:<target>` and stamps nothing; the record goes once a
read is stamped, after every row went out, as for a board."""
failing = f"search:failing:{target}"
provider = provider_of(target)
stale_days = self.options.get("stale_days")
try:
_module, listing = _read(client, target, self.params_for(target), self.options,
self.state, int(stale_days) if stale_days else None)
except Exception as exc:
log.warning("search: %s failed (%s)", target, exc)
_runner.record_failure(self.state, failing, exc)
return
now = datetime.now(UTC)
for row in listing.rows:
description = row.get("description") or ""
yield Item(
source_id=f"{provider}:{row['source_id']}",
location=row["url"],
title=row["title"],
excerpt=description[:300],
body=description,
published_at=row.get("posted") or now,
subscription=label,
kind="role",
raw=_runner.raw_of(row) | {
"company": row.get("company") or "",
"provider": provider,
"search": target,
},
)
_runner.stamp(self.state, "search", target, run_marker, listing,
[f"{provider}:{i}" for i in listing.ids])
self.state.pop(failing, None)
+171
View File
@@ -0,0 +1,171 @@
"""What the JSON API and the config decide, that the reviews of 2026-09-30
found wrong: a board's reach stored as ISO codes, an empty seniority range,
Fetch text answering as the viewing agent, the model a stage will use, and a
starter search on any reader. No network."""
from __future__ import annotations
from collections.abc import Iterator
from datetime import UTC, datetime
from pathlib import Path
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from app import config as config_mod
from app.config import Config, ConfigError, ServerConfig
from app.main import create_app
from app.storage import Store
from desk.models import Item
from prepped import upsert
ROOT = Path(__file__).resolve().parents[1]
@pytest.fixture
def app(tmp_path: Path) -> FastAPI:
return create_app(Config(server=ServerConfig(host="127.0.0.1", port=0),
database=tmp_path / "roledesk.db"))
@pytest.fixture
def store(app: FastAPI) -> Store:
return app.state.store
@pytest.fixture
def client(app: FastAPI) -> Iterator[TestClient]:
with TestClient(app) as c:
yield c
def _role(store: Store, source_id: str = "1", title: str = "Senior Backend Engineer") -> str:
item = Item(source_id=source_id, location=f"https://boards.example/acme/{source_id}",
title=title, body="A teaser.", published_at=datetime(2026, 9, 20, tzinfo=UTC),
raw={})
item.source = "ats"
upsert(store, item)
return item.uid
# ---------------------------------------------------------- a board's reach
@pytest.fixture
def board(store: Store) -> int:
return store.add_subscription("ats", "greenhouse:acme", label="Acme", company="Acme",
board="greenhouse").id
def test_the_reach_countries_are_stored_as_iso_codes(
client: TestClient, store: Store, board: int) -> None:
res = client.post(f"/api/sources/{board}/reach",
json={"mode": "own", "countries": ["UK", "Germany", "dk"]})
assert res.status_code == 200, res.text
sub = store.subscription(board)
assert sub is not None and sub.params["countries"] == ["DE", "DK", "GB"]
def test_a_reach_country_that_is_none_is_refused(
client: TestClient, store: Store, board: int) -> None:
res = client.post(f"/api/sources/{board}/reach",
json={"mode": "own", "countries": ["Atlantis"]})
assert res.status_code == 422 and "Atlantis" in res.text
sub = store.subscription(board)
assert sub is not None and "countries" not in sub.params
def test_a_company_row_names_its_attached_board_in_full(
client: TestClient, store: Store, board: int) -> None:
"""The Replace-board confirm compares the typed `host:slug` with this,
not with `board` (the host) or `ats_slug` (the careers site's)."""
store.add_company("Acme")
row = next(c for c in client.get("/api/companies").json() if c["key"] == "acme")
assert (row["board"], row["target"]) == ("greenhouse", "greenhouse:acme")
# ------------------------------------------------------ the seniority range
@pytest.mark.parametrize("path", ["/api/roles", "/api/roles/facets"])
@pytest.mark.parametrize("query", ["level_from=senior&level_to=junior",
"level_from=exec&track=ic"])
def test_a_seniority_range_with_no_level_is_refused_not_ignored(
client: TestClient, store: Store, path: str, query: str) -> None:
_role(store)
res = client.get(f"{path}?{query}")
assert res.status_code == 422 and "no level" in res.text
def test_a_seniority_range_with_levels_still_filters(client: TestClient, store: Store) -> None:
_role(store)
assert client.get("/api/roles?level_from=junior&level_to=senior").status_code == 200
assert client.get("/api/roles/facets?track=ic").status_code == 200
# --------------------------------------------------------------- Fetch text
def test_fetch_text_answers_with_the_viewing_agents_vote(
client: TestClient, store: Store, monkeypatch: pytest.MonkeyPatch) -> None:
from app.jobs import posting
agent = store.save_agent("Wolf", intent="backend")
uid = _role(store)
store.set_vote(agent.id, uid, 1)
monkeypatch.setattr(posting, "fetch_posting",
lambda s, u: s.set_article_text(u, "The whole posting."))
res = client.post(f"/api/roles/{uid}/text?agent={agent.id}")
assert res.status_code == 200, res.text
assert (res.json()["vote"], res.json()["article_text"]) == (1, "The whole posting.")
# The default agent when none is named, as the other role routes read it.
assert client.post(f"/api/roles/{uid}/text").json()["vote"] == 1
assert client.post(f"/api/roles/{uid}/text?agent=999").status_code == 404
# ------------------------------------------------------------------- stages
def test_a_stage_shows_the_model_that_will_answer(tmp_path: Path) -> None:
config = Config(database=tmp_path / "x.db")
app = create_app(config)
with TestClient(app) as client:
by_name = {s["name"]: s for s in client.get("/api/stages").json()}
assert by_name["extract"]["model"] == config.extract.model
config.agent.model = "claude-opus-4-1"
by_name = {s["name"]: s for s in client.get("/api/stages").json()}
assert by_name["extract"]["model"] == by_name["summarise"]["model"] == "claude-opus-4-1"
assert by_name["enrich"]["model"] == ""
# ------------------------------------------------------------------- config
def test_a_platsbanken_search_loads(tmp_path: Path) -> None:
path = tmp_path / "config.yaml"
path.write_text("searches:\n - {name: Sverige, provider: platsbanken, params: {q: python}}\n",
encoding="utf-8")
assert config_mod.load(path).searches[0].provider == "platsbanken"
def test_a_search_provider_that_is_no_reader_names_the_readers(tmp_path: Path) -> None:
path = tmp_path / "config.yaml"
path.write_text("searches:\n - {name: X, provider: monster}\n", encoding="utf-8")
with pytest.raises(ConfigError, match="jobindex, adzuna, platsbanken"):
config_mod.load(path)
def test_the_example_config_holds_every_default() -> None:
"""Each value in `config.example.yaml` is the default, so a copy changes
nothing; a key added there with another value, or a key the code does not
read, fails here."""
loaded = config_mod.load(ROOT / "config.example.yaml").model_dump(exclude={"path"})
assert loaded == Config().model_dump(exclude={"path"})
def test_the_example_config_names_every_key_of_the_sections_it_documents() -> None:
import yaml
raw = yaml.safe_load((ROOT / "config.example.yaml").read_text("utf-8"))
for name, allowed_missing in (("agent", set()), ("extract", set()), ("enrich", set()),
("embed", set()), ("workers", set()),
("summarise", {"on_ingest"})): # accepted, ignored
fields = set(type(getattr(Config(), name)).model_fields)
# `credentials_database` is filled from `database`, never read from the file.
fields -= {"credentials_database"}
assert fields - set(raw[name]) == allowed_missing, name
+16
View File
@@ -117,3 +117,19 @@ def test_the_read_page_of_its_own_is_gone(client: TestClient) -> None:
other = client.get("/applications").text
assert 'id="rail-toggle"' not in other and 'id="search"' not in other
assert "<main>" in other
STATIC = Path(__file__).resolve().parents[1] / "app" / "static"
def test_no_script_links_to_a_page_that_does_not_exist() -> None:
"""`/messages` was a 404: the notice log lives on Settings, Sources."""
for script in STATIC.glob("*.js"):
assert '"/messages"' not in script.read_text("utf-8"), script.name
def test_the_reader_excerpt_falls_back_in_the_servers_order() -> None:
"""`excerptOf` cuts the same text `_EXCERPT_SOURCE` does: summary, then
description, then teaser, or a teaser-only role's row blanks after Fetch."""
text = (STATIC / "roles.js").read_text("utf-8")
assert "role.summary || role.description || role.teaser" in text
+231 -6
View File
@@ -7,6 +7,7 @@ from __future__ import annotations
import json
import sys
import threading
import types
from concurrent.futures import ThreadPoolExecutor
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -21,8 +22,8 @@ if str(ROOT) not in sys.path:
from desk.models import Item # noqa: E402
from pipeline import Cancelled, RunContext, Tally # noqa: E402
from app.models import Application # noqa: E402
from plugins import readers # noqa: E402
from plugins.readers import _runner # noqa: E402
from plugins import ats, readers # noqa: E402
from plugins.readers import Filters, Reads, _runner # noqa: E402
from plugins.ats import ATSSource # noqa: E402
from app import prep # noqa: E402
from app.jobs import cascade, pull # noqa: E402
@@ -142,7 +143,7 @@ def test_close_boards_closes_what_a_completed_board_no_longer_lists(
result = pull.close_boards(FakeConfig(), store, FakeTally()) # type: ignore[arg-type]
assert result == {"boards": 1, "searches": 0, "failed": 1, "incomplete": 0,
assert result == {"boards": 1, "searches": 0, "unanswered": 1, "incomplete": 0,
"roles": 1, "closed": 1}
assert {r.source_id for r in store.roles()} == {"greenhouse:acme:1", "greenhouse:other:9"}
@@ -163,7 +164,7 @@ def test_close_boards_never_closes_a_board_stamped_by_an_earlier_run(
result = pull.close_boards(FakeConfig(), store, FakeTally()) # type: ignore[arg-type]
assert result == {"boards": 0, "searches": 0, "failed": 1, "incomplete": 0,
assert result == {"boards": 0, "searches": 0, "unanswered": 1, "incomplete": 0,
"roles": 0, "closed": 0}
assert {r.source_id for r in store.roles()} == {"greenhouse:acme:1"}
@@ -256,7 +257,7 @@ def test_a_failing_board_keeps_its_roles_before_stale_days(
_answer(monkeypatch, ValueError("gone"))
result = _pull(acme)
assert _open(acme) == {"workable:acme:1"}
assert (result["failed"], result["closed"]) == (1, 0)
assert (result["unanswered"], result["closed"]) == (1, 0)
since, reason = pull.failing_boards(acme)["workable:acme"]
assert since and reason == "ValueError: gone"
@@ -486,7 +487,7 @@ def test_run_a_settle_does_not_close_on_run_b_ids(
b_result = pull.run_batch(FakeConfig(), acme, **b) # type: ignore[arg-type]
result = pull.close_boards(FakeConfig(), acme, Tally(results=(a_result,))) # type: ignore[arg-type]
assert (result["failed"], result["closed"]) == (1, 0)
assert (result["unanswered"], result["closed"]) == (1, 0)
assert _open(acme) == {"workable:acme:1", "workable:acme:2"}
assert pull.running(acme) == b["run"]
@@ -507,3 +508,227 @@ def test_the_settle_reads_its_marker_off_its_batches_payloads(acme: Store) -> No
context = RunContext(waited_for=(failed,))
result = pull.close_boards(FakeConfig(), acme, Tally(), context=context) # type: ignore[arg-type]
assert result["boards"] == 1 and pull.running(acme) == ""
# -------------------------------------------------------- losses, guarded
def _days_ago(days: float) -> str:
return (datetime.now(UTC) - timedelta(days=days)).isoformat(timespec="seconds")
def _set(store: Store, column: str, value: str, *source_ids: str) -> None:
with store.transaction() as conn:
conn.executemany(f"UPDATE roles SET {column} = ? WHERE source_id = ?",
[(value, i) for i in source_ids])
@pytest.fixture
def big(store: Store) -> Store:
store.add_subscription("ats", "greenhouse:big", label="Big", company="Big",
board="greenhouse")
return store
def test_a_whole_read_of_nothing_after_forty_keeps_the_roles_and_fails(
big: Store, monkeypatch: pytest.MonkeyPatch) -> None:
_answer(monkeypatch, ([_row(str(n)) for n in range(40)], True))
_pull(big)
_answer(monkeypatch, ([], True))
result = _pull(big)
assert len(_open(big)) == 40
assert (result["unanswered"], result["closed"]) == (1, 0)
assert "under 10% of the 40" in pull.failing_boards(big)["greenhouse:big"][1]
def test_a_whole_read_of_nothing_after_five_closes_the_board(
big: Store, monkeypatch: pytest.MonkeyPatch) -> None:
_answer(monkeypatch, ([_row(str(n)) for n in range(5)], True))
_pull(big)
_answer(monkeypatch, ([], True))
assert _pull(big)["closed"] == 5
assert _open(big) == set()
def test_a_suspect_answer_is_accepted_after_stale_days(
big: Store, monkeypatch: pytest.MonkeyPatch) -> None:
_answer(monkeypatch, ([_row(str(n)) for n in range(40)], True))
_pull(big)
_answer(monkeypatch, ([_row("1")], True))
_pull(big)
assert len(_open(big)) == 40
record = json.loads(big.state[f"{pull.FAILING}greenhouse:big"])
big.state[f"{pull.FAILING}greenhouse:big"] = json.dumps(
record | {"since": _days_ago(pull.STALE_DAYS + 1)})
result = _pull(big)
assert _open(big) == {"greenhouse:big:1"}
assert (result["unanswered"], result["closed"]) == (0, 39)
assert pull.failing_boards(big) == {}
assert json.loads(big.state["ats:count:greenhouse:big"])["count"] == 1
def _remote_host(monkeypatch: pytest.MonkeyPatch, remote: str) -> list[str]:
"""A whole-listing host `fake` that filters by country and, per `remote`,
on remote; its feed lists the ids in the returned list."""
listed: list[str] = []
module: object = types.ModuleType("fake")
module.NAME, module.LABEL, module.PATTERNS = "fake", "Fake", () # type: ignore[attr-defined]
module.slug_ok = lambda slug: slug.isalnum() # type: ignore[attr-defined]
module.allowed = lambda slug: lambda h: h == "jobs.fake.example" # type: ignore[attr-defined]
module.careers_url = lambda slug: f"https://jobs.fake.example/{slug}" # type: ignore[attr-defined]
module.READS = Reads( # type: ignore[attr-defined]
scope="board", listing="whole", text="listing", complete="whole", places="one",
robots="test", filters=Filters(country="many", remote=remote)) # type: ignore[arg-type]
module.feed_url = lambda slug, ask: f"https://jobs.fake.example/{slug}" # type: ignore[attr-defined]
module.parse = lambda body, slug: [_row(i) for i in json.loads(body)] # type: ignore[attr-defined]
monkeypatch.setitem(readers.HOSTS, "fake", module)
monkeypatch.setattr(ats, "_get", lambda client, url, host, slug, **kw: json.dumps(
listed).encode())
return listed
def test_a_filtered_read_that_could_not_ask_remote_keeps_remote_roles_until_stale(
store: Store, monkeypatch: pytest.MonkeyPatch) -> None:
listed = _remote_host(monkeypatch, remote="none")
store.add_subscription("ats", "fake:acme", label="Acme", company="Acme", board="fake")
reach = {"fake:acme": {"countries": ["DK"], "remote": True}}
def pull_once() -> dict[str, object]:
run = pull.claim_run(store)
result = pull.run_batch(FakeConfig(), store, targets=["fake:acme"], run=run, # type: ignore[arg-type]
reach=reach)
return pull.close_boards(FakeConfig(), store, Tally(results=(result,))) # type: ignore[arg-type]
listed[:] = ["1", "2", "3", "4"]
pull_once()
_set(store, "remote", "remote", "fake:acme:3", "fake:acme:4")
listed[:] = ["1"]
assert pull_once()["closed"] == 1
assert _open(store) == {"fake:acme:1", "fake:acme:3", "fake:acme:4"}
assert store.state["ats:remote_unasked:fake:acme"] == "1"
_set(store, "last_seen", _days_ago(pull.STALE_DAYS + 1), "fake:acme:4")
assert pull_once()["closed"] == 1
assert _open(store) == {"fake:acme:1", "fake:acme:3"}
def test_a_narrowed_read_closes_what_it_has_not_listed_for_stale_days(
acme: Store, monkeypatch: pytest.MonkeyPatch) -> None:
_answer(monkeypatch, ([_row(i) for i in "123"], True))
_pull(acme)
def fake(*args: object, **kwargs: object) -> _runner.Listing:
return _runner.Listing([_row("1")], True, ["1"], narrowed=True)
monkeypatch.setattr(_runner, "read_listing", fake)
assert _pull(acme)["closed"] == 0
_set(acme, "last_seen", _days_ago(pull.STALE_DAYS + 1), "workable:acme:3")
assert _pull(acme)["closed"] == 1
assert _open(acme) == {"workable:acme:1", "workable:acme:2"}
def test_a_target_that_raises_after_its_read_is_recorded_and_the_batch_goes_on(
store: Store, monkeypatch: pytest.MonkeyPatch) -> None:
targets = [f"greenhouse:{slug}" for slug in ("a", "b", "c")]
for target in targets:
slug = target.partition(":")[2]
store.add_subscription("ats", target, label=slug, company=slug, board="greenhouse")
def fake_fetch(self: ATSSource) -> object:
[(target, _)] = self.targets()
yield _item("greenhouse", target.partition(":")[2], "1")
self.state[f"ats:ok:{target}"] = self.options["run"]
real = Store.upsert_role
def upsert(self: Store, item: Item, sub: object = None) -> bool:
if item.source_id.startswith("greenhouse:b:"):
raise ValueError("a row the store refuses")
return real(self, item, sub) # type: ignore[arg-type]
monkeypatch.setattr(ATSSource, "fetch", fake_fetch)
monkeypatch.setattr(Store, "upsert_role", upsert)
result = pull.run_batch(FakeConfig(), store, targets=targets, run="r") # type: ignore[arg-type]
assert result["roles"] == 2
assert _open(store) == {"greenhouse:a:1", "greenhouse:c:1"}
assert "ats:ok:greenhouse:b" not in store.state
since, reason = pull.failing_boards(store)["greenhouse:b"]
assert since and reason == "ValueError: a row the store refuses"
assert set(pull.failing_boards(store)) == {"greenhouse:b"}
def test_a_board_whose_rows_do_not_store_keeps_its_failing_record(
acme: Store, monkeypatch: pytest.MonkeyPatch) -> None:
since = _days_ago(2)
acme.state[f"{pull.FAILING}workable:acme"] = json.dumps(
{"since": since, "reason": "404 Not Found"})
_answer(monkeypatch, ([_row("1")], True))
def refuse(self: Store, item: Item, sub: object = None) -> bool:
raise ValueError("no")
monkeypatch.setattr(Store, "upsert_role", refuse)
_pull(acme)
assert pull.failing_boards(acme)["workable:acme"] == (since, "ValueError: no")
assert "ats:ok:workable:acme" not in acme.state
# ------------------------------------------------------- the settle speaks
def test_a_settle_names_the_unanswered_board_and_the_one_that_closed_many(
store: Store) -> None:
sub = store.add_subscription("ats", "greenhouse:big", label="Big", company="Big",
board="greenhouse")
store.add_subscription("ats", "greenhouse:gone", label="Gone", company="Gone",
board="greenhouse")
for n in range(80):
store.upsert_role(_item("greenhouse", "big", str(n), company="Big"), sub)
run = pull.claim_run(store)
store.state["ats:ok:greenhouse:big"] = run
store.state["ats:complete:greenhouse:big"] = "1"
store.state["ats:ids:greenhouse:big"] = json.dumps(
[f"greenhouse:big:{n}" for n in range(20)])
store.state[f"{pull.FAILING}greenhouse:gone"] = json.dumps(
{"since": _days_ago(1), "reason": "404 Not Found"})
said: list[object] = []
result = pull.close_boards(FakeConfig(), store, Tally(results=({"roles": 20, "run": run},)), # type: ignore[arg-type]
context=RunContext(on_notice=said.append))
assert (result["unanswered"], result["closed"]) == (1, 60)
[notice] = said
assert notice.code == "pull.settled" # type: ignore[attr-defined]
assert "greenhouse:gone (404 Not Found)" in notice.text # type: ignore[attr-defined]
assert "greenhouse:big 60 of 80" in notice.text # type: ignore[attr-defined]
assert "closed 60 roles" in notice.text # type: ignore[attr-defined]
def test_a_clean_settle_says_nothing(acme: Store, monkeypatch: pytest.MonkeyPatch) -> None:
_answer(monkeypatch, ([_row(i) for i in "12"], True))
said: list[object] = []
results = [pull.run_batch(FakeConfig(), acme, **batch) # type: ignore[arg-type]
for batch in pull._boards(FakeConfig(), acme)] # type: ignore[arg-type]
pull.close_boards(FakeConfig(), acme, Tally(results=tuple(results)), # type: ignore[arg-type]
context=RunContext(on_notice=said.append))
assert said == []
def test_a_finder_that_raises_releases_the_pull_claim(
acme: Store, monkeypatch: pytest.MonkeyPatch) -> None:
def broken(store: Store, subs: object) -> dict[str, object]:
raise RuntimeError("reach broke")
monkeypatch.setattr(pull, "reaches", broken)
# The worker runs the chain's finder once the job is done; here at once.
context = RunContext(on_task=lambda task: task.finder(FakeConfig(), acme))
with pytest.raises(RuntimeError, match="reach broke"):
cascade.run_pull(FakeConfig(), acme, context=context)
assert pull.running(acme) == ""
# And the job itself lets go when the chain cannot be built.
def no_chain(run: str) -> object:
raise RuntimeError("no chain")
monkeypatch.setattr(cascade, "pull_chain", no_chain)
with pytest.raises(RuntimeError, match="no chain"):
cascade.run_pull(FakeConfig(), acme, context=RunContext(on_task=lambda task: True))
assert pull.running(acme) == ""
+132 -2
View File
@@ -26,7 +26,7 @@ if str(ROOT) not in sys.path:
from plugins import ats, readers # noqa: E402
from plugins.readers import _runner # noqa: E402
from plugins.readers import lever # noqa: E402
from plugins.readers._contract import Ask # noqa: E402
from plugins.readers._contract import Ask, Filters, Reads # noqa: E402
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
@@ -253,7 +253,9 @@ def _fake_host(*, detail: Callable[..., Any] | None = None) -> types.ModuleType:
module.slug_ok = lambda slug: slug.isalnum()
module.allowed = lambda slug: lambda h: h == "jobs.fake.example"
module.careers_url = lambda slug: f"https://jobs.fake.example/{slug}"
module.feed_url = lambda slug: f"https://jobs.fake.example/{slug}/list"
module.READS = Reads(scope="board", listing="whole", complete="whole", places="one",
text="detail" if detail is not None else "listing", robots="test")
module.feed_url = lambda slug, ask: f"https://jobs.fake.example/{slug}/list"
module.parse = lambda body, slug: [
{"source_id": i, "title": f"Role {i}", "location": "", "department": "",
"url": f"https://jobs.fake.example/{slug}/{i}", "posted": None,
@@ -590,3 +592,131 @@ def test_the_eu_api_is_refused_for_a_us_board_and_the_reverse() -> None:
with _client(handler) as client, pytest.raises(ats._Refused):
ats._get(client, "https://api.eu.lever.co/v0/postings/x", "lever", "eu/x")
# ------------------------------------------- the guard, asks, remote unasked
def _listing_host(**reads: Any) -> Any:
"""A declared whole-listing board `fake` whose feed lists `module.ids`,
whatever is asked; `reads` override its declaration."""
module: Any = types.ModuleType("fake")
module.NAME, module.LABEL, module.PATTERNS = "fake", "Fake", ()
module.slug_ok = lambda slug: slug.isalnum()
module.allowed = lambda slug: lambda h: h == "jobs.fake.example"
module.careers_url = lambda slug: f"https://jobs.fake.example/{slug}"
module.READS = Reads(**({"scope": "board", "listing": "whole", "text": "listing",
"complete": "whole", "places": "one", "robots": "test"} | reads))
module.ids = []
module.feed_url = lambda slug, ask: f"https://jobs.fake.example/{slug}/list"
module.parse = lambda body, slug: [
{"source_id": i, "title": f"Role {i}", "url": f"https://jobs.fake.example/{i}",
"places": [{"text": "Berlin", "remote": None}]} for i in json.loads(body)]
return module
def _listed(module: Any) -> Callable[..., bytes]:
return lambda url, **kwargs: json.dumps(module.ids).encode()
def _read_and_stamp(state: dict[str, str], module: Any,
limits: _runner.Limits | None = None) -> _runner.Listing:
listing = _runner.read(state, "ats", "fake:acme", module, "acme", _listed(module),
limits=limits)
_runner.stamp(state, "ats", "fake:acme", "r", listing, listing.ids)
return listing
def test_an_unfiltered_whole_read_of_nothing_after_forty_is_suspect() -> None:
"""A Greenhouse-style board, trusted, unfiltered: a changed answer that
parses to no rows is refused rather than read as a board that closed."""
module = _listing_host()
state: dict[str, str] = {}
module.ids = [str(n) for n in range(40)]
_read_and_stamp(state, module)
module.ids = []
with pytest.raises(_runner.Suspect, match="listed 0 roles, under 10% of the 40"):
_runner.read(state, "ats", "fake:acme", module, "acme", _listed(module))
def test_a_whole_read_under_suspect_min_is_not_held() -> None:
module = _listing_host()
state: dict[str, str] = {}
module.ids = [str(n) for n in range(5)]
_read_and_stamp(state, module)
module.ids = []
assert _read_and_stamp(state, module).ids == []
assert json.loads(state["ats:count:fake:acme"])["count"] == 0
def test_the_same_suspect_answer_is_accepted_after_stale_days() -> None:
module = _listing_host()
state: dict[str, str] = {}
module.ids = [str(n) for n in range(40)]
_read_and_stamp(state, module)
module.ids = ["1"]
failing = "ats:failing:fake:acme"
recent = (datetime.now(UTC) - timedelta(days=20)).isoformat(timespec="seconds")
state[failing] = json.dumps({"since": recent, "reason": "Suspect: listed 1 roles"})
limits = _runner.Limits(page_delay=0, stale_days=21)
with pytest.raises(_runner.Suspect):
_runner.read(state, "ats", "fake:acme", module, "acme", _listed(module), limits=limits)
old = (datetime.now(UTC) - timedelta(days=22)).isoformat(timespec="seconds")
state[failing] = json.dumps({"since": old, "reason": "Suspect: listed 1 roles"})
# Outside a pull (no `stale_days`) the guard never gives way.
with pytest.raises(_runner.Suspect):
_runner.read(state, "ats", "fake:acme", module, "acme", _listed(module))
assert _read_and_stamp(state, module, limits).ids == ["1"]
assert json.loads(state["ats:count:fake:acme"])["count"] == 1
def test_keep_size_resets_the_guards_count_unless_told_not_to() -> None:
state = {"ats:count:fake:acme": json.dumps({"asks": "[]", "count": 40})}
ats.keep_size(state, "fake:acme", 40, reset=False)
assert "ats:count:fake:acme" in state
ats.keep_size(state, "fake:acme", 3)
assert "ats:count:fake:acme" not in state
assert state["ats:size:fake:acme"] == "3"
def test_a_filter_that_maps_to_no_host_id_asks_nothing_and_is_incomplete(
monkeypatch: pytest.MonkeyPatch) -> None:
module = _listing_host(filters=Filters(country="ids"))
module.facets = lambda get, slug: {"country": {}}
module.ids = ["1", "2"]
state: dict[str, str] = {}
listing = _runner.read(state, "ats", "fake:acme", module, "acme", _listed(module),
Ask(countries=("DK",)))
assert (listing.rows, listing.ids, listing.complete, listing.filtered) == (
[], [], False, True)
# Through the plugin: stamped, but not complete, so the settle closes nothing.
monkeypatch.setitem(readers.HOSTS, "fake", module)
monkeypatch.setattr(ats, "_get", lambda client, url, host, slug, **kw: b'["1"]')
source = ats.ATSSource({"boards": [{"url": "fake:acme", "label": "Acme",
"params": {"countries": ["DK"]}}], "run": "r"})
assert list(source.fetch()) == []
assert (source.state["ats:ok:fake:acme"], source.state["ats:complete:fake:acme"]) == (
"r", "")
def test_a_filtered_read_that_cannot_ask_remote_says_so() -> None:
reach = {"countries": ["DK"], "remote": True}
blind = _listing_host(filters=Filters(country="many"))
ask = _runner.ask_for(blind, None, reach)
assert (ask.countries, ask.remote) == (("DK",), None)
assert _runner.remote_wanted(blind, None, reach)
listing = _runner.read_listing(blind, "acme", _listed(blind), ask,
remote=_runner.remote_wanted(blind, None, reach))
assert listing.filtered and listing.remote_unasked
state: dict[str, str] = {}
_runner.stamp(state, "ats", "fake:acme", "r", listing, listing.ids)
assert state["ats:remote_unasked:fake:acme"] == "1"
# A host that filters on remote asks it as a series of its own.
sees = _listing_host(filters=Filters(country="many", remote="many"))
ask = _runner.ask_for(sees, None, reach)
listing = _runner.read_listing(sees, "acme", _listed(sees), ask, remote=True)
assert listing.filtered and not listing.remote_unasked
# A params' remote of False asks nothing and wants nothing.
assert not _runner.remote_wanted(blind, {"remote": False}, reach)
+7 -17
View File
@@ -103,8 +103,6 @@ def test_ask_for_narrows_the_params_to_the_declared_filters() -> None:
both = _whole(filters=Filters(country="many", remote="many"))
ask = _runner.ask_for(both, {"countries": ["DE"]}, {"countries": ["DK"], "remote": True})
assert (ask.countries, ask.remote) == (("DE",), True)
# A module without READS reads the whole board.
assert _runner.ask_for(types.ModuleType("old"), {"countries": ["DK"]}) == Ask()
def test_a_searchs_params_go_whole_into_the_ask_with_its_key(
@@ -139,11 +137,12 @@ def test_ids_filters_are_mapped_through_the_facets() -> None:
ask = Ask(countries=("DK", "XX"))
[mapped] = _runner.series(module, ask, {"country": {"DK": "h-1", "SE": "h-2"}})
assert mapped.countries == ("h-1",)
# None known: the host has nothing in reach, and the read is empty.
# None known: nothing is asked, which is no evidence, so the read is
# incomplete and closes nothing.
assert _runner.series(module, Ask(countries=("XX",)), {"country": {"DK": "h-1"}}) == []
listing = _runner.read_listing(module, "acme", Web(), Ask(countries=("XX",)),
facets={"country": {}})
assert (listing.rows, listing.complete, listing.filtered) == ([], True, True)
assert (listing.rows, listing.complete, listing.filtered) == ([], False, True)
def test_facets_are_asked_once_a_day_and_only_for_an_ids_filter() -> None:
@@ -189,17 +188,6 @@ def test_a_body_over_the_cap_is_read_again_at_the_fallback_feed() -> None:
_runner.read_listing(module, "acme", web)
def test_a_module_without_reads_gets_the_older_signatures() -> None:
module: Any = types.ModuleType("old")
module.NAME = "old"
module.feed_url = lambda slug: f"https://jobs.fake.example/{slug}/list"
module.parse = lambda body, slug: [_row(i) for i in json.loads(body)]
listing = _runner.read_listing(module, "acme",
Web({"https://jobs.fake.example/acme/list": b'["1"]'}),
Ask(countries=("DK",)))
assert [r["source_id"] for r in listing.rows] == ["1"] and listing.complete
def test_a_paged_board_with_a_total_is_complete_when_it_read_as_many() -> None:
pages = [["1", "2"], ["3"]]
listing = _runner.read_listing(_paged(pages, total=3), "acme", _pages_web(pages),
@@ -333,9 +321,11 @@ def test_a_suspect_read_far_under_the_last_count_fails() -> None:
web.answers["https://jobs.fake.example/acme/list"] = b'["1"]'
with pytest.raises(_runner.Suspect, match="under 10% of the 20"):
_runner.read(state, "ats", "t", module, "acme", web)
# A reader whose empty answer is trusted reads it as it is.
# A reader whose empty answer is trusted is held too, from a count of
# `SUSPECT_MIN`.
module.READS = module.READS.model_copy(update={"empty": "trusted"})
assert _runner.read(state, "ats", "t", module, "acme", web).ids == ["1"]
with pytest.raises(_runner.Suspect):
_runner.read(state, "ats", "t", module, "acme", web)
def test_the_guard_compares_only_reads_that_asked_the_same() -> None:
+50
View File
@@ -365,6 +365,56 @@ def test_failed_gates_reads_evidence_only() -> None:
"none of remote remote, seniority senior"]
def test_an_exclude_gate_on_title_gates_whole_words_and_names_the_value(
store: Store, config: Config, fake: FakeEmbedder) -> None:
"""The sales and data engineering titles the owner does not want: a
phrase matches whole words of the folded title, so "Data Engineering
Manager" and "Forward Deployed Engineer" stay, and the reason names the
phrase that matched."""
ae = role(store, "1", "Enterprise Account Executive, UK")
de = role(store, "2", "Senior MLOps/Data Engineer")
manager = role(store, "3", "Data Engineering Manager")
fde = role(store, "4", "Forward Deployed Engineer")
se = role(store, "5", "Sales Engineer (SE)")
agent = store.save_agent("No sales", "", [
Gate(field="title", values=["Account Executive", "data engineer"], mode="exclude")])
run(config, store, score.run_score)
scores = store.scores(agent.id)
assert scores[ae] == (0.0, "title Account Executive")
assert scores[de] == (0.0, "title data engineer")
assert all(scores[uid][1] == "" and scores[uid][0] > 0 for uid in (manager, fde, se))
def test_a_favourite_is_gated_by_an_exclude_gate_it_matches(
store: Store, config: Config, fake: FakeEmbedder) -> None:
"""A star skips the `any` group, not the titles the agent leaves out."""
store.add_company("Fave")
director = role(store, "1", "Account Director, Nordics", company="Fave")
engineer = role(store, "2", "Forward Deployed Engineer", company="Fave")
agent = store.save_agent("No sales", "", [
Gate(field="title", values=["account director"], mode="exclude")])
store.set_affinity(agent.id, "fave", 2)
run(config, store, score.run_score)
scores = store.scores(agent.id)
assert scores[director] == (0.0, "title account director")
assert scores[engineer][1] == ""
def test_failed_gates_names_the_value_an_exclude_gate_matched() -> None:
gates = [Gate(field="country", values=["DK"]),
Gate(field="title", values=["account executive", "rvp"], mode="exclude")]
def cand(*states: int) -> Candidate:
return Candidate(uid="u", posted_at=NOW.isoformat(), affinity=2, gates=list(states))
assert score.failed_gates(cand(1, 0), gates) == []
assert score.failed_gates(cand(1, -1), gates) == []
assert score.failed_gates(cand(1, 2), gates) == ["title rvp"]
assert score.failed_gates(cand(0, 1), gates) == ["country not DK", "title account executive"]
def test_a_prefer_gate_adds_its_weight_to_the_score(
store: Store, config: Config, fake: FakeEmbedder) -> None:
remote = role(store, "1", remote="remote")
+53
View File
@@ -371,3 +371,56 @@ def test_a_search_switched_off_mid_pull_ends_the_pull_closed(
assert store.roles() == []
# And a batch that starts after the switch reads nothing.
assert pull.run_batch(FakeConfig(), store, **batches[0])["roles"] == 0 # type: ignore[arg-type]
# ------------------------------------------------- failures and the guard
def _broken_provider(monkeypatch: pytest.MonkeyPatch) -> None:
"""The provider `fake` of `_fake_provider`, answering with an error."""
_fake_provider(monkeypatch, [], complete=True)
def page(get: Any, slug: str, cursor: Any, ask: Ask) -> Any:
raise ValueError("hits renamed")
readers.PROVIDERS["fake"].page = page
def test_a_search_whose_provider_raises_is_recorded_failing(
store: Store, monkeypatch: pytest.MonkeyPatch) -> None:
store.add_subscription("search", "fake:cph", label="Copenhagen", board="fake",
params={"q": "backend"})
_fake_provider(monkeypatch, ["h1"], complete=True)
_pull(store)
_broken_provider(monkeypatch)
result = _pull(store)
assert result["unanswered"] == 1
since, reason = pull.failing_boards(store)["fake:cph"]
assert since and reason == "ValueError: hits renamed"
assert {r.source_id for r in store.roles()} == {"fake:h1"}
# Failing for `STALE_DAYS` closes its roles, as a board's.
old = (datetime.now(UTC) - timedelta(days=pull.STALE_DAYS + 1)).isoformat(
timespec="seconds")
store.state["search:failing:fake:cph"] = json.dumps({"since": old, "reason": reason})
_pull(store)
assert store.roles() == []
# Its next answer removes the record and reopens what it lists.
_fake_provider(monkeypatch, ["h1"], complete=True)
_pull(store)
assert pull.failing_boards(store) == {}
assert {r.source_id for r in store.roles()} == {"fake:h1"}
def test_a_search_that_lists_nothing_after_forty_is_suspect(
store: Store, monkeypatch: pytest.MonkeyPatch) -> None:
store.add_subscription("search", "fake:cph", label="Copenhagen", board="fake",
params={"q": "backend"})
forty = [f"h{n}" for n in range(40)]
_fake_provider(monkeypatch, forty, complete=True)
_pull(store)
_fake_provider(monkeypatch, [], complete=True)
result = _pull(store)
assert (result["unanswered"], result["closed"]) == (1, 0)
assert len(store.roles()) == 40
assert "under 10% of the 40" in pull.failing_boards(store)["fake:cph"][1]
+3 -1
View File
@@ -597,6 +597,7 @@ def test_companies_lists_board_and_open_role_count(
assert res.status_code == 200
[row] = [c for c in res.json() if c["key"] == "acme"]
assert row == {"key": "acme", "name": "Acme", "notes": "", "board": "greenhouse",
"target": "greenhouse:acme",
"subscription_id": sub.id, "on": True, "affinity": 0, "open_roles": 1,
"ats": "greenhouse", "ats_slug": "", "careers_url": "",
"country": "DK", "city": "", "domain": "", "watch": None,
@@ -1449,7 +1450,8 @@ def test_a_company_is_added_by_hand_with_a_reason_and_a_link(
assert res.status_code == 201, res.text
assert res.json() == {
"key": "revolut", "name": "Revolut Ltd", "domain": "", "notes": "wants a login",
"board": "", "subscription_id": None, "on": None, "affinity": 0, "open_roles": 0,
"board": "", "target": "", "subscription_id": None, "on": None, "affinity": 0,
"open_roles": 0,
"ats": "", "ats_slug": "", "careers_url": "https://revolut.example/careers",
"country": "", "city": "", "watch": None, "careers_opened_at": "",
"failing_since": "", "failing_reason": "", "from_search": False,