Merge the pruned README, plugins README, filter survey and config comments (wp-rev-docs-rest)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
commit
80b8416579
13 files changed
+129
-394
No files matched your search
@@ -13,11 +13,11 @@ uv run app job pull # read every board and search, map and fold, embe
|
||||
uv run app serve # http://127.0.0.1:8790
|
||||
```
|
||||
|
||||
`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the chip input's. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents the keys, among them the `extract:` and `summarise:` blocks, which set the default for the two model stages; the switch under Settings, Sources, Runs overrides it without a restart, and `app stages` does the same from the command line.
|
||||
`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the tests of the chip input and the remote chips. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents the keys. Its `extract:`, `summarise:` and `judge:` blocks set the default for the three model stages, and the `fetch_text:` block that of the stage reading role pages. The switches under Settings, Sources, Runs override them without a restart, and `app stages` does the same from the command line.
|
||||
|
||||
## Sources: job boards and company boards
|
||||
|
||||
The Sources sub-tab does all of this; the same actions exist on the command line:
|
||||
The Sources page under Settings does all of this; the same actions exist on the command line:
|
||||
|
||||
```bash
|
||||
uv run app searches add jobindex "Copenhagen golang" --q "golang go-udvikler" --area 1
|
||||
@@ -40,11 +40,11 @@ uv run app agents list # the agents and their pr
|
||||
uv run app prep unmapped # the host words no table maps; prep unlinked lists the place texts no place resolves
|
||||
```
|
||||
|
||||
The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read (its own pages, a host with no module, a site whose robots.txt disallows reading) is still a row, with its careers page and the reason, shown on the Watchlist tab for checking by hand and under Companies as "not readable yet", so the gap is visible and one module at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles, and asks first when there are open roles to lose; a board is read once before it is kept, and one that lists no roles yet is kept too. Setting a board queues a pull right after, so its roles are read by the one already running or the very next one, rather than waiting for the schedule. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
|
||||
The seed is a hand-picked list of European tech employers, not a list of boards. A company whose careers site the app cannot read is still a row, with its careers page and the reason, shown on the Watchlist tab and under Companies as "not readable yet". A row's Set board control (Change once it has a board) attaches, replaces or detaches its board, which is read once before it is kept; replacing or detaching closes the old board's open roles, and so does switching a board or a search off. `companies discover` reads a company's site for a link into one of the hosts the app knows, probes the hosts only when no page names one, and with `--add` sets the board of a new or existing company. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
|
||||
|
||||
`app stages` shows or switches extract, summarise, judge and fetch text, `app seniority` shows per host how often the stated level equals the model's, `app dupes` recomputes which open roles are copies of one posting, and `app prep` maps the host words and folds the copies without a pull.
|
||||
|
||||
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. A freehire query is free text over about 50 hosts whose words are OR over title, company, text and place, so it is noisy: `--country` takes comma-separated ISO codes (`dk,se`) and `--remote` a work mode. A Jobnet query is a loose match without phrases and `--region` one of its five regions (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`). An Akademikernes Jobbank query is OR words, `--region` its `amt` id (2 is Storkøbenhavn) and `--remote` `helt` or `delvist`. A freehire role whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
|
||||
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. A freehire query is free text over about 50 hosts whose words are OR over title, company, text and place, so it is noisy: `--country` takes comma-separated ISO codes (`dk,se`) and `--remote` a work mode. A Jobnet query requires every word and has no phrases, so a search is one term; `--region` is one of its five regions (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`). An Akademikernes Jobbank query is OR words, `--region` its `amt` id (2 is Storkøbenhavn) and `--remote` `helt` or `delvist`. `searches add` runs the search once and saves it only if it finds something. The `embeddings` extra is for the embeddings behind prompt closeness and CV closeness, `agent` serves extract, summarise and judge, and `passkeys` is for the Account tab's passkeys.
|
||||
|
||||
## The tabs
|
||||
|
||||
@@ -52,7 +52,7 @@ Three tabs, and a cog for settings.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Roles** | newsdesk's feed: a filter rail (Viewing as, Show read, Agent match or Newest, first seen, tags, and under more filters company and place chips, seniority from, to and track, function, remote, pay and votes), the open roles with company, place, age, tags and a summary, votes (▲ like ● neither way ▼ pass) and a Save button. A title opens the reader beside the list; on a phone it takes the screen. `j`/`k` move, `Enter` opens, `/` searches |
|
||||
| **Roles** | newsdesk's feed: a filter rail (Viewing as, saved filters, Show read, the order (What I'm looking for, What fits me or Newest), Posted, tags, and under more filters company and place chips, seniority from, to and track, function, remote, pay, votes, Closing soon and Show gated), and the open roles that pass the agent's fetch gate, with company, place, age, tags and a summary, votes (▲ like ● neither way ▼ pass) and a Save button (★). What fits me needs a CV; an agent without one is shown What I'm looking for. A title opens the reader beside the list; on a phone it takes the screen. `j`/`k` move, `Enter` opens, `/` searches |
|
||||
| **Applications** | the saved roles and their status, each change recorded as an event |
|
||||
| **Watchlist** | the companies whose board the app cannot read, for going through their careers pages by hand: Yes or No marks whether one is worth checking (a filter shows the unrated, the yes, the no or all), the careers page opens in a new tab and notes when, and a missing or wrong link is set in the row. A company whose board is failing is listed too. Separate from the affinity a search agent has |
|
||||
|
||||
@@ -60,19 +60,20 @@ The cog opens `/settings`: a rail of pages on the left, the page beside it, stic
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/exclude rules as rows, fetch terms, weights and limits); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
|
||||
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise, judge and fetch text, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
|
||||
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its fetch gate in words, its weights, its analytics filter and its summarise size. Edit turns the card into the form in place; New agent puts a blank card on top. The form is name, prompts, the fetch gate in four parts (World, Location, Role exclusion, Role fetch), CV, languages, likes and dislikes, the analytics filter, the want feed score weights, the CV match weights and the summarise size; other rules are under Advanced. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
|
||||
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what a pull asks its board for and what its read cannot do, the board's size with a Recheck, the All locations and All roles switches, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise, judge and fetch text, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
|
||||
| **Account** | the password, sessions and passkeys. Open on loopback, locked on anything else |
|
||||
|
||||
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its `taste` weight scores a role by how close it is to the roles the agent liked minus the ones it passed.
|
||||
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its job like weight (`taste` in the config) scores a role by how close it is to the roles the agent liked, away from the ones it passed.
|
||||
|
||||
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role whose text fetch text has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role; the switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, bounded to the top of each agent's feed (its `summarise_top`) and the roles liked or saved, so the cost stays flat.
|
||||
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role whose page fetch text has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role. The switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, which reads the top of both orders (each agent's `summarise_top`, 30 to start) and the roles liked or saved, so the cost stays flat.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`documentation/ADR Fetch prep and score.md`](<documentation/ADR Fetch prep and score.md>) and [`documentation/ADR Plugins.md`](<documentation/ADR Plugins.md>): the owner's target design, for the ranking from the fetch gate to the feed and for what a source plugin delivers.
|
||||
- [`docs/decisions.md`](docs/decisions.md): what was decided for roledesk and why.
|
||||
- [`docs/open.md`](docs/open.md): what is left to do, for whoever picks this up next.
|
||||
- [`plugins/README.md`](plugins/README.md): every reader, its limits and the Watchlist; [`docs/filter-survey.md`](docs/filter-survey.md): which source-side filters each host takes.
|
||||
- [`plugins/README.md`](plugins/README.md): every reader, its endpoints and what each states; [`docs/filter-survey.md`](docs/filter-survey.md): the filter parameters each host takes at the source.
|
||||
- `docs/plan-*.md`: plans in progress, folded into `decisions.md` and deleted when their work lands.
|
||||
- [`desk/README.md`](desk/README.md), [`pipeline/README.md`](pipeline/README.md), [`auth/README.md`](auth/README.md), [`deploy/README.md`](deploy/README.md): the copied modules, where each comes from and how to refresh it.
|
||||
- [`docs/hosting.md`](docs/hosting.md), [`docs/release.md`](docs/release.md), [`docs/server.md`](docs/server.md), [`docs/password-protection.md`](docs/password-protection.md), [`docs/llm-backends.md`](docs/llm-backends.md): the boilerplate's deployment docs.
|
||||
+2
-1
@@ -1,12 +1,13 @@
|
||||
"""The command line.
|
||||
|
||||
app serve [--host H] [--port P] [--reload]
|
||||
app job pull|extract|embed|score [--payload JSON]
|
||||
app job pull|fetch_text|prep|embed|score|extract|judge|summarise [--payload JSON]
|
||||
app worker [--workers http=8,cpu=1] [--until-idle]
|
||||
app companies seed|add|edit|checked|remove|by-hand|set-board|detach|discover|check|size
|
||||
app searches add|list|remove
|
||||
app agents list|prompt
|
||||
app stages [extract|summarise|judge|fetch_text on|off]
|
||||
app seniority
|
||||
app dupes
|
||||
app prep [unmapped|unlinked] [--remap]
|
||||
app db init
|
||||
|
||||
+22
-16
@@ -1,22 +1,28 @@
|
||||
"""The work roledesk queues. Importing this package registers every job, every
|
||||
task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
|
||||
|
||||
pull.py read every board and search and store the roles they yield
|
||||
prep.py the `prep` stage: `app.prep` over what a pull stored, then the fold
|
||||
extract.py title and description into the gate columns, on Haiku
|
||||
embed.py one vector per open role, for fit
|
||||
score.py every open role scored for every enabled search agent
|
||||
summarise.py about 60 words on each of the best roles, on Haiku
|
||||
judge.py each agent's best roles against its CV: skills, experience, career
|
||||
cascade.py pull → prep → embed → score → ... → summarise → judge, what `TASKS`
|
||||
offers, the `pull` job a schedule fires, and that schedule
|
||||
stages.py whether extract, summarise and judge run: the switch, else config.yaml
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
added, switched on or off
|
||||
discover.py which ATS a company's careers site runs on, from its pages
|
||||
posting.py one role's posting page fetched as text, on demand
|
||||
reach.py which countries a pull asks each big board for, from the agents
|
||||
limits.py what limits a board's read, in words, for the Companies list
|
||||
pull.py read every board and search and store the roles they yield
|
||||
prep.py the `prep` stage: `app.prep` over what a pull stored, then the fold
|
||||
embed.py one vector per open role, for prompt closeness and CV closeness
|
||||
score.py every open role scored for every enabled search agent, and each
|
||||
agent's cut
|
||||
fetch_text.py the roles in the cut that came without text, read on their own
|
||||
pages, with no model
|
||||
extract.py title and description into the gate columns, on Haiku
|
||||
judge.py each agent's cut against its CV: skills, experience, career
|
||||
summarise.py about 60 words on the top of both orders and the roles liked or
|
||||
saved, on Haiku
|
||||
cascade.py pull → prep → embed → score → fetch_text → prep → embed → score
|
||||
→ extract → score → judge → summarise, what `TASKS` offers, the
|
||||
`pull` and `fetch_text` jobs that start it, and the default schedule
|
||||
stages.py whether extract, summarise, judge and fetch text run: the switch,
|
||||
else config.yaml
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
added, switched on or off
|
||||
discover.py which ATS a company's careers site runs on, from its pages
|
||||
posting.py one role's posting page fetched as text, on demand
|
||||
reach.py which countries a pull asks each big board for, from the agents
|
||||
limits.py what limits a board's read, in words, for the Companies list
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/* The Search agents page: what Agent match on the Roles tab is measured
|
||||
* against, one agent at a time.
|
||||
/* The Search agents page: what the order What I'm looking for on the Roles
|
||||
* tab is measured against, one agent at a time.
|
||||
*
|
||||
* Every agent from `/api/agents` is a card, newest first: its name, its
|
||||
* switch, its prompts, its gates in words, its weights, and a link to the
|
||||
|
||||
+58
-65
@@ -51,11 +51,10 @@ agent:
|
||||
max_tokens: 8192 # the answer's cap per call
|
||||
cli_path: "" # where the `claude` binary is, if not on PATH
|
||||
|
||||
# Title and description in, gate columns out (app/jobs/extract.py). Off by
|
||||
# default: on the API backend every batch of 20 roles is a paid call. On, a
|
||||
# pull reads only the cut of every enabled agent (its limits, see agents
|
||||
# below), after a first score; a second score then applies the fetch gate and
|
||||
# the cut to what was filled. `enabled` is only the
|
||||
# Title and description in, gate columns out: location, remote area, seniority,
|
||||
# languages and deadline (app/jobs/extract.py). Off by default: on the API
|
||||
# backend every batch of 20 roles is a paid call. On, a pull reads the cut of
|
||||
# every enabled agent (its limits, see agents below). `enabled` is only the
|
||||
# default for a fresh database: the switch under Settings, Sources, Runs (or
|
||||
# `app stages extract on|off`) overrides it and needs no restart.
|
||||
extract:
|
||||
@@ -66,12 +65,11 @@ extract:
|
||||
thinking: off # off | adaptive
|
||||
|
||||
# About 60 words on a role: what it is, the stack, where, the pay if stated
|
||||
# (app/jobs/summarise.py). Off by default. On, each pull summarises the top
|
||||
# of every enabled agent's feed that has none yet, the best summarise_top (its
|
||||
# limits) by the want feed score and the best summarise_top by the match
|
||||
# order (CV match, then CV closeness), and every role liked or saved, so the
|
||||
# cost stays bounded however many roles are open. The Summarise button on
|
||||
# a role's page asks for that one role either way. `enabled` is only the
|
||||
# (app/jobs/summarise.py). Off by default. On, each pull summarises the
|
||||
# summarise_top (an agent's limits) best roles of each enabled agent's two
|
||||
# orders, What I'm looking for and What fits me, and every role liked or
|
||||
# saved, when they have no summary yet. The Summarise button on a role's page
|
||||
# asks for that one role either way. `enabled` is only the
|
||||
# default for a fresh database: the switch under Settings, Sources, Runs (or
|
||||
# `app stages summarise on|off`) overrides it and needs no restart.
|
||||
summarise:
|
||||
@@ -84,12 +82,12 @@ summarise:
|
||||
# Only extract reads it.
|
||||
|
||||
# Each search agent's cut read against its CV: skills, experience and career 0
|
||||
# to 10, a total 0 to 100, and 1 to 3 strengths and gaps (app/jobs/judge.py),
|
||||
# for the CV match order on Roles. Off by default. On, each pull judges the cut of
|
||||
# every enabled agent with a CV and a prompt, and only the roles not judged
|
||||
# yet or judged from another CV, prompt, like, avoid or must text, or role
|
||||
# text. The CV rides in every call, so batches are
|
||||
# small. `enabled` is only the default for a fresh database: the switch under
|
||||
# to 10, and 1 to 3 strengths and gaps (app/jobs/judge.py); the CV match, which
|
||||
# orders What fits me on Roles, weights the three. Off by default. On, each pull
|
||||
# judges the cut of every enabled agent with a CV and a prompt, once per change
|
||||
# of its CV, prompts, likes, dislikes or must text, or of the role's text. The
|
||||
# CV rides in every call, so batches are small. `enabled` is only the default
|
||||
# for a fresh database: the switch under
|
||||
# Settings, Sources, Runs (or `app stages judge on|off`) overrides it and needs
|
||||
# no restart. on_ingest is accepted and ignored, as for summarise.
|
||||
judge:
|
||||
@@ -110,25 +108,23 @@ fetch_text:
|
||||
per_run: 1000 # reads per run
|
||||
retry_days: 7 # days before a page that gave no text is read again
|
||||
|
||||
# A role and each agent's prompts as vectors, for the fit term of a score
|
||||
# (app/jobs/embed.py). Needs the `embeddings` extra; without it `embed` posts
|
||||
# a notice and scores leave fit out. Changing the model re-embeds.
|
||||
# multilingual-e5-small, not English-only MiniLM, because the corpus is now
|
||||
# European: measured against the "Copenhagen engineering" agent's intent, it
|
||||
# separates engineering from non-engineering roles better, especially among
|
||||
# Danish-country roles (see docs/decisions.md), for about 3x the CPU time. An
|
||||
# e5 model's "query: " / "passage: " prefixes are added automatically, keyed
|
||||
# off this name.
|
||||
# calibration maps a model's cosines to 0..1 levels, as [lo, hi] for fit and
|
||||
# for taste: lo is 0, hi is 1, clipped outside. multilingual-e5-small has
|
||||
# built-in constants (app/config.py, CALIBRATION); an entry here replaces
|
||||
# them for that model. A model with none is min-max scaled per run instead,
|
||||
# so its fit is a rank, not a level, and the log says so.
|
||||
# A role, each agent's prompts and its CV as vectors, for prompt closeness and
|
||||
# CV closeness (app/jobs/embed.py). Needs the `embeddings` extra; without it
|
||||
# `embed` posts a notice and scores leave closeness out. Changing the model
|
||||
# re-embeds. multilingual-e5-small, not English-only MiniLM, because the corpus
|
||||
# is European (see docs/decisions.md). An e5 model's "query: " / "passage: "
|
||||
# prefixes are added automatically, keyed off this name.
|
||||
# calibration maps a model's cosines to 0..1 levels, as [lo, hi] for fit (prompt
|
||||
# closeness), taste (job like) and cv (CV closeness, fit's band when left out):
|
||||
# lo is 0, hi is 1, clipped outside. multilingual-e5-small has built-in
|
||||
# constants (app/config.py, CALIBRATION); an entry here replaces them for that
|
||||
# model. A model with none is min-max scaled per run instead, so its levels are
|
||||
# ranks, and the log says so.
|
||||
embed:
|
||||
model: intfloat/multilingual-e5-small
|
||||
per_run: 50000
|
||||
calibration: {}
|
||||
# intfloat/multilingual-e5-small: {fit: [0.75, 0.88], taste: [-0.14, 0.19]}
|
||||
# intfloat/multilingual-e5-small: {fit: [0.75, 0.88], taste: [-0.14, 0.19], cv: [0.77, 0.90]}
|
||||
|
||||
# One block per plugin, keyed by the plugin's name. The ats and search plugins
|
||||
# read their boards and searches from the subscriptions table, so they need
|
||||
@@ -141,39 +137,36 @@ sources: {}
|
||||
# search:
|
||||
# adzuna: {app_id: "...", app_key: "..."}
|
||||
|
||||
# Starter search agents, imported once into an empty `agents` table. The gates
|
||||
# are the agent's fetch gate, each {field, values, mode: require|any|exclude}:
|
||||
# a require gate is the region, the any gates are one group a role must match
|
||||
# one of ("in Denmark, or remote"), the place part, and an exclude gate leaves
|
||||
# out the roles that match one of its values (title: the titles to leave
|
||||
# out). A gate rejects only a role whose column is filled, since extract fills
|
||||
# the rest for the roles in the cut. A company set to All locations skips the
|
||||
# any group, one set to All roles the exclude gates on title. Fields are
|
||||
# place, country, remote, remote_from, seniority, function, salary_min,
|
||||
# currency, permit, languages, company and title. weights are the final
|
||||
# score's, the feed's order, default {fit: 0.6, recency: 0.2, affinity: 0.2,
|
||||
# taste: 0.2}; taste is how close a role is to the ones the agent liked minus
|
||||
# the ones it passed. A country is an ISO code, filled from the board's
|
||||
# location line; EU is a role open anywhere in Europe.
|
||||
# limits is the agent's cut, the roles that get fetch text, extract and
|
||||
# judge: the best cut_fit_top by fit level at cut_fit_min or more, then the
|
||||
# best cut_cv_top by CV closeness at cut_cv_min or more that the first list
|
||||
# leaves out, then every role liked or saved. Both levels are calibrated, so
|
||||
# a minimum means the same in every run, and a liked company's roles sit
|
||||
# about 0.05 higher, a favourite's about 0.1. summarise_top is how deep into
|
||||
# each order, want and match, summarise reads. The defaults are
|
||||
# {cut_fit_top: 500, cut_fit_min: 0.6, cut_cv_top: 150, cut_cv_min: 0.6,
|
||||
# summarise_top: 30}. For a broad agent the count stops a list, for a narrow
|
||||
# one the minimum. An agent without a prompt that has text has no fit, so
|
||||
# nothing of it is in the cut by fit.
|
||||
# prompts is a list of {name, text}, each what the agent looks for; a role's
|
||||
# fit is its best prompt's, so one agent can hunt several kinds of role
|
||||
# without blending them into one paragraph. Votes, gates and limits stay the
|
||||
# agent's, shared by its prompts. intent, as in config.yaml, is one prompt
|
||||
# named after the agent.
|
||||
# enabled (default true) leaves an agent out of the scores and the pull when
|
||||
# false. The Search agents sub-tab lists and edits them; the order menu on
|
||||
# Roles picks one by name. Two more, beside the starter in config.yaml:
|
||||
# Starter search agents, imported once into an empty `agents` table. The Search
|
||||
# agents page lists and edits them afterwards, and the Viewing as select on
|
||||
# Roles picks one by name. An agent has:
|
||||
# - name, and enabled (default true; false leaves it out of the scores and the
|
||||
# pull).
|
||||
# - prompts, a list of {name, text}, each what the agent looks for; a role is
|
||||
# ranked by the prompt it is closest to. intent, as in config.yaml, is one
|
||||
# prompt named after the agent.
|
||||
# - gates, its fetch gate, each {field, values, mode: require|any|exclude}: a
|
||||
# require gate is the region (the world), the any gates are one group a role
|
||||
# must match one of ("in Denmark, or remote"; the location), and an exclude
|
||||
# gate leaves out the roles that match one of its values (title: the titles
|
||||
# to leave out). A gate rejects only a role whose column is filled, since
|
||||
# extract fills the rest for the roles in the cut. A company set to All
|
||||
# locations skips the any group, one set to All roles the exclude gates on
|
||||
# title. Fields are place, country, remote, remote_from, seniority,
|
||||
# function, salary_min, currency, permit, languages, company and title. A
|
||||
# country is an ISO code, filled from the board's location line; EU is a
|
||||
# role open anywhere in Europe.
|
||||
# - weights, the want feed score's, which orders What I'm looking for: {fit:
|
||||
# 0.6, recency: 0.2, affinity: 0.2, taste: 0.2}, that is prompt closeness,
|
||||
# newness, company like and job like; taste is how close a role is to the
|
||||
# ones the agent liked minus the ones it passed. cv_skills, cv_experience and
|
||||
# cv_career {30, 25, 30} weight the judge's three numbers in the CV match.
|
||||
# - limits, the analytics filter and the summarise size: {cut_fit_top: 500,
|
||||
# cut_fit_min: 0.6, cut_cv_top: 150, cut_cv_min: 0.6, summarise_top: 30}, the
|
||||
# count and the minimum by prompt closeness and by CV closeness, and how deep
|
||||
# into each order summarise reads. See documentation/ADR Fetch prep and
|
||||
# score.md.
|
||||
# Two more, beside the starter in config.yaml:
|
||||
agents: []
|
||||
# - name: Remote from Europe
|
||||
# prompts:
|
||||
|
||||
+4
-4
@@ -27,10 +27,10 @@ embed:
|
||||
model: intfloat/multilingual-e5-small
|
||||
sources: {}
|
||||
# The starter ranking, imported once into an empty table. Edit it under
|
||||
# Settings, Ranking afterwards; this list is not read again. EU in a country
|
||||
# gate is a role open anywhere in Europe ("Remote - Europe", "EMEA"). Which
|
||||
# roles get fetch text, extract and judge is each agent's cut, its `limits`, see
|
||||
# config.example.yaml, which has two more agents.
|
||||
# Settings, Search agents afterwards; this list is not read again. EU in a
|
||||
# country gate is a role open anywhere in Europe ("Remote - Europe", "EMEA").
|
||||
# Which roles get fetch text, extract and judge is each agent's cut, its
|
||||
# `limits`, see config.example.yaml, which has two more agents.
|
||||
agents:
|
||||
- name: Copenhagen engineering
|
||||
intent: >-
|
||||
|
||||
+2
-92
@@ -1,6 +1,6 @@
|
||||
# Survey: which filters the hosts take at the source
|
||||
|
||||
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. It covers the nine readers that declare a filter (SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken, Adzuna) and Workable, Teamtailor, Jibe and Lever, which declare none. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed. Sections 2 and 3 began as the plan for the fetch terms, which it called F2; F2 has landed (`decisions.md`, Fetch terms), and those sections now say what was recommended and what became of it.
|
||||
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. Twelve readers declare a filter in `READS.filters`: SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken and Adzuna were surveyed; freehire, Jobnet and Jobbank came later and were not, and their parameters are in the reader's docstring. Workable, Teamtailor, Jibe and Lever declare none and were surveyed. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed.
|
||||
|
||||
Marks used in the table:
|
||||
|
||||
@@ -12,7 +12,7 @@ Marks used in the table:
|
||||
|
||||
All request counts are exact: a host with a "live" mark on a claim spent its two requests on it, so anything else about that host is `code`, `front end` or `untested`.
|
||||
|
||||
## 1. Filters per host
|
||||
## Filters per host
|
||||
|
||||
Country and remote are left out; they are in the readers already. "Combine" is how a keyword and a country (or another facet) meet; "unknown" is what an unknown value does.
|
||||
|
||||
@@ -149,93 +149,3 @@ The environment has no key, so nothing was requested. From the public API docume
|
||||
| category | `occupation-field=apaJ_2ja_LuF` with `q=python` answered 497, the first hit `Data/IT` | untested | | untested | live |
|
||||
| place | `region=<län id>` (reader); `municipality=AvNB_uwa_6n6` returned a Stockholm hit | untested | | reader raises on an id outside `REGIONS` | live |
|
||||
| level | `experience=false` was in the same request as `municipality` and `occupation-field`; the answer was 1 hit, so it is not separable | | | | live, inconclusive |
|
||||
|
||||
## 2. Towns
|
||||
|
||||
**How a town reaches the fetch today.** An agent's `any` gate on `place` holds names ("Malmö", "Lund", "Helsingborg", "copenhagen", "odense" in agent 2). In `reach.gate_reach`, each name goes through `gaz.lookup` and `_countries_of`, which turns it into the country the place lies in; a name the gazetteer does not know makes the whole reach everywhere. The reach is a set of ISO codes and a remote flag; it holds no town. `ask_for` narrows it to the reader's declared dimensions, so `Ask.countries` gets `SE` and `DK` and `Ask.places` is empty for every board host, since none declares `place`. The gates then keep only the towns after the pull. A town gate therefore fetches its whole country.
|
||||
|
||||
For agent 2 this costs nothing: its `any` group has DK and SE beside the towns, so the towns are already covered. It costs where the group has towns and no country, and the country is large (Munich as DE).
|
||||
|
||||
**Which hosts could take a town instead.**
|
||||
|
||||
- Could, by what was seen: Eightfold (`location=`, its facet values are town strings; untested), Workable (`location` objects with `city`; untested body), Lever (`location=Stockholm`, live: exact text, matches any of a role's places), Adzuna (`where`, `location0..7`), Jobindex (`geoareaid`), Platsbanken (`municipality`).
|
||||
- Could, with facets: Workday if a tenant has a city facet (none seen: LEGO's is countries only, and `locationMainGroup` holds a country facet), SuccessFactors (`locationsearch=`, untested), Phenom (`location` and `locationData` in the body, untested), Jibe (a city facet is listed; the parameter is not known).
|
||||
- Cannot: SmartRecruiters has `city=` and `region=` in its documentation but it is untested; Oracle lists ten countries and sites only; Teamtailor takes ids.
|
||||
- Only the search providers do so today (`Filters(place="one")` on Adzuna, Jobindex and Platsbanken).
|
||||
|
||||
**Recommendation.** Leave towns as their country for the board hosts, and build `place` only where it saves a lot and the town is the agent's only reach:
|
||||
|
||||
1. The saving is small in the live data. For agent 2, the DK and SE reach already leaves at most 149 roles on any big board (Ericsson), so a town would cut a country read of one or two requests.
|
||||
2. Town strings are the host's own, unlike ISO codes: each host needs a name-to-string table or a facet lookup, and a town the host spells differently returns nothing and, on the closing guard, wrongly closes roles.
|
||||
3. Hosts that state their places per role (all the big ones) leave the exact match to the gates and the gazetteer, which is already right.
|
||||
|
||||
So: the plan was to carry `places` beside `countries` in the reach for a group that has towns and no country, and to pass them only to readers that declare `place`. That was not built. `Reach` (`app/jobs/reach.py`) carries countries and remote, a town in a gate is fetched as its country (`decisions.md`, Fetch terms), and the three search providers take their `place` from the saved params (`PARAMS`), never from a reach. Eightfold and Lever wait for a run that shows the town strings work.
|
||||
|
||||
## 3. What was recommended, and what landed
|
||||
|
||||
Landed: `query` on SmartRecruiters, SuccessFactors, Workday, Phenom, Eightfold and Oracle (`Filters(query="one")`). Not built: the rows below marked "no" or "later", and the rows for Workable, Teamtailor and Jibe, which declare no `query` yet.
|
||||
|
||||
### Declarations per host
|
||||
|
||||
| Host | `query` | `category` | `level` | `place` |
|
||||
|---|---|---|---|---|
|
||||
| SmartRecruiters | yes: `q`, one request per term (multi-word behaviour untested) | no: the parameter is not verified | no | no |
|
||||
| Eightfold | yes: `query` | later: `filter_function`, ids learnt from `filterDef` | no: no tenant filter seen | later: `location=<town string>` after a test |
|
||||
| SuccessFactors | yes: `q` | no: `department` gave a wrong answer for an unknown value and its values need script | no: the "Work Experience" facet is a tenant label | no |
|
||||
| Phenom | yes: `keywords` | later: `selected_fields.category` needs the value list | no: `positionLevel` is empty | no |
|
||||
| Workday | yes: `searchText` | later: `jobFamily` ids from the existing `facets` call | no | no |
|
||||
| Oracle | yes: `keyword` | later: `selectedCategoriesFacet` ids from `categoriesFacet` | no | no |
|
||||
| Workable | not built: `query` answered in the survey | no: the body is not settled (400) | no | no |
|
||||
| Teamtailor | not built: needs a baseline request that confirms the feed filters | no | no | no |
|
||||
| Jibe | not built: `keywords` | later: `categories` | no | no |
|
||||
| Lever | no: no keyword parameter; boards are small and read whole | no | no | later, as above |
|
||||
| Jobindex, Platsbanken, Adzuna | already `query` and `place`; `subid`, `occupation-field` and `category` are available for `category` | | | |
|
||||
|
||||
`level` is not worth declaring on any host: only SuccessFactors and Jibe have a facet for it, and on the tenants seen it is a tenant label or empty (Lever documents `level=` and SmartRecruiters states `experienceLevel` on each role; neither is filtered on). Seniority stays a prep matter (title rules, host word).
|
||||
|
||||
### What F2 built, and why in this order
|
||||
|
||||
The table shows how much terms can narrow big boards, from the live database of 2026-09-30:
|
||||
|
||||
| Board | Host | Open | In Europe | In DK or SE (today's reach) | Roles with a term in the title, Europe |
|
||||
|---|---|---|---|---|---|
|
||||
| Bosch | SmartRecruiters | 4,830 | 1,564 | 15 | 149 |
|
||||
| Delivery Hero | SmartRecruiters | 975 | 314 | 11 | 143 |
|
||||
| Coolblue | SmartRecruiters | 477 | 477 | 0 | 30 |
|
||||
| Wise | SmartRecruiters | 417 | 279 | 0 | 160 |
|
||||
| SAP | SuccessFactors | 807 | 365 | 6 | 128 |
|
||||
| Scania | SuccessFactors | 750 | 532 | 58 | 28 |
|
||||
| Vestas | SuccessFactors | 668 | 327 | 61 | 56 |
|
||||
| Novo Nordisk | SuccessFactors | 428 | 138 | 93 | 45 |
|
||||
| Philips | Phenom | 868 | 194 | 3 | 65 |
|
||||
| Zeiss | Workday | 848 | 264 (295 with no country) | 1 | 59 |
|
||||
| LEGO | Workday | 455 | 175 (45 with none) | 83 | 60 |
|
||||
| Ericsson | Eightfold | 618 | 210 | 149 | 64 |
|
||||
| Nokia | Oracle | 554 | 87 (54 with none) | 0 | 24 |
|
||||
|
||||
"Term in the title" is a count of the words head, director, VP, chief, CTO, CDO, data, AI, machine learning, forward deployed, solutions engineer or architect, professional services, research, innovation, principal, lead and manager in the title. It is a proxy for `fetch_terms`, and a lower bound on what a text filter returns: where the scope could be told, the keyword matched text and not only titles (SmartRecruiters, Eightfold and Oracle by the examples here; on SuccessFactors, Workday, Workable and Jibe it is not determined) (Ericsson: `software` matched a thesis title without it; Nokia: `engineer` matched 468 of 554; Bosch: 1,713 of 4,830). Expect a keyword read to return several times these counts, and a broad term like `engineer` to cut little.
|
||||
|
||||
Findings:
|
||||
|
||||
1. **Narrow reach needs no terms.** At DK and SE every big board is already at most 149 roles (Ericsson), and Bosch is two requests. Terms save requests only on the wide reach, which is the All countries tag; a company set to All roles takes no terms, and a favourite no longer widens the fetch (`decisions.md`).
|
||||
2. **The wide reach is where terms pay.** SmartRecruiters reads Europe (33 countries) as the whole board, because `MAX_SERIES` is 8 for a `one` host: Bosch is 49 requests of 100 for 1,564 European roles. One `q` request per term (about 35% of the board for `engineer`, far less for `director` or `data`) replaces it.
|
||||
3. **The order followed the size of the whole read that terms replace**, then how sure the parameter was:
|
||||
1. SmartRecruiters `q`: Bosch, Delivery Hero, Wise and Coolblue, 6,700 roles in the live DB, one live-verified parameter, AND with country.
|
||||
2. SuccessFactors `q`: SAP, Scania, Vestas and Novo Nordisk, 2,650 roles; verified. The reader already reads a country a request, so a term multiplies the series.
|
||||
3. Workday `searchText` (Zeiss 848, LEGO 455), Phenom `keywords` (Philips 868), Eightfold `query` (Ericsson 618) and Oracle `keyword` (Nokia 554): each verified once, each a small change to the request body or URL.
|
||||
4. **Category comes next, and only for Workday, Oracle, Jibe and Eightfold** (Phenom's needs its value list first). They state their own category ids in a response the reader already reads or can read (`facets`, `categoriesFacet`, `filter.categories`, `filterDef`), and the union across values is verified on Workday and Oracle. But an agent's term (`engineer`) is not a category name, so it needs a mapping from term to category per tenant that nobody has: not before terms have run on real boards.
|
||||
5. **Level is not filtered at the source.** Place is filtered by the search providers; Eightfold and Lever wait for a test that proves a town string, and the rest stay on country.
|
||||
|
||||
### Guards the fetch terms keep
|
||||
|
||||
- A term filter needs the closing guards to hold: a keyword read is not a whole board, so it does not write `ats:size` (a pull no longer rewrites a size at all) and roles outside it are not closed as gone (`narrowed` in `plugins/readers/_runner.py`, `close_boards` in `app/jobs/pull.py`).
|
||||
- Unknown-value behaviour differs: Lever returns nothing, Oracle returns everything, SuccessFactors returned something unrelated, Workable answered 400 to a body whose refused key is not known. The runner treats a facet value as known only when the host's own facet list stated it, as `series` does for ids.
|
||||
- Keywords are text matches everywhere except Adzuna's `title_only`; the agent's own scoring still decides relevance.
|
||||
|
||||
### Open, and cheap to settle with one request each
|
||||
|
||||
- SmartRecruiters: does `q="head of data"` mean all words or any, and are `department`, `function`, `experienceLevel` accepted (one request each)?
|
||||
- Teamtailor: a baseline request to confirm the feed's `query` filter.
|
||||
- Workable: which key of the survey's body was refused (`department` or `location`).
|
||||
- Eightfold: a `location` with a town string, and repeated `filter_function` values.
|
||||
- Adzuna: everything, once the key exists.
|
||||
@@ -16,7 +16,7 @@ In the seed (`seed/companies.json`, 362 companies, 286 with a board), 76 compani
|
||||
| A host that is read, with a gap | Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
|
||||
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
|
||||
|
||||
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures; `own-1.md` and `own-2.md` name `out-own-1.json` and `out-own-2.json`, which do not exist, and `survey-own.json` is the whole record.
|
||||
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures, and `survey-own.json` is the whole record.
|
||||
|
||||
## Decisions to take first
|
||||
|
||||
@@ -30,14 +30,14 @@ Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tes
|
||||
|
||||
| WP | Model | Builds | Needs |
|
||||
|---|---|---|---|
|
||||
| 9 Provisional title | Opus | `title` in `DETAIL_FIELDS`, `upsert_role` keeping a stored title over a provisional one, and their tests; owns `plugins/readers/_runner.py`, `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | `plan-remote.md` WP-A |
|
||||
| 9 Provisional title | Opus | `title` in `DETAIL_FIELDS`, `upsert_role` keeping a stored title over a provisional one, and their tests; owns `plugins/readers/_runner.py`, `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | |
|
||||
| 10 Sitemap reader | Sonnet | `sitemap`: role URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
|
||||
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
|
||||
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
|
||||
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
|
||||
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
|
||||
|
||||
WP9 owns files that `plan-remote.md` WP-A also owns (`app/storage/`, `plugins/`, `tests/`) and `plan-review.md` R1 edits (`_runner.py`, `ats.py`), so it waits until those have merged. WP12 to WP15 own new modules only and can start at once and in parallel. WP10 waits for WP9.
|
||||
WP12 to WP15 own new modules only and can start at once and in parallel. WP10 waits for WP9.
|
||||
|
||||
### WP10 notes: Mambu and Arm
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Own-site batch 1: how each career site can be read
|
||||
|
||||
31 companies, all classified from live responses (2026-09-29). Full data is in `out-own-1.json`.
|
||||
31 companies, all classified from live responses (2026-09-29).
|
||||
|
||||
## Counts per class
|
||||
|
||||
@@ -57,4 +57,4 @@ Discovery misses worth fixing in the app: Lever EU (`api.eu.lever.co`), custom d
|
||||
|
||||
## Method notes
|
||||
|
||||
One request per second per host, `curl -m 20 -L`, browser User-Agent, no login. The Algolia call for new-work used a Referer header matching the site that ships the key. Scratch files are in `wo1/` next to this note.
|
||||
One request per second per host, `curl -m 20 -L`, browser User-Agent, no login. The Algolia call for new-work used a Referer header matching the site that ships the key.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Own-site career pages, batch 2 (30 companies)
|
||||
|
||||
Every request was a live curl on 2026-09-29. Machine-readable results are in out-own-2.json.
|
||||
Every request was a live curl on 2026-09-29.
|
||||
|
||||
## Counts per class
|
||||
|
||||
|
||||
+20
-196
@@ -1,127 +1,12 @@
|
||||
# Where roles come from
|
||||
|
||||
This file lists every source roledesk reads roles from, how each is read and what it cannot deliver, as of the code on main and the live database of 2026-09-30. The contract for writing a reader (a host module or a search provider) is the docstring of `plugins/readers/__init__.py`, with its types and the declaration check in `plugins/readers/_contract.py` and the conformance test in `tests/test_readers_contract.py`, not this file.
|
||||
This file lists every source roledesk reads roles from, how each is read and what it cannot deliver. The contract for writing a reader (a host module or a search provider) is `documentation/ADR Plugins.md`, the owner's draft, and in code the docstring of `plugins/readers/__init__.py`, with its types and the declaration check in `plugins/readers/_contract.py` and the conformance test in `tests/test_readers_contract.py`, not this file.
|
||||
|
||||
The code has two plugins: `ats` (`plugins/ats.py`) reads the board of a company someone listed, with one module per host in `plugins/readers/`; `search` (`plugins/search.py`) runs saved queries on sites that list roles across companies. Below, a module of `ats` is a "host" and a module of `search` is a "provider". A company's list of roles on a host is its "board"; a board is attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
|
||||
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the ADRs state the ones that bear on a reader.
|
||||
|
||||
## Flow
|
||||
|
||||
The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section says what each step leaves behind.
|
||||
|
||||
1. A reader declares what it does (`READS`) and how to ask and parse (`feed_url` and `parse`, or `page` and `rows`, and optionally `detail` and `facets`).
|
||||
2. `pull` queues batches of 10 targets (`app/jobs/pull.py`). In each batch `ats` or `search` calls `_runner.read`: it asks the reader for the facets a filter needs, builds the series of requests from the reach, reads the listing whole or in pages, continues or starts a sweep, and applies the 10% guard.
|
||||
3. The plugin stores every row with `upsert_role` and then stamps the target's state (`self.state`, the `state` table). A search stamps only `ok`, `complete` and `ids`. Keys are `ats:<key>:<target>`:
|
||||
|
||||
| Key | Written by | Read by |
|
||||
|---|---|---|
|
||||
| `ok` | `_runner.stamp`, the run marker, after every row was stored | `close_boards`: a target whose marker is not this run's did not answer |
|
||||
| `complete` | `stamp` | `close_boards`: only a complete read closes |
|
||||
| `ids` | `stamp`, the compound ids the read listed | `close_boards`: closes open roles not in it |
|
||||
| `filtered` | `stamp`, a read by country, place or remote | `close_boards`: keeps roles with a vote or an application |
|
||||
| `remote_unasked` | `stamp`, a filtered read while the agents wanted remote and the host has no remote filter | `close_boards`: keeps remote roles, closes them by age; the Companies line says "remote not asked" |
|
||||
| `narrowed` | `stamp`, a read by fetch terms | `close_boards`: closes only roles not listed for `STALE_DAYS` |
|
||||
| `swept` | `stamp`, a complete read that finished a sweep begun on an earlier pull | `close_boards`: applies the two-strikes rule |
|
||||
| `sweep` | `stamp`, the cursor of a read cut at `MAX_PAGES` | `_runner.read` on the next pull |
|
||||
| `count` | `stamp`, the roles the last read listed; reset by Recheck | `_runner.read`, the 10% guard |
|
||||
| `missing` | `close_boards` (`_close_swept`), the ids the last finished sweep missed | the next finished sweep |
|
||||
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `fetch_text`, the guard's escape after 21 days |
|
||||
| `facets` | `_runner.facets_for`, kept one day | `reach.site_facets`, the Companies line |
|
||||
| `size`, `sizechecked` | `ats.keep_size`: `set_board`, Recheck, `app companies size`, or the first read of an unsized board | `reach.sizes`, which decides whether the board is filtered |
|
||||
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`fetch_text`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `fetch_text.plan` through `detail_due` |
|
||||
|
||||
4. `close_boards`, the settle of the pull, closes what a complete read did not list, closes a board or search that failed for `STALE_DAYS`, releases the run, and raises one `pull.settled` notice in the Problems log when a target did not answer or was read incomplete, or a board closed more than 50 roles or more than half of its open ones.
|
||||
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score` (which writes each agent's cut), `fetch_text` (reads the role pages of the roles in the cut without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `judge`, `summarise`. Extract, judge and summarise are switches under Runs; fetch text is on by default.
|
||||
|
||||
## The three ways a source is read
|
||||
|
||||
**Whole board.** One request returns every role, with its text for most hosts. The read is complete whenever the answer parses, so the pull can close every role the answer no longer lists.
|
||||
|
||||
**Paged.** The list arrives in pages of a fixed size (`READS.page_size`), read one after another `PAGE_DELAY` (0.5 s) apart, at most `MAX_PAGES` (100) pages a pull. The read is complete when the number of distinct roles read equals the total the host states (`complete="stated"`), or, for a host that states none, when the read reaches an empty page (`complete="end"`); a Jobindex search is never complete (`complete="never"`). A board of more than 100 pages is swept: each pull stores what it read and the next continues from the saved cursor, and the read is complete when a sweep reaches the end. Roles closing between pulls shift an offset-paged host's later roles back past the cursor, so a role of a swept board closes only when two finished sweeps in a row missed it. A host with a `ceiling` (Workday 2,000) answers nothing past it: a read that reaches it is incomplete, is not swept and closes nothing, so a filter must bring the board under it.
|
||||
|
||||
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The fetch text stage reads those pages after scoring (`app/jobs/fetch_text.py`):
|
||||
|
||||
- It takes open roles without text in the cut of an enabled agent (the best by fit and by CV closeness, and the liked and saved, see `app/jobs/score.py`), best final score first.
|
||||
- It reads at most `fetch_text.per_board` (200) roles per board and `fetch_text.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
|
||||
- A read that gave no text is tried again after `fetch_text.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
|
||||
- It runs after every pull, after a saved agent, an affinity change or an override switched, not after a vote, and it can be switched off under Runs.
|
||||
|
||||
A role in no agent's cut is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles in the cut do. Four consequences follow from the code:
|
||||
|
||||
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow fetch text encode it again.
|
||||
- A role page may fill only `_runner.DETAIL_FIELDS`: description, location and places, teaser, remote, country, employment, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
|
||||
- Completeness, and so closing, is decided by the list alone. A board of 800 roles closes its roles from the first pull.
|
||||
- SmartRecruiters and Workable list their roles in pages and belong here too, since their list has no text.
|
||||
|
||||
## Overview
|
||||
|
||||
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles fetch text has read.
|
||||
|
||||
| Source | Boards | Open roles | How it is read | Roles per request | Description | Pay | Limit that matters most |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Greenhouse | 87 | 10,457 | whole board | all | in the list | figures | a board over 96 MiB loses its text and pay |
|
||||
| Ashby | 74 | 4,964 | whole board | all | in the list | figures | an unknown board name answers an empty board |
|
||||
| Workday | 17 | 2,041 | list, then each role | 20 | per role (1,280 read) | none | 2,000 roles per board; country arrives per role |
|
||||
| Lever | 26 (25 read) | 1,347 | whole board | all | in the list, the intro paragraph only | figures | text is the opening paragraph |
|
||||
| Oracle Recruiting Cloud | 5 | 937 | list, then each role | 200 | per role (603 read), teaser in the list | none | no pay field |
|
||||
| SuccessFactors | 14 | 749 | list, then each role | the site's own | per role (681 read) | none | default locale only |
|
||||
| Teamtailor | 38 | 611 | paged | 50 | in the list | none | no pay; nine companies without a found site |
|
||||
| Jobindex (search) | 12 saved (8 on) | 562 | paged | 20 | teaser | none | teaser only; at most 200 results per search; never complete |
|
||||
| Platsbanken (search) | 6 saved | 439 | paged | 100 | in the list | none | at most 1,000 results per search |
|
||||
| Phenom | 5 | 389 | list, then each role | 100 | per role (142 read), teaser in the list | none | locale spelling known from three sites |
|
||||
| SmartRecruiters | 11 | 343 | list, then each role | 100 | per role (1 read) | none | one country a request; remote is not filtered |
|
||||
| Personio | 3 | 320 | whole board | all | in the list | figures | no country field |
|
||||
| Eightfold | 3 | 185 | list, then each role | 10 | per role (158 read) | none | 10 per request, so 1,000 roles a pull |
|
||||
| Recruitee | 8 | 176 | whole board | all | in the list | figures | an empty board looks like no board to discovery |
|
||||
| Workable | 8 | 157 | list, then each role | 10 | per role (1 read) | none | answers 429 to bursts |
|
||||
| Jibe | 1 | 137 | paged | 100 | in the list | none | Booking Holdings' roles read with Booking.com's |
|
||||
| Deel | 1 | 96 | whole board | all | in the list | line only | pay has no period |
|
||||
| Gem | 2 | 45 | list, then each role | all | per role (44 read) | none | undocumented query, can break |
|
||||
| Jobylon | 2 | 33 | list, then each role | all | per role (33 read), teaser in the list | none | scraped from the careers widget |
|
||||
| HR Manager | 1 | 7 | paged | 100 | in the list | none | the portal's robots.txt disallows reading; the owner has not decided |
|
||||
| Rippling | 1 | 2 | list, then each role | all | per role | figures | text, date and pay cost one request per role |
|
||||
| Pinpoint | 1 | 2 | whole board | all | in the list | figures | no date |
|
||||
| HiBob | 1 | 2 | whole board | all | in the list | figures | pay never seen filled |
|
||||
| BambooHR | 1 | 2 | list, then each role | all | per role | none | paging of a large board unverified |
|
||||
| Adzuna (search) | 0 | 0 | paged | 50 | teaser | figures | never run live; needs a key |
|
||||
| freehire (search) | 0 | 0 | paged | 100 | in the list | figures | 10,000 results per search; the query is noisy |
|
||||
| Jobnet (search) | 0 | 0 | paged | 100 | in the list | none | 10,000 results per search; needs the header `x-csrf: 1` |
|
||||
| Akademikernes Jobbank (search) | 0 | 0 | paged | 100 | per role | none | not sorted by date, and no total, so never complete |
|
||||
|
||||
"Figures" means yearly `salary_min`, `salary_max` and `currency`, plus a pay line; "line only" means a pay line (`salary_text`) and no figures. Figures are yearly: a monthly figure is multiplied by 12 where the host states the period, and an hourly, daily or weekly one is kept as a line only.
|
||||
|
||||
## Rules every source shares
|
||||
|
||||
| Limit | Value | What it means |
|
||||
|---|---|---|
|
||||
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); fetch text is on by default. |
|
||||
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. Enrich reads one batch per board on the same pool. One pull runs at a time. |
|
||||
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
|
||||
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
|
||||
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request | The same for boards, role pages and searches. A slow board fails for this pull and is asked again next pull. |
|
||||
| Pages per board | 100 a pull | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
|
||||
| Wait | 0.5 s before every page after the first, for boards and searches alike, and before every role page | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
|
||||
| Per-role reads | `fetch_text`, after scoring: 200 per board and 1,000 per run, the roles in the agents' cut | See "List, then each role". Roles in no cut are not read. |
|
||||
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing, and so does a read that asked nothing. A role listed again reopens. A board switched off closes its roles at once. A filtered read keeps roles with a vote or an application, and remote roles when the host could not be asked for remote; a read by fetch terms closes only roles not listed for 21 days. |
|
||||
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. A Jobindex search is never complete, so its roles close by the 21 days only. A search failing for 21 days closes its roles. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, a row that would not store, and a read that lists under 10% of what the same read listed last time (`SUSPECT_SHARE`) once that was at least `SUSPECT_MIN`, 10. A search that fails is recorded the same way and its row says so. |
|
||||
| A board that fails for 21 days | all its open roles close at the next pull, and a suspect answer that has stood that long is accepted | The next answer reopens what it lists. `STALE_DAYS` is the one 21. |
|
||||
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next fetch text run | Nothing is closed on a list cut short by a 429. |
|
||||
|
||||
How far a board is read: a pull asks each board of 300 roles or more for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All locations is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of once per country (33 codes today), unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application. A board's size is set once, when it is added, on Recheck or by `app companies size`, and a pull does not rewrite it; a board with no size yet is sized first in its batch and then read as its size says, and a board whose reader cannot narrow the reach is read whole and sized by that read. A host that cannot filter a dimension ignores it: SmartRecruiters, Phenom, SuccessFactors and Eightfold have no remote filter, so their reach asks for countries only.
|
||||
|
||||
Fetch terms narrow a big board's read by words. Each agent has a list of them (`fetch_terms`, set in the agent editor); the terms of a pull are the union over the agents that are on, and none when one of those agents has none. They go to every board of 300 roles or more, on the narrow and the wide reach alike; a board under 300 roles, one with its own reach and every board of a company set to All roles take none. A host that takes a keyword gets one series per term, unioned by id, each combined with the country series; the survey shows that as an AND on SmartRecruiters, Eightfold, SuccessFactors, Phenom and Workday. When terms times the country series exceed 8, a wide read drops the place part of the reach, whole, and keeps the terms, while a narrow read keeps the places and drops the terms; more than 8 terms read the board unfiltered. The survey found the keyword matching the ad text and not only the title where it could tell, and could not tell on SuccessFactors, Workday, Workable and Jibe (`docs/filter-survey.md`), so a term like `engineer` cuts less than it looks. A read with terms is never the whole board: it records no size, closes no role it did not list, and the 10% guard holds it only to a read with the same terms and places. The search providers take no terms: each saved search has its own query.
|
||||
|
||||
What no source does:
|
||||
|
||||
- Filter on the host's side for a host other than SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom and Eightfold. Greenhouse, Ashby, Recruitee and the rest are read whole. The search providers filter by their own query, place and country.
|
||||
- Filter by category or level at the source. Workday, Oracle, Jibe and Eightfold state category ids, but nothing maps an agent's terms to them yet (`docs/filter-survey.md`).
|
||||
- Read more than one board per company. DFDS is read from its Danish Teamtailor board only.
|
||||
- Use a key taken from a site's script. Maersk, New Work and Welcome to the Jungle are on the Watchlist for that reason.
|
||||
- Read a page that needs a browser to run its script.
|
||||
- Log in anywhere.
|
||||
No reader logs in, runs a page's script or uses a key taken from a site's script. A company that needs one is on the Watchlist tab (`app companies by-hand` lists them), and the readers planned for the rest are in `docs/plan-hosts-wave-2.md`.
|
||||
|
||||
## Hosts read as a whole board
|
||||
|
||||
@@ -130,25 +15,21 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board token, e.g. `greenhouse:trustpilot` |
|
||||
| Companies | 87 boards: Trustpilot, Cognite, Wolt |
|
||||
| List | `boards-api.greenhouse.io/v1/boards/<token>/jobs` with `content=true` and `pay_transparency=true`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1; 2 when the first answer is over 96 MiB |
|
||||
| Description | in the list, full text (`content`) |
|
||||
| Date | `first_published`, else `updated_at`, which the host moves on an edit; trusted as the posting date |
|
||||
| Place and country | every `offices` entry; `location.name` when the role has no office or the name says more, with placeholder names such as "Hybrid" dropped. The host states no country; it is resolved from the places |
|
||||
| Remote | none from the host; prep reads remote, hybrid or onsite from the place words |
|
||||
| Pay | figures from `pay_input_ranges` when there is exactly one range; a range whose title says hour or day is a line only; any other range is taken as yearly. Several ranges give one line per range and no figures |
|
||||
| Found by discovery | yes, from a link to `boards.greenhouse.io`, `job-boards.greenhouse.io` or the embed script; a guessed name is probed first of all hosts |
|
||||
| Limitations | Over 96 MiB the board is read again without `content=true`: titles land, text and pay of the whole board are lost. An `eu.greenhouse.io` link is read from the main API. The pay flag must be sent, or no range comes back. |
|
||||
| Limitations | Over 96 MiB the board is read again without `content=true`: titles land, text and pay of the whole board are lost. It is the only host with a `fallback_url`. An `eu.greenhouse.io` link is read from the main API. The pay flag must be sent, or no range comes back. |
|
||||
|
||||
### Lever
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the site name, or `eu/<site>` on Lever's EU instance, e.g. `lever:spotify`, `lever:eu/tomtom` |
|
||||
| Companies | 26 boards, one switched off: Spotify, TomTom, Pipedrive |
|
||||
| List | `api.lever.co/v0/postings/<site>?mode=json` (`api.eu.lever.co` for `eu/`), the whole board in one answer; no total, no paging parameter sent |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, `description` (else `descriptionPlain`); the module reads no `lists` or `additional` sections, so it is the opening text |
|
||||
| Date | `createdAt`; trusted |
|
||||
| Place and country | `categories.location` and every `allLocations` entry; the host's `country` is handed on when the role has one place, otherwise the country is resolved from the places |
|
||||
@@ -163,9 +44,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board name, which may carry a dot or be a uuid, e.g. `ashby:pleo`, `ashby:jimdo.com` |
|
||||
| Companies | 74 boards: Pleo, Too Good To Go, Lovable |
|
||||
| List | `api.ashbyhq.com/posting-api/job-board/<name>?includeCompensation=true`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text (`descriptionHtml`) |
|
||||
| Date | `publishedAt`; trusted |
|
||||
| Place and country | the primary `location` and every `secondaryLocations` entry; country resolved from the places |
|
||||
@@ -180,9 +59,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `recruitee:bunq` |
|
||||
| Companies | 8 boards: bunq, idealo, Climeworks |
|
||||
| List | `https://<sub>.recruitee.com/api/offers/`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, `description` only (`requirements` is not read) |
|
||||
| Date | `published_at`, else `created_at`; trusted |
|
||||
| Place and country | every `locations` entry as "city, state, country"; `location` or `city` when there are none; a remote offer with no place gets the place "Remote" |
|
||||
@@ -197,9 +74,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `personio:penneo` |
|
||||
| Companies | 3 boards: Penneo, tado, 1KOMMA5° |
|
||||
| List | the XML feed `https://<sub>.jobs.personio.de/xml`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, every `jobDescription` value joined |
|
||||
| Date | `createdAt`; trusted |
|
||||
| Place and country | `office` and every `additionalOffices` office; the feed has no country field, so it is resolved from the places |
|
||||
@@ -214,9 +89,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `pinpoint:xeneta` |
|
||||
| Companies | 1 board: Xeneta |
|
||||
| List | `https://<sub>.pinpointhq.com/postings.json`, the whole board in one answer; no total, no paging keys |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, skills and benefits sections |
|
||||
| Date | none: the list states none and the role page is not read for it; the role counts as posted when first seen |
|
||||
| Place and country | `location.name` (else `city`) plus `province`, one place; country resolved from that line |
|
||||
@@ -231,9 +104,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board slug on `jobs.deel.com`, with case, e.g. `deel:klarna` |
|
||||
| Companies | 1 board: Klarna |
|
||||
| List | the HTML page `https://jobs.deel.com/<slug>`, from which the Next.js server payload is read; the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text (`richtextDescription`) |
|
||||
| Date | the publication's `createdAt`, else the posting's; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every `jobLocations` name as a place, repeated names dropped, each carrying the role's remote flag; country resolved from the places |
|
||||
@@ -248,9 +119,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain of `careers.hibob.com`, e.g. `hibob:dixa` |
|
||||
| Companies | 1 board: Dixa |
|
||||
| List | `https://<sub>.careers.hibob.com/api/job-ad` with the header `companyIdentifier: <sub>`, the whole board in one answer; no total. Declared as paged with `complete="end"` because `page` may send headers; it reads one page |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, requirements, benefits |
|
||||
| Date | `publishedAt`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `site` plus `country`. A company that hires at two sites posts one ad per site with its own id (`places="copies"`); `app/dupes.py` folds them into one role |
|
||||
@@ -267,25 +136,21 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain of `teamtailor.com`, or the whole host of a career site on the company's domain, e.g. `teamtailor:lunar`, `teamtailor:careers.trackunit.com` |
|
||||
| Companies | 38 boards: Lunar, Trackunit, DFDS |
|
||||
| List | the RSS feed `https://<site>/jobs.rss`, 50 per request by offset; no total, so the read goes on to an empty page |
|
||||
| Requests per pull | roles / 50, rounded up, plus 1 for the empty page; the largest board attached has 75 roles, so 3 |
|
||||
| Description | in the list, full text |
|
||||
| Date | `pubDate`; trusted |
|
||||
| Place and country | every `tt:location` as a place "city, country", each remote when `remoteStatus` says so; country resolved from the places |
|
||||
| Remote | `remoteStatus`, handed on as the host's word: fully is remote, hybrid, onsite; `none` and `temporary` stay unknown |
|
||||
| Pay | none: the feed states no pay |
|
||||
| Found by discovery | yes, from a `<sub>.teamtailor.com` link; a page that loads Teamtailor's CDN names the page's own host as the board; a guessed name is probed |
|
||||
| Limitations | A board of more than 100 pages is swept across pulls and completes on the empty page. A page with nothing new ends the read incomplete. Nine seed companies have no board because their careers link is a marketing site, not the career host. |
|
||||
| Limitations | A board of more than 100 pages is swept across pulls and completes on the empty page. A page with nothing new ends the read incomplete. Nine seed companies have no board because their careers link is a marketing site, not the career host. A company has one board, so DFDS, which has a Teamtailor site per country, is read from its Danish one only. |
|
||||
|
||||
### Jibe
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the career site's host, e.g. `jibe:jobs.booking.com` |
|
||||
| Companies | 1 board: Booking.com |
|
||||
| List | `https://<host>/api/jobs?limit=100&page=<n>`, 100 per request, the most the host serves; states `totalCount` |
|
||||
| Requests per pull | roles / 100, rounded up; 2 for Booking.com |
|
||||
| Description | in the list, full text: description and qualifications |
|
||||
| Date | `posted_date`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `full_location` split at ";" into places; the host's `country_code` is handed on when the role has one place |
|
||||
@@ -300,9 +165,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the portal's customer name, e.g. `hrmanager:teknologisk` |
|
||||
| Companies | 1 board: Danish Technological Institute (`hrmanager:teknologisk`, 7 open roles). Seaborg, the one seed company on the portal, is unattached and on the Watchlist as `robots` |
|
||||
| List | the RSS feed `api.hr-manager.net/JobPortal.svc/<customer>/PositionList/rss/?incads=true`, 100 per request by `skip`; no total, so the read goes on to an empty page |
|
||||
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page |
|
||||
| Description | in the list, the whole advertisement |
|
||||
| Date | `pubDate`; not trusted |
|
||||
| Place and country | none: the feed names no place |
|
||||
@@ -313,37 +176,33 @@ What no source does:
|
||||
|
||||
## Hosts read as list, then each role
|
||||
|
||||
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page (`documentation/ADR Plugins.md`, 2.2). "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
|
||||
### SmartRecruiters
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the company identifier, e.g. `smartrecruiters:BoschGroup` |
|
||||
| Companies | 11 boards: Bosch (about 4,800 roles on the site, 13 open in reach), Delivery Hero, Wise |
|
||||
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
|
||||
| Requests per pull | one series per country of the reach (two for Bosch: DK and SE), each roles / 100, rounded up; a board read whole is roles / 100 |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only: 1 of 343 open roles has text |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only |
|
||||
| Date | `releasedDate`; trusted |
|
||||
| Place and country | `fullLocation`, else city and country, with empty parts removed; the host's `country` is handed on |
|
||||
| Remote | `location.remote`, true or false, as the host's word |
|
||||
| Seniority, employment | `experienceLevel` and `typeOfEmployment`, the host's ids |
|
||||
| Pay | none: the list states no pay |
|
||||
| Found by discovery | yes, from a `jobs.`, `careers.` or `api.smartrecruiters.com` link; a guessed name is probed |
|
||||
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
|
||||
| Limitations | 100 pages a pull; a larger board is swept across pulls. Whether the identifier is case-sensitive is unsettled: the module docstring says it is, `docs/decisions.md` says it is not (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
|
||||
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so it cannot be asked for. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text |
|
||||
| Limitations | 100 pages a pull; a larger board is swept across pulls. The identifier is stored with the case it was attached with; whether the host is case-sensitive is unsettled (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
|
||||
|
||||
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until fetch text has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
The reader's Fetch button reads one posting page on demand (see "The search providers"); whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
|
||||
### Workable
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the account name, e.g. `workable:huggingface` |
|
||||
| Companies | 8 boards: Hugging Face, Kahoot!, Starling Bank |
|
||||
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
|
||||
| Requests per pull | roles / 10, rounded up, 0.5 s apart; the largest board has 50 roles, so 5 |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only: 1 of 157 open roles has text |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only |
|
||||
| Date | `published`; trusted |
|
||||
| Place and country | every `locations` entry as "city, region, country", each remote when the role is; `location` when there are none |
|
||||
| Remote | `workplace` (remote, hybrid, on_site) as the host's word, else the `remote` flag |
|
||||
@@ -357,16 +216,14 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the career site's host, or host plus a brand path, e.g. `successfactors:careers.vestas.com`, `successfactors:jobs.teradyne.com/MiR` |
|
||||
| Companies | 14 boards: Novo Nordisk, Vestas, Carlsberg |
|
||||
| List | the site's HTML search page `/search/?startrow=<n>`, parsed in its two known layouts; page size is the site's own (10, 15, 25 or 100) and cannot be set; a total where the pagination label states one |
|
||||
| Requests per pull | roles / the site's page size, rounded up |
|
||||
| Description | per role, the `itemprop="description"` block of the role page |
|
||||
| Date | the list's date column where it parses; the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's location, with a two-letter country code turned into the name; the role page's microdata replaces it |
|
||||
| Remote | none from the host |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from a link to a role, `/go/` page or search on the site's own host; a link to the SAP back end names the host only; not probed. A site on the company's own domain whose pages link only the back end is set by hand |
|
||||
| Filter at the source | `optionsFacetsDD_country=DK`, one country a request; no remote facet. Vestas DK: 64 of 668 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as the form's `q=engineer`, one a request, AND with the country (Vestas DK: 40 of 64) |
|
||||
| Filter at the source | `optionsFacetsDD_country=DK`, one country a request; no remote facet. Fetch terms as the form's `q=engineer`, one a request, AND with the country; whether it matches title or text is not determined (`docs/filter-survey.md`) |
|
||||
| Limitations | Only the default locale's roles are listed: a role that exists only in another locale (Carlsberg's French brand, 37 of 105) is not read. Only `/search/` and `/job/` are read, never `/services/` or the `successfactors.eu` back end, so a company with only the back end (Bang & Olufsen) cannot be read. 100 pages a pull; a larger site is swept across pulls. A third layout or a bot wall that answers 200 fails the board with "not a SuccessFactors search page". Read from this machine only; not yet from the deployed server. |
|
||||
|
||||
### Workday
|
||||
@@ -374,9 +231,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | `<tenant>.<wdN>/<Site>`, site with case, e.g. `workday:lego.wd103/LEGO_External` |
|
||||
| Companies | 17 boards: LEGO Group, Zendesk, Equinor, and since 2026-09-30 Criteo, Darktrace, Novonesis and NVIDIA |
|
||||
| List | a POST to `/wday/cxs/<tenant>/<Site>/jobs`, 20 per request by offset (21 or more answers 400); the first page states the total |
|
||||
| Requests per pull | roles / 20, rounded up; 50 for a board of 1,000 roles |
|
||||
| Description | per role, `jobDescription` from the role's JSON |
|
||||
| Date | the list's "Posted N Days Ago" as a day, possibly one day off; "Posted 30+ Days Ago" and other languages give none; the role's `startDate` replaces it; trusted |
|
||||
| Place and country | the list has `locationsText` ("Copenhagen" or "2 Locations"); the per-role read adds every location and the country, so the country arrives with the role page |
|
||||
@@ -384,7 +239,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| Employment | `timeType`, the host's own word |
|
||||
| Pay | none: no pay field; pay appears only inside the text |
|
||||
| Found by discovery | yes, from a `myworkdayjobs.com` link with a site; not probed |
|
||||
| Filter at the source | the tenant's own facets in `appliedFacets`, ids read by `facets` once a day: the facet whose values are countries, and any value that reads as remote. Two facets intersect, so remote is a series of its own. LEGO DK: 97 of 455 (2026-09-30). A tenant with no country facet is read whole. Which countries is the agents' reach, see above. Fetch terms as `searchText` in the body, one a request, AND with the facets (LEGO: 106 for `engineer`) |
|
||||
| Filter at the source | the tenant's own facets in `appliedFacets`, ids read by `facets` once a day: the facet whose values are countries, and any value that reads as remote. Two facets intersect, so remote is a series of its own. A tenant with no country facet is read whole. Fetch terms as `searchText` in the body, one a request, AND with the facets; whether it matches title or text is not determined (`docs/filter-survey.md`) |
|
||||
| Limitations | 2,000 roles per board (100 pages of 20), the `ceiling`. A tenant has several sites and each is its own board. Every tenant is behind Cloudflare and was read from this machine only. The slug's case is kept (`KEEP_CASE`). |
|
||||
|
||||
### Oracle Recruiting Cloud
|
||||
@@ -392,16 +247,14 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | `<pod host>/<siteNumber>`, e.g. `oracle:fa-evmr-saasfaprod1.fa.ocs.oraclecloud.com/CX_1` |
|
||||
| Companies | 5 boards: Danske Bank, Nokia, Milestone Systems, Sinch, Pioneer Centre for AI |
|
||||
| List | the REST resource `recruitingCEJobRequisitions`, 200 per request by offset, with secondary places expanded; states `TotalJobsCount` |
|
||||
| Requests per pull | roles / 200, rounded up; 3 for Nokia's roughly 600 |
|
||||
| Description | per role: external description, responsibilities and qualifications joined. The list's one-paragraph `ShortDescriptionStr` is stored as the teaser (`READS.teaser=True`), which is shown and embedded until the description arrives |
|
||||
| Date | `PostedDate`; the role's `ExternalPostedStartDate` replaces it when earlier; trusted |
|
||||
| Place and country | the primary place plus every secondary one, each a place of the role |
|
||||
| Remote | `WorkplaceTypeCode` as the host's word (e.g. `ORA_HYBRID`); prep maps it |
|
||||
| Pay | none: the resource states no pay |
|
||||
| Found by discovery | yes, from a candidate site or role link on the pod, or a REST call naming `siteNumber`; not probed |
|
||||
| Filter at the source | `selectedLocationsFacet` with the pod's geography ids, and `selectedWorkplaceTypesFacet=ORA_REMOTE` as a series of its own. `locationsFacet` lists the ten biggest countries only, so a country missing from a full list is read unfiltered: fa-evmr (Nokia) has neither DK nor SE listed and is read whole. Which countries is the agents' reach, see above. Fetch terms as `keyword=` in the finder, one a request; it matches the text (Nokia: 468 of 554 for `engineer`) |
|
||||
| Filter at the source | `selectedLocationsFacet` with the pod's geography ids, and `selectedWorkplaceTypesFacet=ORA_REMOTE` as a series of its own. `locationsFacet` lists the ten biggest countries only, so a country missing from a full list is read unfiltered. Fetch terms as `keyword=` in the finder, one a request; it matches the text |
|
||||
| Limitations | The host is the tenant's Fusion pod, not the company's domain (`jobs.nokia.com` redirects to it). Links are built for locale `en`. The department fields were empty on every role of the first four boards; Pioneer's were not checked. |
|
||||
|
||||
### Phenom
|
||||
@@ -409,16 +262,14 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the career site's host plus its locale root, e.g. `phenom:www.careers.philips.com/global/en` |
|
||||
| Companies | 5 boards: King, Just Eat Takeaway.com, Philips, BCG, Verisure |
|
||||
| List | a POST to `https://<host>/widgets`, 100 per request by offset; states `totalHits` |
|
||||
| Requests per pull | roles / 100, rounded up; 9 for Philips' roughly 880 |
|
||||
| Description | per role, the JSON-LD `JobPosting` description of the role page. The list's `descriptionTeaser` is stored as the teaser (`READS.teaser=True`) |
|
||||
| Date | the list's `dateCreated`, else `postedDate`, which the site moves on a refresh; the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | every place of `multi_location`, each a place of the role |
|
||||
| Remote | per role only, from the page's `checkRemote` |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from a `/<country>/<lang>/search-results` link; a link to Phenom itself names the host only; not probed |
|
||||
| Filter at the source | `selected_fields.country`, a list of English names in one request; no remote. Philips NL and DE: 111 of 873 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `keywords` in the body, one a request, AND with the countries; it matches the text, not the title |
|
||||
| Filter at the source | `selected_fields.country`, a list of English names in one request; no remote. Fetch terms as `keywords` in the body, one a request, AND with the countries; it matches the text, not the title |
|
||||
| Limitations | The locale is built as `<lang>_<country>`, the spelling of the three sites first read; another site may spell it differently. 100 pages a pull; a larger board is swept across pulls. A role page without a description fails the read and is retried on a later run. |
|
||||
|
||||
### Eightfold
|
||||
@@ -426,26 +277,22 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | `<tenant host>/<company domain>`, e.g. `eightfold:jobs.ericsson.com/ericsson.com` |
|
||||
| Companies | 3 boards: Ericsson, Microsoft, Infineon |
|
||||
| List | `https://<host>/api/pcsx/search?domain=<domain>&sort_by=timestamp`, newest first, 10 per request whatever `num` asks; states `count` |
|
||||
| Requests per pull | roles / 10, rounded up, at most 100 |
|
||||
| Description | per role, `jobDescription` from `position_details` |
|
||||
| Date | `creationTs`, else `postedTs`, which the host refreshes in bulk; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location, repeated names dropped, each a place of the role |
|
||||
| Remote | `workLocationOption`: onsite, hybrid, remote |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from an `eightfold.ai` link that carries `domain=`; a bare `eightfold.ai` link names the host only; not probed |
|
||||
| Filter at the source | `location=<country name>`, one country a request; no remote. Ericsson SE: 154 of 618 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `query=`, one a request, AND with the location; it matches title, text and skills (Ericsson SE: 102 of 154 for `software`) |
|
||||
| Limitations | With 10 per request and 100 pages a pull, a board over 1,000 roles is swept across pulls; what the host answers past offset 1,000 was not checked. The country filter brings big boards under the cap: Microsoft (size 2,345) has 29 open roles in reach, and Infineon (999 stored) has none in DK and SE. Verified on three tenants only. |
|
||||
| Filter at the source | `location=<country name>`, one country a request; no remote. Fetch terms as `query=`, one a request, AND with the location; it matches title, text and skills |
|
||||
| Limitations | With 10 per request and 100 pages a pull, a board over 1,000 roles is swept across pulls; what the host answers past offset 1,000 was not checked. The country filter brings big boards under the cap. Verified on three tenants only. |
|
||||
|
||||
### Gem
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board name on `jobs.gem.com`, with case, e.g. `gem:productboard` |
|
||||
| Companies | 2 boards: sennder, Productboard |
|
||||
| List | a GraphQL POST to `jobs.gem.com/api/public/graphql`, the whole board in one answer; no total. Declared as paged with `complete="end"`; it reads one page |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, `descriptionHtml`, one GraphQL POST per role |
|
||||
| Date | per role only, `firstPublishedTsSec`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location as a place (the city, else the location name); `isoCountry` is fetched and not used |
|
||||
@@ -460,9 +307,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the numeric company id, e.g. `jobylon:3143` |
|
||||
| Companies | 2 boards: Einride, Vipps MobilePay |
|
||||
| List | the careers widget's embed page `cdn.jobylon.com/jobs/companies/<id>/embed/v2/`, whose script literal holds the whole board; no total. The list carries a `summary`, stored as the teaser |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, the JSON-LD description of the role page on `emp.jobylon.com` |
|
||||
| Date | the list's day ("Sept. 25, 2026"); the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's place names, which can be cut short ("Houston +3 more"); the per-role read fills every place |
|
||||
@@ -477,9 +322,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board slug on `ats.rippling.com`, with case, e.g. `rippling:prisma-careers` |
|
||||
| Companies | 1 board: Prisma |
|
||||
| List | `api.rippling.com/platform/api/ats/v1/board/<slug>/jobs`, the whole board in one array, one entry per role and place, merged by role; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, `description.role`, one role record per role |
|
||||
| Date | per role only, `createdOn`; trusted |
|
||||
| Place and country | the list's place labels as places; the per-role read replaces them with every work location |
|
||||
@@ -494,9 +337,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `bamboohr:configit` |
|
||||
| Companies | 1 board: Configit |
|
||||
| List | `https://<sub>.bamboohr.com/careers/list`, the whole board in one answer; a list shorter than its `totalCount` fails the board rather than being read as whole |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, `/careers/<id>/detail` |
|
||||
| Date | per role only, `datePosted`; trusted |
|
||||
| Place and country | the list's city and state as one place; the country arrives with the role page |
|
||||
@@ -508,7 +349,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
|
||||
## The search providers
|
||||
|
||||
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
|
||||
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads 5 pages by default and at most 10 (`DEFAULT_PAGES` and `MAX_PAGES` in `plugins/search.py`). A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; Jobbank states the text only on a role's own page, which fetch text reads. The reader's Fetch button shows for a role with a link, no `article_text` and a description under 500 characters. It reads the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not (`app/jobs/posting.py`). For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text; for a Jobbank role it calls the reader's `detail`. A search that fails is recorded as `search:failing:<target>` and skipped for the pull (`plugins/search.py`). No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
|
||||
|
||||
### Jobindex
|
||||
|
||||
@@ -517,7 +358,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| Saved search | `q` in Jobindex's syntax (words are OR, `"a b"` a phrase, `+word` required) and optionally `area`, Jobindex's region id (1 Region Hovedstaden, 8 Aarhus, ...) |
|
||||
| List | the RSS twin of the search page, `jobindex.dk/jobsoegning.rss`, newest first, 20 per page; no total. `READS.complete="never"`: a search is never complete, so its roles close by the 21 days only |
|
||||
| Pages read | 5 by default (100 results), 10 at most (200) |
|
||||
| Text | an excerpt of a couple of hundred characters from the feed, stored as the teaser; 561 of 562 open roles have some text. The full text only through "Fetch the posting", on demand, since every fetch of Jobindex's link counts as a click in its statistics |
|
||||
| Text | an excerpt of a couple of hundred characters from the feed, stored as the teaser. The full text only through "Fetch the posting", on demand, since every fetch of Jobindex's link counts as a click in its statistics |
|
||||
| Date | `pubDate`; trusted |
|
||||
| Place | the item's area, else the search's area; country always Denmark |
|
||||
| Remote | the reader states none; prep reads place and title words |
|
||||
@@ -575,7 +416,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| | |
|
||||
|---|---|
|
||||
| Saved search | `q` and optionally `region`, one of Jobnet's five region ids (`REGIONS` in the module); empty for all of Denmark |
|
||||
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login; this supersedes the earlier note that its search answers 401 outside its own session |
|
||||
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login |
|
||||
| Pages read | 5 by default (500 results), 10 at most (1,000); page 101 at 100 a page answers empty |
|
||||
| Text | the whole text, as HTML in the answer |
|
||||
| Date | `publicationDate`; trusted |
|
||||
@@ -601,20 +442,3 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| Limitations | Academic and research roles, plus some corporate ads. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Cloudflare may answer a challenge instead of the page, which is an error for that role and not an empty text. |
|
||||
|
||||
Jooble was left out: it gives a key only on a manual application and answered a Cloudflare challenge. Jobdanmark was left out too: about a quarter of its newest ads are on Jobnet, freehire carries the rest as source `jobdanmark`, and its list has no text. LinkedIn's guest pages are out because its robots.txt disallows `/` and its terms forbid automated access; LinkedIn enters only as alert mail. Indeed has no public search API.
|
||||
|
||||
## Sources that cannot be read, and why
|
||||
|
||||
The table is the seed's: 76 of the 362 companies in `seed/companies.json` have no board there and carry a careers link the survey opened on 2026-09-29. The live database has moved on, and its Watchlist tab and `app companies by-hand` are the current list. Since the seed was written, Infineon and Microsoft (Eightfold) and Novonesis, Darktrace and Criteo (Workday) have been attached, so they are struck from the company lists below; the seed still names them.
|
||||
|
||||
| Reason | Seed | Live 2026-09-30 | What it means | Companies |
|
||||
|---|---|---|---|---|
|
||||
| `unread` | 35 | 35 | readable by a program; no module reads it yet | Zalando, Ørsted, Siemens, Google DeepMind, Amazon, Apple, ASML, Netcompany, Tryg, DNB |
|
||||
| `robots` | 10 | 7 | the site's robots.txt disallows the page a board would be read from | Bang & Olufsen (SuccessFactors), Pandora, Terma, Zenegy, Umbraco, Snyk, Seaborg (HR Manager) |
|
||||
| `group` | 8 | 8 | the roles are inside another company's board, with no field that separates them | Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio, Native Instruments, Visma |
|
||||
| `blocked` | 7 | 8 | the site answers a program with a challenge | Tradeshift, Personio, Adjust, Revolut, Silverfin, HashiCorp, Meta |
|
||||
| `gone` | 4 | 5 | no careers page | wefox, WeTransfer, Featurespace, Unbabel |
|
||||
| `key` | 3 | 3 | the only listing answers with a key the page's script carries | Maersk, New Work SE, Yousign (Welcome to the Jungle) |
|
||||
| `added` | – | 14 | added by hand or by an import and not looked at yet | Danish Centre for AI Innovation, 2021.AI, McKinsey, AMD, Ingka Group and nine more |
|
||||
| no reason | 9 | 9 | on Teamtailor, but the careers link is a marketing site and the career host is not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial |
|
||||
|
||||
The live counts are companies with no `ats` board subscription; the company lists are the seed's, less the five attached companies. What is planned for the `unread` companies and for the gaps in hosts already read (SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, Avature for Siemens) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
|
||||
@@ -268,7 +268,7 @@ def test_cv_match_orders_the_unjudged_by_closeness_then_score(store: Store) -> N
|
||||
assert [r.uid for r in store.roles(agent=owner.id, by_cv=True)] == [
|
||||
judged, close, far, none_b, none_a]
|
||||
assert [r.uid for r in store.roles_page(agent=owner.id)] == [
|
||||
far, none_b, none_a, close, judged], "Agent match ignores it"
|
||||
far, none_b, none_a, close, judged], "What I'm looking for ignores it"
|
||||
found = store.role(close, owner.id)
|
||||
assert found is not None and found.cv_fit == 0.9
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Where a remote role may be worked from: the `remote:` tokens, a remote
|
||||
role's area in prep, the chips and gates that read it, the model's answer and
|
||||
migration 27. The place texts are the live boards' (docs/plan-remote.md)."""
|
||||
migration 27. The place texts are the live boards'."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
Reference in new issue
Block a user