Test migration 32, the cut, saved filters and the override rescore, and bring the READMEs and config comments to fetch text and the cut
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
617356e60c
commit
8b6c6d5ac3
9 files changed
+260
-37
No files matched your search
@@ -9,7 +9,7 @@ Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeli
|
||||
```bash
|
||||
uv sync --all-extras
|
||||
uv run app companies seed # 362 European employers from seed/companies.json, 286 with a board; a re-run adds new rows and boards
|
||||
uv run app job pull # read every board and search, map and fold, embed and score, read the best roles' pages, extract, rescore, summarise
|
||||
uv run app job pull # read every board and search, map and fold, embed and score, fetch the text of the roles in the cut, extract, rescore, judge, summarise
|
||||
uv run app serve # http://127.0.0.1:8790
|
||||
```
|
||||
|
||||
@@ -42,7 +42,7 @@ uv run app prep unmapped # the host words no table
|
||||
|
||||
The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read (its own pages, a host with no module, a site whose robots.txt disallows reading) is still a row, with its careers page and the reason, shown on the Watchlist tab for checking by hand and under Companies as "not readable yet", so the gap is visible and one module at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles, and asks first when there are open roles to lose; a board is read once before it is kept, and one that lists no roles yet is kept too. Setting a board queues a pull right after, so its roles are read by the one already running or the very next one, rather than waiting for the schedule. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
|
||||
|
||||
`app stages` shows or switches extract, summarise and enrich, `app seniority` shows per host how often the stated level equals the model's, `app dupes` recomputes which open roles are copies of one posting, and `app prep` maps the host words and folds the copies without a pull.
|
||||
`app stages` shows or switches extract, summarise, judge and fetch text, `app seniority` shows per host how often the stated level equals the model's, `app dupes` recomputes which open roles are copies of one posting, and `app prep` maps the host words and folds the copies without a pull.
|
||||
|
||||
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. A freehire query is free text over about 50 hosts whose words are OR over title, company, text and place, so it is noisy: `--country` takes comma-separated ISO codes (`dk,se`) and `--remote` a work mode. A Jobnet query is a loose match without phrases and `--region` one of its five regions (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`). An Akademikernes Jobbank query is OR words, `--region` its `amt` id (2 is Storkøbenhavn) and `--remote` `helt` or `delvist`. A freehire role whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
|
||||
|
||||
@@ -60,13 +60,13 @@ The cog opens `/settings`: a rail of pages on the left, the page beside it, stic
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/prefer rules as rows, fetch terms, weights and limits); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
|
||||
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise and enrich, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
|
||||
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/exclude rules as rows, fetch terms, weights and limits); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
|
||||
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise, judge and fetch text, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
|
||||
| **Account** | the password, sessions and passkeys. Open on loopback, locked on anything else |
|
||||
|
||||
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its `taste` weight scores a role by how close it is to the roles the agent liked minus the ones it passed.
|
||||
|
||||
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role that `enrich` has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role; the switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, bounded to the top roles of each agent by that agent's own `limits` so the cost stays flat.
|
||||
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role whose text fetch text has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role; the switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, bounded to the top of each agent's feed (its `summarise_top`) and the roles liked or saved, so the cost stays flat.
|
||||
|
||||
## Docs
|
||||
|
||||
|
||||
+1
-1
@@ -272,7 +272,7 @@ def build_batch(config: Config, store: Store, limit: int = 5,
|
||||
def _mark(value: Any) -> int | None:
|
||||
if isinstance(value, bool) or not isinstance(value, int | float):
|
||||
return None
|
||||
return max(0, min(100, int(round(value))))
|
||||
return max(0, min(100, round(value)))
|
||||
|
||||
|
||||
def _lines(value: Any) -> list[str]:
|
||||
|
||||
+7
-6
@@ -770,7 +770,8 @@ Status = Literal["saved", "applied", "screen", "technical", "onsite", "offer",
|
||||
"rejected", "withdrawn"]
|
||||
STATUSES: tuple[str, ...] = get_args(Status)
|
||||
|
||||
#: -1 mute, 0 neutral, 1 interested, 2 favourite (skips the `any` gates).
|
||||
#: -1 mute, 0 neutral, 1 interested, 2 favourite. Affinity raises a role's
|
||||
#: score and its levels for the cut; it skips no part of the fetch gate.
|
||||
AFFINITY = range(-1, 3)
|
||||
|
||||
#: Why a company is on the list checked by hand. `unread` is a host the app
|
||||
@@ -1038,7 +1039,7 @@ def capped_career(career: int, broken: int = 0) -> int:
|
||||
|
||||
|
||||
def cv_score(skills: int, experience: int, career: int,
|
||||
weights: "Weights | None" = None, broken: int = 0) -> float | None:
|
||||
weights: Weights | None = None, broken: int = 0) -> float | None:
|
||||
"""The one number of a CV match, 0 to 100: the three numbers of the
|
||||
judge averaged by the agent's `cv_skills`, `cv_experience` and
|
||||
`cv_career` weights (30, 25 and 30 by default), career as
|
||||
@@ -1562,10 +1563,10 @@ class Agent(Model):
|
||||
and limits. The prompts are rows of `agent_prompts`, read alongside; their
|
||||
cached vectors are columns this does not declare.
|
||||
|
||||
`fetch_terms` are words a role must match for a big board read with the
|
||||
wide reach to send it (`app/jobs/reach.py`): the pull's terms are the
|
||||
union over the enabled agents, and an agent with none means no board is
|
||||
narrowed by words. They decide what is fetched, not what ranks.
|
||||
`fetch_terms` are words a role must match for a big board to send it,
|
||||
where its host can filter on them (`app/jobs/reach.py`): the pull's terms
|
||||
are the union over the enabled agents, and an agent with none means no
|
||||
board is narrowed by words. They decide what is fetched, not what ranks.
|
||||
|
||||
`cv_text`, `cv_languages`, `like_text`, `avoid_text` and `must_text` are
|
||||
what `judge` reads a role against: the CV, the languages it states with
|
||||
|
||||
+3
-3
@@ -53,9 +53,9 @@ agent:
|
||||
|
||||
# Title and description in, gate columns out (app/jobs/extract.py). Off by
|
||||
# default: on the API backend every batch of 20 roles is a paid call. On, a
|
||||
# pull reads only the best roles of every enabled agent, as deep and as low
|
||||
# as that agent's own limits say (see agents below), after a first score; a
|
||||
# second score then applies the gates to what was filled. `enabled` is only the
|
||||
# pull reads only the cut of every enabled agent (its limits, see agents
|
||||
# below), after a first score; a second score then applies the fetch gate and
|
||||
# the cut to what was filled. `enabled` is only the
|
||||
# default for a fresh database: the switch under Settings, Sources, Runs (or
|
||||
# `app stages extract on|off`) overrides it and needs no restart.
|
||||
extract:
|
||||
|
||||
+20
-20
@@ -4,7 +4,7 @@ This file lists every source roledesk reads roles from, how each is read and wha
|
||||
|
||||
The code has two plugins: `ats` (`plugins/ats.py`) reads the board of a company someone listed, with one module per host in `plugins/readers/`; `search` (`plugins/search.py`) runs saved queries on sites that list roles across companies. Below, a module of `ats` is a "host" and a module of `search` is a "provider". A company's list of roles on a host is its "board"; a board is attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
|
||||
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `enrich:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
|
||||
|
||||
## Flow
|
||||
|
||||
@@ -26,13 +26,13 @@ The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section say
|
||||
| `sweep` | `stamp`, the cursor of a read cut at `MAX_PAGES` | `_runner.read` on the next pull |
|
||||
| `count` | `stamp`, the roles the last read listed; reset by Recheck | `_runner.read`, the 10% guard |
|
||||
| `missing` | `close_boards` (`_close_swept`), the ids the last finished sweep missed | the next finished sweep |
|
||||
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `enrich`, the guard's escape after 21 days |
|
||||
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `fetch_text`, the guard's escape after 21 days |
|
||||
| `facets` | `_runner.facets_for`, kept one day | `reach.site_facets`, the Companies line |
|
||||
| `size`, `sizechecked` | `ats.keep_size`: `set_board`, Recheck, `app companies size`, or the first read of an unsized board | `reach.sizes`, which decides whether the board is filtered |
|
||||
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`enrich`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `enrich.plan` through `detail_due` |
|
||||
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`fetch_text`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `fetch_text.plan` through `detail_due` |
|
||||
|
||||
4. `close_boards`, the settle of the pull, closes what a complete read did not list, closes a board or search that failed for `STALE_DAYS`, releases the run, and raises one `pull.settled` notice in the Problems log when a target did not answer or was read incomplete, or a board closed more than 50 roles or more than half of its open ones.
|
||||
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score`, `enrich` (reads the role pages of the best-ranked roles without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `summarise`. Extract and summarise are switches under Runs; enrich is on by default.
|
||||
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score` (which writes each agent's cut), `fetch_text` (reads the role pages of the roles in the cut without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `judge`, `summarise`. Extract, judge and summarise are switches under Runs; fetch text is on by default.
|
||||
|
||||
## The three ways a source is read
|
||||
|
||||
@@ -40,23 +40,23 @@ The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section say
|
||||
|
||||
**Paged.** The list arrives in pages of a fixed size (`READS.page_size`), read one after another `PAGE_DELAY` (0.5 s) apart, at most `MAX_PAGES` (100) pages a pull. The read is complete when the number of distinct roles read equals the total the host states (`complete="stated"`), or, for a host that states none, when the read reaches an empty page (`complete="end"`); a Jobindex search is never complete (`complete="never"`). A board of more than 100 pages is swept: each pull stores what it read and the next continues from the saved cursor, and the read is complete when a sweep reaches the end. Roles closing between pulls shift an offset-paged host's later roles back past the cursor, so a role of a swept board closes only when two finished sweeps in a row missed it. A host with a `ceiling` (Workday 2,000) answers nothing past it: a read that reaches it is incomplete, is not swept and closes nothing, so a filter must bring the board under it.
|
||||
|
||||
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The `enrich` stage reads those pages after scoring (`app/jobs/enrich.py`):
|
||||
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The fetch text stage reads those pages after scoring (`app/jobs/fetch_text.py`):
|
||||
|
||||
- It takes open roles without text that an enabled agent with a prompt ranks ungated within its best `enrich.top` (500) at `enrich.min_score` (0.5) or above, best score first.
|
||||
- It reads at most `enrich.per_board` (200) roles per board and `enrich.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
|
||||
- A read that gave no text is tried again after `enrich.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
|
||||
- It runs after every pull, after a saved agent and after a star change, not after a vote, and it can be switched off under Runs.
|
||||
- It takes open roles without text in the cut of an enabled agent (the best by fit and by CV closeness, and the liked and saved, see `app/jobs/score.py`), best final score first.
|
||||
- It reads at most `fetch_text.per_board` (200) roles per board and `fetch_text.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
|
||||
- A read that gave no text is tried again after `fetch_text.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
|
||||
- It runs after every pull, after a saved agent, an affinity change or an override switched, not after a vote, and it can be switched off under Runs.
|
||||
|
||||
A role below the bar for every agent is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles that rank do. Four consequences follow from the code:
|
||||
A role in no agent's cut is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles in the cut do. Four consequences follow from the code:
|
||||
|
||||
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow `enrich` encode it again.
|
||||
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow fetch text encode it again.
|
||||
- A role page may fill only `_runner.DETAIL_FIELDS`: description, location and places, teaser, remote, country, employment, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
|
||||
- Completeness, and so closing, is decided by the list alone. A board of 800 roles closes its roles from the first pull.
|
||||
- SmartRecruiters and Workable list their roles in pages and belong here too, since their list has no text.
|
||||
|
||||
## Overview
|
||||
|
||||
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles `enrich` has read.
|
||||
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles fetch text has read.
|
||||
|
||||
| Source | Boards | Open roles | How it is read | Roles per request | Description | Pay | Limit that matters most |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -95,20 +95,20 @@ Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open
|
||||
|
||||
| Limit | Value | What it means |
|
||||
|---|---|---|
|
||||
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); enrich is on by default. |
|
||||
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); fetch text is on by default. |
|
||||
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. Enrich reads one batch per board on the same pool. One pull runs at a time. |
|
||||
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
|
||||
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
|
||||
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request | The same for boards, role pages and searches. A slow board fails for this pull and is asked again next pull. |
|
||||
| Pages per board | 100 a pull | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
|
||||
| Wait | 0.5 s before every page after the first, for boards and searches alike, and before every role page | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
|
||||
| Per-role reads | `enrich`, after scoring: 200 per board and 1,000 per run, ranked roles first | See "List, then each role". Roles the agents do not rank are not read. |
|
||||
| Per-role reads | `fetch_text`, after scoring: 200 per board and 1,000 per run, the roles in the agents' cut | See "List, then each role". Roles in no cut are not read. |
|
||||
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing, and so does a read that asked nothing. A role listed again reopens. A board switched off closes its roles at once. A filtered read keeps roles with a vote or an application, and remote roles when the host could not be asked for remote; a read by fetch terms closes only roles not listed for 21 days. |
|
||||
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. A Jobindex search is never complete, so its roles close by the 21 days only. A search failing for 21 days closes its roles. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, a row that would not store, and a read that lists under 10% of what the same read listed last time (`SUSPECT_SHARE`) once that was at least `SUSPECT_MIN`, 10. A search that fails is recorded the same way and its row says so. |
|
||||
| A board that fails for 21 days | all its open roles close at the next pull, and a suspect answer that has stood that long is accepted | The next answer reopens what it lists. `STALE_DAYS` is the one 21. |
|
||||
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next enrich run | Nothing is closed on a list cut short by a 429. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next fetch text run | Nothing is closed on a list cut short by a 429. |
|
||||
|
||||
How far a board is read: a pull asks each board of 300 roles or more for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of once per country (33 codes today), unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application. A board's size is set once, when it is added, on Recheck or by `app companies size`, and a pull does not rewrite it; a board with no size yet is sized first in its batch and then read as its size says, and a board whose reader cannot narrow the reach is read whole and sized by that read. A host that cannot filter a dimension ignores it: SmartRecruiters, Phenom, SuccessFactors and Eightfold have no remote filter, so their reach asks for countries only.
|
||||
|
||||
@@ -313,7 +313,7 @@ What no source does:
|
||||
|
||||
## Hosts read as list, then each role
|
||||
|
||||
For every host below the pull reads the list only; the description arrives when `enrich` reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
|
||||
### SmartRecruiters
|
||||
|
||||
@@ -323,7 +323,7 @@ For every host below the pull reads the list only; the description arrives when
|
||||
| Companies | 11 boards: Bosch (about 4,800 roles on the site, 13 open in reach), Delivery Hero, Wise |
|
||||
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
|
||||
| Requests per pull | one series per country of the reach (two for Bosch: DK and SE), each roles / 100, rounded up; a board read whole is roles / 100 |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by `enrich` for ranked roles only: 1 of 343 open roles has text |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only: 1 of 343 open roles has text |
|
||||
| Date | `releasedDate`; trusted |
|
||||
| Place and country | `fullLocation`, else city and country, with empty parts removed; the host's `country` is handed on |
|
||||
| Remote | `location.remote`, true or false, as the host's word |
|
||||
@@ -333,7 +333,7 @@ For every host below the pull reads the list only; the description arrives when
|
||||
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
|
||||
| Limitations | 100 pages a pull; a larger board is swept across pulls. Whether the identifier is case-sensitive is unsettled: the module docstring says it is, `docs/decisions.md` says it is not (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
|
||||
|
||||
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until `enrich` has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until fetch text has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
|
||||
### Workable
|
||||
|
||||
@@ -343,7 +343,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| Companies | 8 boards: Hugging Face, Kahoot!, Starling Bank |
|
||||
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
|
||||
| Requests per pull | roles / 10, rounded up, 0.5 s apart; the largest board has 50 roles, so 5 |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by `enrich` for ranked roles only: 1 of 157 open roles has text |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only: 1 of 157 open roles has text |
|
||||
| Date | `published`; trusted |
|
||||
| Place and country | every `locations` entry as "city, region, country", each remote when the role is; `location` when there are none |
|
||||
| Remote | `workplace` (remote, hybrid, on_site) as the host's word, else the `remote` flag |
|
||||
@@ -592,7 +592,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| Saved search | `q` and optionally `region` (an `amt` id from the site's own search form: `REGIONS` in the module, 2 is Storkøbenhavn) and `remote` (`helt` fully remote, `delvist` partly) |
|
||||
| List | the RSS feed `jobbank.dk/job/rss/`, 100 items a page; an empty page ends it. It states no total and is not sorted by date |
|
||||
| Pages read | at most the module's `ceiling`, which keeps a run under about 20 pages |
|
||||
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by `enrich` for the roles an agent ranks high. The listing carries none |
|
||||
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by fetch text for the roles in the cut. The listing carries none |
|
||||
| Date | `datePosted` from the role's page; trusted |
|
||||
| Place | the region in the item's description line, and `jobLocation.addressLocality` from the page; country Denmark |
|
||||
| Remote | not stated by a row; the search's `remote` filter narrows the list |
|
||||
|
||||
@@ -210,7 +210,7 @@ def test_the_role_carries_its_judgement_and_languages(
|
||||
|
||||
|
||||
def test_an_agent_without_languages_flags_nothing(client: TestClient, store: Store) -> None:
|
||||
owner, uid = setup(store)
|
||||
owner, _uid = setup(store)
|
||||
store.save_agent("Data", prompts=owner.prompts, agent_id=owner.id, cv_languages=[])
|
||||
rows = client.get("/api/roles", params={"agent": owner.id}).json()["items"]
|
||||
assert {r["lang_flag"] for r in rows} == {""}
|
||||
|
||||
+70
-1
@@ -23,7 +23,7 @@ from pipeline import FINAL, JOBS, SCHEDULES, TASKS, JobQueue, Jobs, RunContext,
|
||||
from plugins.ats import ATSSource # noqa: E402
|
||||
from app.config import CALIBRATION, AgentSeed, Config, ConfigError # noqa: E402
|
||||
from app.jobs import cascade, embed, score # noqa: E402
|
||||
from app.models import Candidate, Gate, Limits, Prompt, Weights # noqa: E402
|
||||
from app.models import Application, Candidate, Gate, Limits, Prompt, Weights # noqa: E402
|
||||
from app.storage import Store # noqa: E402
|
||||
from prepped import upsert # noqa: E402
|
||||
|
||||
@@ -991,6 +991,75 @@ def test_without_an_embedder_the_prompts_are_skipped_and_fit_is_out(
|
||||
assert store.scores(agent.id) == {uid: (0.0, "no prompt vector")}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- cut
|
||||
|
||||
def test_the_affinity_bonus_is_the_share_it_has_in_the_final_score() -> None:
|
||||
w = Weights()
|
||||
assert score.bonus(0, w) == 0
|
||||
assert score.bonus(1, w) == pytest.approx(0.2 / 3 / 1.2) # about 0.056
|
||||
assert score.bonus(2, w) == pytest.approx(0.4 / 3 / 1.2) # about 0.111
|
||||
assert score.bonus(2, Weights(fit=0, recency=0, affinity=0, taste=0)) == 0
|
||||
|
||||
|
||||
def _cands(*rows: tuple[str, str, int]) -> list[Candidate]:
|
||||
"""`(uid, posted_at day, affinity)` as candidates."""
|
||||
return [Candidate(uid=uid, posted_at=f"2026-09-{day}T00:00:00+00:00", affinity=affinity)
|
||||
for uid, day, affinity in rows]
|
||||
|
||||
|
||||
def test_the_cut_is_the_best_by_fit_then_by_cv_then_the_kept() -> None:
|
||||
cands = _cands(("a", "10", 0), ("b", "11", 0), ("c", "12", 0), ("d", "13", 0),
|
||||
("e", "14", 0), ("f", "15", 0), ("g", "16", 0))
|
||||
fits = {"a": 0.9, "b": 0.8, "c": 0.7, "d": 0.65, "e": 0.3, "f": 0.2}
|
||||
near = {"a": 0.99, "b": 0.1, "c": 0.1, "d": 0.75, "e": 0.8, "f": 0.5}
|
||||
passed = {"a", "b", "c", "d", "e", "f"}
|
||||
limits = Limits(cut_fit_top=2, cut_fit_min=0.6, cut_cv_top=2, cut_cv_min=0.6)
|
||||
cut = score.cut_of(cands, passed, fits, near, {"f", "g", "b"}, Weights(), limits)
|
||||
# a and b by fit; e and d the closest to the CV that those leave out (a,
|
||||
# the closest of all, is in already); f kept; g kept but gated, so out.
|
||||
assert cut == {"a": "fit", "b": "fit", "e": "cv", "d": "cv", "f": "kept"}
|
||||
|
||||
|
||||
def test_the_minimum_stops_a_list_before_its_count() -> None:
|
||||
cands = _cands(("a", "10", 0), ("b", "11", 0))
|
||||
limits = Limits(cut_fit_top=500, cut_fit_min=0.6, cut_cv_top=150, cut_cv_min=0.6)
|
||||
cut = score.cut_of(cands, {"a", "b"}, {"a": 0.61, "b": 0.59}, {"b": 0.59}, set(),
|
||||
Weights(), limits)
|
||||
assert cut == {"a": "fit"}
|
||||
|
||||
|
||||
def test_affinity_lifts_a_level_over_the_minimum_but_is_no_pass() -> None:
|
||||
cands = _cands(("liked", "10", 1), ("fave", "10", 2), ("plain", "10", 0),
|
||||
("weak", "10", 2))
|
||||
fits = {"liked": 0.56, "fave": 0.5, "plain": 0.59, "weak": 0.1}
|
||||
cut = score.cut_of(cands, {c.uid for c in cands}, fits, {}, set(), Weights(),
|
||||
Limits(cut_fit_min=0.6))
|
||||
assert cut == {"liked": "fit", "fave": "fit"}
|
||||
|
||||
|
||||
def test_a_tie_goes_to_the_newer_role_then_the_lower_uid() -> None:
|
||||
cands = _cands(("b", "10", 0), ("a", "10", 0), ("old", "09", 0), ("new", "11", 0))
|
||||
fits = dict.fromkeys(("a", "b", "old", "new"), 0.8)
|
||||
cut = score.cut_of(cands, {c.uid for c in cands}, fits, {}, set(), Weights(),
|
||||
Limits(cut_fit_top=3, cut_fit_min=0))
|
||||
assert set(cut) == {"new", "a", "b"}
|
||||
|
||||
|
||||
def test_a_like_and_a_save_put_a_role_in_the_cut_on_the_next_score(
|
||||
store: Store, config: Config, fake: FakeEmbedder) -> None:
|
||||
liked = role(store, "1", "Sales Manager", body="sales")
|
||||
saved = role(store, "2", "Office Manager", body="office")
|
||||
other = role(store, "3", "Barista", body="coffee")
|
||||
agent = store.save_agent("Backend", "python backend",
|
||||
limits=Limits(cut_fit_min=1, cut_cv_min=1))
|
||||
store.set_vote(agent.id, liked, 1)
|
||||
store.save_application(Application(role_uid=saved, role_title="Office Manager"))
|
||||
run(config, store, embed.run_embed)
|
||||
run(config, store, score.run_score)
|
||||
assert store.cut(agent.id) == {liked: "kept", saved: "kept"}
|
||||
assert other not in store.cut(agent.id)
|
||||
|
||||
|
||||
def test_a_disabled_agent_is_not_scored(
|
||||
store: Store, config: Config, fake: FakeEmbedder) -> None:
|
||||
role(store, "1")
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime, timedelta
|
||||
@@ -1778,3 +1779,116 @@ def test_summarise_waits_on_the_top_of_each_enabled_agent_by_its_own_limits(
|
||||
conn.execute("UPDATE agents SET limits = '{}'")
|
||||
assert {r.uid for r in store.summarise_waiting(10)} == {a, b, c}
|
||||
assert store.agents()[0].limits == Limits()
|
||||
|
||||
|
||||
# ------------------------------------------------------------- migration 32
|
||||
|
||||
def _v31(path: Path) -> None:
|
||||
"""Take a file `Store` made back to v31: no cut columns, no saved
|
||||
filters."""
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("DROP INDEX idx_scores_cut")
|
||||
conn.execute("ALTER TABLE scores DROP COLUMN cut")
|
||||
conn.execute("ALTER TABLE scores DROP COLUMN fit")
|
||||
conn.execute("DROP TABLE saved_filters")
|
||||
conn.execute("PRAGMA user_version = 31")
|
||||
|
||||
|
||||
def test_v32_resets_the_limits_drops_prefer_gates_and_renames_enrich(tmp_path: Path) -> None:
|
||||
"""An agent stored before the cut, with the old limits keys, a prefer
|
||||
gate and a `weight` on every gate, loads after migrating; the switch and
|
||||
the queued jobs of `enrich` follow the rename."""
|
||||
from pipeline import Jobs
|
||||
|
||||
path = tmp_path / "v31.db"
|
||||
store = Store(path)
|
||||
upsert(store, posting("a"))
|
||||
Jobs(path)
|
||||
old_gates = [{"field": "country", "values": ["DK"], "mode": "require", "weight": 1.0},
|
||||
{"field": "remote", "values": ["hybrid", "remote"], "mode": "prefer",
|
||||
"weight": 0.05},
|
||||
{"field": "title", "values": ["account executive"], "mode": "exclude",
|
||||
"weight": 1.0}]
|
||||
old_limits = {"extract_top": 500, "extract_min": 0.55, "summarise_top": 30,
|
||||
"summarise_min": 0.65, "judge_top": 150, "judge_min": 0.55,
|
||||
"judge_cv_top": 50}
|
||||
_v31(path)
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("INSERT INTO agents (name, gates, limits, created_at) VALUES (?,?,?,?)",
|
||||
("Wolf", json.dumps(old_gates), json.dumps(old_limits), "2026-10-01"))
|
||||
conn.execute("INSERT INTO agents (name, created_at) VALUES ('Bare', '2026-10-01')")
|
||||
conn.execute("INSERT INTO state (key, value) VALUES ('stage:enrich', 'off')")
|
||||
for kind, state in (("enrich", "queued"), ("enrich_batch", "queued"),
|
||||
("enrich", "done"), ("score", "queued")):
|
||||
conn.execute("INSERT INTO jobs (kind, state, created_at) VALUES (?,?,?)",
|
||||
(kind, state, "2026-10-01"))
|
||||
conn.execute("INSERT INTO schedules (name, kind, cadence, created_at) "
|
||||
"VALUES ('texts', 'enrich', 'every 1d', '2026-10-01')")
|
||||
|
||||
migrated = Store(path)
|
||||
|
||||
assert migrated.migrations_applied == [32]
|
||||
by_name = {a.name: a for a in migrated.agents()}
|
||||
assert by_name["Wolf"].limits == Limits() == by_name["Bare"].limits
|
||||
assert [(g.field, g.mode) for g in by_name["Wolf"].gates] == [
|
||||
("country", "require"), ("title", "exclude")]
|
||||
assert migrated.state["stage:fetch_text"] == "off" and "stage:enrich" not in migrated.state
|
||||
with sqlite3.connect(path) as conn:
|
||||
assert conn.execute("SELECT kind, state FROM jobs ORDER BY id").fetchall() == [
|
||||
("enrich", "done"), ("score", "queued")]
|
||||
assert conn.execute("SELECT kind FROM schedules WHERE name = 'texts'").fetchone() == (
|
||||
"fetch_text",)
|
||||
stored = json.loads(conn.execute(
|
||||
"SELECT gates FROM agents WHERE name = 'Wolf'").fetchone()[0])
|
||||
assert all("weight" not in g for g in stored)
|
||||
agent = by_name["Wolf"]
|
||||
migrated.replace_scores(agent.id, [(posting("a").uid, 0.5, "", "", 0.7, 0.6, "fit")])
|
||||
assert migrated.cut(agent.id) == {posting("a").uid: "fit"}
|
||||
assert migrated.save_filter(agent.id, "Remote", "remote=remote").name == "Remote"
|
||||
|
||||
|
||||
def test_v32_runs_twice_and_keeps_limits_set_since(tmp_path: Path) -> None:
|
||||
path = tmp_path / "v31.db"
|
||||
store = Store(path)
|
||||
agent = store.save_agent("Wolf", limits=Limits(cut_fit_top=900))
|
||||
_v31(path)
|
||||
assert Store(path).migrations_applied == [32]
|
||||
assert Store(path).agent(agent.id).limits == Limits() # type: ignore[union-attr]
|
||||
narrow = Limits(cut_fit_top=900, summarise_top=10)
|
||||
store.save_agent("Wolf", agent_id=agent.id, limits=narrow)
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("PRAGMA user_version = 31") # the columns are there already
|
||||
again = Store(path)
|
||||
assert again.migrations_applied == [32]
|
||||
assert again.agent(agent.id).limits == narrow # type: ignore[union-attr]
|
||||
|
||||
|
||||
def test_a_fresh_file_has_the_cut_columns_and_saved_filters(store: Store) -> None:
|
||||
with store.transaction() as conn:
|
||||
columns = {r[1] for r in conn.execute("PRAGMA table_info(scores)")}
|
||||
tables = {r[0] for r in conn.execute("SELECT name FROM sqlite_master")}
|
||||
assert {"fit", "cut", "cv_fit"} <= columns and "saved_filters" in tables
|
||||
|
||||
|
||||
# ------------------------------------------------------------ saved filters
|
||||
|
||||
def test_saved_filters_are_per_agent_named_once_regardless_of_case(store: Store) -> None:
|
||||
one, two = store.save_agent("One"), store.save_agent("Two")
|
||||
remote = store.save_filter(one.id, " Remote only ", "remote=remote&order=newest")
|
||||
store.save_filter(one.id, "berlin", "places=place:berlin")
|
||||
store.save_filter(two.id, "Remote only", "remote=remote")
|
||||
|
||||
assert (remote.name, remote.query) == ("Remote only", "remote=remote&order=newest")
|
||||
assert [f.name for f in store.saved_filters(one.id)] == ["berlin", "Remote only"]
|
||||
with pytest.raises(ValueError, match="already saved"):
|
||||
store.save_filter(one.id, "REMOTE ONLY", "x=1")
|
||||
with pytest.raises(ValueError, match="needs a name"):
|
||||
store.save_filter(one.id, " ", "x=1")
|
||||
with pytest.raises(KeyError):
|
||||
store.save_filter(999, "Nobody", "x=1")
|
||||
|
||||
assert store.delete_filter(remote.id) and not store.delete_filter(remote.id)
|
||||
assert [f.name for f in store.saved_filters(one.id)] == ["berlin"]
|
||||
store.delete_agent(one.id)
|
||||
assert store.saved_filters(one.id) == []
|
||||
assert [f.name for f in store.saved_filters(two.id)] == ["Remote only"]
|
||||
@@ -1874,3 +1874,42 @@ def test_saving_queues_no_summary_when_the_stage_is_off_or_it_has_one(
|
||||
store.upsert_role(posting("3"))
|
||||
client.post("/api/applications", json={"role_uid": posting("3").uid})
|
||||
assert _summary_jobs(queue) == []
|
||||
|
||||
|
||||
# ------------------------------------------------------------ saved filters
|
||||
|
||||
def test_saved_filters_are_listed_saved_and_deleted_per_agent(
|
||||
client: TestClient, store: Store) -> None:
|
||||
one, two = store.save_agent("One"), store.save_agent("Two")
|
||||
url = f"/api/agents/{one.id}/filters"
|
||||
|
||||
made = client.post(url, json={"name": "Remote", "query": "remote=remote&order=newest"})
|
||||
assert made.status_code == 201
|
||||
row = made.json()
|
||||
assert set(row) == {"id", "name", "query", "created_at"}
|
||||
assert (row["name"], row["query"]) == ("Remote", "remote=remote&order=newest")
|
||||
client.post(url, json={"name": "berlin", "query": "places=place:berlin"})
|
||||
assert [f["name"] for f in client.get(url).json()] == ["berlin", "Remote"]
|
||||
assert client.get(f"/api/agents/{two.id}/filters").json() == []
|
||||
|
||||
assert client.post(url, json={"name": "remote", "query": "x=1"}).status_code == 409
|
||||
assert client.post(url, json={"name": " ", "query": "x=1"}).status_code == 422
|
||||
assert client.post("/api/agents/999/filters", json={"name": "A"}).status_code == 404
|
||||
assert client.get("/api/agents/999/filters").status_code == 404
|
||||
|
||||
assert client.delete(f"/api/filters/{row['id']}").status_code == 204
|
||||
assert client.delete(f"/api/filters/{row['id']}").status_code == 404
|
||||
assert [f["name"] for f in client.get(url).json()] == ["berlin"]
|
||||
|
||||
|
||||
def test_an_override_queues_a_rescore_and_fetch_text_either_way(
|
||||
client: TestClient, store: Store, queue: Any) -> None:
|
||||
"""All countries and All roles change the agents' fetch gates for the
|
||||
roles already stored, so either switch, on or off, rescores."""
|
||||
states = ("queued", "running", "done", "failed", "cancelled")
|
||||
store.add_company("Acme")
|
||||
assert client.post("/api/companies/acme/all-roles", json={"on": False}).status_code == 200
|
||||
found = queue.jobs.last("fetch_text", states=states)
|
||||
assert found is not None
|
||||
assert [queue.jobs.get(i).kind for i in found.depends_on] == ["score"]
|
||||
assert queue.jobs.last("pull", states=states) is None
|
||||
Reference in new issue
Block a user