Compare commits
60
Commits
97c3046f53
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
920457d326 | ||
|
|
2262cda8f1 | ||
|
|
1913101061 | ||
|
|
849adb197d | ||
|
|
136047023e | ||
|
|
911dfe8584 | ||
|
|
8b875b377b | ||
|
|
c6730062fd | ||
|
|
49929372c7 | ||
|
|
f141cffc3c | ||
|
|
02ea191b53 | ||
|
|
1976f6a353 | ||
|
|
2159c7925f | ||
|
|
0c4f3c9bbe | ||
|
|
38c31c830b | ||
|
|
4a9bcb2b1c | ||
|
|
e61185e4b9 | ||
|
|
e8157d9509 | ||
|
|
b4774f1da9 | ||
|
|
8f774b5147 | ||
|
|
d2f18cb3c7 | ||
|
|
62cf1c738c | ||
|
|
c3c527dc6c | ||
|
|
c4c17fb019 | ||
|
|
6b12d10bee | ||
|
|
9da9bd7942 | ||
|
|
7f37270eb3 | ||
|
|
af1efd4fd6 | ||
|
|
937573e048 | ||
|
|
408c5ed73e | ||
|
|
8c3c51dd9f | ||
|
|
9907816cc0 | ||
|
|
80b8416579 | ||
|
|
56a0bb8558 | ||
|
|
9b120a3b94 | ||
|
|
c3bc2cd7f3 | ||
|
|
85aed0807b | ||
|
|
19fb73caa0 | ||
|
|
0b92f19f4d | ||
|
|
b64d77d224 | ||
|
|
1b24de6980 | ||
|
|
caed82d416 | ||
|
|
92ad8c2c1b | ||
|
|
8be59a7ee4 | ||
|
|
2c2b9b6bee | ||
|
|
7ac63dea7c | ||
|
|
3388b97cf4 | ||
|
|
b6d0a54f39 | ||
|
|
1127d06f03 | ||
|
|
d80f26a930 | ||
|
|
1f444aa077 | ||
|
|
1b5a38b61c | ||
|
|
6d82287402 | ||
|
|
463e53ac32 | ||
|
|
b6e27fcb26 | ||
|
|
744d14160d | ||
|
|
6cad9713ac | ||
|
|
a0cb8c6a71 | ||
|
|
9b70f64a20 | ||
|
|
8f37f51254 |
No files matched your search
@@ -43,4 +43,4 @@ A larger rework is planned first, as a doc in `docs/` committed on main with wor
|
||||
|
||||
# Rule:
|
||||
|
||||
`docs/architecture.md` is the owner's document: the target design of the flow from board to feed and its key decisions, with no description of today's code. What is built is tested against it. Change it only when the owner makes a key decision; the way there goes into a plan, the detailed reasoning into `decisions.md`.
|
||||
`documentation/` holds the owner's ADRs (`ADR Fetch prep and score.md`, `ADR Plugins.md`): the target design and its key decisions, with no description of today's code. Each stands alone and may refer only to the other ADRs, never to `docs/` or the code's files. What is built is tested against them. Change them only when the owner makes a key decision; the way there goes into a plan in `docs/`, the detailed reasoning into `decisions.md`.
|
||||
@@ -13,11 +13,11 @@ uv run app job pull # read every board and search, map and fold, embe
|
||||
uv run app serve # http://127.0.0.1:8790
|
||||
```
|
||||
|
||||
`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the chip input's. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents the keys, among them the `extract:` and `summarise:` blocks, which set the default for the two model stages; the switch under Settings, Sources, Runs overrides it without a restart, and `app stages` does the same from the command line.
|
||||
`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the tests of the chip input and the remote chips. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents the keys. Its `extract:`, `summarise:` and `judge:` blocks set the default for the three model stages, and the `fetch_text:` block that of the stage reading role pages. The switches under Settings, Sources, Runs override them without a restart, and `app stages` does the same from the command line.
|
||||
|
||||
## Sources: job boards and company boards
|
||||
|
||||
The Sources sub-tab does all of this; the same actions exist on the command line:
|
||||
The Sources page under Settings does all of this; the same actions exist on the command line:
|
||||
|
||||
```bash
|
||||
uv run app searches add jobindex "Copenhagen golang" --q "golang go-udvikler" --area 1
|
||||
@@ -40,11 +40,11 @@ uv run app agents list # the agents and their pr
|
||||
uv run app prep unmapped # the host words no table maps; prep unlinked lists the place texts no place resolves
|
||||
```
|
||||
|
||||
The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read (its own pages, a host with no module, a site whose robots.txt disallows reading) is still a row, with its careers page and the reason, shown on the Watchlist tab for checking by hand and under Companies as "not readable yet", so the gap is visible and one module at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles, and asks first when there are open roles to lose; a board is read once before it is kept, and one that lists no roles yet is kept too. Setting a board queues a pull right after, so its roles are read by the one already running or the very next one, rather than waiting for the schedule. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
|
||||
The seed is a hand-picked list of European tech employers, not a list of boards. A company whose careers site the app cannot read is still a row, with its careers page and the reason, shown on the Watchlist tab and under Companies as "not readable yet". A row's Set board control (Change once it has a board) attaches, replaces or detaches its board, which is read once before it is kept; replacing or detaching closes the old board's open roles, and so does switching a board or a search off. `companies discover` reads a company's site for a link into one of the hosts the app knows, probes the hosts only when no page names one, and with `--add` sets the board of a new or existing company. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
|
||||
|
||||
`app stages` shows or switches extract, summarise, judge and fetch text, `app seniority` shows per host how often the stated level equals the model's, `app dupes` recomputes which open roles are copies of one posting, and `app prep` maps the host words and folds the copies without a pull.
|
||||
|
||||
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. A freehire query is free text over about 50 hosts whose words are OR over title, company, text and place, so it is noisy: `--country` takes comma-separated ISO codes (`dk,se`) and `--remote` a work mode. A Jobnet query is a loose match without phrases and `--region` one of its five regions (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`). An Akademikernes Jobbank query is OR words, `--region` its `amt` id (2 is Storkøbenhavn) and `--remote` `helt` or `delvist`. A freehire role whose employer is a board roledesk reads is a copy of that board's role and is hidden behind it. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
|
||||
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. A Platsbanken query is free text and `--region` the id of a län (`CaRE_1nn_cSU` is Skåne). An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. A freehire query is free text over about 50 hosts whose words are OR over title, company, text and place, so it is noisy: `--country` takes comma-separated ISO codes (`dk,se`) and `--remote` a work mode. A Jobnet query requires every word and has no phrases, so a search is one term; `--region` is one of its five regions (`HovedstadenOgBornholm`, `Midtjylland`, `Syddanmark`, `OevrigeSjaelland`, `Nordjylland`). An Akademikernes Jobbank query is OR words, `--region` its `amt` id (2 is Storkøbenhavn) and `--remote` `helt` or `delvist`. `searches add` runs the search once and saves it only if it finds something. The `embeddings` extra is for the embeddings behind prompt closeness and CV closeness, `agent` serves extract, summarise and judge, and `passkeys` is for the Account tab's passkeys.
|
||||
|
||||
## The tabs
|
||||
|
||||
@@ -52,7 +52,7 @@ Three tabs, and a cog for settings.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Roles** | newsdesk's feed: a filter rail (Viewing as, Show read, Agent match or Newest, first seen, tags, and under more filters company and place chips, seniority from, to and track, function, remote, pay and votes), the open roles with company, place, age, tags and a summary, votes (▲ like ● neither way ▼ pass) and a Save button. A title opens the reader beside the list; on a phone it takes the screen. `j`/`k` move, `Enter` opens, `/` searches |
|
||||
| **Roles** | newsdesk's feed: a filter rail (Viewing as, saved filters, Show read, the order (What I'm looking for, What fits me or Newest), Posted, tags, and under more filters company and place chips, seniority from, to and track, function, remote, pay, votes, Closing soon and Show gated), and the open roles that pass the agent's fetch gate, with company, place, age, tags and a summary, votes (▲ like ● neither way ▼ pass) and a Save button (★). What fits me needs a CV; an agent without one is shown What I'm looking for. A title opens the reader beside the list; on a phone it takes the screen. `j`/`k` move, `Enter` opens, `/` searches |
|
||||
| **Applications** | the saved roles and their status, each change recorded as an event |
|
||||
| **Watchlist** | the companies whose board the app cannot read, for going through their careers pages by hand: Yes or No marks whether one is worth checking (a filter shows the unrated, the yes, the no or all), the careers page opens in a new tab and notes when, and a missing or wrong link is set in the row. A company whose board is failing is listed too. Separate from the affinity a search agent has |
|
||||
|
||||
@@ -60,19 +60,20 @@ The cog opens `/settings`: a rail of pages on the left, the page beside it, stic
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its rules in words, and its weights. Edit turns the card into the form in place (name, prompts as rows, a Where block for the location rules, the other require/any/exclude rules as rows, fetch terms, weights and limits); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
|
||||
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what the pull asks its board for, as a reach line such as "follows agents: DK, SE, remote · all roles", the board's size with a Recheck, the All countries and All roles overrides, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise, judge and fetch text, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
|
||||
| **Search agents** | a card per agent, newest first: name, an On/Off switch, its prompts, its fetch gate in words, its weights, its analytics filter and its summarise size. Edit turns the card into the form in place; New agent puts a blank card on top. The form is name, prompts, the fetch gate in four parts (World, Location, Role exclusion, Role fetch), CV, languages, likes and dislikes, the analytics filter, the want feed score weights, the CV match weights and the summarise size; other rules are under Advanced. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it |
|
||||
| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex, Adzuna, Platsbanken, freehire, Jobnet or Akademikernes Jobbank: new, try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, what a pull asks its board for and what its read cannot do, the board's size with a Recheck, the All locations and All roles switches, following a company, setting or changing a company's board), runs (Run the pull now, the switches of extract, summarise, judge and fetch text, what is running, the schedule, and the Problems log) and unmapped (the place texts no place resolves and the host words no table maps, to copy) |
|
||||
| **Account** | the password, sessions and passkeys. Open on loopback, locked on anything else |
|
||||
|
||||
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its `taste` weight scores a role by how close it is to the roles the agent liked minus the ones it passed.
|
||||
Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its job like weight (`taste` in the config) scores a role by how close it is to the roles the agent liked, away from the ones it passed.
|
||||
|
||||
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role whose text fetch text has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role; the switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, bounded to the top of each agent's feed (its `summarise_top`) and the roles liked or saved, so the cost stays flat.
|
||||
The reader, `/roles/{uid}`, shows the full posting: the same vote and Save controls, the company's own affinity, the tags and the summary when there is one. Opening a role marks it read, and so does a vote. A role whose text is a teaser or missing (a search role, or a board role whose page fetch text has not read yet) gets a Fetch button that reads the posting page once and stores the text; for a Jobindex role it follows the "Se jobbet" link to the posting itself. A Summarise button asks for a short summary of that one role. The switch under Settings, Sources, Runs (default from `config.yaml`'s `summarise:` block) turns summarising on for the pull itself, which reads the top of both orders (each agent's `summarise_top`, 30 to start) and the roles liked or saved, so the cost stays flat.
|
||||
|
||||
## Docs
|
||||
|
||||
- [`documentation/ADR Fetch prep and score.md`](<documentation/ADR Fetch prep and score.md>) and [`documentation/ADR Plugins.md`](<documentation/ADR Plugins.md>): the owner's target design, for the ranking from the fetch gate to the feed and for what a source plugin delivers.
|
||||
- [`docs/decisions.md`](docs/decisions.md): what was decided for roledesk and why.
|
||||
- [`docs/open.md`](docs/open.md): what is left to do, for whoever picks this up next.
|
||||
- [`plugins/README.md`](plugins/README.md): every reader, its limits and the Watchlist; [`docs/filter-survey.md`](docs/filter-survey.md): which source-side filters each host takes.
|
||||
- [`plugins/README.md`](plugins/README.md): every reader, its endpoints and what each states; [`docs/filter-survey.md`](docs/filter-survey.md): the filter parameters each host takes at the source.
|
||||
- `docs/plan-*.md`: plans in progress, folded into `decisions.md` and deleted when their work lands.
|
||||
- [`desk/README.md`](desk/README.md), [`pipeline/README.md`](pipeline/README.md), [`auth/README.md`](auth/README.md), [`deploy/README.md`](deploy/README.md): the copied modules, where each comes from and how to refresh it.
|
||||
- [`docs/hosting.md`](docs/hosting.md), [`docs/release.md`](docs/release.md), [`docs/server.md`](docs/server.md), [`docs/password-protection.md`](docs/password-protection.md), [`docs/llm-backends.md`](docs/llm-backends.md): the boilerplate's deployment docs.
|
||||
+2
-1
@@ -1,12 +1,13 @@
|
||||
"""The command line.
|
||||
|
||||
app serve [--host H] [--port P] [--reload]
|
||||
app job pull|extract|embed|score [--payload JSON]
|
||||
app job pull|fetch_text|prep|embed|score|extract|judge|summarise [--payload JSON]
|
||||
app worker [--workers http=8,cpu=1] [--until-idle]
|
||||
app companies seed|add|edit|checked|remove|by-hand|set-board|detach|discover|check|size
|
||||
app searches add|list|remove
|
||||
app agents list|prompt
|
||||
app stages [extract|summarise|judge|fetch_text on|off]
|
||||
app seniority
|
||||
app dupes
|
||||
app prep [unmapped|unlinked] [--remap]
|
||||
app db init
|
||||
|
||||
+3
-3
@@ -55,8 +55,8 @@ class ExtractConfig(ModelTaskConfig):
|
||||
class SummariseConfig(ModelTaskConfig):
|
||||
"""About 60 words on a role, for the list and the read page. See
|
||||
app/jobs/summarise.py. Off by default like extract; on, a pull summarises
|
||||
the best `summarise_top` final scores of each enabled agent (its
|
||||
`limits`), so the cost is bounded whatever the backlog, and every role
|
||||
the top `summarise_top` (its `limits`) of each enabled agent's want order
|
||||
and of its match order, so the cost is bounded whatever the backlog, and every role
|
||||
that is saved or liked, whatever its rank. The read page's Summarise button asks for one
|
||||
role whether this is on or not."""
|
||||
|
||||
@@ -70,7 +70,7 @@ class SummariseConfig(ModelTaskConfig):
|
||||
class JudgeConfig(ModelTaskConfig):
|
||||
"""Each search agent's cut read against its CV, as three numbers and a
|
||||
total. See app/jobs/judge.py. Off by default like extract; on, a pull
|
||||
judges the cut of each enabled agent with a CV and a prompt, and only
|
||||
judges the cut of each enabled agent with a CV, and only
|
||||
the roles whose judgement is missing or was read from another CV or
|
||||
text."""
|
||||
|
||||
|
||||
+23
-4
@@ -82,6 +82,8 @@ class Copy:
|
||||
text: str = ""
|
||||
#: The link that opens this row on its host.
|
||||
url: str = ""
|
||||
#: Whether the row is a remote role (`roles.remote` is 'remote').
|
||||
remote: bool = False
|
||||
|
||||
|
||||
def _country(c: Copy) -> str:
|
||||
@@ -109,6 +111,12 @@ def same_posting(a: Copy, b: Copy) -> bool:
|
||||
return not ta or not tb or bool(ta & tb)
|
||||
|
||||
|
||||
def _same_country(a: Copy, b: Copy) -> bool:
|
||||
"""Whether two rows can be in one country: equal, or either unknown."""
|
||||
ca, cb = _country(a), _country(b)
|
||||
return not ca or not cb or ca == cb
|
||||
|
||||
|
||||
def keep_order(c: Copy) -> tuple[bool, bool, bool, str, str]:
|
||||
"""The sort key that picks the kept copy of a group, smallest first: a
|
||||
board over a search provider, then one with a description, then the
|
||||
@@ -191,9 +199,19 @@ def fold(rows: Iterable[Copy]) -> Folded:
|
||||
|
||||
Within a `(company_key, title_key)` the kept rows of `assign` are grouped
|
||||
again, in `keep_order`: a row joins the first group whose members are all
|
||||
of its host and `similar` to it. Only a row with a link can be folded, as
|
||||
its link is what the kept role keeps. Every row of a merged group points
|
||||
at its first row, the rows that pointed at a folded one included."""
|
||||
of its host, in its country (equal, or either unknown, as `same_posting`
|
||||
reads it), `similar` to it and remote as it is or not. Only a row with
|
||||
a link can be folded, as its link is what the kept role keeps. Every row
|
||||
of a merged group points at its first row, the rows that pointed at a
|
||||
folded one included.
|
||||
|
||||
One country, as ADR Fetch prep and score 1.4 has it: a job posted once
|
||||
per office in several countries is several roles, one per country, which
|
||||
keeps a different opening with an alike text (a federal sales role in
|
||||
the US and one in Australia) from merging. A remote listing and an office listing stay two roles: the kept role
|
||||
takes a copy's places and not its remote, so the fetch gate, which reads
|
||||
a remote role by its area, would reject the remote listing with an
|
||||
office role elsewhere, and the office one with a remote role elsewhere."""
|
||||
rows = list(rows)
|
||||
result = assign(rows)
|
||||
by_uid = {r.uid: r for r in rows}
|
||||
@@ -207,7 +225,8 @@ def fold(rows: Iterable[Copy]) -> Folded:
|
||||
groups: list[list[Copy]] = []
|
||||
for row in sorted(part, key=keep_order):
|
||||
for group in groups:
|
||||
if all(m.host == row.host and similar(m.text, row.text) for m in group):
|
||||
if all(m.host == row.host and m.remote == row.remote
|
||||
and _same_country(m, row) and similar(m.text, row.text) for m in group):
|
||||
group.append(row)
|
||||
break
|
||||
else:
|
||||
|
||||
+22
-16
@@ -1,22 +1,28 @@
|
||||
"""The work roledesk queues. Importing this package registers every job, every
|
||||
task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
|
||||
|
||||
pull.py read every board and search and store the roles they yield
|
||||
prep.py the `prep` stage: `app.prep` over what a pull stored, then the fold
|
||||
extract.py title and description into the gate columns, on Haiku
|
||||
embed.py one vector per open role, for fit
|
||||
score.py every open role scored for every enabled search agent
|
||||
summarise.py about 60 words on each of the best roles, on Haiku
|
||||
judge.py each agent's best roles against its CV: skills, experience, career
|
||||
cascade.py pull → prep → embed → score → ... → summarise → judge, what `TASKS`
|
||||
offers, the `pull` job a schedule fires, and that schedule
|
||||
stages.py whether extract, summarise and judge run: the switch, else config.yaml
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
added, switched on or off
|
||||
discover.py which ATS a company's careers site runs on, from its pages
|
||||
posting.py one role's posting page fetched as text, on demand
|
||||
reach.py which countries a pull asks each big board for, from the agents
|
||||
limits.py what limits a board's read, in words, for the Companies list
|
||||
pull.py read every board and search and store the roles they yield
|
||||
prep.py the `prep` stage: `app.prep` over what a pull stored, then the fold
|
||||
embed.py one vector per open role, for prompt closeness and CV closeness
|
||||
score.py every open role scored for every enabled search agent, and each
|
||||
agent's cut
|
||||
fetch_text.py the roles in the cut that came without text, read on their own
|
||||
pages, with no model
|
||||
extract.py title and description into the gate columns, on Haiku
|
||||
judge.py each agent's cut against its CV: skills, experience, career
|
||||
summarise.py about 60 words on the top of both orders and the roles liked or
|
||||
saved, on Haiku
|
||||
cascade.py pull → prep → embed → score → fetch_text → prep → embed → score
|
||||
→ extract → score → judge → summarise, what `TASKS` offers, the
|
||||
`pull` and `fetch_text` jobs that start it, and the default schedule
|
||||
stages.py whether extract, summarise, judge and fetch text run: the switch,
|
||||
else config.yaml
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
added, switched on or off
|
||||
discover.py which ATS a company's careers site runs on, from its pages
|
||||
posting.py one role's posting page fetched as text, on demand
|
||||
reach.py which countries a pull asks each big board for, from the agents
|
||||
limits.py what limits a board's read, in words, for the Companies list
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
+3
-2
@@ -22,8 +22,9 @@ it filled none, `step` passes over all three. Extract reads the cut; the
|
||||
score after it applies the fetch gate and the cut again with the columns
|
||||
extract filled, so a role can drop out or enter (one that enters is
|
||||
extracted on the next pull). Judge reads each agent's cut as recomputed,
|
||||
against its CV. Summarise runs last, on the top of each agent's feed by
|
||||
final score and the roles liked or saved: it is no part of any score.
|
||||
against its CV. Summarise runs last, on the top of each agent's feed in
|
||||
the want order and in the match order, and the roles liked or saved: it is
|
||||
no part of any score.
|
||||
|
||||
A batch that failed, or a stage somebody killed, does not stop the chain: the
|
||||
work comes round on the next pull, and a rescore of what did arrive is worth
|
||||
|
||||
+2
-1
@@ -9,7 +9,8 @@ refused batch, and failing with a notice when no model is reachable are
|
||||
|
||||
`extract` is also the afterburner for the level and the remote area. A role
|
||||
is waiting while it is open, has not been given up on, has never been
|
||||
answered (`extracted_at IS NULL`), has no model level (`seniority_model`
|
||||
answered or was answered before a better text came (`extracted_at IS
|
||||
NULL`, which `Store.upsert_role` sets again), has no model level (`seniority_model`
|
||||
empty), was never asked where a remote hire may live (`remote_where_model`
|
||||
NULL) or for its languages and deadline (`languages_required` NULL), and is
|
||||
in the cut of an enabled agent (`scores.cut`, see `app.jobs.score`). That
|
||||
|
||||
+55
-91
@@ -2,21 +2,19 @@
|
||||
|
||||
A `ModelTask` beside `summarise`, after the `/rank` of
|
||||
github.com/MadsLorentzen/ai-job-search: the model reads a posting against a
|
||||
written profile and scores skills, experience and career 0 to 100, with one
|
||||
to three strengths and gaps. The total is `app.models.cv_score`, the three
|
||||
numbers under the agent's `cv_*` weights, worked out when a role is served and
|
||||
not stored, so the weights are no part of `input_key`; it is what the feed's
|
||||
CV match sorts by. The model also reports a verdict on each of the
|
||||
agent's must-haves (one per line of its must text, numbered M1..Mn in the
|
||||
prompt); it does not apply them. The rule is code's: a "broken" one makes
|
||||
career count as at most `app.models.BROKEN_CAP`, when the role is served. Culture and location are left out: the first
|
||||
is guessed from ad text, the second is what the fetch gate does.
|
||||
written profile and scores skills, experience and career 0 to 10, with one
|
||||
to three strengths and gaps. A model's rating is an estimate, so the scale is
|
||||
coarse. The total is `app.models.cv_score`, the three numbers under the
|
||||
agent's `cv_*` weights times 10, so 0 to 100, worked out when a role is served
|
||||
and not stored, so the weights are no part of `input_key`; it is what the
|
||||
feed's CV match sorts by. Culture and location are left out: the first is
|
||||
guessed from ad text, the second is what the fetch gate does.
|
||||
|
||||
The work is pairs, `(agent, role)`: an enabled agent with a CV and a prompt,
|
||||
and an open role in its cut (`scores.cut`, see `app.jobs.score`: the best by
|
||||
The work is pairs, `(agent, role)`: an enabled agent with a CV, with or
|
||||
without a prompt, and an open role in its cut (`scores.cut`, see `app.jobs.score`: the best by
|
||||
fit, the best by CV closeness that those leave out, and the liked and saved)
|
||||
with some text, that has no judgement yet, or one read from another CV,
|
||||
prompts, like, avoid or must text or role text than today's (`input_key`).
|
||||
prompts, languages, like or avoid text or role text than today's (`input_key`).
|
||||
The cut is the only selection: the pull judges it after extract and the
|
||||
rescore that follows, so it reads the cut as recomputed. A pair is named by the composite key `"<agent_id>:<uid>"`,
|
||||
which is what `desk.modeltask` hands round as a uid, so `desk/` is untouched.
|
||||
@@ -49,9 +47,7 @@ from pipeline import NULL, RunContext, Task, job
|
||||
from pipeline.task import Batch
|
||||
from app.config import Config
|
||||
from app.jobs import stages
|
||||
from app.models import (
|
||||
LANGUAGE_LEVELS, MUST_VERDICTS, Agent, CvLanguage, cv_languages, must_lines,
|
||||
)
|
||||
from app.models import LANGUAGE_LEVELS, Agent, CvLanguage, _language_name, cv_languages
|
||||
from app.storage import Store
|
||||
|
||||
#: How much of a posting goes in one item: summarise's 8000, since the
|
||||
@@ -63,42 +59,39 @@ EXCERPT = 8000
|
||||
#: 2: the verdict on each numbered must-have (`musts`), the cap in code.
|
||||
#: 3: what the person likes (`like_text`) beside what they do not, and the
|
||||
#: career rubric reading both.
|
||||
FORMAT = 3
|
||||
#: 4: the three numbers 0 to 10 rather than 0 to 100.
|
||||
#: 5: the person's languages, read into skills.
|
||||
#: 6: no must-haves: the profile is the CV, languages, likes and dislikes.
|
||||
FORMAT = 6
|
||||
|
||||
#: How many strengths and gaps are kept, and how long each may be.
|
||||
MOST, LONGEST = 3, 300
|
||||
|
||||
SYSTEM = """You judge job postings for one person: how well each fits their CV and what they want.
|
||||
|
||||
Below the rules is the person: their CV, what they want, what they like and
|
||||
what they do not like (the work, the workplace, situations), and their
|
||||
must-haves and deal-breakers when they gave any. Each item is one posting with a key, its title, company, place
|
||||
and text. Answer one judgement per item.
|
||||
Below the rules is the person: their CV, the languages they speak and how
|
||||
well, what they want, what they like and what they do not like (the work,
|
||||
the workplace, situations). Each item is one posting with a key, its title,
|
||||
company, place and text. Answer one judgement per item.
|
||||
|
||||
Score three dimensions, each 0 to 100:
|
||||
- skills: 80-100 the core requirements are the person's primary skills; 60-79
|
||||
most requirements match, with 1-2 gaps that are learnable; 40-59 a partial
|
||||
match, significant upskilling needed; 0-39 a fundamental mismatch.
|
||||
Score three dimensions, each a whole number from 0 to 10:
|
||||
- skills: 8-10 the core requirements are the person's primary skills; 6-7
|
||||
most requirements match, with 1-2 gaps that are learnable; 4-5 a partial
|
||||
match, significant upskilling needed; 0-3 a fundamental mismatch. A
|
||||
language the posting requires counts as a requirement: one the person
|
||||
speaks below the level asked, or not at all when their languages are
|
||||
listed, lowers skills and is a gap. A language the posting does not
|
||||
mention changes nothing.
|
||||
- experience: match on the function and nature of the work performed, not the
|
||||
literal job title, and on its level. 80-100 direct experience in the same
|
||||
domain and role type; 60-79 related, with transferable skills clear; 40-59
|
||||
adjacent, the person would need to make the case; 0-39 unrelated.
|
||||
literal job title, and on its level. 8-10 direct experience in the same
|
||||
domain and role type; 6-7 related, with transferable skills clear; 4-5
|
||||
adjacent, the person would need to make the case; 0-3 unrelated.
|
||||
- career: against what the person wants (the prompts), what they like and
|
||||
what they do not like. 80-100 strongly
|
||||
aligned with their direction, with clear growth; 60-79 a good role, partly
|
||||
aligned; 40-59 a decent job that does not build toward their goals; 0-39 a
|
||||
dead end or a step backwards. Judge whether the tasks and the setting would
|
||||
energise them, not only whether they could do them.
|
||||
|
||||
Their must-haves and deal-breakers are numbered M1, M2 and so on. Give a
|
||||
verdict on every one of them, in `musts`, for each item:
|
||||
- "met": the posting text says it is met.
|
||||
- "broken": only when the posting text clearly contradicts it.
|
||||
- "unclear": the text does not say. Never infer from silence: a posting that
|
||||
is silent on a must-have is "unclear", not "broken".
|
||||
Judge a must-have by its own words, not by how well the role fits otherwise.
|
||||
Answer musts as [] when there are none. The verdicts do not change the three
|
||||
scores, which follow their rubrics; name a broken must-have as a gap.
|
||||
what they do not like. 8-10 strongly aligned with their direction, with
|
||||
clear growth; 6-7 a good role, partly aligned; 4-5 a decent job that does
|
||||
not build toward their goals; 0-3 a dead end or a step backwards. Judge
|
||||
whether the tasks and the setting would energise them, not only whether
|
||||
they could do them.
|
||||
|
||||
Then 1 to 3 strengths and 1 to 3 gaps, each one short sentence grounded in
|
||||
the posting text. State the gaps honestly: a prestigious role that fits
|
||||
@@ -123,21 +116,8 @@ SCHEMA: dict[str, Any] = {
|
||||
"career": {"type": "integer"},
|
||||
"strengths": {"type": "array", "items": {"type": "string"}},
|
||||
"gaps": {"type": "array", "items": {"type": "string"}},
|
||||
"musts": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {"type": "string"},
|
||||
"verdict": {"type": "string", "enum": list(MUST_VERDICTS)},
|
||||
},
|
||||
"required": ["id", "verdict"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
},
|
||||
"required": ["key", "skills", "experience", "career", "strengths", "gaps",
|
||||
"musts"],
|
||||
"required": ["key", "skills", "experience", "career", "strengths", "gaps"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
@@ -178,18 +158,19 @@ def _text(row: Any) -> str:
|
||||
|
||||
def input_key(agent: Agent, text: str) -> str:
|
||||
"""A hash of everything a judgement reads: the CV, the prompts, the
|
||||
like, avoid and must texts, and the role's text as sent. A judgement stored
|
||||
under another key is read again."""
|
||||
languages, the like and avoid texts, and the role's text as sent. A judgement
|
||||
stored under another key is read again."""
|
||||
blob = json.dumps([FORMAT, agent.cv_text, [[p.name, p.text] for p in agent.prompts],
|
||||
agent.like_text, agent.avoid_text, agent.must_text, text], ensure_ascii=False)
|
||||
[[l.language, l.level] for l in agent.cv_languages],
|
||||
agent.like_text, agent.avoid_text, text], ensure_ascii=False)
|
||||
return hashlib.sha1(blob.encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def _judged(agent: Agent) -> bool:
|
||||
"""Whether `judge` reads for this agent at all: enabled, with a CV and a
|
||||
prompt with text (without one nothing of its cut is in by fit)."""
|
||||
return agent.enabled and bool(agent.cv_text.strip()) \
|
||||
and any(p.text.strip() for p in agent.prompts)
|
||||
"""Whether `judge` reads for this agent at all: enabled, with a CV. An
|
||||
agent without a prompt is judged too: its cut is the CV list and the
|
||||
kept, and the profile goes without `<wants>` (`system`)."""
|
||||
return agent.enabled and bool(agent.cv_text.strip())
|
||||
|
||||
|
||||
def waiting(store: Store, keys: Sequence[str] | None = None,
|
||||
@@ -227,6 +208,9 @@ def pending(store: Store) -> int:
|
||||
def system(agent: Agent) -> str:
|
||||
"""The rubric, and the person as this agent describes them."""
|
||||
parts = [SYSTEM, "", "The person.", "", f"<cv>\n{agent.cv_text.strip()}\n</cv>"]
|
||||
if agent.cv_languages:
|
||||
spoken = ", ".join(f"{_language_name(l.language)} {l.level}" for l in agent.cv_languages)
|
||||
parts.append(f"\n<languages title=\"the languages they speak, and how well\">\n{spoken}\n</languages>")
|
||||
wants = "\n".join(f"- {p.name}: {' '.join(p.text.split())}"
|
||||
for p in agent.prompts if p.text.strip())
|
||||
if wants:
|
||||
@@ -237,9 +221,6 @@ def system(agent: Agent) -> str:
|
||||
if agent.avoid_text.strip():
|
||||
parts.append("\n<dislikes title=\"what they do not like\">\n"
|
||||
f"{agent.avoid_text.strip()}\n</dislikes>")
|
||||
if musts := must_lines(agent.must_text):
|
||||
lines = "\n".join(f"M{n}: {line}" for n, line in enumerate(musts, start=1))
|
||||
parts.append(f"\n<must title=\"must-haves and deal-breakers\">\n{lines}\n</must>")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
@@ -269,10 +250,16 @@ def build_batch(config: Config, store: Store, limit: int = 5,
|
||||
|
||||
# -------------------------------------------------------------------- answer
|
||||
|
||||
#: The top of the judge's scale.
|
||||
TOP = 10
|
||||
|
||||
|
||||
def _mark(value: Any) -> int | None:
|
||||
"""One of the three numbers, rounded and held to 0..`TOP`; None for
|
||||
anything not a number."""
|
||||
if isinstance(value, bool) or not isinstance(value, int | float):
|
||||
return None
|
||||
return max(0, min(100, round(value)))
|
||||
return max(0, min(TOP, round(value)))
|
||||
|
||||
|
||||
def _lines(value: Any) -> list[str]:
|
||||
@@ -281,25 +268,6 @@ def _lines(value: Any) -> list[str]:
|
||||
return [line for line in found if line][:MOST]
|
||||
|
||||
|
||||
def _musts(agent: Agent | None, value: Any) -> list[dict[str, str]]:
|
||||
"""The verdicts on the agent's must-haves as they are stored: one
|
||||
`{id, text, verdict}` per line of its must text, in order. A must-have
|
||||
the model left out, or gave a verdict outside "met", "broken", "unclear",
|
||||
is "unclear": nothing is broken by silence. An id that is not one of the
|
||||
agent's is ignored, the first answer for an id counts."""
|
||||
said: dict[str, str] = {}
|
||||
for one in value if isinstance(value, list) else []:
|
||||
if isinstance(one, dict):
|
||||
said.setdefault(str(one.get("id") or "").strip().upper(),
|
||||
str(one.get("verdict") or "").strip().lower())
|
||||
found = []
|
||||
for n, line in enumerate(must_lines(agent.must_text if agent else ""), start=1):
|
||||
verdict = said.get(f"M{n}", "")
|
||||
found.append({"id": f"M{n}", "text": line,
|
||||
"verdict": verdict if verdict in MUST_VERDICTS else "unclear"})
|
||||
return found
|
||||
|
||||
|
||||
def apply_batch(store: Store, answer: dict[str, Any],
|
||||
batch: dict[str, Any] | None = None,
|
||||
config: Config | None = None) -> JudgeResult:
|
||||
@@ -308,7 +276,6 @@ def apply_batch(store: Store, answer: dict[str, Any],
|
||||
all three numbers; the pair stays waiting."""
|
||||
keys: dict[str, str] = dict((batch or {}).get("keys") or {})
|
||||
stats = JudgeResult()
|
||||
agents: dict[int, Agent | None] = {}
|
||||
for one in answer.get("judgements") or []:
|
||||
key = str(one.get("key") or "")
|
||||
pair = pair_of(key)
|
||||
@@ -317,11 +284,8 @@ def apply_batch(store: Store, answer: dict[str, Any],
|
||||
stats.rejected.append(key or "?")
|
||||
continue
|
||||
skills, experience, career = (int(m) for m in marks if m is not None)
|
||||
if pair[0] not in agents:
|
||||
agents[pair[0]] = store.agent(pair[0])
|
||||
if not store.set_judgement(pair[0], pair[1], keys.pop(key), skills, experience, career,
|
||||
_lines(one.get("strengths")), _lines(one.get("gaps")),
|
||||
_musts(agents[pair[0]], one.get("musts"))):
|
||||
_lines(one.get("strengths")), _lines(one.get("gaps"))):
|
||||
stats.rejected.append(key)
|
||||
continue
|
||||
stats.judged += 1
|
||||
|
||||
+17
-3
@@ -10,7 +10,10 @@ both; nothing here decides how a board is read, so the words never disagree
|
||||
with the pull as long as they read the keys the pull writes.
|
||||
|
||||
A word with a number carries it after a colon ("without text: 12"), and the
|
||||
filter matches the phrase before the colon (`WORDS`).
|
||||
filter matches the phrase before the colon (`WORDS`). One word is outside
|
||||
`WORDS`: the listings the last read dropped for want of an id, a title or a
|
||||
link ("3 listings dropped: no title or link", `_runner.valid`), shown on the
|
||||
board's row and not offered by the filter.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -49,7 +52,7 @@ WORDS: dict[str, str] = {
|
||||
|
||||
#: The `state` keys `words` reads, `<prefix>:<key>:<target>`.
|
||||
KEYS = ("failing", "ok", "complete", "swept", "sweep", "filtered", "remote_unasked",
|
||||
"narrowed", "detailfailed")
|
||||
"narrowed", "detailfailed", "dropped")
|
||||
|
||||
|
||||
class Texts(NamedTuple):
|
||||
@@ -84,7 +87,8 @@ def texts(store: Store) -> dict[int, Texts]:
|
||||
def words(module: ModuleType, state: Mapping[str, str], target: str,
|
||||
sub: Subscription | None, size: int | None, *, plan: Plan | None = None,
|
||||
line: str = "", counts: Texts | None = None) -> list[str]:
|
||||
"""The limits of one board, in the order of `WORDS`, those that apply.
|
||||
"""The limits of one board, in the order of `WORDS`, those that apply, the
|
||||
rows the last read dropped after "read incomplete".
|
||||
|
||||
`state` holds the board's keys (`state_of`), `size` its kept size,
|
||||
`plan` and `sub` the reach it is read with (for "read whole"), `line` the
|
||||
@@ -105,6 +109,8 @@ def words(module: ModuleType, state: Mapping[str, str], target: str,
|
||||
out.append("failing")
|
||||
if state.get(key("ok")) and not state.get(key("complete")):
|
||||
out.append("read incomplete")
|
||||
if dropped := _count(state.get(key("dropped"))):
|
||||
out.append(f"{dropped} listing{'' if dropped == 1 else 's'} dropped: no title or link")
|
||||
if state.get(key("swept")) == "1" or key("sweep") in state:
|
||||
out.append("swept")
|
||||
if state.get(key("filtered")) == "1":
|
||||
@@ -130,6 +136,14 @@ def words(module: ModuleType, state: Mapping[str, str], target: str,
|
||||
return out
|
||||
|
||||
|
||||
def _count(value: str | None) -> int:
|
||||
"""A number kept in state, 0 for none or one that does not parse."""
|
||||
try:
|
||||
return max(0, int(value or 0))
|
||||
except ValueError:
|
||||
return 0
|
||||
|
||||
|
||||
def _whole(reads: Reads, sub: Subscription | None, size: int | None, plan: Plan | None) -> bool:
|
||||
"""Whether the reach leaves the board unread by place: its reader cannot
|
||||
filter, or the board is under `SMALL` roles (the line's two "read whole"
|
||||
|
||||
+49
-37
@@ -37,16 +37,23 @@ not swept clears them.
|
||||
A read narrowed by the agents' fetch terms (`ats:narrowed:<target>`) lists
|
||||
the roles matching a word, which says nothing of the roles it did not list,
|
||||
so it closes by list nothing: a role of such a board closes once the board
|
||||
has not listed it for `STALE_DAYS`, the rule searches have.
|
||||
has not listed it for `STALE_DAYS`, the rule searches have. The same age
|
||||
rule is the floor under every board (ADR Fetch prep and score 1.3): an
|
||||
incomplete read (a ceiling, a page cap without a sweep) and a filtered read,
|
||||
which spares the roles acted on and the remote ones it could not ask for,
|
||||
close by age what no read has listed for `STALE_DAYS` (`_close_aged`), so no
|
||||
role stays open for good.
|
||||
|
||||
A search closes the same way when it listed everything that matches (see
|
||||
`plugins/search.py` for how each provider knows). One that did not closes its
|
||||
roles only once no search has seen them for `STALE_DAYS`. A board or a
|
||||
search that has failed on every fetch for `STALE_DAYS` (`ats:failing:<target>`,
|
||||
`search:failing:<target>`) closes all of its roles, and its next answer
|
||||
reopens what it lists. A target that raises in its batch after its read, a
|
||||
row that does not store, is recorded as failing the same way and the batch
|
||||
goes on with the next one (`run_batch`).
|
||||
A search never closes by its list, even one stamped complete because it
|
||||
read as many results as its provider stated: a search is never a company's
|
||||
whole list, so a role that drops out of its results may still be open. Its
|
||||
roles close once no search has listed them for `STALE_DAYS`
|
||||
(`Store.close_unseen`). A board or a search that has failed on every fetch
|
||||
for `STALE_DAYS` (`ats:failing:<target>`, `search:failing:<target>`) closes
|
||||
all of its roles, and its next answer reopens what it lists. A target that
|
||||
raises in its batch after its read, a row that does not store, is recorded
|
||||
as failing the same way and the batch goes on with the next one
|
||||
(`run_batch`).
|
||||
|
||||
The settle says what needs a look in one notice (`settled`): the targets
|
||||
that did not answer or were read incomplete, and the boards that closed more
|
||||
@@ -302,11 +309,12 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
|
||||
had already parsed stamped — see the module docstring. A board whose
|
||||
`ats:ok:<target>` was not written during *this* run — it failed, or its
|
||||
whole batch job never got to it — is never closed by its list, and a board
|
||||
that answered incompletely is not either. A search closes by its list only
|
||||
when that list was complete, and otherwise by `STALE_DAYS`. A board or a
|
||||
search that has failed on every fetch since `STALE_DAYS` ago closes all
|
||||
its roles (`failing_boards`). How a read that answered closes is
|
||||
`_close_read`'s.
|
||||
that answered incompletely is not either. A search never closes by its
|
||||
list, complete or not: its roles close once no search has listed them
|
||||
for `STALE_DAYS` (`Store.close_unseen`, run when a search answered). A
|
||||
board or a search that has failed on every fetch since `STALE_DAYS` ago
|
||||
closes all its roles (`failing_boards`). How a board's read that
|
||||
answered closes is `_close_read`'s.
|
||||
|
||||
The stamps are compared with this run's own marker, the one its batches
|
||||
carried, never with the stored `pull:run`, which a later run may have
|
||||
@@ -350,18 +358,22 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
|
||||
unanswered.append(sub.target)
|
||||
since = failing.get(sub.target, ("", ""))[0]
|
||||
shut = store.close_subscription(sub.id) if since and since < stale else 0
|
||||
elif source == "search":
|
||||
# Never a company's whole list, complete or not: a search's
|
||||
# roles close by age alone (`close_unseen`, after the loop).
|
||||
searched += 1
|
||||
continue
|
||||
else:
|
||||
if source == "search":
|
||||
searched += 1
|
||||
else:
|
||||
answered += 1
|
||||
answered += 1
|
||||
read = _close_read(store, source, sub, stale)
|
||||
if read is None:
|
||||
if source == "ats":
|
||||
incomplete += 1
|
||||
if not store.state.get(f"ats:sweep:{sub.target}"):
|
||||
partial.append(sub.target)
|
||||
continue
|
||||
incomplete += 1
|
||||
if not store.state.get(f"ats:sweep:{sub.target}"):
|
||||
partial.append(sub.target)
|
||||
# An incomplete read closes nothing by its list; what no
|
||||
# read has listed for STALE_DAYS still goes.
|
||||
read = _close_aged(store, source, sub.id, json.loads(
|
||||
store.state.get(f"{source}:ids:{sub.target}", "[]")), stale)
|
||||
shut = read
|
||||
closed += shut
|
||||
before = open_before.get(sub.id, 0)
|
||||
@@ -380,8 +392,9 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
|
||||
|
||||
|
||||
def _close_read(store: Store, source: str, sub: Any, stale: str) -> int | None:
|
||||
"""Close what one target's read of this run says has closed, and return
|
||||
how many; None when the read was incomplete, which closes nothing.
|
||||
"""Close what one board's read of this run says has closed, and return
|
||||
how many; None when the read was incomplete, which closes nothing. A
|
||||
search's read never comes here (see `close_boards`).
|
||||
|
||||
A read narrowed by terms closes only the roles it has not listed for
|
||||
`STALE_DAYS` (`_close_aged`). A read that finished a sweep closes what
|
||||
@@ -389,8 +402,9 @@ def _close_read(store: Store, source: str, sub: Any, stale: str) -> int | None:
|
||||
clears that record and closes what it did not list, all of it unfiltered
|
||||
(`Store.mark_closed`), filtered sparing the roles acted on
|
||||
(`_close_unlisted`). A filtered read that could not ask for remote roles
|
||||
(`remote_unasked`) spares the remote ones as well, and closes those by
|
||||
age instead."""
|
||||
(`remote_unasked`) spares the remote ones as well. What a filtered read
|
||||
spares closes by age (`_close_aged`), as on an incomplete read
|
||||
(`close_boards`)."""
|
||||
state, target = store.state, sub.target
|
||||
if not state.get(f"{source}:complete:{target}", ""):
|
||||
return None
|
||||
@@ -399,8 +413,7 @@ def _close_read(store: Store, source: str, sub: Any, stale: str) -> int | None:
|
||||
return _close_aged(store, source, sub.id, seen_ids, stale)
|
||||
is_filtered = bool(state.get(f"{source}:filtered:{target}", ""))
|
||||
spare_remote = is_filtered and bool(state.get(f"{source}:remote_unasked:{target}", ""))
|
||||
aged = (_close_aged(store, source, sub.id, seen_ids, stale, remote_only=True)
|
||||
if spare_remote else 0)
|
||||
aged = _close_aged(store, source, sub.id, seen_ids, stale) if is_filtered else 0
|
||||
if state.get(f"{source}:swept:{target}", ""):
|
||||
return aged + _close_swept(store, source, sub, seen_ids, is_filtered,
|
||||
spare_remote=spare_remote)
|
||||
@@ -479,20 +492,19 @@ def _close_unlisted(store: Store, source: str, seen_ids: list[str], subscription
|
||||
|
||||
|
||||
def _close_aged(store: Store, source: str, subscription_id: int, seen_ids: list[str],
|
||||
stale: str, *, remote_only: bool = False) -> int:
|
||||
stale: str) -> int:
|
||||
"""Close the open roles of one board that its read did not list and that
|
||||
no read has listed since `stale` (`roles.last_seen`, which `upsert_role`
|
||||
writes on every listing): the rule for what a read cannot speak for. A
|
||||
read narrowed by terms closes all such roles, as a search does; with
|
||||
`remote_only`, a filtered read that could not ask for remote closes only
|
||||
its remote roles so, sparing those acted on as its list does."""
|
||||
where = " AND remote = 'remote'" + _ACTED_ON if remote_only else ""
|
||||
writes on every listing): the rule for what a read cannot speak for, a
|
||||
read narrowed by terms, an incomplete one, or what a filtered read
|
||||
spares. A role acted on is not spared here: three weeks unlisted is the
|
||||
one evidence a filtered or incomplete read gives."""
|
||||
with store.transaction() as conn:
|
||||
return conn.execute(
|
||||
f"""UPDATE roles SET closed_at = ?
|
||||
"""UPDATE roles SET closed_at = ?
|
||||
WHERE source = ? AND subscription_id = ? AND closed_at IS NULL
|
||||
AND last_seen < ?
|
||||
AND source_id NOT IN (SELECT value FROM json_each(?)){where}""",
|
||||
AND source_id NOT IN (SELECT value FROM json_each(?))""",
|
||||
(store.now(), source, subscription_id, stale,
|
||||
json.dumps(list(seen_ids)))).rowcount
|
||||
|
||||
|
||||
+48
-17
@@ -10,7 +10,7 @@ on. Computed once per pull (`plan`) and handed to each board's batch as the
|
||||
empty, or holds any other member (a function, a remote value other than
|
||||
`remote`, a place the gazetteer does not know), can take a role from
|
||||
anywhere, so the base reach is then everywhere.
|
||||
- **Wide reach, for the companies set to All countries**
|
||||
- **Wide reach, for the companies set to All locations**
|
||||
(`companies.all_countries`, global): the union over the enabled agents of
|
||||
their `require` reach, the intersection of each one's `require` gates on
|
||||
country and place, Europe today, expanded to its countries through the
|
||||
@@ -272,7 +272,7 @@ class Plan:
|
||||
def why(self, sub: Subscription) -> tuple[Reach, str]:
|
||||
"""The reach `sub` is read with, and why: `override`, `small`,
|
||||
`unknown` (a board never read whole), `countries` (its company is
|
||||
set to All countries) or `agents`."""
|
||||
set to All locations) or `agents`."""
|
||||
if OVERRIDE_KEYS & set(sub.params or {}):
|
||||
return EVERYWHERE, "override"
|
||||
size = self.sizes.get(sub.target)
|
||||
@@ -283,9 +283,18 @@ class Plan:
|
||||
return self._big(sub)
|
||||
|
||||
def _big(self, sub: Subscription) -> tuple[Reach, str]:
|
||||
"""The reach of `sub` as a board of `SMALL` roles or more."""
|
||||
"""The reach of `sub` as a board of `SMALL` roles or more. A host
|
||||
that cannot filter on remote while the agents want remote roles is
|
||||
asked for the wide reach, the agents' world (ADR Fetch prep and
|
||||
score 1.2): asked for "DK, SE" alone it would never list a remote
|
||||
role filed under Germany, and the gate sorts the rest out. Only
|
||||
where the host takes the world in one read, a list of countries or
|
||||
their ids: on a host that takes one country a request, Europe is
|
||||
the whole board, so the location read stays and remote is not asked."""
|
||||
if company_key(sub.company or sub.label) in self.all_countries:
|
||||
return self.wide, "countries"
|
||||
if self.base.remote and not _filters_remote(sub) and _takes_world(sub, self.wide):
|
||||
return self.wide, "world"
|
||||
return self.base, "agents"
|
||||
|
||||
def narrows_roles(self, sub: Subscription) -> bool:
|
||||
@@ -296,7 +305,7 @@ class Plan:
|
||||
def for_subscription(self, sub: Subscription) -> dict[str, Any] | None:
|
||||
"""The `reach` a board's batch hands `ask_for`; None for none. With
|
||||
fetch terms, and unless its company is set to All roles, a big board
|
||||
carries them, and `wide` when its company is set to All countries, for
|
||||
carries them, and `wide` when its company is set to All locations, for
|
||||
`ask_for` to decide which side gives way over `MAX_SERIES` (see the
|
||||
module docstring). A board with no size carries
|
||||
`unsized` and `small`, and the reach of a big board unless its params
|
||||
@@ -305,15 +314,34 @@ class Plan:
|
||||
if why == "unknown":
|
||||
found, why = self._big(sub)
|
||||
out = found.params()
|
||||
if self.terms and why in ("countries", "agents") and self.narrows_roles(sub):
|
||||
if self.terms and why in ("countries", "agents", "world") and self.narrows_roles(sub):
|
||||
out = (out or {}) | {"terms": list(self.terms)}
|
||||
if why == "countries":
|
||||
if why in ("countries", "world"):
|
||||
out["wide"] = True
|
||||
if sub.target not in self.sizes:
|
||||
out = (out or {}) | {"unsized": True, "small": SMALL}
|
||||
return out
|
||||
|
||||
|
||||
def _filters_remote(sub: Subscription) -> bool:
|
||||
"""Whether `sub`'s host declares a remote filter at all. A host that
|
||||
declares one by ids may still lack the facet on a site, which the
|
||||
runner finds out when it asks (`_runner.ask_for`)."""
|
||||
module = readers.HOSTS.get(sub.target.partition(":")[0])
|
||||
return module is None or module.READS.filters.remote != "none"
|
||||
|
||||
|
||||
def _takes_world(sub: Subscription, wide: Reach) -> bool:
|
||||
"""Whether `sub`'s host can be asked for the wide reach in one read: a
|
||||
country filter that takes a list (`many`, `ids`), or one value a request
|
||||
with the countries within `MAX_SERIES`; never everywhere."""
|
||||
module = readers.HOSTS.get(sub.target.partition(":")[0])
|
||||
if module is None or wide.everywhere or not wide.countries:
|
||||
return False
|
||||
reach = module.READS.filters.country
|
||||
return reach in ("many", "ids") or (reach == "one" and len(wide.countries) <= _runner.MAX_SERIES)
|
||||
|
||||
|
||||
def sizes(store: Store, subs: Iterable[Subscription]) -> dict[str, int]:
|
||||
"""Each board's size as kept (`ats:size:<target>`); a board never sized
|
||||
has none."""
|
||||
@@ -360,7 +388,7 @@ def plan(store: Store, subs: Iterable[Subscription] | None = None) -> Plan:
|
||||
base = wide = EVERYWHERE
|
||||
subs = list(store.subscriptions(source="ats") if subs is None else subs)
|
||||
rows = store.companies()
|
||||
# All countries never reads less than the board would without it: the
|
||||
# All locations never reads less than the board would without it: the
|
||||
# base's remote, which a `require` of Europe does not name, is kept.
|
||||
return Plan(base, wide | base, sizes(store, subs),
|
||||
all_countries=frozenset(c.key for c in rows if c.all_countries),
|
||||
@@ -380,8 +408,8 @@ def reaches(store: Store, subs: Iterable[Subscription]) -> Mapping[str, dict[str
|
||||
|
||||
#: Why a board is read whole, as the Companies list says it.
|
||||
_WHOLE = {"small": f"under {SMALL} roles", "unknown": "size not known yet",
|
||||
"override": "set here", "countries": "all countries (everywhere)",
|
||||
"agents": "an agent takes any place"}
|
||||
"override": "set here", "countries": "all locations (everywhere)",
|
||||
"world": "the agents' world (everywhere)", "agents": "an agent takes any place"}
|
||||
|
||||
|
||||
def _codes(codes: Iterable[str]) -> list[str]:
|
||||
@@ -452,8 +480,8 @@ def line(found: Plan, sub: Subscription) -> str:
|
||||
"follows agents: DK, SE, remote · follows agents' terms: engineer,
|
||||
product", "follows agents: DK, SE, remote · all roles" (a company set to
|
||||
All roles, or a reader without a query, or terms over `MAX_SERIES`),
|
||||
"all countries (Europe) · follows agents' terms: engineer, product,
|
||||
data", "all countries (Europe) · all roles", "set here: DE · all roles",
|
||||
"all locations (Europe) · follows agents' terms: engineer, product,
|
||||
data", "all locations (Europe) · all roles", "set here: DE · all roles",
|
||||
"all places, too many countries to ask · follows agents' terms:
|
||||
engineer", or, for a board read whole, why: "under 300 roles: read
|
||||
whole", "Greenhouse cannot filter: read whole". A board with no size yet
|
||||
@@ -485,23 +513,26 @@ def line(found: Plan, sub: Subscription) -> str:
|
||||
else "")
|
||||
if why == "countries":
|
||||
named = found.wide_names or f"{len(found.wide.countries)} countries"
|
||||
return f"all countries ({named}){unasked} · {roles}"
|
||||
return f"all locations ({named}){unasked} · {roles}"
|
||||
if why == "world":
|
||||
named = found.wide_names or f"{len(found.wide.countries)} countries"
|
||||
return f"follows agents' world ({named}){unasked} · {roles}"
|
||||
if why == "override":
|
||||
return f"set here: {_places(ask)}{unasked} · {roles}"
|
||||
return f"follows agents: {_places(ask)}{unasked} · {roles}"
|
||||
if ask.terms:
|
||||
if given.everywhere:
|
||||
what = "all countries (everywhere)" if why == "countries" else _WHOLE[why]
|
||||
what = "all locations (everywhere)" if why == "countries" else _WHOLE[why]
|
||||
return f"{what} · {roles}"
|
||||
return f"all places, too many countries to ask · {roles}"
|
||||
if why in ("agents", "countries") and given.everywhere:
|
||||
if why in ("agents", "countries", "world") and given.everywhere:
|
||||
return f"{_WHOLE[why]}: read whole"
|
||||
if why in ("agents", "countries", "override") and not module.READS.filters.declared():
|
||||
if why in ("agents", "countries", "world", "override") and not module.READS.filters.declared():
|
||||
return f"{module.LABEL} cannot filter: read whole"
|
||||
reads = module.READS.filters
|
||||
if why in ("agents", "countries") and reads.country == "one" \
|
||||
if why in ("agents", "countries", "world") and reads.country == "one" \
|
||||
and len(given.countries) > _runner.MAX_SERIES:
|
||||
return "too many countries to ask: read whole"
|
||||
if why in ("agents", "countries"):
|
||||
if why in ("agents", "countries", "world"):
|
||||
return f"{module.LABEL} cannot ask for these: read whole"
|
||||
return f"{_WHOLE[why]}: read whole"
|
||||
+19
-9
@@ -9,16 +9,19 @@ good for a given search. Each agent first applies its fetch gate:
|
||||
lifts
|
||||
any gates one group, the place part: a role that matches none of
|
||||
them, and has the column filled for at least one, is
|
||||
gated ("none of country DK / remote remote"); skipped for
|
||||
a company set to All countries
|
||||
gated ("none of country DK / remote remote"), unless its
|
||||
location is unknown; skipped for a company set to All
|
||||
locations
|
||||
exclude gates a role that matches one of the values keeps its row, with
|
||||
score 0 and the value ("title account executive"); those
|
||||
on `title` are skipped for a company set to All roles
|
||||
|
||||
A gate rejects on evidence, never on absence: a role whose column is empty
|
||||
is not gated by it. `extract` is what fills an empty column, and it reads
|
||||
only the roles in the cut, so gating on an empty column would keep those
|
||||
roles out of extract's reach for good. A hidden company (`affinity = -1`)
|
||||
is not gated by it, and a place the gazetteer could not resolve, or one
|
||||
that holds the agent's places (Øresund for Copenhagen), is no evidence.
|
||||
`extract` is what fills an empty column, and it reads only the roles in
|
||||
the cut, so gating on an empty column would keep those roles out of
|
||||
extract's reach for good. A hidden company (`affinity = -1`)
|
||||
is not scored at all; a favourite passes the gate like any other company.
|
||||
|
||||
The final score, the feed's order, is the weighted sum of four terms over
|
||||
@@ -42,8 +45,8 @@ all there is, when it is gated as "no vector". An agent with no votes scores
|
||||
every role without `taste`; the direction is newsdesk's
|
||||
`centroid(liked) - centroid(disliked)`. An agent with no prompt that has
|
||||
text has no `fit` either, so nothing of it enters the cut by fit and its
|
||||
scores do not decide what summarise reads (`Store._summarise_top`); a run
|
||||
names such agents in one notice.
|
||||
want order does not decide what summarise reads (`Store._summarise_top`); a
|
||||
run names such agents in one notice.
|
||||
|
||||
Beside the score, `fit` is stored as it is, and for an agent with a CV,
|
||||
`cv_fit`: how close the role is to the CV, the best cosine between the role
|
||||
@@ -181,7 +184,11 @@ def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
|
||||
it fails, every exclude gate it matches, and the `any` group when it
|
||||
matches none of them. A gate with an empty column (state -1) neither
|
||||
passes nor fails, so the group fails only when some member had evidence.
|
||||
A company set to All countries skips the `any` group, the place part,
|
||||
A role whose location is unknown, every `country` and `place` member
|
||||
empty, passes the group whatever its remote says: a hybrid role with no
|
||||
place may still be in one of the agent's towns, and extract reads it
|
||||
only in the cut. A group of remote alone still fails a hybrid role.
|
||||
A company set to All locations skips the `any` group, the place part,
|
||||
and one set to All roles the exclude gates on `title`, the titles to
|
||||
leave out; the require gates, the region, apply to every company."""
|
||||
states = list(zip(gates, candidate.gates, strict=True))
|
||||
@@ -190,7 +197,10 @@ def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
|
||||
if g.mode == "exclude" and state > 0
|
||||
and not (candidate.all_roles and g.field == "title")]
|
||||
group = [(g, state) for g, state in states if g.mode == "any"]
|
||||
if not candidate.all_countries and group and not any(state == 1 for _, state in group) \
|
||||
located = [state for g, state in group if g.field in ("country", "place")]
|
||||
unknown = bool(located) and all(state == -1 for state in located)
|
||||
if not candidate.all_countries and group and not unknown \
|
||||
and not any(state == 1 for _, state in group) \
|
||||
and any(state == 0 for _, state in group):
|
||||
reasons.append("none of " + ", ".join(_wanted(g) for g, _ in group))
|
||||
return reasons
|
||||
|
||||
@@ -7,8 +7,10 @@ answer), and running it is `desk.modeltask`'s.
|
||||
|
||||
A role is waiting while it has no summary, has some text, is open and no copy,
|
||||
and is either in the top of an enabled agent's feed, the best `summarise_top`
|
||||
final scores among the roles its fetch gate passes (its `limits`), or one the
|
||||
owner cares about whatever its rank: saved (it has an application, the star)
|
||||
(its `limits`, 30 by default) by the want feed score and the best
|
||||
`summarise_top` by the match order (CV match, then CV closeness) among the
|
||||
roles its fetch gate passes, so at most twice that many, or one the owner
|
||||
cares about whatever its rank: saved (it has an application, the star)
|
||||
or voted up by an enabled agent. Summarise is no part of any score and runs
|
||||
last in the pull, once the final scores are set. The limit keeps the cost
|
||||
flat: a pull summarises the roles somebody will look at, not the 30k that are
|
||||
|
||||
+26
-72
@@ -1008,76 +1008,30 @@ def language_flag(required: list[RoleLanguage],
|
||||
return "", ""
|
||||
|
||||
|
||||
#: What career counts as at most when a posting breaks one of the agent's
|
||||
#: must-haves: the top of the rubric's 0-39 band for a dead end. The model
|
||||
#: reports whether each must-have is met; this cap is applied in code, when a
|
||||
#: role is served (`cv_score`, `Store.judgement`, `store._cv_sql`).
|
||||
BROKEN_CAP = 39
|
||||
|
||||
#: A must-have's verdict, as the judge answers it.
|
||||
MUST_VERDICTS = ("met", "broken", "unclear")
|
||||
|
||||
_BULLET = re.compile(r"^\s*(?:[-*\u2022]+|\(?\d+[.)])\s+")
|
||||
|
||||
|
||||
def must_lines(text: str) -> list[str]:
|
||||
"""The agent's must-haves text read as one must-have per line: a leading
|
||||
bullet (`-`, `*`, `•`) or number (`1.`, `2)`) is dropped, so are empty
|
||||
lines."""
|
||||
found = []
|
||||
for line in text.splitlines():
|
||||
line = " ".join(_BULLET.sub("", line, count=1).split())
|
||||
if line:
|
||||
found.append(line)
|
||||
return found
|
||||
|
||||
|
||||
def capped_career(career: int, broken: int = 0) -> int:
|
||||
"""`career` as it counts: at most `BROKEN_CAP` when any must-have is
|
||||
broken."""
|
||||
return min(career, BROKEN_CAP) if broken else career
|
||||
|
||||
|
||||
def cv_score(skills: int, experience: int, career: int,
|
||||
weights: Weights | None = None, broken: int = 0) -> float | None:
|
||||
"""The one number of a CV match, 0 to 100: the three numbers of the
|
||||
judge averaged by the agent's `cv_skills`, `cv_experience` and
|
||||
`cv_career` weights (30, 25 and 30 by default), career as
|
||||
`capped_career` has it. None when all three weights are 0."""
|
||||
weights: Weights | None = None) -> float | None:
|
||||
"""The one number of a CV match, 0 to 100 for sorting and display: the
|
||||
judge's three numbers, each 0 to 10, averaged by the agent's
|
||||
`cv_skills`, `cv_experience` and `cv_career` weights (30, 25 and 30 by
|
||||
default), times 10. None when all three weights are 0."""
|
||||
w = weights or Weights()
|
||||
total = w.cv_skills + w.cv_experience + w.cv_career
|
||||
if not total:
|
||||
return None
|
||||
career = capped_career(career, broken)
|
||||
return round((w.cv_skills * skills + w.cv_experience * experience
|
||||
+ w.cv_career * career) / total, 1)
|
||||
|
||||
|
||||
class Must(Model):
|
||||
"""One must-have of an agent as the judge read it against a role: its
|
||||
`id` (`M1`, by line), the line's `text` and the `verdict`."""
|
||||
|
||||
id: str
|
||||
text: str = ""
|
||||
verdict: Literal["met", "broken", "unclear"] = "unclear"
|
||||
return round(10 * (w.cv_skills * skills + w.cv_experience * experience
|
||||
+ w.cv_career * career) / total, 1)
|
||||
|
||||
|
||||
class Judgement(Model):
|
||||
"""What `judge` read of one role for one agent (`role_judgements`): the
|
||||
three dimensions 0 to 100 as the model gave them, `cv_score`
|
||||
(`cv_score`) and 1 to 3 strengths and gaps grounded in the posting.
|
||||
`musts` is the verdict on each must-have and `broken` how many are
|
||||
broken; any makes `career_capped` the lower of `career` and
|
||||
`BROKEN_CAP`. `cv_score` and `career_capped` are not read from the
|
||||
table: `Store.judgement` sets them from the agent's weights; `cv_score`
|
||||
is None when all three are 0."""
|
||||
three dimensions 0 to 10 as the model gave them, `cv_score`
|
||||
(`cv_score`, 0 to 100) and 1 to 3 strengths and gaps grounded in the
|
||||
posting. `cv_score` is not read from the table: `Store.judgement` sets
|
||||
it from the agent's weights; it is None when all three are 0."""
|
||||
|
||||
skills: int
|
||||
experience: int
|
||||
career: int
|
||||
career_capped: int | None = None
|
||||
broken: int = 0
|
||||
musts: Annotated[list[Must], BeforeValidator(lambda v: _json(v, "[]"))] = []
|
||||
cv_score: float | None = None
|
||||
strengths: JsonStrings = []
|
||||
gaps: JsonStrings = []
|
||||
@@ -1387,7 +1341,8 @@ class Company(Model):
|
||||
#: not on that list.
|
||||
hand_reason: str = ""
|
||||
#: The two fetch overrides (`app/jobs/reach.py`), global rather than an
|
||||
#: agent's, since fetching is shared. `all_countries` 1: every big board
|
||||
#: agent's, since fetching is shared. `all_countries` 1 (All locations,
|
||||
#: as the switch is called; the column keeps its name): every big board
|
||||
#: of the company is read with the agents' wide reach, Europe today,
|
||||
#: rather than their places. `all_roles` 1: no role-type narrowing, no
|
||||
#: fetch terms today, on any of its boards.
|
||||
@@ -1459,7 +1414,7 @@ class Gate(Model):
|
||||
fails it (the row is kept with the reason); the `any` gates together are
|
||||
one group a role must match at least one of ("in Denmark, or remote");
|
||||
`exclude` gates a role that matches any of its values ("title account
|
||||
executive"). A company set to All countries skips the `any` group and one
|
||||
executive"). A company set to All locations skips the `any` group and one
|
||||
set to All roles skips the `exclude` gates on `title` (`score.failed_gates`).
|
||||
A gate rejects on evidence only: a role whose column is empty is not
|
||||
rejected, since `extract` fills the column later for the roles in the
|
||||
@@ -1497,14 +1452,16 @@ class Limits(Model):
|
||||
leaves out, then every role liked or saved. Both levels are calibrated
|
||||
(`score.scale`) with the company's affinity as a small bonus, so a
|
||||
minimum means the same in every run. For a broad agent the count stops a
|
||||
list, for a narrow one the minimum. Summarise reads the best
|
||||
`summarise_top` by final score, and every role liked or saved."""
|
||||
list, for a narrow one the minimum. Summarise reads the top
|
||||
`summarise_top` of the want feed score (the score order) and the top
|
||||
`summarise_top` of the match order (CV match, then CV closeness), so at
|
||||
most twice as many, and every role liked or saved."""
|
||||
|
||||
cut_fit_top: int = Field(500, ge=0)
|
||||
cut_fit_min: float = Field(0.6, ge=0, le=1)
|
||||
cut_cv_top: int = Field(150, ge=0)
|
||||
cut_cv_min: float = Field(0.6, ge=0, le=1)
|
||||
summarise_top: int = Field(50, ge=0)
|
||||
summarise_top: int = Field(30, ge=0)
|
||||
|
||||
|
||||
class Prompt(Model):
|
||||
@@ -1568,14 +1525,11 @@ class Agent(Model):
|
||||
are the union over the enabled agents, and an agent with none means no
|
||||
board is narrowed by words. They decide what is fetched, not what ranks.
|
||||
|
||||
`cv_text`, `cv_languages`, `like_text`, `avoid_text` and `must_text` are
|
||||
what `judge` reads a role against: the CV, the languages it states with
|
||||
a level, what the person likes and does not like (the work, the
|
||||
workplace, situations; both feed the career score), and their
|
||||
must-haves and deal-breakers (set through the API only, not on the
|
||||
form; soft, read by the model only; a hard rule is a gate). An agent without
|
||||
a CV is not judged, and one without languages flags none
|
||||
(`language_flag`)."""
|
||||
`cv_text`, `cv_languages`, `like_text` and `avoid_text` are what `judge`
|
||||
reads a role against: the CV, the languages it states with a level, and
|
||||
what the person likes and does not like (the work, the workplace,
|
||||
situations; both feed the career score). An agent without a CV is not
|
||||
judged, and one without languages flags none (`language_flag`)."""
|
||||
|
||||
id: int
|
||||
name: str
|
||||
@@ -1588,7 +1542,6 @@ class Agent(Model):
|
||||
cv_languages: CvLanguages = []
|
||||
like_text: str = ""
|
||||
avoid_text: str = ""
|
||||
must_text: str = ""
|
||||
enabled: bool = True
|
||||
created_at: str
|
||||
|
||||
@@ -1616,7 +1569,8 @@ class Candidate(Model):
|
||||
uid: str
|
||||
posted_at: str
|
||||
affinity: int = 0
|
||||
#: `companies.all_countries`: the `any` group (the place part) is skipped.
|
||||
#: `companies.all_countries`, All locations: the `any` group (the place
|
||||
#: part) is skipped.
|
||||
all_countries: bool = False
|
||||
#: `companies.all_roles`: the `exclude` gates on `title` are skipped.
|
||||
all_roles: bool = False
|
||||
|
||||
+7
-4
@@ -283,7 +283,8 @@ _SKANE = Region(
|
||||
|
||||
_HOVEDSTADEN = Region(
|
||||
"hovedstaden", "Capital Region of Denmark",
|
||||
_split("Capital Region of Denmark, Capital Region Denmark, Region Hovedstaden, Hovedstaden"),
|
||||
_split("""Capital Region of Denmark, Capital Region Denmark, Region Hovedstaden, Hovedstaden,
|
||||
Storkøbenhavn, Nordsjælland"""),
|
||||
_split("""København, Copenhagen, Frederiksberg, Albertslund, Allerød, Ballerup,
|
||||
Bornholm, Brøndby, Dragør, Egedal, Fredensborg, Frederikssund, Furesø, Gentofte,
|
||||
Gladsaxe, Glostrup, Gribskov, Halsnæs, Helsingør, Herlev, Hillerød, Hvidovre,
|
||||
@@ -303,7 +304,8 @@ _HOVEDSTADEN = Region(
|
||||
# The bare word is left out of `names` since "New Zealand" would match it.
|
||||
_ZEALAND = Region(
|
||||
"zealand", "Zealand",
|
||||
_split("Region Zealand, Region Sjælland, Zealand Denmark, Sjælland, Sjaelland"),
|
||||
_split("""Region Zealand, Region Sjælland, Zealand Denmark, Sjælland, Sjaelland,
|
||||
Østsjælland, Vestsjælland, Sydsjælland"""),
|
||||
_split("""Faxe, Greve, Guldborgsund, Holbæk, Kalundborg, Køge, Lejre, Lolland, Næstved,
|
||||
Odsherred, Ringsted, Roskilde, Slagelse, Solrød, Sorø, Stevns, Vordingborg,
|
||||
Nykøbing Falster, Nykøbing F, Nykøbing Sjælland, Nakskov, Maribo, Sakskøbing,
|
||||
@@ -316,7 +318,7 @@ _JUTLAND = Region(
|
||||
"jutland", "Jutland",
|
||||
_split("""Jutland, Jylland, Central Jutland, Midtjylland, Region Midtjylland,
|
||||
North Jutland, Nordjylland, Region Nordjylland, South Jutland, Sønderjylland,
|
||||
East Jutland, West Jutland, Vestjylland, Østjylland"""),
|
||||
East Jutland, West Jutland, Vestjylland, Østjylland, Sydøstjylland, Sydvestjylland"""),
|
||||
_split("""Aarhus, Aalborg, Esbjerg, Randers, Kolding, Horsens, Vejle, Silkeborg, Herning,
|
||||
Viborg, Billund, Fredericia, Holstebro, Skive, Ringkøbing, Videbæk, Lem, Risskov,
|
||||
Åbyhøj, Brabrand, Viby J, Aabenraa, Haderslev, Sønderborg, Grenaa, Grenå,
|
||||
@@ -882,7 +884,8 @@ _PLACELESS = frozenset((
|
||||
"remote", "hybrid", "anywhere", "worldwide", "global", "distributed", "home", "based",
|
||||
"from", "work", "working", "wfh", "office", "offices", "onsite", "on", "site", "in",
|
||||
"multiple", "several", "various", "locations", "location", "flexible", "or", "and", "any",
|
||||
"first", "field", "travel", "tbd", "n", "a", "other", "job", "jobs", "position"))
|
||||
"first", "field", "travel", "tbd", "n", "a", "other", "job", "jobs", "position",
|
||||
"udlandet", "ovrige"))
|
||||
|
||||
|
||||
def placeless(text: str) -> bool:
|
||||
|
||||
+6
-2
@@ -37,8 +37,9 @@ from app.storage import Store
|
||||
#: Raised when the rules in this package change the way a stored word maps,
|
||||
#: so every open role is mapped again as it is after a table changes. 3: the
|
||||
#: remote area, and remote as the model's before the host's word. 4: the
|
||||
#: deadline, the host's before the model's.
|
||||
VERSION = 4
|
||||
#: deadline, the host's before the model's. 5: `remote_text`, the title or
|
||||
#: text saying remote while no field does.
|
||||
VERSION = 5
|
||||
|
||||
#: The state key the last run's `fingerprint` is kept under.
|
||||
STATE = "prep:tables"
|
||||
@@ -61,6 +62,7 @@ def fingerprint(tables: Tables = TABLES) -> str:
|
||||
"""A hash of every host table, the shared remote words, the gazetteer the
|
||||
code describes and `VERSION`."""
|
||||
blob = json.dumps({"version": VERSION, "tables": tables, "remote": words.REMOTE_WORDS,
|
||||
"remote_text": [words.REMOTE_TEXT, words.REMOTE_TEXT_NOT],
|
||||
"places": repr(places.seed())}, sort_keys=True)
|
||||
return hashlib.sha1(blob.encode()).hexdigest()
|
||||
|
||||
@@ -114,6 +116,7 @@ def map_role(row: Mapping[str, Any], gaz: places.Gazetteer, folds: Mapping[str,
|
||||
"seniority_title": titled,
|
||||
"employment": words.employment(host, row["employment_host"], tables),
|
||||
"remote": remote,
|
||||
"remote_text": int(words.remote_in_text(row["title"], row["text"] or "")),
|
||||
"remote_area": list(area(gaz, texts, place_ids, _model_names(row["remote_where_model"]))
|
||||
if remote == "remote" else ()),
|
||||
"country": country(gaz, place_ids, row["country_host"]),
|
||||
@@ -125,6 +128,7 @@ def _changed(row: Mapping[str, Any], found: Mapping[str, Any]) -> bool:
|
||||
level = row["seniority_model"] or found["seniority_stated"] or found["seniority_title"]
|
||||
return bool(row["prep_due"]) or level != row["seniority"] \
|
||||
or found["remote"] != row["remote"] \
|
||||
or found["remote_text"] != row["remote_text"] \
|
||||
or found["remote_area"] != json.loads(row["remote_area"] or "[]") \
|
||||
or (found["country"] and found["country"] != row["country"]) \
|
||||
or found["deadline"] != row["deadline"] \
|
||||
|
||||
@@ -58,6 +58,39 @@ _NOT_REMOTE_WORK = re.compile(r"\bremote[\s-]+(?:sensing|control\w*|monitoring)\
|
||||
re.IGNORECASE)
|
||||
|
||||
|
||||
#: Phrases in a title or text that say the role is worked remotely, and the
|
||||
#: phrases that take one back. Both are read on the lowercased text with its
|
||||
#: hyphens and runs of spaces squashed to one space. A match marks the role as
|
||||
#: possibly remote (`remote_in_text`); it decides nothing by itself.
|
||||
REMOTE_TEXT: tuple[str, ...] = (
|
||||
"fully remote", "100% remote", "100 % remote", "remote first", "remote work",
|
||||
"remote working", "work remotely", "working remotely", "remote position", "remote role",
|
||||
"remote job", "remote team", "remote friendly", "remote possible", "remote option",
|
||||
"remote ok", "work from home", "working from home", "work from anywhere", "home based",
|
||||
"home office", "homeoffice", "hjemmearbejde", "fjernarbejde", "distansarbete",
|
||||
)
|
||||
REMOTE_TEXT_NOT: tuple[str, ...] = (
|
||||
"no remote", "not remote", "not a remote", "non remote", "not be remote", "cannot be remote",
|
||||
"no home office", "kein homeoffice", "not available remotely", "not open to remote",
|
||||
"remote work is not", "remote working is not", "is not a remote", "remote sensing",
|
||||
"remote control", "remote monitoring",
|
||||
)
|
||||
|
||||
_REMOTE_TEXT = re.compile("|".join(rf"\b{re.escape(p)}\b" for p in REMOTE_TEXT))
|
||||
_REMOTE_TEXT_NOT = re.compile("|".join(rf"\b{re.escape(p)}\b" for p in REMOTE_TEXT_NOT))
|
||||
_SQUASHED = re.compile(r"[\s-]+")
|
||||
|
||||
|
||||
def remote_in_text(title: str, text: str) -> bool:
|
||||
"""Whether the title or the text says the role is worked remotely: a
|
||||
phrase of `REMOTE_TEXT` left once those of `REMOTE_TEXT_NOT` are taken
|
||||
out. Cheap evidence, no model: a role marked so passes a location gate
|
||||
on remote until extract has read it (ADR Fetch prep and score 1.1), so a
|
||||
false mark costs a cut slot at most, and a missed one nothing new."""
|
||||
folded = _SQUASHED.sub(" ", f"{title or ''} {text or ''}".lower())
|
||||
return bool(_REMOTE_TEXT.search(_REMOTE_TEXT_NOT.sub(" ", folded)))
|
||||
|
||||
|
||||
def _line_word(text: str) -> str:
|
||||
"""`remote_of`'s reading of a line, else its Danish and Swedish words."""
|
||||
if found := remote_of(text):
|
||||
|
||||
+78
-107
@@ -1,19 +1,23 @@
|
||||
/* The Search agents page: what Agent match on the Roles tab is measured
|
||||
* against, one agent at a time.
|
||||
/* The Search agents page: what the order What I'm looking for on the Roles
|
||||
* tab is measured against, one agent at a time.
|
||||
*
|
||||
* Every agent from `/api/agents` is a card, newest first: its name, its
|
||||
* switch, its prompts, its gates in words, its weights, and a link to the
|
||||
* Roles tab in its order. The prompts are the body, since fit is measured
|
||||
* against the one a role fits best; the weights only share out the terms.
|
||||
* The form edits the prompts as rows, and the fetch gate in three parts: the
|
||||
* region (the scope select), the place (a short list of lines a role passes by
|
||||
* matching any one of: countries, towns, remote) and the role (the fetch terms
|
||||
* and the titles to leave out), over a row per remaining rule under Advanced;
|
||||
* a gate the three parts cannot say lands in a row, so a load and a save lose
|
||||
* nothing. A gate is a field, values and a mode (must, one of, not), with no
|
||||
* weight. After it come the feed order (the four weights of the final score and
|
||||
* the CV match weights), the cut (the best N by fit and the best M by CV
|
||||
* closeness, each with a minimum level) and the size of summarise. Edit turns a card
|
||||
* Roles tab in its order. The prompts are the body, since prompt closeness is
|
||||
* measured against the one a role is closest to; the weights only share out the terms.
|
||||
* The form edits the prompts as rows, and the fetch gate in four parts, in the
|
||||
* ADR's order: the world (the scope select), the location (a short list of
|
||||
* lines a role passes by matching any one of: countries, towns, remote), the
|
||||
* role exclusion (the titles to leave out) and the role fetch (the fetch
|
||||
* terms), and nothing else: seniority, pay and the like are filters on Roles.
|
||||
* A gate is a field, values and a mode (must, one of, not), with no weight.
|
||||
* A stored gate the four parts cannot say, which only the config or the API
|
||||
* writes, is named in a note under the fetch gate and sent back as it came,
|
||||
* so a load and a save lose nothing. After the fetch gate come the analytics
|
||||
* filter (the best N by prompt closeness and the best M by CV closeness, each
|
||||
* with a minimum level), the want feed score weights (prompt closeness,
|
||||
* newness, company like, job like), the CV match weights and the size of
|
||||
* summarise. Edit turns a card
|
||||
* into the form from the page's `<template>`, in place, and New agent puts a
|
||||
* blank card on top in the same state. One card edits at a time, so the
|
||||
* form's ids stay unique. Votes, liked companies and scores are the agent's,
|
||||
@@ -28,10 +32,7 @@
|
||||
* text (typed, or read from a .md/.txt file by the browser), the languages it
|
||||
* states as rows of a code and a level, and the two soft texts, what the
|
||||
* person likes and what they do not like (the work, the workplace,
|
||||
* situations), which the judge reads for the career score. The must-have text
|
||||
* is no longer on the form: a save leaves `must_text` out, so the stored one
|
||||
* is kept (it is set through the API, and still caps career at 39 in code
|
||||
* when the judge reports one broken).
|
||||
* situations), which the judge reads for the career score.
|
||||
* Every save sends the CV, languages and the two texts, and the card's switch
|
||||
* re-posts the row as the server gave it, so neither drops them.
|
||||
*
|
||||
@@ -47,13 +48,14 @@
|
||||
const DEFAULT_WEIGHTS = { fit: 0.6, recency: 0.2, affinity: 0.2, taste: 0.2,
|
||||
cv_skills: 30, cv_experience: 25, cv_career: 30 };
|
||||
// What the models read after a pull, per agent (`app.models.Limits`): the
|
||||
// cut is the best `cut_fit_top` roles by fit at `cut_fit_min` or more, then
|
||||
// the best `cut_cv_top` by CV closeness at `cut_cv_min` or more that the
|
||||
// first list leaves out, and every role liked or saved. Fetch text, extract
|
||||
// and judge read the cut; summarise reads the top `summarise_top` of the
|
||||
// feed and the roles liked or saved.
|
||||
// analytics filter is the best `cut_fit_top` roles by prompt closeness at
|
||||
// `cut_fit_min` or more, then the best `cut_cv_top` by CV closeness at
|
||||
// `cut_cv_min` or more that the first list leaves out, and every role liked
|
||||
// or saved: the cut. Fetch text, extract and judge read the cut; summarise
|
||||
// reads the top `summarise_top` of each feed order, want and match, and the
|
||||
// roles liked or saved.
|
||||
const DEFAULT_LIMITS = { cut_fit_top: 500, cut_fit_min: 0.6, cut_cv_top: 150, cut_cv_min: 0.6,
|
||||
summarise_top: 50 };
|
||||
summarise_top: 30 };
|
||||
// The levels of a language, the order of the server's ladder, and how big a
|
||||
// CV file the browser reads into the box.
|
||||
const LEVELS = ["basic", "conversational", "professional", "fluent", "native"];
|
||||
@@ -61,9 +63,8 @@
|
||||
const NEW = "new";
|
||||
// Europe as the starter agents spell it (static/remote.js).
|
||||
const EUROPE = rh.EUROPE;
|
||||
const FIELDS = ["place", "country", "remote", "remote_from", "seniority", "function", "salary_min",
|
||||
"currency", "permit", "languages", "company", "title"];
|
||||
const MODES = [["require", "must"], ["any", "one of"], ["exclude", "not"]];
|
||||
// A gate's mode in the words of the note on the gates the form keeps.
|
||||
const MODES = { require: "must", any: "one of", exclude: "not" };
|
||||
|
||||
/** @type {object[]} every agent, as the server lists them */
|
||||
let agents = [];
|
||||
@@ -233,7 +234,7 @@
|
||||
|
||||
function weightsLine(row) {
|
||||
const w = Object.assign({}, DEFAULT_WEIGHTS, row.weights);
|
||||
return `fit ${w.fit} · newness ${w.recency} · companies ${w.affinity} · liked ${w.taste}`
|
||||
return `prompt closeness ${w.fit} · newness ${w.recency} · company like ${w.affinity} · job like ${w.taste}`
|
||||
+ ` · CV match: skills ${w.cv_skills}, experience ${w.cv_experience}, career ${w.cv_career}`;
|
||||
}
|
||||
|
||||
@@ -263,8 +264,8 @@
|
||||
}
|
||||
|
||||
// The region and place in words, one sentence led by the region, then the lines a
|
||||
// role may match, with the one-of rows among them since they are the same
|
||||
// group: "Within Europe: keeps roles anywhere in Denmark, in Malmö, Lund or
|
||||
// role may match, with any other one-of gates among them since they are the
|
||||
// same group: "Within Europe: keeps roles anywhere in Denmark, in Malmö, Lund or
|
||||
// Helsingborg, or remote, and leaves out the rest." The lines add up, which
|
||||
// the first wording ("Roles in ...") left the owner to guess.
|
||||
function whereSentence(where, others) {
|
||||
@@ -299,18 +300,18 @@
|
||||
return sentences.join(" ");
|
||||
}
|
||||
|
||||
// What the models read after a pull: the cut, then the stages that read
|
||||
// it, then summarise's own size. A stage that is off says so, with a link to
|
||||
// What the models read after a pull: the analytics filter's cut, then the
|
||||
// stages that read it, then summarise's own size. A stage that is off says so, with a link to
|
||||
// its switch under Runs: the numbers are only what it would read once it is
|
||||
// on. The stage names are those of `/api/stages`.
|
||||
function limitsLine(row) {
|
||||
const l = Object.assign({}, DEFAULT_LIMITS, row.limits);
|
||||
const p = el("p", "agent-limits");
|
||||
const lists = [];
|
||||
if (l.cut_fit_top) lists.push(`the best ${l.cut_fit_top} by fit at ${l.cut_fit_min} or more`);
|
||||
if (l.cut_cv_top) lists.push(`the best ${l.cut_cv_top} by closeness to the CV at ${l.cut_cv_min} or more`);
|
||||
if (l.cut_fit_top) lists.push(`the best ${l.cut_fit_top} by prompt closeness at ${l.cut_fit_min} or more`);
|
||||
if (l.cut_cv_top) lists.push(`the best ${l.cut_cv_top} by CV closeness at ${l.cut_cv_min} or more`);
|
||||
const cut = lists.length ? `${lists.join(", then ")}, and liked and saved roles` : "liked and saved roles only";
|
||||
p.append(`The cut: ${cut}.`);
|
||||
p.append(`Analytics filter: ${cut}.`);
|
||||
const off = (name, would) => {
|
||||
const a = link("/settings/sources#stages", `${name} is off`);
|
||||
a.title = "Switch it on under Runs.";
|
||||
@@ -323,7 +324,7 @@
|
||||
p.append(` ${said[0].toUpperCase()}${said.slice(1)} ${on.length > 1 ? "read" : "reads"} it`);
|
||||
}
|
||||
for (const [key, name] of reads) if (stageOn[key] === false) off(name, "reads the cut");
|
||||
const summarised = `the top ${l.summarise_top} of the feed and liked and saved roles`;
|
||||
const summarised = `the top ${l.summarise_top} of each feed order and liked and saved roles`;
|
||||
if (stageOn.summarise === false) off("summarise", `summarises ${summarised}`);
|
||||
else p.append(` · summarises ${summarised}`);
|
||||
return p;
|
||||
@@ -345,7 +346,7 @@
|
||||
|
||||
const prompts = promptsOf(row).filter((p) => p.text.trim());
|
||||
if (!prompts.length) {
|
||||
div.appendChild(el("p", "agent-intent empty", "Nothing described yet, so fit has nothing to measure against. Edit says what this agent looks for."));
|
||||
div.appendChild(el("p", "agent-intent empty", "Nothing described yet, so prompt closeness has nothing to measure against. Edit says what this agent looks for."));
|
||||
}
|
||||
for (const prompt of prompts) {
|
||||
// A lone prompt named after its agent, as a migrated intent is, says its name once.
|
||||
@@ -381,7 +382,7 @@
|
||||
// ---------------------------------------------------------------- the form
|
||||
|
||||
// Tell the form something changed that no field typed: a chip or a row
|
||||
// added or removed. It marks the form dirty and redraws the gates view.
|
||||
// added or removed. It marks the form dirty and redraws the gates' sentence.
|
||||
function changed(node) {
|
||||
node.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
}
|
||||
@@ -556,9 +557,9 @@
|
||||
return box;
|
||||
}
|
||||
|
||||
// A language as a row, after the rule rows: a code, a level, Remove. The
|
||||
// code is two letters, which the server turns into the language; its name
|
||||
// is the row's title where the browser knows it.
|
||||
// A language as a row: a code, a level, Remove. The code is two letters,
|
||||
// which the server turns into the language; its name is the row's title
|
||||
// where the browser knows it.
|
||||
function languageRow(language, remove) {
|
||||
const row = el("div", "rule-row");
|
||||
const code = el("input");
|
||||
@@ -584,32 +585,32 @@
|
||||
return { node: row, read: () => ({ language: code.value.trim().toLowerCase(), level: level.value }) };
|
||||
}
|
||||
|
||||
// A gate as a row: field, mode, values and Remove. Values untouched since the load are sent as they came, so a
|
||||
// value with a comma in it survives a save.
|
||||
function ruleRow(gate, remove) {
|
||||
const row = el("div", "rule-row");
|
||||
const field = select(FIELDS.map((f) => [f, f.replace("_", " ")]), gate.field, "Field");
|
||||
const mode = select(MODES, gate.mode, "Mode");
|
||||
const values = el("input");
|
||||
values.type = "text";
|
||||
values.required = true;
|
||||
values.spellcheck = false;
|
||||
values.placeholder = "senior, staff";
|
||||
values.value = gate.values.join(", ");
|
||||
values.setAttribute("aria-label", "Values, separated by commas");
|
||||
const loaded = values.value;
|
||||
const drop = linky("Remove");
|
||||
drop.addEventListener("click", () => remove(row));
|
||||
row.append(field, mode, values, drop);
|
||||
// The gates the fetch gate's four parts say, in the order the starter
|
||||
// agents have them: the world, the location lines and the titles left out,
|
||||
// then `kept`, the stored gates the parts cannot say, as they came. `parts`
|
||||
// holds what the form reads: `scope` (europe, countries, everywhere) and
|
||||
// its `scoped` countries, the `within` countries, the `towns`, `remote`
|
||||
// (none, scope, from) and its `from` areas, and the `titles`.
|
||||
function formGates(parts, kept = []) {
|
||||
const out = [];
|
||||
const { scope, scoped, within, towns: lines, remote, from, titles } = parts;
|
||||
if (scope === "europe") out.push({ field: "country", values: rh.europeWith(within), mode: "require" });
|
||||
if (scope === "countries" && scoped.length) out.push({ field: "country", values: scoped, mode: "require" });
|
||||
if (within.length) out.push({ field: "country", values: within, mode: "any" });
|
||||
if (remote === "scope") out.push({ field: "remote", values: ["remote"], mode: "any" });
|
||||
if (remote === "from" && from.length) out.push({ field: "remote_from", values: from, mode: "any" });
|
||||
if (lines.length) out.push({ field: "place", values: lines, mode: "any" });
|
||||
if (titles.length) out.push({ field: "title", values: titles, mode: "exclude" });
|
||||
return out.concat(kept);
|
||||
}
|
||||
|
||||
function read() {
|
||||
const parts = values.value === loaded
|
||||
? gate.values
|
||||
: values.value.split(",").map((v) => v.trim()).filter(Boolean)
|
||||
.map((v) => (field.value === "salary_min" ? Number(v) : v));
|
||||
return { field: field.value, values: parts, mode: mode.value };
|
||||
}
|
||||
return { node: row, read };
|
||||
// The note on the gates the form keeps: "The form cannot edit these rules,
|
||||
// and Save keeps them as stored: must: with seniority senior; not: at Acme."
|
||||
function keptNote(kept) {
|
||||
if (!kept.length) return "";
|
||||
const said = kept.map((g) => `${MODES[g.mode] || g.mode}: ${phrase(g)}`).join("; ");
|
||||
return `The form cannot edit ${kept.length > 1 ? "these rules" : "this rule"}, and Save keeps `
|
||||
+ `${kept.length > 1 ? "them" : "it"} as stored: ${said}.`;
|
||||
}
|
||||
|
||||
function form(row) {
|
||||
@@ -710,51 +711,18 @@
|
||||
remoteFrom.node.hidden = remoteMode.value !== "from";
|
||||
};
|
||||
offer();
|
||||
$("details.advanced", f).open = rest.length > 0;
|
||||
const kept = $("#a-kept", f);
|
||||
kept.textContent = keptNote(rest);
|
||||
kept.hidden = !rest.length;
|
||||
|
||||
const rules = [];
|
||||
const dropRule = (node) => {
|
||||
rules.splice(rules.findIndex((r) => r.node === node), 1);
|
||||
node.remove();
|
||||
changed(f);
|
||||
};
|
||||
const addRule = (gate) => {
|
||||
const entry = ruleRow(gate, dropRule);
|
||||
rules.push(entry);
|
||||
$("#a-rules", f).appendChild(entry.node);
|
||||
return entry;
|
||||
};
|
||||
rest.forEach(addRule);
|
||||
$("#a-add-rule", f).addEventListener("click", () => {
|
||||
$("select", addRule({ field: "seniority", values: [], mode: "require" }).node).focus();
|
||||
changed(f);
|
||||
});
|
||||
|
||||
// The gates the fetch gate's three parts say, in the order the starter
|
||||
// agents have them: the region, the place lines, and the titles left out.
|
||||
function whereGates() {
|
||||
const out = [];
|
||||
const within = countries.values();
|
||||
const towns = places.values();
|
||||
const from = remoteFrom.values();
|
||||
if (scope.value === "europe") out.push({ field: "country", values: rh.europeWith(within), mode: "require" });
|
||||
if (scope.value === "countries" && scoped.values().length) {
|
||||
out.push({ field: "country", values: scoped.values(), mode: "require" });
|
||||
}
|
||||
if (within.length) out.push({ field: "country", values: within, mode: "any" });
|
||||
if (remoteMode.value === "scope") out.push({ field: "remote", values: ["remote"], mode: "any" });
|
||||
if (remoteMode.value === "from" && from.length) out.push({ field: "remote_from", values: from, mode: "any" });
|
||||
if (towns.length) out.push({ field: "place", values: towns, mode: "any" });
|
||||
const left = titles.values();
|
||||
if (left.length) out.push({ field: "title", values: left, mode: "exclude" });
|
||||
return out;
|
||||
}
|
||||
// The gates Save sends: the fetch gate's, then the rows. The sentence
|
||||
// under it says them all, as the card will, rows included.
|
||||
const gates = () => whereGates().concat(rules.map((r) => r.read()));
|
||||
// The gates Save sends: the four parts', then the kept ones. The sentence
|
||||
// under them says them all, as the card will, the kept ones included.
|
||||
const gates = () => formGates({
|
||||
scope: scope.value, scoped: scoped.values(), within: countries.values(), towns: places.values(),
|
||||
remote: remoteMode.value, from: remoteFrom.values(), titles: titles.values(),
|
||||
}, rest);
|
||||
const view = () => {
|
||||
offer();
|
||||
$("#a-gates", f).value = JSON.stringify(gates(), null, 2);
|
||||
$("#a-where-sum", f).textContent = rulesLine(gates()) || "Passes every role.";
|
||||
};
|
||||
view();
|
||||
@@ -978,6 +946,9 @@
|
||||
}
|
||||
}
|
||||
|
||||
// For tests/js/agents.test.mjs: what Save sends and how it is said.
|
||||
window.agentGates = { formGates, keptNote, rulesLine };
|
||||
|
||||
document.addEventListener("DOMContentLoaded", async () => {
|
||||
if (!$("#agents-list")) return;
|
||||
$("#a-new").addEventListener("click", () => open(null));
|
||||
|
||||
+10
-22
@@ -579,10 +579,12 @@ kbd {
|
||||
font-size: .8rem; padding: .18rem .3rem;
|
||||
border: 1px solid var(--line);
|
||||
}
|
||||
/* The three order labels share one line: Agent match is the longest and
|
||||
would otherwise wrap in a 240px rail. Cells grow to the width, each by its
|
||||
own text. */
|
||||
#sort button { white-space: nowrap; }
|
||||
/* The three order labels share one row. What I'm looking for is longer than
|
||||
a 240px rail has room for on one line, so a cell may wrap its words; the
|
||||
cells split the width by their text, the short ones keeping theirs. */
|
||||
#sort button { min-width: 0; line-height: 1.2; }
|
||||
#sort button:nth-child(2), #sort button:nth-child(3) { flex-grow: 0; white-space: nowrap; }
|
||||
#sort button:first-child { flex-basis: 0; flex-grow: 1; }
|
||||
/* Square everything first; only the ends are ends. Every cell but the first
|
||||
eats its neighbour's border, so the joins are one rule wide rather than two. */
|
||||
.seg > * { border-radius: 0; }
|
||||
@@ -976,14 +978,6 @@ body[data-reader="open"] .company-strip { display: none; }
|
||||
.reader-cv .cv-total { margin: 0 0 .45rem; font-variant-numeric: tabular-nums; }
|
||||
.reader-cv .cv-total b { font-size: 1.25rem; }
|
||||
.reader-cv .cv-total span { display: inline-block; margin-left: .4rem; color: var(--dim); font-size: .88rem; }
|
||||
/* The must-haves, one line each with its verdict's mark. */
|
||||
.cv-musts { list-style: none; margin: 0 0 .55rem; padding: 0; font-size: .92rem; line-height: 1.4; }
|
||||
.cv-musts li { display: flex; gap: .45rem; }
|
||||
.cv-musts li + li { margin-top: .15rem; }
|
||||
.cv-musts .mark { flex: none; width: 1rem; text-align: center; font-weight: 700; }
|
||||
.cv-musts .must-met .mark { color: var(--good); }
|
||||
.cv-musts .must-broken .mark { color: var(--error); }
|
||||
.cv-musts .must-unclear .mark { color: var(--faint); }
|
||||
.cv-lists { display: grid; grid-template-columns: 1fr 1fr; gap: .5rem 1.2rem; }
|
||||
.cv-lists ul { margin: 0; padding-left: 1.1rem; font-size: .92rem; line-height: 1.45; }
|
||||
.cv-lists li + li { margin-top: .2rem; }
|
||||
@@ -1210,7 +1204,7 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
|
||||
.affinity button[data-value="-1"][aria-pressed="true"] {
|
||||
color: var(--accent-fg); background: var(--error); border-color: var(--error);
|
||||
}
|
||||
/* The two fetch overrides, All countries and All roles, global and apart
|
||||
/* The two fetch overrides, All locations and All roles, global and apart
|
||||
from the stances: green, the colour the app gives what widens (`--good`). */
|
||||
.affinity button[data-value="countries"][aria-pressed="true"],
|
||||
.affinity button[data-value="roles"][aria-pressed="true"] {
|
||||
@@ -1306,8 +1300,6 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
|
||||
background: var(--bg); border: 1px solid var(--line);
|
||||
border-radius: .3rem; padding: .35rem .5rem; width: 100%;
|
||||
}
|
||||
/* Prose in a textarea is read in the page face; JSON is in the mono one. */
|
||||
.agent-form textarea.code { font-family: var(--mono); font-size: .85rem; }
|
||||
.agent-form .hint { color: var(--dim); font-size: .8rem; overflow-wrap: anywhere; }
|
||||
.weights { display: flex; gap: .75rem; flex-wrap: wrap; border: 1px solid var(--line); border-radius: .3rem; }
|
||||
/* A label that wraps to two lines pushes its box down; the boxes stay in line. */
|
||||
@@ -1322,14 +1314,13 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
|
||||
}
|
||||
/* The fieldsets that hold rows: a column, the Add button under the rows. */
|
||||
.agent-form fieldset.prompts, .agent-form fieldset.where, .agent-form fieldset.titles,
|
||||
.agent-form fieldset.terms,
|
||||
.agent-form fieldset.rules {
|
||||
.agent-form fieldset.terms {
|
||||
display: flex; flex-direction: column; align-items: stretch; gap: .5rem; min-width: 0;
|
||||
border: 1px solid var(--line); border-radius: .3rem;
|
||||
}
|
||||
.agent-form fieldset > button { align-self: flex-start; }
|
||||
/* A prompt or a rule as a row of a list, after `.crow`: the name and Remove
|
||||
on line one, the text spanning the row under them. A rule row wraps like
|
||||
/* A prompt or a language as a row of a list, after `.crow`: the name and Remove
|
||||
on line one, the text spanning the row under them. A language row wraps like
|
||||
`.crow .board-edit`, so a phone takes it in two lines rather than scrolling. */
|
||||
.prompt-row {
|
||||
display: grid; grid-template-columns: 1fr auto; align-items: center;
|
||||
@@ -1359,9 +1350,6 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
|
||||
.agent-form .chips input[type="text"] { flex: 1 1 6rem; width: auto; min-width: 0; }
|
||||
.agent-form .chips select { flex: 0 1 12rem; min-width: 0; max-width: 100%; }
|
||||
.chips .tag span { color: var(--faint); }
|
||||
.agent-form details summary { cursor: pointer; color: var(--dim); font-size: .9rem; }
|
||||
.agent-form details[open] summary { margin-bottom: .5rem; }
|
||||
.agent-form .advanced-body { display: flex; flex-direction: column; gap: .6rem; }
|
||||
|
||||
/* ================================================================ */
|
||||
/* companies and sources — the Sources page's sections, */
|
||||
|
||||
+16
-16
@@ -16,7 +16,7 @@
|
||||
* Beside `api` on `window`, the few helpers more than one page script uses:
|
||||
* `count`, `cell`, `debounce`, `safeUrl`, and the company list with its
|
||||
* affinity control (`loadCompanies`, `companies`, `affinityControl`,
|
||||
* `setAffinity`) and the two fetch overrides beside it, All countries and
|
||||
* `setAffinity`) and the two fetch overrides beside it, All locations and
|
||||
* All roles (`overridesControl`). One
|
||||
* copy each, since two copies of a helper is one of them
|
||||
* being fixed.
|
||||
@@ -189,26 +189,26 @@
|
||||
/* The two fetch overrides, one more pair of toggles in the affinity
|
||||
buttons' fragment beside them, in a group of their own since they are
|
||||
global rather than an agent's stance (app/jobs/reach.py). The fetch gate
|
||||
has three parts, region, place and role, and a company follows every
|
||||
agent's on all three unless it is set to:
|
||||
- All countries: the place part is skipped. The company's roles are
|
||||
fetched and kept from the whole region (Europe), not only the agents'
|
||||
places, for every agent. The region itself still holds.
|
||||
- All roles: the role part is skipped: no fetch terms and no titles to
|
||||
leave out.
|
||||
has four parts, world, location, role exclusion and role fetch, and a
|
||||
company follows every agent's unless it is set to:
|
||||
- All locations: the location gate is skipped. The company's roles are
|
||||
fetched and kept from the whole world gate (Europe, say), not only the
|
||||
agents' locations, for every agent. The world gate itself still holds.
|
||||
- All roles: role exclusion and role fetch are skipped: no titles to
|
||||
leave out and no fetch terms.
|
||||
Turning either on or off queues a rescore on the server, as the other
|
||||
switches that change what passes do, and a pull reads the board anew.
|
||||
`company` is the shared list's row; `onChange` runs after the server
|
||||
accepted a switch, with the company it answered. */
|
||||
const OVERRIDES = [
|
||||
{ value: "countries", field: "all_countries", path: "all-countries", text: "All countries",
|
||||
on: "All countries is on: this company's roles are fetched and kept from the whole "
|
||||
+ "region (Europe), not only the agents' places, for every agent. The region still "
|
||||
+ "holds. Turning it off puts the agents' places back and rescores.",
|
||||
off: "All countries: fetch and keep this company's roles from the whole region "
|
||||
+ "(Europe) instead of only the agents' places, for every agent. Turning it on "
|
||||
+ "rescores.",
|
||||
said: (on) => (on ? "all countries in the region" : "the agents' places") },
|
||||
{ value: "countries", field: "all_countries", path: "all-countries", text: "All locations",
|
||||
on: "All locations is on: this company's roles are fetched and kept from the whole "
|
||||
+ "world gate (Europe, say), not only the agents' locations, for every agent. The "
|
||||
+ "world gate still holds. Turning it off puts the agents' locations back and rescores.",
|
||||
off: "All locations: fetch and keep this company's roles from the whole world gate "
|
||||
+ "(Europe, say) instead of only the agents' locations, for every agent. Turning it "
|
||||
+ "on rescores.",
|
||||
said: (on) => (on ? "all locations in the world gate" : "the agents' locations") },
|
||||
{ value: "roles", field: "all_roles", path: "all-roles", text: "All roles",
|
||||
on: "All roles is on: this company's roles are fetched with no fetch terms and kept "
|
||||
+ "whatever their title, for every agent. Turning it off puts the agents' fetch "
|
||||
|
||||
+11
-7
@@ -5,7 +5,7 @@
|
||||
* joined with each company's board subscription — its id and whether it is
|
||||
* on — so nothing here fetches `/api/sources` as well. The affinity buttons
|
||||
* are `affinityControl`, the same ones the Roles tab and the read page draw,
|
||||
* and beside them `overridesControl`, the two fetch overrides All countries
|
||||
* and beside them `overridesControl`, the two fetch overrides All locations
|
||||
* and All roles, which are global and drawn with or without an agent.
|
||||
*
|
||||
* Liked and hidden are per search agent, so the `#c-agent` select above the
|
||||
@@ -25,10 +25,10 @@
|
||||
* from app/jobs/reach.py), so the row shows whether the company follows the
|
||||
* agents' filters or overrides them. Its editor has a second line that sets
|
||||
* the board's own reach: the agents', or its own countries and remote,
|
||||
* posted to `/api/sources/{id}/reach`. Fetching from the whole region (All
|
||||
* countries) and no fetch terms or titles to leave out (All roles) are the
|
||||
* posted to `/api/sources/{id}/reach`. Fetching from the whole world gate (All
|
||||
* locations) and no fetch terms or titles to leave out (All roles) are the
|
||||
* company's two overrides instead, and the board's own countries win over
|
||||
* All countries.
|
||||
* All locations.
|
||||
*
|
||||
* Beside that line stands the board's size, set when it was added and
|
||||
* changed only by a Recheck ("999 roles · checked 30 Sep"). Recheck, beside
|
||||
@@ -326,8 +326,8 @@
|
||||
note.className = "muted small";
|
||||
note.textContent = [
|
||||
company.all_countries
|
||||
? "All countries is on: roles are fetched and kept from the whole region, and countries set here win over it."
|
||||
: "All countries, beside the stars, fetches and keeps this company's roles from the whole region (Europe), not only the agents' places.",
|
||||
? "All locations is on: roles are fetched and kept from the whole world gate, and countries set here win over it."
|
||||
: "All locations, beside the stars, fetches and keeps this company's roles from the whole world gate (Europe, say), not only the agents' locations.",
|
||||
company.all_roles
|
||||
? "All roles is on: no fetch terms and no titles to leave out."
|
||||
: "All roles drops the fetch terms and the titles to leave out.",
|
||||
@@ -420,9 +420,13 @@
|
||||
const row = document.createElement("div");
|
||||
row.className = "crow";
|
||||
if (company.affinity === -1) row.dataset.hidden = "true";
|
||||
// By key, as the feed's own company links are, so the Roles page opens
|
||||
// the company view with its strip and gate note. A name would be a
|
||||
// substring there, which also lists other companies whose names hold it
|
||||
// and misses roles spelt otherwise.
|
||||
const link = document.createElement("a");
|
||||
link.className = "name";
|
||||
link.href = `/?company=${encodeURIComponent(company.name)}`;
|
||||
link.href = `/?company_keys=${encodeURIComponent(company.key)}`;
|
||||
link.title = "This company's open roles";
|
||||
link.textContent = company.name;
|
||||
row.appendChild(link);
|
||||
|
||||
+12
-10
@@ -1,7 +1,8 @@
|
||||
/* Where a remote role may be worked from, as the pages say it: the words of
|
||||
* the badge, the picks of the agent's "Remote, open to someone in" chips, and
|
||||
* how an agent's gates share out between the fetch gate's blocks and Other rules. No
|
||||
* DOM in it, so `tests/js/remote.test.mjs` can walk it without a browser.
|
||||
* how an agent's gates share out between the fetch gate's blocks and the rest
|
||||
* the form keeps as stored. No DOM in it, so `tests/js/remote.test.mjs` can
|
||||
* walk it without a browser.
|
||||
*
|
||||
* The area is a place token in three spellings, one per layer: `remote:<place
|
||||
* token>` on the Roles rail and in the address, `<place token>` in a
|
||||
@@ -61,18 +62,19 @@
|
||||
}
|
||||
|
||||
/** Share out an agent's gates between the fetch gate's blocks (region,
|
||||
place, titles to leave out) and Other rules. Each line takes the first
|
||||
gate it can say whole; the rest are rows. A gate has a field, values and
|
||||
a mode (require, any, exclude) and no weight; a gate with a mode the
|
||||
form does not know, such as an old prefer, stays a row.
|
||||
place, titles to leave out) and the rest, which the form cannot edit
|
||||
and keeps as stored. Each line takes the first gate it can say whole.
|
||||
A gate has a field, values and a mode (require, any, exclude) and no
|
||||
weight; a gate with a mode the form does not know, such as an old
|
||||
prefer, is in the rest.
|
||||
|
||||
The scope line is the first plain require gate on country: `europe`
|
||||
when it is exactly Europe and the countries line (`europeWith`),
|
||||
`countries` with its codes as chips otherwise, `everywhere` when there
|
||||
is none. Any other require gate on country is a row. The remote line is
|
||||
`scope` for the one-of on `remote: [remote]` (remote, within the
|
||||
scope), `from` for a one-of on `remote_from`, else `none`; when both
|
||||
are there `remote_from` wins and the plain one stays a row. */
|
||||
is none. Any other require gate on country is in the rest. The remote
|
||||
line is `scope` for the one-of on `remote: [remote]` (remote, within
|
||||
the scope), `from` for a one-of on `remote_from`, else `none`; when
|
||||
both are there `remote_from` wins and the plain one is in the rest. */
|
||||
function distribute(gates) {
|
||||
const where = { scope: null, countries: null, places: null, remote: { mode: "none", values: [] }, titles: null };
|
||||
const rest = [];
|
||||
|
||||
+292
-164
@@ -10,10 +10,11 @@
|
||||
*
|
||||
* A row is newsdesk's `.row`: the score gutter and number (hidden under
|
||||
* Newest, since the score is not what put the row there, and the CV
|
||||
* match under CV match, a dash for a role not judged yet), the title,
|
||||
* match under What fits me, a dash for a role not judged yet), the title,
|
||||
* company · place · badges · age · tags, the summary or the description's
|
||||
* opening, and ▼ ● ▲ ★ from judge.js. A passed role and a hidden company's
|
||||
* are dimmed, never dropped, so a stray tap stays visible to be undone.
|
||||
* opening, and ▼ ● ▲ ★ from judge.js. A passed role is dimmed, never
|
||||
* dropped, so a stray tap stays visible to be undone. A hidden company's roles
|
||||
* are never in the feed.
|
||||
*
|
||||
* Opening a role pushes `/roles/{uid}`; closing goes back. Stepping with j/k
|
||||
* while it is open replaces that entry rather than stacking one per role, so
|
||||
@@ -29,17 +30,22 @@
|
||||
* and each company affinity. One is always picked while any is enabled: the
|
||||
* one in `?agent=`, else the one this browser viewed last (localStorage),
|
||||
* else the enabled one with the lowest id, which is the server's default too.
|
||||
* The feed's first request carries the first two before the agents are in,
|
||||
* and waits for them when there is neither, since the server gates nothing
|
||||
* for a request that names no agent (`start`).
|
||||
* Picking another reloads the feed and the counts under the reader, which
|
||||
* stays open and is fetched again for that agent's vote.
|
||||
*
|
||||
* Order is Agent match (`order=score`, the agent's final score moved by its
|
||||
* votes), Newest (`order=newest`) or CV match (`order=cv`, the roles the
|
||||
* agent's `judge` read against its CV first, by that total), the same three
|
||||
* whichever agent is viewed, and all carry the agent's votes and hidden
|
||||
* companies. CV match needs a CV: an agent without one is shown Agent
|
||||
* match, as an agentless page is shown Newest. Every order keeps to the
|
||||
* roles the agent's fetch gate passes, which a note under Viewing as says;
|
||||
* Show gated (`gated=true`) adds the ones it rejects, with the reason.
|
||||
* Order is What I'm looking for (`order=score`, the agent's want feed score
|
||||
* moved by its votes), Newest (`order=newest`) or What fits me (`order=cv`,
|
||||
* the roles the agent's `judge` read against its CV first, by that total),
|
||||
* the same three whichever agent is viewed, and all carry the agent's votes.
|
||||
* What fits me needs a CV: an agent without one is shown What I'm looking
|
||||
* for, as an agentless page is shown Newest. Every order keeps to the
|
||||
* roles the agent's fetch gate passes, which a note under Viewing as says,
|
||||
* and for a phone, whose rail is shut, the end of the list and the empty
|
||||
* state; Show gated (`gated=true`) adds the ones it rejects, with the
|
||||
* reason. The reader says when the gate rejects the role open.
|
||||
*
|
||||
* Saved filters belong to the viewing agent (`/api/agents/{id}/filters`):
|
||||
* pills in the rail, newsdesk's `#active` row, each the query string
|
||||
@@ -75,14 +81,14 @@
|
||||
* screen with that company as its one company chip (`?company_keys=`); a
|
||||
* plain click filters in place and pushes. With one company filtered a strip
|
||||
* over the feed names it, counts its roles, links its careers page and
|
||||
* carries its ⊘ ▲ ★ for the viewing agent. `?company=`, `?company_keys=` and
|
||||
* `?agent=` are how the Companies and Search agents pages link in.
|
||||
* carries its ⊘ ▲ ★ for the viewing agent. `?company_keys=` and `?agent=`
|
||||
* are how Settings › Sources and Search agents link in.
|
||||
*
|
||||
* Company and Place in the rail are chip inputs (chips.js) on the suggest
|
||||
* routes: a chip is a company key or a place token (`country:DK`,
|
||||
* `region:skane`), several joined by OR. `?company=` and `?place=`, the
|
||||
* substrings of the links from other pages, still filter and show as a chip
|
||||
* of their own to take off; nothing in the rail types a new one. Seniority
|
||||
* substrings of older links and saved filters, still filter and show as a
|
||||
* chip of their own to take off; nothing in the rail types a new one. Seniority
|
||||
* (a From and a To rung and a Track, `level_from`, `level_to` and `track`)
|
||||
* and Pay carry the counts of `/api/roles/facets`, asked after each reload
|
||||
* with the feed's own parameters, so each says what a click would add. The
|
||||
@@ -90,8 +96,11 @@
|
||||
* an old `?seniority=` is ignored.
|
||||
*
|
||||
* The list is paged: 60 rows, then 60 more each time the end of the list comes
|
||||
* within 600px of the screen, with the offset the rows already shown. Back to
|
||||
* this page from another tab puts the same number of rows and the scroll
|
||||
* within 600px of the screen. The server sorts afresh on every request and a
|
||||
* vote or a read moves a role in its list or out of it, so the next page
|
||||
* starts a row earlier for each and a row already on screen is dropped
|
||||
* (`pageAsk`, `pageMerge`): none is skipped and none is shown twice. Back
|
||||
* to this page from another tab puts the same number of rows and the scroll
|
||||
* position back, from sessionStorage.
|
||||
*/
|
||||
(function () {
|
||||
@@ -128,10 +137,11 @@
|
||||
};
|
||||
const TRACKS = { "": "Both", ic: "Individual", people: "People" };
|
||||
|
||||
// What the list is in the order of. Agent match is the viewing agent's
|
||||
// score, moved by its votes; Newest is first seen; CV match is
|
||||
// the agent's `judge` total, the roles not judged after by score.
|
||||
const SORTS = { score: "Agent match", cv: "CV match", time: "Newest" };
|
||||
// What the list is in the order of. What I'm looking for is the viewing
|
||||
// agent's want feed score, moved by its votes; Newest is the posting date;
|
||||
// What fits me is the agent's `judge` total, the roles not judged after by
|
||||
// CV closeness.
|
||||
const SORTS = { score: "What I\u2019m looking for", cv: "What fits me", time: "Newest" };
|
||||
// The route's `order` for each.
|
||||
const ORDERS = { score: "score", time: "newest", cv: "cv" };
|
||||
// How many days ahead a deadline earns a badge: today to today + 4, the
|
||||
@@ -154,15 +164,16 @@
|
||||
|
||||
// Everything `clearFilters` has to put back, in one place, so a new filter
|
||||
// added here cannot be forgotten there. `company` and `place` are the
|
||||
// substrings of links from other pages; `companyKeys` and `places` are the
|
||||
// rail's chips. `levelFrom`, `levelTo` and `track` are the seniority range;
|
||||
// `seniorityUnknown` and `payUnknown` are the "and roles with no level / no
|
||||
// pay" boxes, asked only while a range, a track or a pay option is picked.
|
||||
// substrings of older links and saved filters; `companyKeys` and `places`
|
||||
// are the rail's chips. `levelFrom`, `levelTo` and `track` are the seniority
|
||||
// range; `seniorityUnknown` and `payUnknown` are the "and roles with no
|
||||
// level / no pay" boxes, asked only while a range, a track or a pay option
|
||||
// is picked.
|
||||
function blankFilter() {
|
||||
return {
|
||||
sort: "score", agent: null, period: "all", tags: new Set(), showRead: true,
|
||||
company: "", place: "", function: "", remote: "", judged: "", pay: "",
|
||||
showMuted: false, gated: false, companyKeys: [], places: [],
|
||||
gated: false, companyKeys: [], places: [],
|
||||
levelFrom: "", levelTo: "", track: "", seniorityUnknown: true, payUnknown: false,
|
||||
closing: false,
|
||||
};
|
||||
@@ -174,7 +185,7 @@
|
||||
counted. */
|
||||
function filtersOn(f = state.filter, { more = false } = {}) {
|
||||
const folded = [f.company.trim(), f.place.trim(), f.function, f.remote, f.pay, f.judged,
|
||||
f.showMuted, f.closing, f.agent !== null && f.gated].filter(Boolean).length
|
||||
f.closing, f.agent !== null && f.gated].filter(Boolean).length
|
||||
+ f.companyKeys.length + f.places.length
|
||||
+ [f.levelFrom, f.levelTo, f.track].filter(Boolean).length;
|
||||
if (more) return folded;
|
||||
@@ -191,6 +202,10 @@
|
||||
tags: [], // [{tag, count, unread}] from /api/filters
|
||||
items: [], // the rows on screen, in order
|
||||
total: 0, // how many roles the filters match in all
|
||||
offset: 0, // how far into the server's list the pages have read
|
||||
more: false, // the server's list goes on past `offset`
|
||||
moved: 0, // votes and reads no page has been asked after yet (`pageAsk`)
|
||||
movedOnce: 0, // those one page has been asked after
|
||||
loaded: false, // the first page has landed
|
||||
loading: false, // a page is in flight, so the pager holds off
|
||||
selected: 0, // which row the keyboard is on
|
||||
@@ -225,13 +240,14 @@
|
||||
/** The viewing agent, once the agents are in. */
|
||||
const viewed = () => state.agents.find((a) => a.id === state.filter.agent) || null;
|
||||
|
||||
/** Whether CV match has a CV to go by: the viewed agent has one, or
|
||||
/** Whether What fits me has a CV to go by: the viewed agent has one, or
|
||||
the agents are not in yet, when the address is taken at its word. */
|
||||
const hasCv = () => !state.agentsLoaded || Boolean(viewed()?.cv_text?.trim());
|
||||
|
||||
/** The order the list is in. Agent match needs an agent to score by: with
|
||||
none enabled the server answers newest first, and so does this. Best
|
||||
CV match needs that agent's CV too, and without one is Agent match. */
|
||||
/** The order the list is in. What I'm looking for needs an agent to score
|
||||
by: with none enabled the server answers newest first, and so does this.
|
||||
What fits me needs that agent's CV too, and without one is What I'm
|
||||
looking for. */
|
||||
function sortShown() {
|
||||
const { sort, agent } = state.filter;
|
||||
if (sort === "time" || agent === null) return "time";
|
||||
@@ -249,7 +265,6 @@
|
||||
for (const key of ["function", "remote", "judged", "pay"]) if (f[key]) p.set(key, f[key]);
|
||||
const days = PERIODS[f.period].days;
|
||||
if (days) p.set("days", String(days));
|
||||
if (f.showMuted) p.set("show_muted", "true");
|
||||
if (!f.showRead) p.set("include_read", "false");
|
||||
f.tags.forEach((t) => p.append("tags", t));
|
||||
f.companyKeys.forEach((key) => p.append("company_keys", key));
|
||||
@@ -286,7 +301,6 @@
|
||||
f[key] = q.get(key) || "";
|
||||
}
|
||||
f.period = PERIOD_OF_DAYS[q.get("days")] || "all";
|
||||
f.showMuted = q.get("show_muted") === "true";
|
||||
f.showRead = q.get("include_read") !== "false";
|
||||
f.gated = q.get("gated") === "true" || q.get("gated") === "1";
|
||||
f.tags = new Set(q.getAll("tags"));
|
||||
@@ -387,7 +401,6 @@
|
||||
written into the entry landed on, so the entry says what is shown. */
|
||||
function restore() {
|
||||
const before = addressFor(pathFor(null));
|
||||
const muted = state.filter.showMuted;
|
||||
readUrl(location.search, { view: false });
|
||||
$("#search").value = state.q;
|
||||
writeUrl();
|
||||
@@ -397,25 +410,79 @@
|
||||
labelPlaces();
|
||||
if (addressFor(pathFor(null)) === before) return;
|
||||
reload();
|
||||
if (state.filter.showMuted !== muted) loadFilters().catch(() => {});
|
||||
renderRail();
|
||||
renderActive();
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------- loading
|
||||
|
||||
/** The most rows one request may ask for (`limit` on /api/roles). */
|
||||
const MOST = 1000;
|
||||
|
||||
/* Where the next page starts and how many rows it asks for. The server
|
||||
sorts afresh on every request, and a vote moves a role in its order
|
||||
(`_VOTED_SCORE`) or out of the list (Judged), as a read does with Show
|
||||
read off. Each row that leaves the part already read lets a row from
|
||||
below up into it, which a page asked from `offset` would never show, so
|
||||
the page starts one row earlier for each such change and is that much
|
||||
longer; the rows it repeats are dropped (`pageMerge`). `moved` holds
|
||||
each change for two pages, the one asked after it and the next
|
||||
(`state.moved` and `state.movedOnce`), since a vote still in flight
|
||||
when a page is asked may land after the server has read that page. */
|
||||
function pageAsk(offset, moved, page = PAGE) {
|
||||
const back = Math.min(offset, moved, MOST - page);
|
||||
return { offset: offset - back, limit: page + back };
|
||||
}
|
||||
|
||||
/* An answer onto the rows on screen: the rows not on screen yet after
|
||||
them, in the server's order, so the role a pass sank is not shown a
|
||||
second time; how far into the server's list the pages have now read;
|
||||
and whether it goes on. A short page is its end whatever `total` says,
|
||||
since the count and the page are two queries. */
|
||||
function pageMerge(items, ask, answer) {
|
||||
const shown = new Set(items.map((r) => r.uid));
|
||||
const offset = ask.offset + answer.items.length;
|
||||
return {
|
||||
items: items.concat(answer.items.filter((r) => !shown.has(r.uid))),
|
||||
offset,
|
||||
more: answer.items.length >= ask.limit && offset < answer.total,
|
||||
};
|
||||
}
|
||||
|
||||
// For the tests, which page through a list sorted afresh without a DOM.
|
||||
window.feedPaging = { pageAsk, pageMerge };
|
||||
|
||||
let feedRequest = 0;
|
||||
|
||||
/* The feed has nothing to show, so the empty state offers a retry rather
|
||||
than leaving the feed blank for good. */
|
||||
function feedFailed() {
|
||||
state.feedError = true;
|
||||
state.loaded = false;
|
||||
renderFeed();
|
||||
}
|
||||
|
||||
/* A reload starts the list again; an append asks for the rows after the
|
||||
ones shown. `keep` asks for that many in one go, which is how the list is
|
||||
put back as it was. The latch is let go only by the newest request: a
|
||||
reload overtaking an append must not let a second append start with an
|
||||
offset into the list being replaced. */
|
||||
ones read (`pageAsk`). `keep` asks for that many in one go, which is how
|
||||
the list is put back as it was. The latch is let go only by the newest
|
||||
request: a reload overtaking an append must not let a second append
|
||||
start with an offset into the list being replaced.
|
||||
|
||||
Nothing is asked while no agent is named and the agents have not said
|
||||
whether there is one, since the server would list every open role,
|
||||
ungated (`start`); the empty state offers the retry instead. */
|
||||
async function loadFeed({ append = false, keep = 0 } = {}) {
|
||||
if (append && (state.loading || state.items.length >= state.total)) return;
|
||||
if (state.filter.agent === null && !state.agentsLoaded) {
|
||||
if (!append) feedFailed();
|
||||
return;
|
||||
}
|
||||
if (append && (state.loading || !state.more)) return;
|
||||
const ask = append ? pageAsk(state.offset, state.moved + state.movedOnce)
|
||||
: { offset: 0, limit: Math.min(MOST, Math.max(PAGE, keep)) };
|
||||
const params = feedParams();
|
||||
params.set("offset", String(append ? state.items.length : 0));
|
||||
params.set("limit", String(Math.min(1000, Math.max(PAGE, keep))));
|
||||
params.set("offset", String(ask.offset));
|
||||
params.set("limit", String(ask.limit));
|
||||
const moved = state.moved;
|
||||
const request = ++feedRequest;
|
||||
state.loading = true;
|
||||
let data;
|
||||
@@ -423,23 +490,25 @@
|
||||
data = await api(`/api/roles?${params}`);
|
||||
} catch (err) {
|
||||
// An append failure leaves what is already on screen; a first load or
|
||||
// a reload has nothing to show, so the empty state offers a retry
|
||||
// rather than leaving the feed blank for good.
|
||||
if (!append && request === feedRequest) {
|
||||
state.feedError = true;
|
||||
state.loaded = false;
|
||||
renderFeed();
|
||||
}
|
||||
// a reload has nothing to show (`feedFailed`).
|
||||
if (!append && request === feedRequest) feedFailed();
|
||||
return;
|
||||
} finally {
|
||||
if (request === feedRequest) state.loading = false;
|
||||
}
|
||||
if (request !== feedRequest) return;
|
||||
state.feedError = false;
|
||||
// The changes made before this page was asked are counted once more, for
|
||||
// the next; those made while it was in flight wait for theirs.
|
||||
state.movedOnce = moved;
|
||||
state.moved -= moved;
|
||||
|
||||
const held = append ? state.items[state.selected]?.uid : state.readingUid;
|
||||
const page = pageMerge(append ? state.items : [], ask, data);
|
||||
state.total = data.total;
|
||||
state.items = append ? state.items.concat(data.items) : data.items;
|
||||
state.items = page.items;
|
||||
state.offset = page.offset;
|
||||
state.more = page.more;
|
||||
state.loaded = true;
|
||||
const at = held ? state.items.findIndex((r) => r.uid === held) : -1;
|
||||
state.selected = at >= 0 ? at : append ? state.selected : 0;
|
||||
@@ -514,12 +583,11 @@
|
||||
if (state.readingUid) refreshReader(state.readingUid);
|
||||
}
|
||||
|
||||
// The tag counts. Asked with `show_muted` and the agent only, the filters
|
||||
// the route takes, so picking a tag does not renumber the others.
|
||||
// The tag counts. Asked with the agent only, the filter the route takes, so
|
||||
// picking a tag does not renumber the others.
|
||||
let filtersRequest = 0;
|
||||
async function loadFilters() {
|
||||
const params = new URLSearchParams();
|
||||
if (state.filter.showMuted) params.set("show_muted", "true");
|
||||
if (state.filter.agent !== null) params.set("agent", String(state.filter.agent));
|
||||
const request = ++filtersRequest;
|
||||
const data = await api(`/api/filters?${params}`);
|
||||
@@ -547,7 +615,7 @@
|
||||
it: pushed, so back takes the filter off and brings the role back, or
|
||||
replacing the entry for text being typed (`replace`). A filter that
|
||||
changes nothing only closes the reader. */
|
||||
function setFilter(patch = {}, { counts = false, replace = false } = {}) {
|
||||
function setFilter(patch = {}, { replace = false } = {}) {
|
||||
Object.assign(state.filter, patch);
|
||||
if (state.readingUid && addressFor(pathFor(null)) === pathFor(null) + location.search) {
|
||||
closeReader();
|
||||
@@ -556,7 +624,6 @@
|
||||
route();
|
||||
}
|
||||
reload();
|
||||
if (counts) loadFilters().catch(() => {});
|
||||
renderRail();
|
||||
renderActive();
|
||||
}
|
||||
@@ -568,7 +635,7 @@
|
||||
// Text half typed into a chip field goes too, and with it its list.
|
||||
companyChips?.clear();
|
||||
placeChips?.clear();
|
||||
setFilter({}, { counts: true });
|
||||
setFilter();
|
||||
}
|
||||
|
||||
/* The other order over the same filter: the entry is replaced, since an
|
||||
@@ -593,11 +660,9 @@
|
||||
|
||||
/** Where a company's name links: the list on screen, every filter and the
|
||||
search kept, with this company as its one company filter, in place of
|
||||
any company chips and of `?company=`. A company this agent hid turns
|
||||
on Show hidden companies, or its own name would lead to an empty list. */
|
||||
any company chips and of `?company=`. */
|
||||
function companyHref(key) {
|
||||
const f = Object.assign({}, state.filter, { companyKeys: [key], company: "" });
|
||||
if (companies.get(key)?.affinity === -1) f.showMuted = true;
|
||||
return addressFor(pathFor(null), f);
|
||||
}
|
||||
|
||||
@@ -627,10 +692,9 @@
|
||||
const link = e.currentTarget;
|
||||
link.href = companyHref(link.dataset.company);
|
||||
companyNames.set(link.dataset.company, link.textContent.trim());
|
||||
const counts = state.filter.showMuted;
|
||||
readUrl(new URL(link.href).search, { view: false });
|
||||
$("#search").value = state.q;
|
||||
setFilter({}, { counts: counts !== state.filter.showMuted });
|
||||
setFilter();
|
||||
}
|
||||
|
||||
/* The strip's way back to every company: back, when the entry under this
|
||||
@@ -669,8 +733,8 @@
|
||||
|
||||
renderAgent();
|
||||
|
||||
// The order, the same three whatever is viewed. Agent match needs an
|
||||
// agent to score by, and CV match that agent's CV, so without one
|
||||
// The order, the same three whatever is viewed. What I'm looking for needs
|
||||
// an agent to score by, and What fits me that agent's CV, so without one
|
||||
// each is greyed and says why.
|
||||
const none = state.agentsLoaded && !state.agents.length;
|
||||
const why = {
|
||||
@@ -714,7 +778,6 @@
|
||||
setValue("f-remote", f.remote);
|
||||
setValue("f-pay", f.pay);
|
||||
setValue("f-judged", f.judged);
|
||||
setValue("f-show-muted", f.showMuted);
|
||||
setValue("f-closing", f.closing);
|
||||
setValue("f-gated", f.gated);
|
||||
renderFacets();
|
||||
@@ -746,7 +809,7 @@
|
||||
input.set(want);
|
||||
}
|
||||
|
||||
/* `?company=` or `?place=`, the substring a link from another page
|
||||
/* `?company=` or `?place=`, the substring an older link or saved filter
|
||||
brought: a plain chip saying what it matches, which takes it off and
|
||||
pushes, as a filter does. */
|
||||
function renderText(id, text, noun, key) {
|
||||
@@ -852,7 +915,7 @@
|
||||
gate.hidden = state.filter.agent === null;
|
||||
gate.textContent = state.filter.gated
|
||||
? "Filtered by this agent's fetch gate, and Show gated adds the roles it rejects."
|
||||
: "Already filtered by this agent's fetch gate: region, places and role.";
|
||||
: "Already filtered by this agent's fetch gate: world, location and role exclusion. Hidden companies are never shown.";
|
||||
if (agents.length) {
|
||||
// Rebuilt only when the agents changed, since the rail is drawn again on
|
||||
// every filter and a rebuild shuts a select that is open.
|
||||
@@ -950,7 +1013,6 @@
|
||||
async function applySaved(id) {
|
||||
const saved = state.saved.find((x) => x.id === id);
|
||||
if (!saved) return;
|
||||
const muted = state.filter.showMuted;
|
||||
const query = new URLSearchParams(saved.query);
|
||||
readUrl(`?${saved.query}`, { view: false });
|
||||
const sort = query.get("sort");
|
||||
@@ -961,7 +1023,7 @@
|
||||
await labelPlaces({ write: false });
|
||||
if (filtersOn(state.filter, { more: true })) $("#f-more").open = true;
|
||||
if (overlayRail() && state.railOpen) toggleRail();
|
||||
setFilter({}, { counts: muted !== state.filter.showMuted });
|
||||
setFilter();
|
||||
}
|
||||
|
||||
async function deleteSaved(id) {
|
||||
@@ -1236,7 +1298,7 @@
|
||||
}
|
||||
|
||||
/* The gutter's number, of 100, and what it says: the agent's score, or
|
||||
under CV match the CV match. A role `judge` has not read yet has an
|
||||
under What fits me the CV match. A role `judge` has not read yet has an
|
||||
empty bar and, faint, how close it is to the CV (`cv_fit`), which is
|
||||
what orders it there, else a dash. */
|
||||
function gutterOf(role) {
|
||||
@@ -1300,12 +1362,10 @@
|
||||
el.querySelector(".company")?.addEventListener("click", companyClick);
|
||||
}
|
||||
|
||||
/* A passed role and a hidden company's are dimmed, never dropped: the pass
|
||||
has to stay visible to be undone. */
|
||||
/* A passed role is dimmed, never dropped: the pass has to stay visible to be
|
||||
undone. */
|
||||
function paintRow(el, role) {
|
||||
const company = companies.get(role.company_key);
|
||||
el.dataset.read = String(Boolean(role.read_at));
|
||||
el.classList.toggle("muted", Boolean(company && company.affinity === -1));
|
||||
el.classList.toggle("passed", role.vote === -1);
|
||||
el.classList.toggle("gated", Boolean(role.gated_reason));
|
||||
}
|
||||
@@ -1350,31 +1410,67 @@
|
||||
empty.innerHTML = `<h2>Could not load the feed</h2>
|
||||
<p>The server did not answer. <button type="button" class="linky" data-retry
|
||||
>Try again</button>.</p>`;
|
||||
// Without the agents the feed may not know whose it is, so they are
|
||||
// asked again with it.
|
||||
const retry = empty.querySelector("[data-retry]");
|
||||
if (retry) retry.onclick = () => loadFeed();
|
||||
if (retry) retry.onclick = () => (state.agentsLoaded ? loadFeed() : start());
|
||||
} else {
|
||||
// An agent's list is what its fetch gate passes, which a phone,
|
||||
// whose rail is shut, learns here and at the end of the list.
|
||||
const filtered = state.q || filtersOn();
|
||||
const gate = gateKept() ? `${esc(gateOwner())} fetch gate` : "";
|
||||
empty.innerHTML = filtered
|
||||
? `<h2>Nothing matches</h2>
|
||||
<p>No open role fits these filters.
|
||||
<p>No open role ${gate ? `that passes ${gate} ` : ""}fits these filters.
|
||||
<button type="button" class="linky" data-clear>Clear them</button> and start again.</p>`
|
||||
: `<h2>No roles yet</h2>
|
||||
<p>Nothing has been pulled yet. <a href="/settings/sources#runs">Runs</a> starts one;
|
||||
<a href="/settings/sources#job-boards">Job boards</a> and
|
||||
<a href="/settings/sources#company-boards">Company boards</a> decide what it brings in.</p>`;
|
||||
: gate
|
||||
? `<h2>No roles yet</h2>
|
||||
<p>No open role passes ${gate} yet. <a href="/settings/sources#runs">Runs</a>
|
||||
starts a pull; <a href="/settings/sources#job-boards">Job boards</a> and
|
||||
<a href="/settings/sources#company-boards">Company boards</a> decide what it
|
||||
brings in, <a href="/settings/agents">Search agents</a> what the gate passes.</p>`
|
||||
: `<h2>No roles yet</h2>
|
||||
<p>Nothing has been pulled yet. <a href="/settings/sources#runs">Runs</a> starts one;
|
||||
<a href="/settings/sources#job-boards">Job boards</a> and
|
||||
<a href="/settings/sources#company-boards">Company boards</a> decide what it brings in.</p>`;
|
||||
const clear = empty.querySelector("[data-clear]");
|
||||
if (clear) clear.onclick = () => { state.q = ""; $("#search").value = ""; clearFilters(); };
|
||||
}
|
||||
}
|
||||
|
||||
// The end of the list says which end it is: a list that has run out and
|
||||
// one that is still loading look the same otherwise.
|
||||
// one that is still loading look the same otherwise. Whether it has run
|
||||
// out is the server's to say (`pageMerge`), since the rows on screen
|
||||
// keep the ones a vote or a read has since taken out of its list.
|
||||
const end = $("#feed-end");
|
||||
const n = state.items.length;
|
||||
end.hidden = !state.loaded || !n;
|
||||
end.textContent = n < state.total
|
||||
? "Loading more…"
|
||||
: `${number.format(n)} ${n === 1 ? "role" : "roles"} — that is all of them.`;
|
||||
end.textContent = state.more ? "Loading more…" : `${listSaid(n)} — that is all of them.`;
|
||||
}
|
||||
|
||||
/** Whether the list keeps to the viewing agent's fetch gate: there is an
|
||||
agent, and Show gated is off. */
|
||||
const gateKept = () => state.filter.agent !== null && !state.filter.gated;
|
||||
|
||||
/** Whose fetch gate, for a sentence: the viewing agent's name, or "this
|
||||
agent's" until the agents are in. */
|
||||
function gateOwner() {
|
||||
const agent = viewed();
|
||||
return agent ? `${agent.name}'s` : "this agent's";
|
||||
}
|
||||
|
||||
/** The rows on screen in words, for the end of the list: how many and,
|
||||
with an agent viewing, that they are what its fetch gate passes (and
|
||||
the filters keep), or with Show gated that they hold what it rejects. */
|
||||
function listSaid(n) {
|
||||
const roles = `${number.format(n)} ${n === 1 ? "role" : "roles"}`;
|
||||
if (state.filter.agent === null) return roles;
|
||||
const gate = `${gateOwner()} fetch gate`;
|
||||
const filtered = Boolean(state.q) || filtersOn() > (state.filter.gated ? 1 : 0);
|
||||
if (state.filter.gated) {
|
||||
return `${roles}${filtered ? " in these filters" : ""}, the ones ${gate} rejects included`;
|
||||
}
|
||||
return `${roles} ${n === 1 ? "passes" : "pass"} ${gate}${filtered ? " and these filters" : ""}`;
|
||||
}
|
||||
|
||||
/* With exactly one company filtered, which is where its name leads, a
|
||||
@@ -1432,43 +1528,48 @@
|
||||
}
|
||||
|
||||
/* Which parts of the viewing agent's fetch gate apply to a company's
|
||||
roles: All countries lifts the place part (the Where lines), All roles
|
||||
the role part (fetch terms, titles to leave out); the region always
|
||||
holds. A favourite only ranks higher, so it is not named here. Nothing
|
||||
with no agent, when no fetch gate applies. */
|
||||
roles: All locations lifts the location gate, All roles the role
|
||||
exclusion and the fetch terms; the world gate always holds. A favourite
|
||||
only ranks higher, so it is not named here. A hidden company has no
|
||||
roles in the feed at all. Nothing with no agent, when no fetch gate
|
||||
applies. */
|
||||
function gateNote(company) {
|
||||
if (state.filter.agent === null) return "";
|
||||
const countries = Boolean(company?.all_countries);
|
||||
if (company?.affinity === -1) return "Hidden for this agent: its roles are never shown.";
|
||||
const locations = Boolean(company?.all_countries);
|
||||
const roles = Boolean(company?.all_roles);
|
||||
if (countries && roles) {
|
||||
return "All countries and All roles: every role in the agent's region, "
|
||||
+ "not filtered by place or by role.";
|
||||
if (locations && roles) {
|
||||
return "All locations and All roles: every role in the agent's world, "
|
||||
+ "not filtered by location, role exclusion or fetch terms.";
|
||||
}
|
||||
if (countries) {
|
||||
return "All countries: every role in the agent's region, not filtered by place. "
|
||||
+ "The agent's fetch terms and titles to leave out still apply.";
|
||||
if (locations) {
|
||||
return "All locations: every role in the agent's world, not filtered by location. "
|
||||
+ "The agent's role exclusion still applies.";
|
||||
}
|
||||
if (roles) {
|
||||
return "All roles: not filtered by role, no fetch terms and no titles left out. "
|
||||
+ "The agent's places still apply.";
|
||||
return "All roles: no titles left out and no fetch terms. "
|
||||
+ "The agent's locations still apply.";
|
||||
}
|
||||
return "The roles this agent's fetch gate passes: its region, places and role. "
|
||||
+ "All countries lifts the places for this company, All roles the role.";
|
||||
return "The roles this agent's fetch gate passes: its world, locations and role exclusion. "
|
||||
+ "All locations lifts the locations for this company, All roles the role exclusion.";
|
||||
}
|
||||
|
||||
/* A company seen only through a search has no row until its first change
|
||||
creates one, so the list is reloaded then. A change dims or undims its
|
||||
rows at once; where they rank waits on the `score` it queued, whose id
|
||||
the affinity route answers with (`updated.score_job`), and the feed
|
||||
reloads when that has finished. */
|
||||
creates one, so the list is reloaded then. Hiding takes the company's
|
||||
rows out at once, since a hidden company is never in the feed; where the
|
||||
others rank waits on the `score` it queued, whose id the affinity route
|
||||
answers with (`updated.score_job`), and the feed reloads when that has
|
||||
finished. A list the company filter has come off meanwhile has its rows
|
||||
asked again in the new order, as `renderAffinity` does. */
|
||||
async function affinityChanged(key, company, updated) {
|
||||
if (!company) await loadCompanies(state.filter.agent).catch(() => {});
|
||||
renderFeed();
|
||||
if (updated?.affinity === -1) reload(); else renderFeed();
|
||||
const status = () => $("#company-strip .strip-status");
|
||||
if (status()) status().textContent = "Scoring again; the list reloads when that is done.";
|
||||
await scored(updated?.score_job);
|
||||
if (status()) status().textContent = "";
|
||||
if (state.filter.companyKeys.includes(key)) reload();
|
||||
else loadFeed({ keep: state.items.length });
|
||||
}
|
||||
|
||||
/* Wait for the `score` job `job`, the way agents.js's `rescore` waits:
|
||||
@@ -1568,6 +1669,8 @@
|
||||
async function markRead(uid) {
|
||||
const item = state.items.find((r) => r.uid === uid);
|
||||
if (item && item.read_at) return;
|
||||
// With Show read off the server's list loses it, see `pageAsk`.
|
||||
if (!state.filter.showRead) state.moved += 1;
|
||||
if (item) { item.read_at = new Date().toISOString(); renderFeed(); }
|
||||
try {
|
||||
const role = await api(`/api/roles/${encodeURIComponent(uid)}/read`, { method: "POST" });
|
||||
@@ -1641,33 +1744,20 @@
|
||||
try { return languageNames?.of(code) || code; } catch (err) { return code; }
|
||||
}
|
||||
|
||||
const MUST_MARKS = { met: ["✓", "met"], broken: ["✗", "broken"], unclear: ["?", "unclear"] };
|
||||
|
||||
/* The agent's `judge` reading against its CV: the total and its three
|
||||
numbers, career as the server counted it (capped at 39 when a must-have
|
||||
is broken, with what the model said), the must-haves with a verdict
|
||||
each, then the strengths and gaps it named. A box as the summary is,
|
||||
since it is the same kind of thing: a model's reading, not the posting. */
|
||||
numbers, each of 10, then the strengths and gaps it named. A box as the
|
||||
summary is, since it is the same kind of thing: a model's reading, not
|
||||
the posting. */
|
||||
function cvMatch(r) {
|
||||
const j = r.judgement;
|
||||
if (!j) return "";
|
||||
const list = (items, none) => (items?.length
|
||||
? `<ul>${items.map((x) => `<li>${esc(x)}</li>`).join("")}</ul>`
|
||||
: `<p class="muted small">${none}</p>`);
|
||||
const counted = j.career_capped ?? j.career;
|
||||
const career = counted < j.career
|
||||
? `career ${counted} (capped: breaks a must-have; model said ${j.career})` : `career ${j.career}`;
|
||||
const musts = Array.isArray(j.musts) && j.musts.length
|
||||
? `<ul class="cv-musts">${j.musts.map((m) => {
|
||||
const [mark, word] = MUST_MARKS[m.verdict] || MUST_MARKS.unclear;
|
||||
return `<li class="must-${word}"><span class="mark" role="img"
|
||||
aria-label="${word}" title="${word}">${mark}</span><span>${esc(m.text)}</span></li>`;
|
||||
}).join("")}</ul>` : "";
|
||||
return `<section class="reader-summary reader-cv">
|
||||
<h2 class="reader-label">CV match</h2>
|
||||
<p class="cv-total"><b>${Math.round(j.cv_score)}</b> of 100
|
||||
<span>skills ${j.skills} · experience ${j.experience} · ${career}</span></p>
|
||||
${musts}
|
||||
<span>skills ${j.skills}/10 · experience ${j.experience}/10 · career ${j.career}/10</span></p>
|
||||
<div class="cv-lists">
|
||||
<div><h3 class="reader-label">Strengths</h3>${list(j.strengths, "None named.")}</div>
|
||||
<div><h3 class="reader-label">Gaps</h3>${list(j.gaps, "None named.")}</div>
|
||||
@@ -1738,6 +1828,10 @@
|
||||
const waiting = state.summarising === state.readingUid;
|
||||
const teaser = loaded && !r.article_text && out && (r.description || "").length < TEASER;
|
||||
|
||||
// Under the title, in the copy note's style: that the viewing agent's
|
||||
// fetch gate rejects the role and why (`gated_reason`), since a link
|
||||
// from elsewhere, a saved role on Applications say, opens one the feed
|
||||
// never lists.
|
||||
el.innerHTML = `
|
||||
<div class="reader-bar">
|
||||
<span class="judge"></span>
|
||||
@@ -1761,6 +1855,9 @@
|
||||
<h1>${esc(r.title || "")}</h1>
|
||||
${loaded && r.dup_of ? `<p class="reader-note reader-copy">A copy of
|
||||
<a href="${esc(pathFor(r.dup_of))}">a posting</a> the feed lists once.</p>` : ""}
|
||||
${loaded && state.filter.agent !== null && r.gated_reason
|
||||
? `<p class="reader-note reader-copy">Rejected by ${esc(gateOwner())} fetch gate:
|
||||
${esc(r.gated_reason)}. The feed lists it only with Show gated.</p>` : ""}
|
||||
${pay ? `<p class="reader-salary">${esc(pay.range)}</p>
|
||||
${pay.detail ? `<p class="muted small">${esc(pay.detail)}</p>` : ""}
|
||||
${pay.note ? `<p class="muted small">${esc(pay.note)}</p>` : ""}` : ""}
|
||||
@@ -1812,15 +1909,22 @@
|
||||
/* The company's ⊘ ▲ ★, the one place a company is judged outside the
|
||||
Companies page, and its two fetch overrides beside them. A company seen
|
||||
only through a search has no row until its first change creates one, so
|
||||
the list is reloaded then, and the overrides, which need a row, wait for it. A
|
||||
change dims or undims its rows in the feed. */
|
||||
the list is reloaded then, and the overrides, which need a row, wait for it.
|
||||
Hiding reloads the feed, which no longer holds the company's roles. Any
|
||||
other stance moves every role of the company once the `score` it queued
|
||||
has run, up into the rows already read as well as down out of them,
|
||||
which no page asked from an offset makes up for (`pageAsk`); so the rows
|
||||
on screen are asked again then, as many, in the new order. */
|
||||
function renderAffinity(slot, r) {
|
||||
const company = companies.get(r.company_key);
|
||||
if (!r.company_key) { slot.replaceChildren(); return; }
|
||||
slot.replaceChildren(affinityControl(r.company_key, company ? company.affinity : 0, async () => {
|
||||
slot.replaceChildren(affinityControl(r.company_key, company ? company.affinity : 0, async (updated) => {
|
||||
if (!company) await loadCompanies(state.filter.agent).catch(() => {});
|
||||
renderAffinity(slot, r);
|
||||
if (updated?.affinity === -1) { reload(); return; }
|
||||
renderFeed();
|
||||
await scored(updated?.score_job);
|
||||
loadFeed({ keep: state.items.length });
|
||||
}, state.filter.agent));
|
||||
if (company) {
|
||||
slot.append(" ", overridesControl(r.company_key, company,
|
||||
@@ -2109,8 +2213,6 @@
|
||||
});
|
||||
$("#f-seniority-unknown").addEventListener("change",
|
||||
(e) => setFilter({ seniorityUnknown: e.target.checked }));
|
||||
$("#f-show-muted").addEventListener("change",
|
||||
(e) => setFilter({ showMuted: e.target.checked }, { counts: true }));
|
||||
$("#f-gated").addEventListener("change", (e) => setFilter({ gated: e.target.checked }));
|
||||
$("#f-closing").addEventListener("change", (e) => setFilter({ closing: e.target.checked }));
|
||||
|
||||
@@ -2132,6 +2234,16 @@
|
||||
new IntersectionObserver((entries) => {
|
||||
if (entries.some((e) => e.isIntersecting)) loadFeed({ append: true });
|
||||
}, { rootMargin: "600px" }).observe($("#feed-end"));
|
||||
// ▼ ● ▲ and ★, which likes, on a row or in the reader move the role in
|
||||
// the server's order, or out of its list under Judged or with Show read
|
||||
// off, which the next page makes up for (`pageAsk`). The keys press the
|
||||
// same buttons. Counted at the click, failed or undone ones too, since a
|
||||
// page that starts a row too early only repeats one.
|
||||
for (const pane of [$("#roles-list"), $("#reader")]) {
|
||||
pane.addEventListener("click", (e) => {
|
||||
if (e.target.closest(".judge button")) state.moved += 1;
|
||||
});
|
||||
}
|
||||
|
||||
// Kept on the way out, whichever way that is, so back finds it.
|
||||
window.addEventListener("pagehide", keep);
|
||||
@@ -2151,6 +2263,60 @@
|
||||
});
|
||||
}
|
||||
|
||||
/* Whose feed it is, then the feed. The server keeps to a fetch gate only
|
||||
for an agent it is told of, and without one lists every open role, so
|
||||
the first request carries the agent whenever one is known before the
|
||||
agents are in: the address's, else the one this browser viewed last
|
||||
(`storedAgent`). `chooseAgent` checks it against the enabled ones when
|
||||
they land, and the feed reloads if it picks another. With neither, the
|
||||
feed waits for the agents: with none enabled the list of every open
|
||||
role is the right one, and when they cannot be had the feed's error
|
||||
state offers to try again (`loadFeed`), rather than every open role
|
||||
passing for the agent's list.
|
||||
|
||||
The rest fills in when it arrives and the feed does not wait on it: the
|
||||
companies, the tag counts, the saved filters. Each one drawn before is
|
||||
greyed or missing, which is what "no agent" (renderAgent), an unjoined
|
||||
company (paintRow) and the tag row being empty already mean, so nothing
|
||||
here is a new state to handle. `keep` and `y` put the list back as it
|
||||
was (`kept`). Run again by the error state's retry when the agents did
|
||||
not come. */
|
||||
async function start({ keep = 0, y = null } = {}) {
|
||||
if (state.filter.agent === null) state.filter.agent = storedAgent();
|
||||
const agentsIn = loadAgents().then(() => { chooseAgent(); return true; }, () => false);
|
||||
const trackedIn = loadTracked().catch(() => {});
|
||||
// What the reader's fetch and then the feed's first request go out with,
|
||||
// so the agents landing later know whether either is to be asked again.
|
||||
const routed = state.filter.agent;
|
||||
route();
|
||||
if (state.filter.agent === null) await agentsIn;
|
||||
const asked = state.filter.agent;
|
||||
const sortAsked = sortShown();
|
||||
const enhance = (async () => {
|
||||
const [ready] = await Promise.all([agentsIn, trackedIn]);
|
||||
if (!ready) return;
|
||||
await loadCompanies(state.filter.agent).catch(() => {});
|
||||
renderRail();
|
||||
renderActive();
|
||||
loadFilters().catch(() => {});
|
||||
loadSaved().catch(() => {});
|
||||
// Reload for the score and votes when the agent chosen is not the one
|
||||
// asked for (a remembered one disabled since), and either way bring
|
||||
// the reader's affinity control and judge buttons up to date, since
|
||||
// `companies` was still empty when it first painted. What fits me
|
||||
// asked of an agent with no CV is What I'm looking for.
|
||||
if (state.filter.agent !== asked || sortShown() !== sortAsked) reload();
|
||||
else renderFeed();
|
||||
if (state.readingUid) {
|
||||
if (state.filter.agent !== routed) refreshReader(state.readingUid);
|
||||
else renderReader();
|
||||
}
|
||||
})();
|
||||
await loadFeed({ keep });
|
||||
if (y !== null && !state.readingUid) window.scrollTo(0, y);
|
||||
await enhance;
|
||||
}
|
||||
|
||||
document.addEventListener("DOMContentLoaded", async () => {
|
||||
readUrl();
|
||||
const was = kept();
|
||||
@@ -2161,44 +2327,6 @@
|
||||
// Named before the first request, which a token the server does not
|
||||
// know would turn into a 422 rather than a list.
|
||||
await labelPlaces();
|
||||
|
||||
// The feed is the page; agents and companies are enhancements that fill
|
||||
// in when they arrive, not something it waits on — a slow or failed
|
||||
// /api/agents or /api/companies must not leave it blank. Each one drawn
|
||||
// here is greyed or missing until `enhance` lands, which is what "no
|
||||
// agent" (renderAgent), an unjoined company (paintRow) and the tag row
|
||||
// being empty already mean, so nothing here is a new state to handle.
|
||||
const before = state.filter.agent;
|
||||
const sortBefore = sortShown();
|
||||
const enhance = (async () => {
|
||||
try {
|
||||
await Promise.all([loadTracked(), loadAgents()]);
|
||||
chooseAgent();
|
||||
await loadCompanies(state.filter.agent);
|
||||
} catch (err) {
|
||||
return;
|
||||
}
|
||||
renderRail();
|
||||
renderActive();
|
||||
loadFilters().catch(() => {});
|
||||
loadSaved().catch(() => {});
|
||||
// The feed's first request went out with whatever agent `readUrl` had
|
||||
// (often none): reload for the score and votes once the real one is
|
||||
// known, and either way bring the reader's affinity control and judge
|
||||
// buttons up to date, since `companies` was still empty when it first
|
||||
// painted.
|
||||
// CV match asked of an agent with no CV is Agent match.
|
||||
if (state.filter.agent !== before || sortShown() !== sortBefore) reload();
|
||||
else renderFeed();
|
||||
if (state.readingUid) {
|
||||
if (state.filter.agent !== before) refreshReader(state.readingUid);
|
||||
else renderReader();
|
||||
}
|
||||
})();
|
||||
|
||||
route();
|
||||
await loadFeed({ keep: was ? was.count : 0 });
|
||||
if (was && !state.readingUid) window.scrollTo(0, was.y || 0);
|
||||
await enhance;
|
||||
await start(was ? { keep: was.count, y: was.y || 0 } : {});
|
||||
});
|
||||
})();
|
||||
+66
-13
@@ -55,6 +55,10 @@ CREATE TABLE IF NOT EXISTS roles (
|
||||
country TEXT NOT NULL DEFAULT '',
|
||||
remote TEXT NOT NULL DEFAULT ''
|
||||
CHECK (remote IN ('', 'remote', 'hybrid', 'onsite')),
|
||||
-- 1 when the title or text says the role is worked remotely while no
|
||||
-- field does (prep, app.prep.words.remote_in_text): the location gate
|
||||
-- lets such a role through until extract has read it.
|
||||
remote_text INTEGER NOT NULL DEFAULT 0,
|
||||
salary_min INTEGER,
|
||||
salary_max INTEGER,
|
||||
currency TEXT NOT NULL DEFAULT '',
|
||||
@@ -328,9 +332,8 @@ END;
|
||||
-- as JSON: a key it lacks means the `Limits` default. fetch_terms is a JSON
|
||||
-- list of words that narrow what big boards send (app/jobs/reach.py); empty
|
||||
-- means no narrowing. cv_text, cv_languages (a JSON list of {language,
|
||||
-- level}), like_text (what they like: the work, the workplace, situations),
|
||||
-- avoid_text (what they do not like) and must_text (must-haves and
|
||||
-- deal-breakers, set through the API only) are what judge reads a role
|
||||
-- level}), like_text (what they like: the work, the workplace, situations)
|
||||
-- and avoid_text (what they do not like) are what judge reads a role
|
||||
-- against; an agent with no CV is not judged.
|
||||
CREATE TABLE IF NOT EXISTS agents (
|
||||
id INTEGER PRIMARY KEY,
|
||||
@@ -344,7 +347,6 @@ CREATE TABLE IF NOT EXISTS agents (
|
||||
cv_text TEXT NOT NULL DEFAULT '',
|
||||
cv_languages TEXT NOT NULL DEFAULT '[]',
|
||||
avoid_text TEXT NOT NULL DEFAULT '',
|
||||
must_text TEXT NOT NULL DEFAULT '',
|
||||
like_text TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
@@ -431,13 +433,12 @@ CREATE TABLE IF NOT EXISTS agent_cv_vectors (
|
||||
|
||||
-- One agent's judge reading of a role against its CV. Its own table, since
|
||||
-- replace_scores rewrites scores every run and a judgement outlives that
|
||||
-- and the role closing. skills, experience and career are 0 to 100,
|
||||
-- cv_score their weighted total, strengths and gaps JSON lists. input_key
|
||||
-- is a hash of the CV, prompts, avoid text and the role's text it read: a
|
||||
-- different one is read again. musts is a JSON list of {id, text, verdict},
|
||||
-- the model's verdict on each of the agent's must-haves (one per line of its
|
||||
-- must text), broken how many are "broken": career counts as at most 39 then
|
||||
-- (app.models.BROKEN_CAP), applied when the role is served. tries counts
|
||||
-- and the role closing. skills, experience and career are 0 to 10 (0 to
|
||||
-- 100 until migration 33), cv_score their weighted total times 10 at the
|
||||
-- default weights (not read: app.models.cv_score is worked out when a role
|
||||
-- is served), strengths and gaps JSON lists. input_key
|
||||
-- is a hash of the CV, prompts, languages, like and avoid texts and the
|
||||
-- role's text it read: a different one is read again. tries counts
|
||||
-- replies that left the role without one since; at 3 it is given up until the agent is saved again. A
|
||||
-- row with judged_at NULL holds only tries.
|
||||
CREATE TABLE IF NOT EXISTS role_judgements (
|
||||
@@ -449,8 +450,6 @@ CREATE TABLE IF NOT EXISTS role_judgements (
|
||||
cv_score REAL,
|
||||
strengths TEXT NOT NULL DEFAULT '[]',
|
||||
gaps TEXT NOT NULL DEFAULT '[]',
|
||||
musts TEXT NOT NULL DEFAULT '[]',
|
||||
broken INTEGER NOT NULL DEFAULT 0,
|
||||
input_key TEXT NOT NULL DEFAULT '',
|
||||
tries INTEGER NOT NULL DEFAULT 0,
|
||||
judged_at TEXT,
|
||||
@@ -758,6 +757,15 @@ UPDATE roles SET seniority = COALESCE(NULLIF(seniority_model, ''),
|
||||
(32, "the cut: scores.fit and scores.cut, agents' limits reset to the cut's defaults, "
|
||||
"prefer gates gone, saved_filters, and enrich renamed fetch text",
|
||||
lambda conn: _add_cut(conn)),
|
||||
# A Python step: JSON limits, and a count for the log.
|
||||
(33, "the judge on 0 to 10: every judgement cleared to be made again, and summarise_top "
|
||||
"50 becomes 30", lambda conn: _judge_out_of_ten(conn)),
|
||||
# A Python step: it drops the columns only where they are there. The
|
||||
# judge reads every role again (judge.FORMAT).
|
||||
(34, "agents.must_text, role_judgements.musts and broken dropped: the profile is the "
|
||||
"CV, languages, likes and dislikes", lambda conn: _drop_musts(conn)),
|
||||
(35, "roles.remote_text: the text says remote while no field does, read by prep for the "
|
||||
"location gate", lambda conn: _add_remote_text(conn)),
|
||||
)
|
||||
|
||||
#: What a file created from `SCHEMA` today is already at.
|
||||
@@ -1007,6 +1015,15 @@ def _add_posting_url(conn: sqlite3.Connection) -> None:
|
||||
conn.execute("ALTER TABLE roles ADD COLUMN posting_url TEXT NOT NULL DEFAULT ''")
|
||||
|
||||
|
||||
def _add_remote_text(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 35's step: `roles.remote_text`, 0 until the next prep,
|
||||
which maps every open role again since its rules changed. Runs twice
|
||||
without harm."""
|
||||
columns = {r[1] for r in conn.execute("PRAGMA table_info(roles)")}
|
||||
if "remote_text" not in columns:
|
||||
conn.execute("ALTER TABLE roles ADD COLUMN remote_text INTEGER NOT NULL DEFAULT 0")
|
||||
|
||||
|
||||
def _add_remote_area(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 27's step: `roles.remote_model` empty and
|
||||
`roles.remote_where_model` NULL for every role, so `extract` reads the
|
||||
@@ -1191,6 +1208,42 @@ def _add_cut(conn: sqlite3.Connection) -> None:
|
||||
"dropped", reset, dropped)
|
||||
|
||||
|
||||
def _judge_out_of_ten(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 33's step. The judge scores 0 to 10 rather than 0 to 100
|
||||
(`judge.FORMAT` 4), so every judgement is cleared: its numbers, the
|
||||
verdicts, strengths and gaps, `input_key`, `judged_at` and `tries`, and no
|
||||
old number is shown on the new scale. The rows stay, as rows of no
|
||||
judgement, and the judge reads every pair again. An agent whose
|
||||
`summarise_top` is 50, migration 32's default, takes 30, the new one.
|
||||
Runs twice without harm, also after migration 34 dropped the verdicts."""
|
||||
columns = {r[1] for r in conn.execute("PRAGMA table_info(role_judgements)")}
|
||||
verdicts = ", musts = '[]', broken = 0" if {"musts", "broken"} <= columns else ""
|
||||
cleared = conn.execute(
|
||||
f"""UPDATE role_judgements
|
||||
SET skills = NULL, experience = NULL, career = NULL, cv_score = NULL,
|
||||
strengths = '[]', gaps = '[]'{verdicts},
|
||||
input_key = '', tries = 0, judged_at = NULL""").rowcount
|
||||
lowered = conn.execute(
|
||||
"""UPDATE agents SET limits = json_set(limits, '$.summarise_top', 30)
|
||||
WHERE json_valid(limits) AND json_type(limits) = 'object'
|
||||
AND json_extract(limits, '$.summarise_top') = 50""").rowcount
|
||||
log.info("migration 33: %d judgements cleared for the 0 to 10 scale, %d agents' "
|
||||
"summarise_top 50 set to 30", cleared, lowered)
|
||||
|
||||
|
||||
def _drop_musts(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 34's step: the must-haves are gone, the agent's text
|
||||
(`agents.must_text`) and the judge's verdicts on it
|
||||
(`role_judgements.musts`, `broken`), and with them the cap on career.
|
||||
Runs twice without harm."""
|
||||
for table, columns in (("agents", ("must_text",)),
|
||||
("role_judgements", ("musts", "broken"))):
|
||||
present = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
|
||||
for column in columns:
|
||||
if column in present:
|
||||
conn.execute(f"ALTER TABLE {table} DROP COLUMN {column}")
|
||||
|
||||
|
||||
def _add_all_roles(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 21's step. The column; the tag for every company that is a
|
||||
favourite of any agent, since favourite no longer widens what a pull
|
||||
|
||||
+226
-151
@@ -11,7 +11,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import sqlite3
|
||||
from collections import Counter
|
||||
from collections.abc import Callable, Iterable, Iterator, Mapping, MutableMapping, Sequence
|
||||
from collections.abc import Callable, Iterable, Iterator, MutableMapping, Sequence
|
||||
from contextlib import closing, contextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
@@ -26,7 +26,7 @@ from app.models import (
|
||||
AFFINITY, HAND_REASONS, LEVELS, STATUSES, Agent, Application,
|
||||
ApplicationEvent, Candidate, Company, CvLanguage, Gate, Judgement, Limits, Prompt, Role,
|
||||
RoleCopy, RolePlace, RoleRow, SavedFilter, ScoredRole, cv_languages as normal_languages, cv_score,
|
||||
BROKEN_CAP, capped_career, language_flag, role_languages,
|
||||
language_flag, role_languages,
|
||||
Subscription, Weights, check_prompts, company_fold, company_key, country_of, excerpt, place_key,
|
||||
fetch_terms as normal_terms, host_word, iso_day, posted_at, posted_host, remote_of, title_key,
|
||||
)
|
||||
@@ -65,7 +65,8 @@ _EXTRACT_TEXT = ("place", "country", "currency", "permit")
|
||||
GIVE_UP_AFTER = 3
|
||||
|
||||
#: A role `extract` has yet to read: still open, no copy of another
|
||||
#: (`rebuild_dupes`), not given up, and either never answered, without a
|
||||
#: (`rebuild_dupes`), not given up, and either never answered (or answered
|
||||
#: before a better text came, `upsert_role`), without a
|
||||
#: model level, never asked where a remote hire may live
|
||||
#: (`remote_where_model` NULL), or never asked for its languages and deadline
|
||||
#: (`languages_required` NULL). The last three make `extract` the
|
||||
@@ -83,6 +84,15 @@ _EXTRACT_WAITING = ("(extracted_at IS NULL OR seniority_model = '' OR remote_whe
|
||||
_IN_CUT = ("r.uid IN (SELECT s.uid FROM scores s JOIN agents a ON a.id = s.agent_id "
|
||||
"AND a.enabled = 1 WHERE s.cut <> '')")
|
||||
|
||||
#: A role one agent's feed shows, as a `WHERE` on `r` taking the agent's id
|
||||
#: twice: its fetch gate let the role through (`_roles_from` with
|
||||
#: `fetch_gate`), and the company is not one it hid. With the open and kept
|
||||
#: copy test, what the rail's counts for that agent cover.
|
||||
_SHOWN = ("EXISTS (SELECT 1 FROM scores s WHERE s.uid = r.uid AND s.agent_id = ? "
|
||||
"AND s.gated_reason = '') "
|
||||
"AND r.company_key NOT IN (SELECT company_key FROM agent_companies "
|
||||
"WHERE agent_id = ? AND affinity = -1)")
|
||||
|
||||
#: The order within one agent's cut, over its scores as `s`: in by fit, best
|
||||
#: fit first, then in by CV closeness, closest first, then kept.
|
||||
_CUT_ORDER = ("CASE s.cut WHEN 'fit' THEN 0 WHEN 'cv' THEN 1 ELSE 2 END, "
|
||||
@@ -130,7 +140,7 @@ _VOTED_SCORE = ("MIN(1.0, s.score * CASE v.vote WHEN -1 THEN 0.5 "
|
||||
_WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?"
|
||||
|
||||
#: The agent's `judge` readings beside every role, the same way: `j.skills`,
|
||||
#: `j.experience`, `j.career` and `j.broken` are NULL for a role not judged and with no
|
||||
#: `j.experience` and `j.career` are NULL for a role not judged and with no
|
||||
#: agent.
|
||||
_WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agent_id = ? "
|
||||
"AND j.judged_at IS NOT NULL")
|
||||
@@ -138,20 +148,17 @@ _WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agen
|
||||
|
||||
|
||||
def _cv_sql(weights: Weights | None) -> str:
|
||||
"""`cv_score` as SQL for one agent's weights, unrounded (`app.models.cv_score`
|
||||
rounds it): NULL for a role not judged, and for all three weights 0. Career
|
||||
counts as at most `BROKEN_CAP` when the judgement holds a broken
|
||||
must-have (`j.broken`), as `app.models.capped_career` has it. The
|
||||
numbers are the model's validated, finite floats, so they are written in
|
||||
rather than bound, the select's placeholders being then no concern."""
|
||||
"""`cv_score` as SQL for one agent's weights, 0 to 100 from the judge's
|
||||
three numbers 0 to 10, unrounded (`app.models.cv_score` rounds it): NULL
|
||||
for a role not judged, and for all three weights 0. The weights are
|
||||
validated, finite floats, so they are written in rather than bound, the
|
||||
select's placeholders being then no concern."""
|
||||
w = weights or Weights()
|
||||
total = w.cv_skills + w.cv_experience + w.cv_career
|
||||
if not total:
|
||||
return "NULL"
|
||||
career = (f"(CASE WHEN j.broken > 0 AND j.career > {BROKEN_CAP} THEN {BROKEN_CAP} "
|
||||
"ELSE j.career END)")
|
||||
return (f"(({w.cv_skills!r} * j.skills + {w.cv_experience!r} * j.experience "
|
||||
f"+ {w.cv_career!r} * {career}) / {total!r})")
|
||||
return (f"(10.0 * ({w.cv_skills!r} * j.skills + {w.cv_experience!r} * j.experience "
|
||||
f"+ {w.cv_career!r} * j.career) / {total!r})")
|
||||
|
||||
|
||||
def _by_cv(expression: str) -> str:
|
||||
@@ -288,14 +295,22 @@ def _gate_sql(gate: Gate, gaz: places.Gazetteer) -> tuple[str, str, list[Any]]:
|
||||
`params` belong to `match`; `empty` takes none.
|
||||
|
||||
`country` and `place` match over the gazetteer, as the feed's Place
|
||||
filter does (`_places_sql`): a country gate is met by a role whose
|
||||
filter does (`_places_sql`), and also on a role place that holds one of
|
||||
the gate's places (`wider`): a country gate is met by a role whose
|
||||
`country` is one of its codes or that has a place in one of them, so
|
||||
`DK` is met by `Copenhagen; London`. Its column is empty only when the
|
||||
role has neither. A `require` gate on country reads a remote role whose
|
||||
`DK` is met by `Copenhagen; London`, and by Øresund or Europe alone,
|
||||
which are no evidence the role is outside Denmark; a town in Germany
|
||||
is, also beside Europe. Its column is empty only when the role has neither a country
|
||||
nor a resolved place. A `require` gate on country reads a remote role whose
|
||||
area is stated through the area alone, as the remote chips do
|
||||
(`Gazetteer.overlap`): "Home Based - Worldwide" and "Germany (Remote)"
|
||||
meet Europe, "Remote US" does not, and neither is empty. A place gate's
|
||||
value is a place token, bare text a `place:`.
|
||||
value is a place token, bare text a `place:`. Its column is empty when
|
||||
the role has no place the gazetteer resolved, as "Storkøbenhavn" or
|
||||
"Headquarters" are not: that is no evidence the role is elsewhere. Such
|
||||
a role can still match, by its country or by whole words of its line
|
||||
for a town the gazetteer does not know, so a place gate's `match` is
|
||||
tested before its `empty` (`Store.score_candidates`).
|
||||
|
||||
`remote_from` takes place tokens without their `remote:` (`country:DK`,
|
||||
`anywhere`) and matches as the remote chips do. It is empty for a role
|
||||
@@ -313,7 +328,8 @@ def _gate_sql(gate: Gate, gaz: places.Gazetteer) -> tuple[str, str, list[Any]]:
|
||||
return f"(r.remote = '' OR (r.remote = 'remote' AND NOT {_HAS_AREA}))", match, params
|
||||
if gate.field == "country":
|
||||
tokens = [f"country:{v}" for v in values]
|
||||
match, params = _places_sql(tokens, gaz, table="r", codes=[v.upper() for v in values])
|
||||
match, params = _places_sql(tokens, gaz, table="r", codes=[v.upper() for v in values],
|
||||
wider=True)
|
||||
empty = f"(r.country = '' AND {_unplaced(gaz)})"
|
||||
if gate.mode != "require":
|
||||
return empty, match, params
|
||||
@@ -322,8 +338,8 @@ def _gate_sql(gate: Gate, gaz: places.Gazetteer) -> tuple[str, str, list[Any]]:
|
||||
return (f"({empty} AND NOT {_HAS_AREA})",
|
||||
f"(CASE WHEN {_HAS_AREA} THEN {area} ELSE {match} END)", [*more, *params])
|
||||
if gate.field == "place":
|
||||
match, params = _places_sql(values, gaz, table="r")
|
||||
return "r.place = ''", match, params
|
||||
match, params = _places_sql(values, gaz, table="r", wider=True)
|
||||
return _unplaced(gaz), match, params
|
||||
if gate.field == "title":
|
||||
# Whole words of the folded title: " account executive " inside
|
||||
# " enterprise account executive uk ". A fold drops `%` and `_`, so a
|
||||
@@ -339,19 +355,33 @@ def _gate_sql(gate: Gate, gaz: places.Gazetteer) -> tuple[str, str, list[Any]]:
|
||||
return ("r.languages IN ('', '[]')",
|
||||
f"NOT EXISTS (SELECT 1 FROM json_each(r.languages) "
|
||||
f"WHERE LOWER(value) NOT IN ({marks}))", values)
|
||||
if gate.field == "remote" and "remote" in values:
|
||||
# A role whose text says remote while no field does is possibly
|
||||
# remote (`remote_text`, prep): it meets a gate on remote until
|
||||
# extract has read it, and is not empty for it.
|
||||
return (f"({column} = '' AND r.remote_text = 0)",
|
||||
f"(LOWER({column}) IN ({marks}) OR ({column} = '' AND r.remote_text = 1))", values)
|
||||
return f"{column} = ''", f"LOWER({column}) IN ({marks})", values
|
||||
|
||||
|
||||
def _places_sql(tokens: Sequence[str], gaz: places.Gazetteer, *, table: str = "roles",
|
||||
codes: Sequence[str] = (), scope: Sequence[Gate] = ()) -> tuple[str, list[Any]]:
|
||||
codes: Sequence[str] = (), scope: Sequence[Gate] = (),
|
||||
wider: bool = False) -> tuple[str, list[Any]]:
|
||||
"""Any of the place tokens over `table`, as a join over the gazetteer: a
|
||||
role with a place under one of the places the tokens name (`country:DK`
|
||||
every place in Denmark, `region:skane` every town in Skåne, `place:malmo`
|
||||
Malmö by any spelling), or whose `country` is one of their countries (a
|
||||
role whose board gave a country and no place). `codes` are more values of
|
||||
`country` to accept. A `place:` token the gazetteer does not know is
|
||||
matched on whole words of the role's folded line (`places.match_sql`),
|
||||
so a town nobody has entered yet is still found.
|
||||
`country` to accept. `wider` also takes a role placed only at places
|
||||
that hold one of them (`Gazetteer.overlap`): Øresund, Europe or Anywhere
|
||||
alone meet `country:DK`, as the fetch gate reads them (`_gate_sql`); the
|
||||
feed's Place filter leaves it off. A role that also names a place
|
||||
elsewhere is not taken: `London; Europe` is in London, and `Aarhus;
|
||||
Denmark` does not meet Copenhagen. The `country` column is not widened,
|
||||
as Sweden would then meet Malmö for a role in Stockholm. A `place:`
|
||||
token the gazetteer does not know is matched on whole words of the
|
||||
role's folded line (`places.match_sql`), so a town nobody has entered
|
||||
yet is still found.
|
||||
|
||||
A `remote:` token is a remote role whose area overlaps its places
|
||||
(`Gazetteer.overlap`), `remote:anywhere` one whose area is Anywhere.
|
||||
@@ -359,6 +389,7 @@ def _places_sql(tokens: Sequence[str], gaz: places.Gazetteer, *, table: str = "r
|
||||
`scope`, the viewing agent's `require` gates on country; with none,
|
||||
every remote role."""
|
||||
ids: set[int] = set()
|
||||
around: set[int] = set()
|
||||
areas: set[int] = set()
|
||||
found = {c for c in codes if c}
|
||||
clauses: list[str] = []
|
||||
@@ -381,15 +412,26 @@ def _places_sql(tokens: Sequence[str], gaz: places.Gazetteer, *, table: str = "r
|
||||
continue
|
||||
under = gaz.under(named)
|
||||
ids |= under
|
||||
if wider:
|
||||
around |= gaz.overlap(named)
|
||||
found |= gaz.codes(under)
|
||||
first: list[tuple[str, list[Any]]] = []
|
||||
if ids:
|
||||
clauses.insert(0, f"{table}.uid IN (SELECT uid FROM role_places "
|
||||
"WHERE place_id IN (SELECT value FROM json_each(?)))")
|
||||
params.insert(0, json.dumps(sorted(ids)))
|
||||
first.append((f"{table}.uid IN (SELECT uid FROM role_places "
|
||||
"WHERE place_id IN (SELECT value FROM json_each(?)))",
|
||||
[json.dumps(sorted(ids))]))
|
||||
if found:
|
||||
clauses.insert(1 if ids else 0,
|
||||
f"{table}.country IN (SELECT value FROM json_each(?))")
|
||||
params.insert(1 if ids else 0, json.dumps(sorted(found)))
|
||||
first.append((f"{table}.country IN (SELECT value FROM json_each(?))",
|
||||
[json.dumps(sorted(found))]))
|
||||
if around - ids:
|
||||
first.append((f"({table}.uid IN (SELECT uid FROM role_places "
|
||||
"WHERE place_id IN (SELECT value FROM json_each(?))) "
|
||||
f"AND {table}.uid NOT IN (SELECT uid FROM role_places "
|
||||
"WHERE place_id IS NOT NULL "
|
||||
"AND place_id NOT IN (SELECT value FROM json_each(?))))",
|
||||
[json.dumps(sorted(around - ids)), json.dumps(sorted(around))]))
|
||||
clauses[:0] = [sql for sql, _ in first]
|
||||
params[:0] = [value for _, values in first for value in values]
|
||||
if areas:
|
||||
remote.append(_areas_sql(areas, table))
|
||||
for sql, values in remote:
|
||||
@@ -508,7 +550,8 @@ def rebuild_dupes(conn: sqlite3.Connection) -> DupeCounts:
|
||||
|
||||
A copy the host published for another place (`dupes.Folded.placed`)
|
||||
leaves its places on the kept role as `role_places` rows with the copy's
|
||||
link, see `_sync_copy_places`."""
|
||||
link, see `_sync_copy_places`. Its remote stays its own, which is why a
|
||||
remote listing is never folded into an office one (`dupes.fold`)."""
|
||||
# Migration 18 runs this on a file that has no `teaser` and no
|
||||
# `role_places` yet: there it groups as before and folds no places.
|
||||
current = any(r[1] == "teaser" for r in conn.execute("PRAGMA table_info(roles)"))
|
||||
@@ -516,7 +559,7 @@ def rebuild_dupes(conn: sqlite3.Connection) -> DupeCounts:
|
||||
rows = conn.execute(
|
||||
f"""SELECT uid, company_key, title_key, country, place, {_HOST} AS host,
|
||||
description != '' AS described, posted_at,
|
||||
SUBSTR({text}, 1, {dupes.TEXT_CHARS}), original_url
|
||||
SUBSTR({text}, 1, {dupes.TEXT_CHARS}), original_url, remote = 'remote'
|
||||
FROM roles
|
||||
WHERE closed_at IS NULL AND company_key != ''
|
||||
AND (company_key, title_key) IN (
|
||||
@@ -527,7 +570,7 @@ def rebuild_dupes(conn: sqlite3.Connection) -> DupeCounts:
|
||||
folded = made(
|
||||
dupes.Copy(uid=r[0], company_key=r[1], title_key=r[2], country=r[3], place=r[4],
|
||||
host=r[5], has_description=bool(r[6]), posted_at=r[7] or None,
|
||||
text=r[8] or "", url=r[9] or "")
|
||||
text=r[8] or "", url=r[9] or "", remote=bool(r[10]))
|
||||
for r in rows)
|
||||
new = folded.dup_of
|
||||
old: dict[str, str | None] = dict.fromkeys(new)
|
||||
@@ -819,8 +862,9 @@ class Store:
|
||||
the way a board's own structured field is (docs/decisions.md, "a
|
||||
guess is not a board field"). So once a role has been through
|
||||
`extract` (`extracted_at` is set), this upsert leaves `function`
|
||||
alone for good, whatever the plugin's raw guess now says — see
|
||||
`set_extracted` for the other half of the rule.
|
||||
alone, whatever the plugin's raw guess now says, until a better text
|
||||
puts the role back in extract's queue (below) — see `set_extracted`
|
||||
for the other half of the rule.
|
||||
|
||||
The role's places (`role_places`) are the texts of `raw["places"]`,
|
||||
else `raw["place"]` cut at `;` (`gazetteer.texts`), written unresolved
|
||||
@@ -832,9 +876,13 @@ class Store:
|
||||
under the same keep-filled rule. `teaser` is kept-filled too.
|
||||
|
||||
A role is embedded from its description, else its teaser, else its
|
||||
title (`app.jobs.embed.document`). When a pull brings a better one of
|
||||
these than the role had, its vector is dropped so the next `embed`
|
||||
reads it.
|
||||
title (`app.jobs.embed.document`), and `extract` reads the same. When
|
||||
a pull brings a better one of these than the role had, its vector is
|
||||
dropped so the next `embed` reads it, and `extracted_at` and
|
||||
`extract_tries` are reset so `extract` reads it again (if it is still
|
||||
in a cut). That read overwrites the model's own columns and fills
|
||||
what is still empty (`set_extracted`); until it comes, `function` is
|
||||
the plugin's guess's again, as for any role not yet read.
|
||||
"""
|
||||
if not item.source:
|
||||
raise ValueError(f"item {item.source_id!r} has no source; the runner sets it")
|
||||
@@ -921,14 +969,17 @@ class Store:
|
||||
if before is not None:
|
||||
conn.execute("UPDATE roles SET prep_due = prep_due + 1 WHERE uid = ?",
|
||||
(row["uid"],))
|
||||
# `embed` encodes a role once, from the best text it has. When a
|
||||
# later pull brings a better one, as a host that states the text
|
||||
# on the role's own page does, the vector goes, so the next
|
||||
# `embed` reads it.
|
||||
# `embed` encodes a role once, from the best text it has, and
|
||||
# `extract` reads it once. When a later pull brings a better one,
|
||||
# as fetch text does for a host that states the text on the
|
||||
# role's own page, the vector goes and the role waits for
|
||||
# extract again, so both read the new text.
|
||||
if before is not None and _text_rank(row["description"] or before["description"],
|
||||
row["teaser"] or before["teaser"]) \
|
||||
> _text_rank(before["description"], before["teaser"]):
|
||||
conn.execute("DELETE FROM vectors WHERE uid=?", (row["uid"],))
|
||||
conn.execute("UPDATE roles SET extracted_at = NULL, extract_tries = 0 "
|
||||
"WHERE uid=?", (row["uid"],))
|
||||
return before is None
|
||||
|
||||
def role(self, uid: str, agent_id: int | None = None) -> ScoredRole | None:
|
||||
@@ -1134,7 +1185,7 @@ class Store:
|
||||
place: str | None = None, open_only: bool = True,
|
||||
q: str | None = None, function: str | None = None,
|
||||
remote: str | None = None, days: int | None = None,
|
||||
hide_muted: bool = False, include_read: bool = True,
|
||||
include_read: bool = True,
|
||||
tags: Sequence[str] = (), tag_mode: str = "any",
|
||||
agent: int | None = None, by_score: bool = True, gated: bool = False,
|
||||
fetch_gate: bool = False, judged: str | None = None, pay: str | None = None,
|
||||
@@ -1161,18 +1212,18 @@ class Store:
|
||||
|
||||
`agent` says whose votes, hidden companies and scores are meant. Each
|
||||
row carries that agent's `vote` and `voted_at`; `judged` (a key of
|
||||
`JUDGED`: liked, unjudged or passed) filters on them. `hide_muted`
|
||||
excludes the companies that agent hid, before `limit` is applied, in
|
||||
SQL rather than after the fact: a company that has published a lot,
|
||||
all at once — a fresh board added, a re-pull that finally reached
|
||||
it — can otherwise fill the whole page with roles that then all turn
|
||||
out to be from the one company someone hid, leaving nothing to show
|
||||
for a default query that would otherwise have plenty. With no agent
|
||||
`JUDGED`: liked, unjudged or passed) filters on them. The companies
|
||||
that agent hid are always left out, in every order and count: a
|
||||
hidden company's roles are never shown. That is done before `limit`,
|
||||
in SQL rather than after the fact: a company that has published a
|
||||
lot, all at once — a fresh board added, a re-pull that finally
|
||||
reached it — could otherwise fill the whole page with roles that then
|
||||
all turn out to be from the one company someone hid. With no agent
|
||||
every vote is None, nothing is hidden and `judged` reads as such.
|
||||
|
||||
`include_read` set False excludes a role `set_read` or `set_vote` has
|
||||
stamped, the same way and for the same reason as `hide_muted`:
|
||||
before `limit`, not after. `tags` keeps only roles carrying at least
|
||||
stamped, the same way and for the same reason as the hidden
|
||||
companies: before `limit`, not after. `tags` keeps only roles carrying at least
|
||||
one of them (`tag_mode="any"`, the default) or every one
|
||||
(`tag_mode="all"`), also before `limit`.
|
||||
|
||||
@@ -1201,7 +1252,7 @@ class Store:
|
||||
the whole page rather than one per role."""
|
||||
sql, params = self._roles_from(
|
||||
company=company, place=place, open_only=open_only, q=q, function=function,
|
||||
remote=remote, days=days, hide_muted=hide_muted, include_read=include_read,
|
||||
remote=remote, days=days, include_read=include_read,
|
||||
tags=tags, tag_mode=tag_mode, agent=agent, by_score=by_score, gated=gated,
|
||||
fetch_gate=fetch_gate,
|
||||
judged=judged, pay=pay, company_keys=company_keys, places=places,
|
||||
@@ -1253,7 +1304,7 @@ class Store:
|
||||
place: str | None = None, open_only: bool = True,
|
||||
q: str | None = None, function: str | None = None,
|
||||
remote: str | None = None, days: int | None = None,
|
||||
hide_muted: bool = False, include_read: bool = True,
|
||||
include_read: bool = True,
|
||||
tags: Sequence[str] = (), tag_mode: str = "any",
|
||||
agent: int | None = None, by_score: bool = True, gated: bool = False,
|
||||
fetch_gate: bool = False, judged: str | None = None, pay: str | None = None,
|
||||
@@ -1277,7 +1328,7 @@ class Store:
|
||||
column itself never crosses out of SQLite for a list."""
|
||||
sql, params = self._roles_from(
|
||||
company=company, place=place, open_only=open_only, q=q, function=function,
|
||||
remote=remote, days=days, hide_muted=hide_muted, include_read=include_read,
|
||||
remote=remote, days=days, include_read=include_read,
|
||||
tags=tags, tag_mode=tag_mode, agent=agent, by_score=by_score, gated=gated,
|
||||
fetch_gate=fetch_gate,
|
||||
judged=judged, pay=pay, company_keys=company_keys, places=places,
|
||||
@@ -1333,8 +1384,7 @@ class Store:
|
||||
def count_roles(self, company: str | None = None, place: str | None = None,
|
||||
open_only: bool = True, q: str | None = None,
|
||||
function: str | None = None, remote: str | None = None,
|
||||
days: int | None = None, hide_muted: bool = False,
|
||||
include_read: bool = True, tags: Sequence[str] = (),
|
||||
days: int | None = None, include_read: bool = True, tags: Sequence[str] = (),
|
||||
tag_mode: str = "any", agent: int | None = None, by_score: bool = True,
|
||||
gated: bool = False, fetch_gate: bool = False,
|
||||
judged: str | None = None,
|
||||
@@ -1345,7 +1395,7 @@ class Store:
|
||||
"""How many roles `roles` would list with the same filters and no limit."""
|
||||
sql, params = self._roles_from(
|
||||
company=company, place=place, open_only=open_only, q=q, function=function,
|
||||
remote=remote, days=days, hide_muted=hide_muted, include_read=include_read,
|
||||
remote=remote, days=days, include_read=include_read,
|
||||
tags=tags, tag_mode=tag_mode, agent=agent, by_score=by_score, gated=gated,
|
||||
fetch_gate=fetch_gate,
|
||||
judged=judged, pay=pay, company_keys=company_keys, places=places,
|
||||
@@ -1357,8 +1407,7 @@ class Store:
|
||||
def role_facets(self, company: str | None = None, place: str | None = None,
|
||||
open_only: bool = True, q: str | None = None,
|
||||
function: str | None = None, remote: str | None = None,
|
||||
days: int | None = None, hide_muted: bool = False,
|
||||
include_read: bool = True, tags: Sequence[str] = (),
|
||||
days: int | None = None, include_read: bool = True, tags: Sequence[str] = (),
|
||||
tag_mode: str = "any", agent: int | None = None, by_score: bool = True,
|
||||
gated: bool = False, fetch_gate: bool = False,
|
||||
judged: str | None = None,
|
||||
@@ -1375,7 +1424,7 @@ class Store:
|
||||
and then `""`, no pay stated. A value no role has counts 0."""
|
||||
shared: dict[str, Any] = {
|
||||
"company": company, "place": place, "open_only": open_only, "q": q,
|
||||
"function": function, "remote": remote, "days": days, "hide_muted": hide_muted,
|
||||
"function": function, "remote": remote, "days": days,
|
||||
"include_read": include_read, "tags": tags, "tag_mode": tag_mode, "agent": agent,
|
||||
"by_score": by_score, "gated": gated, "fetch_gate": fetch_gate, "judged": judged,
|
||||
"company_keys": company_keys, "places": places, "closing": closing}
|
||||
@@ -1394,7 +1443,7 @@ class Store:
|
||||
|
||||
def _roles_from(self, *, company: str | None, place: str | None, open_only: bool,
|
||||
q: str | None, function: str | None, remote: str | None,
|
||||
days: int | None, hide_muted: bool, include_read: bool,
|
||||
days: int | None, include_read: bool,
|
||||
tags: Sequence[str], tag_mode: str, agent: int | None,
|
||||
by_score: bool, gated: bool, judged: str | None,
|
||||
fetch_gate: bool = False,
|
||||
@@ -1408,7 +1457,7 @@ class Store:
|
||||
(see `_WITH_VOTES`) and its judgements as `j` (`_WITH_JUDGEMENTS`);
|
||||
its scores as `s` when ordering by them or with `fetch_gate`, and
|
||||
then a role the agent's fetch gate rejects is left out unless
|
||||
`gated`. A copy of another role
|
||||
`gated`. A company the agent hid is always left out. A copy of another role
|
||||
(`rebuild_dupes`) is never listed: its kept role stands for it and
|
||||
counts its copies. `closing` keeps the roles whose deadline is
|
||||
today or within `CLOSING_DAYS` after it."""
|
||||
@@ -1451,7 +1500,7 @@ class Store:
|
||||
params += list(seniority)
|
||||
if open_only:
|
||||
where.append("closed_at IS NULL")
|
||||
if hide_muted and agent is not None:
|
||||
if agent is not None:
|
||||
where.append("company_key NOT IN (SELECT company_key FROM agent_companies "
|
||||
"WHERE agent_id = ? AND affinity = -1)")
|
||||
params.append(agent)
|
||||
@@ -1499,14 +1548,18 @@ class Store:
|
||||
if found is not None else []
|
||||
|
||||
def tag_counts(self, agent_id: int | None = None,
|
||||
show_muted: bool = False) -> list[tuple[str, int, int]]:
|
||||
fetch_gate: bool = False) -> list[tuple[str, int, int]]:
|
||||
"""`(tag, open_count, unread_count)` over open roles, most-used tag
|
||||
first and then alphabetically. What `/api/filters` serves; the
|
||||
companies `agent_id` hid are left out unless `show_muted`, the same
|
||||
default as `roles`, and with no agent nothing is hidden."""
|
||||
companies `agent_id` hid are left out, as in `roles`, and with no
|
||||
agent nothing is hidden. With `fetch_gate`, only the roles the
|
||||
agent's feed shows count (`_SHOWN`)."""
|
||||
where = "r.closed_at IS NULL AND r.dup_of IS NULL"
|
||||
params: list[Any] = []
|
||||
if not show_muted and agent_id is not None:
|
||||
if agent_id is not None and fetch_gate:
|
||||
where += f" AND {_SHOWN}"
|
||||
params += [agent_id, agent_id]
|
||||
elif agent_id is not None:
|
||||
where += (" AND r.company_key NOT IN (SELECT company_key FROM agent_companies "
|
||||
"WHERE agent_id = ? AND affinity = -1)")
|
||||
params.append(agent_id)
|
||||
@@ -1578,8 +1631,9 @@ class Store:
|
||||
f"""SELECT uid, prep_due, source, {_HOST} AS host, title, company, company_key, place,
|
||||
seniority_host, employment_host, remote_host, country_host,
|
||||
seniority_model, seniority_stated, seniority_title, seniority,
|
||||
employment, remote, country, remote_model, remote_where_model,
|
||||
deadline_host, deadline_model, deadline,
|
||||
employment, remote, remote_text, country, remote_model,
|
||||
remote_where_model, deadline_host, deadline_model, deadline,
|
||||
SUBSTR(COALESCE(NULLIF(description, ''), teaser), 1, 6000) AS text,
|
||||
json_extract(raw, '$.places') AS raw_places,
|
||||
(SELECT json_group_array(place_id) FROM (
|
||||
SELECT place_id FROM role_remote rr WHERE rr.uid = roles.uid
|
||||
@@ -1605,8 +1659,8 @@ class Store:
|
||||
def write_prep(self, mapped: Sequence[dict[str, Any]]) -> int:
|
||||
"""Write what prep mapped, one dict per role with `uid`, `due` (the
|
||||
`prep_due` it read), `company_key`, `seniority_stated`,
|
||||
`seniority_title`, `employment`, `remote`, `remote_area`, `country`
|
||||
and `deadline` (`YYYY-MM-DD`, None for none, written as it is). `seniority` is the model's level, else the stated, else
|
||||
`seniority_title`, `employment`, `remote`, `remote_text`, `remote_area`,
|
||||
`country` and `deadline` (`YYYY-MM-DD`, None for none, written as it is). `seniority` is the model's level, else the stated, else
|
||||
the title's. `remote` is written as prep composed it (the model's,
|
||||
else the host's word, else the place line) and `remote_area`, place
|
||||
ids, replaces the role's `role_remote`. An empty `country` leaves the
|
||||
@@ -1622,6 +1676,7 @@ class Store:
|
||||
NULLIF(:seniority_stated, ''), :seniority_title),
|
||||
employment = :employment,
|
||||
remote = :remote,
|
||||
remote_text = :remote_text,
|
||||
country = COALESCE(NULLIF(:country, ''), country),
|
||||
deadline = :deadline,
|
||||
prep_due = CASE WHEN prep_due = :due THEN 0 ELSE prep_due END
|
||||
@@ -1664,29 +1719,43 @@ class Store:
|
||||
# -------------------------------------------------------------- extract
|
||||
|
||||
def _summarise_top(self) -> tuple[str, list[Any]]:
|
||||
"""The `WHERE` fragment for a role among the best `summarise_top`
|
||||
final scores of an enabled agent with a prompt, open, no copy and
|
||||
passing that agent's fetch gate: the top of its feed. The rank is
|
||||
numbered per agent and the agent's limits ride out of the window
|
||||
subquery beside it, so each row is held to the limit of the agent
|
||||
that ranked it; a key the JSON lacks is the `Limits` default. The
|
||||
role is `r`. An agent without a prompt with text counts for nothing
|
||||
here: its scores have no fit, so they say how fresh a role is, not
|
||||
whether it is wanted."""
|
||||
return ("""r.uid IN (SELECT uid FROM (
|
||||
SELECT s.uid, a.limits, ROW_NUMBER() OVER (
|
||||
PARTITION BY s.agent_id
|
||||
ORDER BY s.score DESC, top.posted_at DESC, s.uid) AS n
|
||||
FROM scores s
|
||||
JOIN agents a ON a.id = s.agent_id AND a.enabled = 1
|
||||
AND EXISTS (SELECT 1 FROM agent_prompts p
|
||||
WHERE p.agent_id = a.id
|
||||
AND TRIM(p.text, ' ' || char(9, 10, 13)) <> '')
|
||||
JOIN roles top ON top.uid = s.uid AND top.closed_at IS NULL
|
||||
AND top.dup_of IS NULL
|
||||
WHERE s.gated_reason = '')
|
||||
WHERE n <= COALESCE(json_extract(limits, '$.summarise_top'), ?))""",
|
||||
[Limits().summarise_top])
|
||||
"""The `WHERE` fragment for a role in the top of an enabled agent's
|
||||
feed in either order, open, no copy, passing that agent's fetch gate
|
||||
and not of a company it hid: the best `summarise_top` (its `limits`)
|
||||
by the want feed score, the score order of `roles` with the vote
|
||||
applied (`_VOTED_SCORE`), and the best `summarise_top` by the match
|
||||
order, `cv_score` under the agent's weights, then `cv_fit`, then
|
||||
score (`_by_cv`). So at most twice `summarise_top` per agent. An
|
||||
agent without a prompt with text gives only its match top (its
|
||||
scores have no fit, so they say how fresh a role is, not whether it
|
||||
is wanted), one without a CV only its want top. The role is `r`; no
|
||||
agent that counts is a fragment that holds for no role."""
|
||||
arms: list[str] = []
|
||||
params: list[Any] = []
|
||||
for agent in self.agents(enabled_only=True):
|
||||
top = agent.limits.summarise_top
|
||||
orders = []
|
||||
if any(p.text.strip() for p in agent.prompts):
|
||||
orders.append(f"{_VOTED_SCORE} DESC")
|
||||
if agent.cv_text.strip():
|
||||
orders.append(_by_cv(_cv_sql(agent.weights)) + f"{_VOTED_SCORE} DESC")
|
||||
for order in orders:
|
||||
arms.append(f"""SELECT uid FROM (
|
||||
SELECT s.uid FROM scores s
|
||||
JOIN roles top ON top.uid = s.uid AND top.closed_at IS NULL
|
||||
AND top.dup_of IS NULL
|
||||
LEFT JOIN role_votes v ON v.uid = s.uid AND v.agent_id = s.agent_id
|
||||
LEFT JOIN role_judgements j ON j.uid = s.uid AND j.agent_id = s.agent_id
|
||||
AND j.judged_at IS NOT NULL
|
||||
WHERE s.agent_id = ? AND s.gated_reason = ''
|
||||
AND top.company_key NOT IN (
|
||||
SELECT company_key FROM agent_companies
|
||||
WHERE agent_id = s.agent_id AND affinity = -1)
|
||||
ORDER BY {order}, top.posted_at DESC, s.uid LIMIT ?)""")
|
||||
params += [agent.id, top]
|
||||
if not arms:
|
||||
return "0", []
|
||||
return f"r.uid IN ({' UNION '.join(arms)})", params
|
||||
|
||||
def _extract_scope(self, uids: Sequence[str] | None) -> tuple[str, list[Any]]:
|
||||
"""The `WHERE` for a role `extract` may read: open, never answered,
|
||||
@@ -1845,9 +1914,9 @@ class Store:
|
||||
some text to summarise. Given `uids`, those of them and nothing else,
|
||||
open or not and given up or not, because somebody asked for them.
|
||||
Otherwise open, not given up, and in the top of an enabled agent's
|
||||
feed (`_summarise_top`) or one the owner cares about whatever its
|
||||
rank: saved (it has an application) or voted up by an enabled
|
||||
agent."""
|
||||
feed in either order (`_summarise_top`) or one the owner cares about
|
||||
whatever its rank: saved (it has an application) or voted up by an
|
||||
enabled agent."""
|
||||
where = ["r.summary = ''",
|
||||
"(TRIM(r.article_text) <> '' OR TRIM(r.description) <> '')"]
|
||||
if uids is not None:
|
||||
@@ -1875,9 +1944,10 @@ class Store:
|
||||
return int(row["n"]) if row else 0
|
||||
|
||||
def summarise_waiting(self, limit: int, uids: Sequence[str] | None = None) -> list[Role]:
|
||||
"""Roles `summarise` has yet to write, newest first: the best
|
||||
`summarise_top` final scores of every enabled agent, so the backlog
|
||||
is bounded by the agents rather than by the number of open roles,
|
||||
"""Roles `summarise` has yet to write, newest first: the top
|
||||
`summarise_top` of every enabled agent's want order and of its match
|
||||
order (`_summarise_top`), so the backlog is bounded by the agents
|
||||
rather than by the number of open roles,
|
||||
and every open role that is saved or voted up by an enabled agent,
|
||||
whatever its rank; given `uids`, those still without a summary,
|
||||
whatever their rank. A batch asks again when it runs."""
|
||||
@@ -1955,33 +2025,28 @@ class Store:
|
||||
|
||||
def set_judgement(self, agent_id: int, uid: str, input_key: str, skills: int,
|
||||
experience: int, career: int, strengths: Sequence[str],
|
||||
gaps: Sequence[str], musts: Sequence[Mapping[str, str]] = ()) -> bool:
|
||||
gaps: Sequence[str]) -> bool:
|
||||
"""Store `judge`'s reading of one role for one agent, replacing the
|
||||
one before: the three numbers as the model gave them, the verdict on
|
||||
each of the agent's must-haves (`musts`, `{id, text, verdict}`) and how
|
||||
many are broken, `cv_score` (`app.models.cv_score`, at the default
|
||||
weights and with the cap: the table's column is not read, see
|
||||
`judgement`) and its tries back at 0. False for an unknown role or
|
||||
agent."""
|
||||
broken = sum(1 for m in musts if m.get("verdict") == "broken")
|
||||
one before: the three numbers as the model gave them, `cv_score`
|
||||
(`app.models.cv_score`, at the default weights: the table's column is
|
||||
not read, see `judgement`) and its tries back at 0. False for an
|
||||
unknown role or agent."""
|
||||
try:
|
||||
with self.transaction() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO role_judgements (uid, agent_id, skills, experience, career,
|
||||
cv_score, strengths, gaps, musts, broken,
|
||||
cv_score, strengths, gaps,
|
||||
input_key, tries, judged_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,0,?)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,0,?)
|
||||
ON CONFLICT(uid, agent_id) DO UPDATE SET
|
||||
skills=excluded.skills, experience=excluded.experience,
|
||||
career=excluded.career, cv_score=excluded.cv_score,
|
||||
strengths=excluded.strengths, gaps=excluded.gaps,
|
||||
musts=excluded.musts, broken=excluded.broken,
|
||||
input_key=excluded.input_key, tries=0, judged_at=excluded.judged_at""",
|
||||
(uid, agent_id, skills, experience, career,
|
||||
cv_score(skills, experience, career, broken=broken),
|
||||
cv_score(skills, experience, career),
|
||||
json.dumps(list(strengths), ensure_ascii=False),
|
||||
json.dumps(list(gaps), ensure_ascii=False),
|
||||
json.dumps([dict(m) for m in musts], ensure_ascii=False), broken,
|
||||
input_key, self.now()))
|
||||
except sqlite3.IntegrityError:
|
||||
return False
|
||||
@@ -2009,8 +2074,7 @@ class Store:
|
||||
def judgement(self, agent_id: int | None, uid: str) -> Judgement | None:
|
||||
"""The agent's judgement of one role, None when it has none, its
|
||||
`cv_score` the three numbers under the agent's weights as they are
|
||||
now, not as stored, and `career_capped` the model's career as the
|
||||
broken must-haves leave it (`BROKEN_CAP`)."""
|
||||
now, not as stored."""
|
||||
if agent_id is None:
|
||||
return None
|
||||
row = self._row("SELECT * FROM role_judgements WHERE agent_id = ? AND uid = ? "
|
||||
@@ -2019,9 +2083,8 @@ class Store:
|
||||
return None
|
||||
found = Judgement.from_row(row)
|
||||
agent = self.agent(agent_id)
|
||||
found.career_capped = capped_career(found.career, found.broken)
|
||||
found.cv_score = cv_score(found.skills, found.experience, found.career,
|
||||
agent.weights if agent else None, found.broken)
|
||||
agent.weights if agent else None)
|
||||
return found
|
||||
|
||||
def set_article_text(self, uid: str, text: str) -> Role | None:
|
||||
@@ -2143,14 +2206,13 @@ class Store:
|
||||
cv_text: str | None = None,
|
||||
cv_languages: Sequence[CvLanguage] | None = None,
|
||||
avoid_text: str | None = None,
|
||||
must_text: str | None = None,
|
||||
like_text: str | None = None) -> Agent:
|
||||
"""Create an agent, or with `agent_id` replace that one's fields and
|
||||
prompts. Without `prompts`, a non-empty `intent` is its one prompt,
|
||||
named after the agent. Without `fetch_terms` an update keeps the
|
||||
stored ones and a new agent has none, so a caller that does not know
|
||||
them (the CLI's prompt edit) loses none; `cv_text`, `cv_languages`,
|
||||
`like_text`, `avoid_text` and `must_text` are kept the same way. A prompt whose text is
|
||||
`like_text` and `avoid_text` are kept the same way. A prompt whose text is
|
||||
unchanged keeps its cached vector, whatever its name or place, so an
|
||||
edit re-embeds only the prompts it changed. A save gives every role
|
||||
`judge` gave up on for the agent its tries back, since what it is
|
||||
@@ -2171,7 +2233,6 @@ class Store:
|
||||
(limits or Limits()).model_dump_json(), int(enabled), terms,
|
||||
None if cv_text is None else cv_text.strip(), spoken,
|
||||
None if avoid_text is None else avoid_text.strip(),
|
||||
None if must_text is None else must_text.strip(),
|
||||
None if like_text is None else like_text.strip())
|
||||
try:
|
||||
with self.transaction() as conn:
|
||||
@@ -2179,10 +2240,9 @@ class Store:
|
||||
agent_id = conn.execute(
|
||||
"""INSERT INTO agents (name, gates, weights, limits, enabled,
|
||||
fetch_terms, cv_text, cv_languages, avoid_text,
|
||||
must_text, like_text, created_at)
|
||||
like_text, created_at)
|
||||
VALUES (?,?,?,?,?,COALESCE(?, '[]'),COALESCE(?, ''),
|
||||
COALESCE(?, '[]'),COALESCE(?, ''),COALESCE(?, ''),
|
||||
COALESCE(?, ''),?)""",
|
||||
COALESCE(?, '[]'),COALESCE(?, ''),COALESCE(?, ''),?)""",
|
||||
(*fields, self.now())).lastrowid
|
||||
elif not conn.execute(
|
||||
"""UPDATE agents SET name=?, gates=?, weights=?, limits=?, enabled=?,
|
||||
@@ -2190,7 +2250,6 @@ class Store:
|
||||
cv_text=COALESCE(?, cv_text),
|
||||
cv_languages=COALESCE(?, cv_languages),
|
||||
avoid_text=COALESCE(?, avoid_text),
|
||||
must_text=COALESCE(?, must_text),
|
||||
like_text=COALESCE(?, like_text)
|
||||
WHERE id=?""",
|
||||
(*fields, agent_id)).rowcount:
|
||||
@@ -2288,7 +2347,7 @@ class Store:
|
||||
def score_candidates(self, agent_id: int, gates: Sequence[Gate]) -> list[Candidate]:
|
||||
"""Every open role of a company this agent has not hidden, copies
|
||||
(`rebuild_dupes`) aside, with the agent's affinity for the company,
|
||||
the company's All countries and All roles, and how the role meets
|
||||
the company's All locations and All roles, and how the role meets
|
||||
each gate: 1 match, 0 no match, -1 the column is empty. The gates are
|
||||
evaluated here, in SQL, so a `require` gate is the `WHERE` it reads
|
||||
as, written as a column so a gated role keeps its row and reason."""
|
||||
@@ -2304,7 +2363,11 @@ class Store:
|
||||
empty, match, values = _gate_sql(part, gaz)
|
||||
whens.append(f"WHEN {match} THEN {k}")
|
||||
params += values
|
||||
columns.append(f"CASE WHEN {empty} THEN -1 {' '.join(whens)} ELSE 0 END AS g{n}")
|
||||
# A place gate's match goes first: an unresolved place can still
|
||||
# meet one of its values (`_gate_sql`).
|
||||
case = (f"{' '.join(whens)} WHEN {empty} THEN -1" if gate.field == "place"
|
||||
else f"WHEN {empty} THEN -1 {' '.join(whens)}")
|
||||
columns.append(f"CASE {case} ELSE 0 END AS g{n}")
|
||||
select = ", ".join(["r.uid", "r.posted_at",
|
||||
"COALESCE(ac.affinity, 0) AS affinity",
|
||||
"COALESCE(c.all_countries, 0) AS all_countries",
|
||||
@@ -2467,7 +2530,7 @@ class Store:
|
||||
return self._company(key)
|
||||
|
||||
def set_all_countries(self, key: str, on: bool) -> Company:
|
||||
"""Switch a company's All countries override: its big boards are read
|
||||
"""Switch a company's All locations override: its big boards are read
|
||||
with the agents' wide reach, Europe today, rather than their places,
|
||||
from the next pull (`app/jobs/reach.py`). Global, not per agent.
|
||||
Raises KeyError for an unknown company."""
|
||||
@@ -2536,7 +2599,7 @@ class Store:
|
||||
def merge_companies(self, keep: str, other: str) -> Company:
|
||||
"""Fold company `other` into `keep`, in one transaction: its roles and
|
||||
subscriptions move to `keep`, each agent's stance moves unless the
|
||||
agent has one on `keep` already, the All countries and All roles
|
||||
agent has one on `keep` already, the All locations and All roles
|
||||
overrides carry over, its aliases and its name become aliases of
|
||||
`keep`, and its row goes. Copies are not recomputed (`rebuild_dupes`).
|
||||
Raises KeyError for an unknown company and ValueError for merging one
|
||||
@@ -2583,25 +2646,30 @@ class Store:
|
||||
"SELECT company_key, COUNT(*) AS n FROM roles "
|
||||
"WHERE closed_at IS NULL AND dup_of IS NULL GROUP BY company_key")}
|
||||
|
||||
def open_companies(self, agent_id: int | None = None
|
||||
def open_companies(self, agent_id: int | None = None, fetch_gate: bool = False
|
||||
) -> list[tuple[str, list[str], int, int]]:
|
||||
"""`(company_key, names, open roles, affinity)` for every company with
|
||||
an open role, what the Company field suggests from. `names` are the
|
||||
company row's name and then every spelling its roles carry, commonest
|
||||
first, so a search finds a company by any of them. A company
|
||||
`agent_id` hid is left out; `affinity` is that agent's, 0 without
|
||||
one. Copies are not counted, as in the feed."""
|
||||
one. Copies are not counted, as in the feed. With `fetch_gate`, only
|
||||
the roles the agent's feed shows count (`_SHOWN`), and a company with
|
||||
none of them is left out."""
|
||||
gate, params = "", [agent_id]
|
||||
if agent_id is not None and fetch_gate:
|
||||
gate, params = f"AND {_SHOWN}", [agent_id, agent_id, agent_id]
|
||||
rows = self._rows(
|
||||
"""SELECT r.company_key AS key, r.company AS name, COUNT(*) AS n,
|
||||
c.name AS own, COALESCE(ac.affinity, 0) AS affinity
|
||||
FROM roles r
|
||||
LEFT JOIN companies c ON c.key = r.company_key
|
||||
LEFT JOIN agent_companies ac ON ac.company_key = r.company_key
|
||||
AND ac.agent_id = ?
|
||||
WHERE r.closed_at IS NULL AND r.dup_of IS NULL AND r.company_key != ''
|
||||
AND COALESCE(ac.affinity, 0) != -1
|
||||
GROUP BY r.company_key, r.company
|
||||
ORDER BY r.company_key, n DESC, r.company""", (agent_id,))
|
||||
f"""SELECT r.company_key AS key, r.company AS name, COUNT(*) AS n,
|
||||
c.name AS own, COALESCE(ac.affinity, 0) AS affinity
|
||||
FROM roles r
|
||||
LEFT JOIN companies c ON c.key = r.company_key
|
||||
LEFT JOIN agent_companies ac ON ac.company_key = r.company_key
|
||||
AND ac.agent_id = ?
|
||||
WHERE r.closed_at IS NULL AND r.dup_of IS NULL AND r.company_key != ''
|
||||
AND COALESCE(ac.affinity, 0) != -1 {gate}
|
||||
GROUP BY r.company_key, r.company
|
||||
ORDER BY r.company_key, n DESC, r.company""", params)
|
||||
found: dict[str, tuple[list[str], list[int], int]] = {}
|
||||
for r in rows:
|
||||
names, counts, _ = found.setdefault(
|
||||
@@ -2612,24 +2680,31 @@ class Store:
|
||||
return [(key, names, sum(counts), affinity)
|
||||
for key, (names, counts, affinity) in found.items()]
|
||||
|
||||
def open_places(self) -> list[tuple[str, str, int]]:
|
||||
def open_places(self, agent_id: int | None = None) -> list[tuple[str, str, int]]:
|
||||
"""`(place, country, open roles)` for every place line an open role
|
||||
carries, what `places.suggest` reads. Copies are not counted, as in
|
||||
the feed."""
|
||||
the feed. Given `agent_id`, only the roles that agent's feed shows
|
||||
(`_SHOWN`)."""
|
||||
gate, params = (f"AND {_SHOWN}", [agent_id, agent_id]) if agent_id is not None \
|
||||
else ("", [])
|
||||
return [(str(r["place"]), str(r["country"]), int(r["n"])) for r in self._rows(
|
||||
"SELECT place, country, COUNT(*) AS n FROM roles "
|
||||
"WHERE closed_at IS NULL AND dup_of IS NULL AND place != '' "
|
||||
"GROUP BY place, country")]
|
||||
"SELECT r.place, r.country, COUNT(*) AS n FROM roles r "
|
||||
f"WHERE r.closed_at IS NULL AND r.dup_of IS NULL AND r.place != '' {gate} "
|
||||
"GROUP BY r.place, r.country", params)]
|
||||
|
||||
def open_remote_areas(self) -> list[tuple[tuple[int, ...], int]]:
|
||||
def open_remote_areas(self, agent_id: int | None = None
|
||||
) -> list[tuple[tuple[int, ...], int]]:
|
||||
"""`(area, open roles)` for every remote area an open remote role
|
||||
holds, the area as place ids in order: what `places.suggest` counts
|
||||
the remote tokens from. Copies are not counted, as in the feed."""
|
||||
the remote tokens from. Copies are not counted, as in the feed.
|
||||
Given `agent_id`, only the roles that agent's feed shows (`_SHOWN`)."""
|
||||
gate, params = (f"AND {_SHOWN}", [agent_id, agent_id]) if agent_id is not None \
|
||||
else ("", [])
|
||||
found: dict[str, list[int]] = {}
|
||||
for r in self._rows(
|
||||
"SELECT rr.uid, rr.place_id FROM role_remote rr JOIN roles r ON r.uid = rr.uid "
|
||||
"WHERE r.closed_at IS NULL AND r.dup_of IS NULL AND r.remote = 'remote' "
|
||||
"ORDER BY rr.uid, rr.rowid"):
|
||||
f"WHERE r.closed_at IS NULL AND r.dup_of IS NULL AND r.remote = 'remote' {gate} "
|
||||
"ORDER BY rr.uid, rr.rowid", params):
|
||||
found.setdefault(str(r[0]), []).append(int(r[1]))
|
||||
return list(Counter(tuple(ids) for ids in found.values()).items())
|
||||
|
||||
|
||||
+12
-11
@@ -46,8 +46,9 @@
|
||||
<h3 id="viewing-as">Viewing as <a class="linky" href="/settings/agents">agents</a></h3>
|
||||
<select id="agent" class="rail-select agent-pick" aria-labelledby="viewing-as" hidden></select>
|
||||
<p id="agent-none" class="rail-note" hidden></p>
|
||||
{# Every order keeps to the agent's fetch gate (docs/architecture.md,
|
||||
"5 Feed"), so the filters below only narrow what it passed. #}
|
||||
{# Every order keeps to the agent's fetch gate (documentation/ADR Fetch
|
||||
prep and score.md, "5 Feed"), so the filters below only narrow what
|
||||
it passed. #}
|
||||
<p id="agent-gate" class="rail-note" hidden></p>
|
||||
</section>
|
||||
{# The agent's saved filters (`/api/agents/{id}/filters`), newsdesk's
|
||||
@@ -74,11 +75,13 @@
|
||||
</section>
|
||||
<section>
|
||||
<h3>Order</h3>
|
||||
{# Agent match is the viewing agent's final score, Newest is first
|
||||
seen, CV match the agent's `judge` total against its CV; all carry
|
||||
that agent's votes and keep to its fetch gate. #}
|
||||
{# What I'm looking for is the viewing agent's want feed score, Newest
|
||||
the posting date, What fits me the agent's `judge` total against its
|
||||
CV; all carry that agent's votes and keep to its fetch gate. The
|
||||
period counts back over the posting date too, the date a row shows
|
||||
as its age. #}
|
||||
<div class="seg" id="sort" role="group" aria-label="Order"></div>
|
||||
<label class="rail-field" for="period">First seen</label>
|
||||
<label class="rail-field" for="period">Posted</label>
|
||||
<select id="period" class="rail-select"></select>
|
||||
</section>
|
||||
<section>
|
||||
@@ -92,8 +95,9 @@
|
||||
<div class="rail-fields">
|
||||
{# Company and Place are chip inputs (static/chips.js), each pick a
|
||||
company key or a place token from the suggest routes. The chips
|
||||
of `?company=` and `?place=`, the substrings other pages link
|
||||
with, sit under them to be taken off; nothing types a new one. #}
|
||||
of `?company=` and `?place=`, the substrings of older links and
|
||||
saved filters, sit under them to be taken off; nothing types a
|
||||
new one. #}
|
||||
<div id="f-company"></div>
|
||||
<div id="f-company-text" class="rail-legacy" hidden></div>
|
||||
<div id="f-place"></div>
|
||||
@@ -149,9 +153,6 @@
|
||||
<label class="check">
|
||||
<input type="checkbox" id="f-closing"> Closing soon
|
||||
</label>
|
||||
<label class="check">
|
||||
<input type="checkbox" id="f-show-muted"> Show hidden companies
|
||||
</label>
|
||||
{# The roles the agent's fetch gate rejects, each with the reason,
|
||||
in any order; offered only with an agent. #}
|
||||
<label class="check" id="show-gated" hidden
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||
|
||||
{# A card per agent: `static/agents.js` fills the list from `/api/agents`,
|
||||
newest first. A card shows the agent's prompts, its fetch gate in words, its
|
||||
switch, its weights, its cut and its summarise size; Edit turns that card into the form below, in place, and New puts
|
||||
a blank one on top. The form is a `<template>` because only one card edits
|
||||
switch, its weights, its analytics filter and its summarise size; Edit turns
|
||||
that card into the form below, in place, and New puts a blank one on top. The form is a `<template>` because only one card edits
|
||||
at a time, so its ids stay unique. #}
|
||||
{% block content %}
|
||||
<h1>Search agents</h1>
|
||||
<p class="lede">A search agent is what the Roles tab ranks by: describe each kind of role you want as a prompt, set the fetch gate (where, what role), and it measures every open role against the prompt each fits best. Several agents rank the same roles differently, each with its own votes and liked companies; Viewing as on Roles picks one, and its votes are that agent's.</p>
|
||||
<p class="lede">A search agent is what the Roles tab ranks by: describe each kind of role you want as a prompt, set the fetch gate (world, location, role exclusion, role fetch), and it measures every open role's closeness to the prompt it is closest to. Several agents rank the same roles differently, each with its own votes and liked companies; Viewing as on Roles picks one, and its votes are that agent's.</p>
|
||||
|
||||
<div class="agents-head">
|
||||
<p id="agents-count" class="count" role="status"></p>
|
||||
@@ -24,24 +24,24 @@
|
||||
<legend>What you are looking for</legend>
|
||||
<div id="a-prompts" class="prompt-rows"></div>
|
||||
<button type="button" id="a-add-prompt" class="secondary">Add prompt</button>
|
||||
<span class="hint">A prompt is one kind of role in your own words: the work, the level, the setting. A role is ranked by the prompt it fits best, so three kinds of role are three prompts, not one blended paragraph. Votes and liked companies count for the whole agent.</span>
|
||||
<span class="hint">A prompt is one kind of role in your own words: the work, the level, the setting. A role is ranked by the prompt it is closest to, so three kinds of role are three prompts, not one blended paragraph. Votes and liked companies count for the whole agent.</span>
|
||||
</fieldset>
|
||||
<fieldset class="where">
|
||||
<legend>The fetch gate</legend>
|
||||
<span class="hint">Hard rules, a yes or no per role, applied at every stage: at the host where it can filter, before anything is scored, again after extract has read the posting, and in the feed. A role that fails is never read by a model and never shown. A rule rejects on evidence only: a role whose location is not known yet passes until extract reads it. A company set to All countries skips the place part and one set to All roles skips the role part, for every agent; both are switched under Sources.</span>
|
||||
<h3>Region</h3>
|
||||
<span class="hint">Hard rules, a yes or no per role, in four parts. A role that fails is never read by a model and never shown. A rule rejects on evidence only: a role whose location is not known yet passes until extract reads it. Two company switches under Sources override parts for every agent: All locations bypasses Location, All roles bypasses Role exclusion and Role fetch.</span>
|
||||
<h3>World</h3>
|
||||
<div class="where-line">
|
||||
<span>Only roles in:</span>
|
||||
<select id="a-scope" aria-label="Only roles in">
|
||||
<select id="a-scope" aria-label="World: only roles in">
|
||||
<option value="europe">Europe</option>
|
||||
<option value="countries">Chosen countries</option>
|
||||
<option value="everywhere">Everywhere</option>
|
||||
</select>
|
||||
<div id="a-scope-countries"></div>
|
||||
</div>
|
||||
<span class="hint">Every role must be in this, and no company switch lifts it. A company set to All countries is fetched and kept from the whole region.</span>
|
||||
<h3>Place</h3>
|
||||
<span class="hint">A role passes when it matches at least one line: a country, a town or remote, so a line more means more roles. With every line empty a role can be anywhere in the region.</span>
|
||||
<span class="hint">The outer countries. Every role must be in them, and no company switch lifts this. All locations falls back to it.</span>
|
||||
<h3>Location</h3>
|
||||
<span class="hint">A role passes when it matches at least one line: a country, a town or remote, so a line more means more roles. With every line empty a role can be anywhere in the world gate.</span>
|
||||
<div class="where-line">
|
||||
<span>Anywhere in</span>
|
||||
<div id="a-countries"></div>
|
||||
@@ -55,28 +55,30 @@
|
||||
<span>Remote:</span>
|
||||
<select id="a-remote" aria-label="Remote">
|
||||
<option value="none">no</option>
|
||||
<option value="scope">anywhere within the region</option>
|
||||
<option value="scope">anywhere within the world gate</option>
|
||||
<option value="from">only when open to someone in:</option>
|
||||
</select>
|
||||
<div id="a-remote-from"></div>
|
||||
</div>
|
||||
<span class="hint">A remote role passes on top of the lines above. A remote role whose area is not stated yet passes until extract reads the posting.</span>
|
||||
<h3>Role</h3>
|
||||
<div class="where-line">
|
||||
<span>Only roles matching</span>
|
||||
<div id="a-terms"></div>
|
||||
</div>
|
||||
<span class="hint">Fetch terms, separated by a comma or Enter: <code>machine learning, data science</code> is two terms, and <code>machine learning</code> is searched as one. They are asked of the host, which then sends only roles with one of the terms anywhere in the ad, not only the title. They apply on boards of 300 or more roles whose host takes a query, and only there; smaller boards and other hosts are read whole. Empty means no narrowing.</span>
|
||||
<h3>Role exclusion</h3>
|
||||
<div class="where-line">
|
||||
<span>Leave out titles with</span>
|
||||
<div id="a-titles"></div>
|
||||
</div>
|
||||
<span class="hint">Phrases, separated by a comma or Enter. A phrase matches whole words anywhere in the title, whatever the case: <code>account executive</code> takes "Enterprise Account Executive, UK", and <code>data engineer</code> leaves "Data Engineering Manager" alone. A shorter phrase takes more: <code>sales</code> takes Sales Engineer too. A host cannot be asked this, so it applies once the roles are in, to liked companies as well. A role left out keeps its row, and Show gated on Roles lists it with the phrase that took it.</span>
|
||||
<span class="hint">Phrases, separated by a comma or Enter. A phrase matches whole words anywhere in the title, whatever the case: <code>account executive</code> takes "Enterprise Account Executive, UK", and <code>data engineer</code> leaves "Data Engineering Manager" alone. A shorter phrase takes more: <code>sales</code> takes Sales Engineer too. No host can be asked this, so it is applied to every role after the fetch, liked companies included. A role left out keeps its row, and Show gated on Roles lists it with the phrase that took it.</span>
|
||||
<h3>Role fetch</h3>
|
||||
<div class="where-line">
|
||||
<span>Only roles matching</span>
|
||||
<div id="a-terms"></div>
|
||||
</div>
|
||||
<span class="hint">Fetch terms, separated by a comma or Enter: <code>machine learning, data science</code> is two terms, and <code>machine learning</code> is searched as one. A host that takes a query sends only roles with one of the terms anywhere in the ad, but only for boards of 300 or more roles; smaller boards and other hosts are read whole. The terms are ignored after the fetch, since prompt closeness is the finer filter. Empty means no narrowing.</span>
|
||||
<p id="a-where-sum" class="where-sum" role="status"></p>
|
||||
<p id="a-kept" class="hint" hidden></p>
|
||||
</fieldset>
|
||||
<fieldset class="terms cv">
|
||||
<legend>Your CV</legend>
|
||||
<span class="hint">Optional. With a CV, the judge stage reads each of this agent's best roles against it and scores skills, experience and career, which CV match on Roles sorts by. A role is read at most once per change of the CV, the prompts or the texts below. The CV stays in this app and goes only to the model you set up.</span>
|
||||
<span class="hint">Optional. With a CV, the judge stage reads each of this agent's best roles against it and rates skills, experience and career from 0 to 10, which the CV match on Roles is made of. A role is read at most once per change of the CV, the prompts or the texts below. The CV stays in this app and goes only to the model you set up.</span>
|
||||
<label>CV
|
||||
<textarea id="a-cv-text" rows="10" placeholder="Paste your CV or career summary here"></textarea></label>
|
||||
<label>or read it from a file
|
||||
@@ -93,50 +95,36 @@
|
||||
<textarea id="a-like-text" rows="4" placeholder="Building a team from scratch, close work with the product side, small companies, a say in the roadmap"></textarea></label>
|
||||
<label>What I don't like
|
||||
<textarea id="a-avoid-text" rows="4" placeholder="Constant on-call, sales targets, large consultancies, open-plan offices"></textarea></label>
|
||||
<span class="hint">The work, the workplace, situations; read by the CV match for the career score. Rules that must hold go under the fetch gate or Other rules instead.</span>
|
||||
<span class="hint">The work, the workplace, situations; read by the judge for the career rating. Rules that must hold go under the fetch gate instead.</span>
|
||||
</fieldset>
|
||||
<fieldset class="weights limits">
|
||||
<legend>Analytics filter</legend>
|
||||
<label>Prompt closeness: the best <input type="number" id="a-cut-fit-top" min="0" step="10" value="500"></label>
|
||||
<label>each at least <input type="number" id="a-cut-fit-min" min="0" max="1" step="0.05" value="0.6"></label>
|
||||
<label>CV closeness: the best <input type="number" id="a-cut-cv-top" min="0" step="10" value="150"></label>
|
||||
<label>not already in, each at least <input type="number" id="a-cut-cv-min" min="0" max="1" step="0.05" value="0.6"></label>
|
||||
</fieldset>
|
||||
<span class="hint">Of the roles that pass the fetch gate, the analytics filter keeps the best N by prompt closeness and, of the rest, the best M by CV closeness. Liked and saved roles always pass, whatever their closeness. The roles that pass, the cut, get fetch text, extract and judge, so it sets what the models read. Closeness is a level from 0 to 1, the same in every run and free to compute; a liked company adds about 0.056 to it and a favourite about 0.111. For a broad agent the count stops a list, for a narrow one the minimum does, so roles that do not fit are not read to fill the count. An agent with no prompt has no prompt list, and one with no CV no CV list.</span>
|
||||
<fieldset class="weights">
|
||||
<legend>The feed order</legend>
|
||||
<label>Fit to the prompts <input type="number" id="a-fit" min="0" step="0.05" value="0.6"></label>
|
||||
<legend>Want feed score</legend>
|
||||
<label>Prompt closeness <input type="number" id="a-fit" min="0" step="0.05" value="0.6"></label>
|
||||
<label>Newness <input type="number" id="a-recency" min="0" step="0.05" value="0.2"></label>
|
||||
<label>Companies you like <input type="number" id="a-affinity" min="0" step="0.05" value="0.2"></label>
|
||||
<label>Roles you liked <input type="number" id="a-taste" min="0" step="0.05" value="0.2"></label>
|
||||
<label>Company like <input type="number" id="a-affinity" min="0" step="0.05" value="0.2"></label>
|
||||
<label>Job like <input type="number" id="a-taste" min="0" step="0.05" value="0.2"></label>
|
||||
</fieldset>
|
||||
<span class="hint">The final score, which orders What I'm looking for on Roles: the weighted sum of the four over the sum of the weights. Newness halves every 14 days; Roles you liked is a role's closeness to the roles you liked, away from those you passed. Changing the weights re-sorts and reads nothing.</span>
|
||||
<span class="hint">Orders What I'm looking for on Roles: the weighted sum of the four over the sum of the weights. Newness halves every 14 days; job like is a role's closeness to the roles you liked, away from those you passed. Changing the weights re-sorts and reads nothing.</span>
|
||||
<fieldset class="weights">
|
||||
<legend>How much each counts in the CV match</legend>
|
||||
<legend>CV match weights</legend>
|
||||
<label>Skills <input type="number" id="a-cv-skills" min="0" step="5" value="30"></label>
|
||||
<label>Experience <input type="number" id="a-cv-experience" min="0" step="5" value="25"></label>
|
||||
<label>Career <input type="number" id="a-cv-career" min="0" step="5" value="30"></label>
|
||||
</fieldset>
|
||||
<span class="hint">The judge's three numbers are averaged with these shares into the CV match. Changing them re-sorts at once and asks the model for nothing; 0 leaves a number out, all three 0 means no CV match.</span>
|
||||
<fieldset class="weights limits">
|
||||
<legend>The cut</legend>
|
||||
<label>Fit to the prompts: the best <input type="number" id="a-cut-fit-top" min="0" step="10" value="500"></label>
|
||||
<label>each at least <input type="number" id="a-cut-fit-min" min="0" max="1" step="0.05" value="0.6"></label>
|
||||
<label>Closeness to the CV: the best <input type="number" id="a-cut-cv-top" min="0" step="10" value="150"></label>
|
||||
<label>not already in, each at least <input type="number" id="a-cut-cv-min" min="0" max="1" step="0.05" value="0.6"></label>
|
||||
</fieldset>
|
||||
<span class="hint">Of the roles that pass the fetch gate, the cut holds the best by fit and, of the rest, the best by closeness to the CV. Liked and saved roles are always in, whatever their levels. The cut gets fetch text, extract and judge, so it sets what the models read. Both are levels from 0 to 1, the same in every run and free to compute; a liked company adds about 0.05 to them and a favourite about 0.1. For a broad agent the count stops a list, for a narrow one the minimum does, so roles that do not fit are not read to fill the count. An agent with no prompt has no fit list, and one with no CV no closeness list.</span>
|
||||
<span class="hint">The judge rates skills, experience and career from 0 to 10; the CV match, from 0 to 100, is those three weighted by these shares. It orders What fits me on Roles. Changing the weights re-sorts at once and asks the model for nothing; 0 leaves a number out, all three 0 means no CV match.</span>
|
||||
<fieldset class="weights limits">
|
||||
<legend>After the ranking</legend>
|
||||
<label>Summarise the top <input type="number" id="a-summarise-top" min="0" step="5" value="50"></label>
|
||||
<label>Summarise the top <input type="number" id="a-summarise-top" min="0" step="5" value="30"> of each feed order</label>
|
||||
</fieldset>
|
||||
<span class="hint">Summarise writes a short summary and tags for the top of the feed by the final score, and for every role you liked or saved, whatever its score. It is no part of any score.</span>
|
||||
<details class="advanced">
|
||||
<summary>Advanced</summary>
|
||||
<div class="advanced-body">
|
||||
<fieldset class="rules">
|
||||
<legend>Other rules</legend>
|
||||
<div id="a-rules" class="rule-rows"></div>
|
||||
<button type="button" id="a-add-rule" class="secondary">Add rule</button>
|
||||
<span class="hint">Must: a role must meet it. One of: a role must meet one of these or a Place line. Not: a role that meets it is left out. Values are separated by commas; salary is a number.</span>
|
||||
</fieldset>
|
||||
<label>Rules as sent
|
||||
<textarea id="a-gates" class="code" rows="6" spellcheck="false" readonly>[]</textarea>
|
||||
<span class="hint">Read only: what Save sends as the agent's gates, from the fetch gate and Other rules.</span></label>
|
||||
</div>
|
||||
</details>
|
||||
<span class="hint">Summarise writes a short summary and tags for the top N of What I'm looking for and the top N of What fits me, at most twice N roles, and for every role you liked or saved. It is no part of any score.</span>
|
||||
<div class="agent-actions">
|
||||
<button type="submit">Save and score</button>
|
||||
<button type="button" id="a-cancel" class="secondary">Cancel</button>
|
||||
|
||||
@@ -104,7 +104,7 @@
|
||||
|
||||
<section class="card" id="company-boards" data-section>
|
||||
<h2>Company boards</h2>
|
||||
<p class="lede">Every company roledesk has seen, and how many of its roles are open. Switching a board off stops reading it and closes its roles; hiding a company keeps reading it but leaves its roles out of the list and unscored. Like or favourite raises a company's roles a little in the cut and in the feed order, in each search agent's own ranking (a liked company about 0.05, a favourite about 0.1), and passes nothing the fetch gate rejects. All countries and All roles are for every agent: the first skips the place part of the fetch gate, the second the fetch terms and the titles to leave out.</p>
|
||||
<p class="lede">Every company roledesk has seen, and how many of its roles are open. Switching a board off stops reading it and closes its roles; hiding a company keeps reading it but leaves its roles out of the list and unscored. Like or favourite raises a company's roles a little in the cut and in the feed order, in each search agent's own ranking (a liked company about 0.056, a favourite about 0.111 on the start weights), and passes nothing the fetch gate rejects. All locations and All roles are for every agent: the first skips the location gate and keeps the world gate, the second skips the role exclusion and the fetch terms.</p>
|
||||
|
||||
<form id="board-form" class="agent-form source-form" autocomplete="off">
|
||||
<h3>Follow a company</h3>
|
||||
@@ -132,7 +132,7 @@
|
||||
<option value="favourite">Favourite</option>
|
||||
<option value="liked">Liked</option>
|
||||
<option value="hidden">Hidden</option>
|
||||
<option value="all_countries">All countries</option>
|
||||
<option value="all_countries">All locations</option>
|
||||
<option value="all_roles">All roles</option>
|
||||
<option value="on">On</option>
|
||||
<option value="off">Off</option>
|
||||
|
||||
+38
-30
@@ -70,7 +70,6 @@ def _feed_filters(
|
||||
function: str = "",
|
||||
remote: str = "",
|
||||
days: int | None = None,
|
||||
show_muted: bool = False,
|
||||
include_read: bool = True,
|
||||
tags: list[str] = Query(default=[]),
|
||||
tag_mode: Literal["any", "all"] = "any",
|
||||
@@ -96,7 +95,8 @@ def _feed_filters(
|
||||
them (`levels_between`), which is what the store filters on. `order=cv`
|
||||
is the score order with the agent's judged roles first by `cv_score`
|
||||
(`by_cv`). A request that names an agent keeps to its fetch gate in
|
||||
every order, newest included (`fetch_gate`), unless `gated`."""
|
||||
every order, newest included (`fetch_gate`), unless `gated`, and never
|
||||
shows a company the agent hid."""
|
||||
for token in tokens:
|
||||
if not places.known(token):
|
||||
raise HTTPException(422, f"No place is known as {token!r}.")
|
||||
@@ -115,7 +115,7 @@ def _feed_filters(
|
||||
return {
|
||||
"q": q or None, "company": company or None, "place": place or None,
|
||||
"function": function or None, "remote": remote or None, "days": days,
|
||||
"hide_muted": not show_muted, "include_read": include_read, "tags": tags,
|
||||
"include_read": include_read, "tags": tags,
|
||||
"tag_mode": tag_mode, "agent": _agent_id(store, agent),
|
||||
"by_score": agent is not None and order in ("score", "cv"),
|
||||
"by_cv": agent is not None and order == "cv", "gated": gated,
|
||||
@@ -155,7 +155,8 @@ def role_facets(store: StoreDep, filters: FeedFilters) -> Facets:
|
||||
def get_role(uid: str, store: StoreDep, agent: int | None = None) -> ScoredRole:
|
||||
"""One role, open or closed, for the read page, with `agent`'s vote and
|
||||
score, the prompt that gave its fit, its `judgement` against the agent's
|
||||
CV and the language flag (the default agent's when absent;
|
||||
CV (`skills`, `experience` and `career` 0 to 10, `cv_score` 0 to 100)
|
||||
and the language flag (the default agent's when absent;
|
||||
None when there is no agent). `copies` lists the open copies of a kept
|
||||
role (`RoleCopy`, for "Also posted"); a copy itself opens too, with
|
||||
`dup_of` naming its kept role."""
|
||||
@@ -235,9 +236,9 @@ def list_roles(
|
||||
`Store.roles` for what each filter does.
|
||||
|
||||
`agent` says whose votes, hidden companies and scores are meant. Each
|
||||
role carries that agent's `vote`; a company it hid is left out unless
|
||||
`show_muted` is set; `judged` keeps the liked, the unjudged or the
|
||||
passed. Every order keeps to the roles that agent's fetch gate passes,
|
||||
role carries that agent's `vote`; a company it hid is always left out,
|
||||
in every order, the total and the facets; `judged` keeps the liked, the
|
||||
unjudged or the passed. Every order keeps to the roles that agent's fetch gate passes,
|
||||
a gated one only with `gated=1`: with `order=score` (the default) best
|
||||
first by final score, carrying `score` and `gated_reason`; with
|
||||
`order=newest` newest first, carrying `gated_reason` and no score; with
|
||||
@@ -255,11 +256,11 @@ def list_roles(
|
||||
as well. `closing=1` keeps the roles whose `deadline` is today or in
|
||||
the next four days (Closing soon).
|
||||
|
||||
Each row carries `deadline`, the agent's `cv_score` (None when not
|
||||
judged), and `lang_flag` and `lang_note`, the posting's languages
|
||||
Each row carries `deadline`, the agent's `cv_score` (0 to 100, None
|
||||
when not judged), and `lang_flag` and `lang_note`, the posting's languages
|
||||
against the agent's (`app.models.language_flag`).
|
||||
|
||||
`company` and `place` are substrings, kept for links from other pages.
|
||||
`company` and `place` are substrings, kept for older links and saved filters.
|
||||
The rail's fields are `company_keys` (repeated, exact keys) and
|
||||
`places` (repeated place tokens, `app.places`: `country:DK`,
|
||||
`region:skane`, `place:malmo`, matched on whole folded words), several
|
||||
@@ -290,14 +291,15 @@ class FiltersOut(BaseModel):
|
||||
|
||||
|
||||
@router.get("/filters")
|
||||
def list_filters(store: StoreDep, show_muted: bool = False,
|
||||
agent: int | None = None) -> FiltersOut:
|
||||
def list_filters(store: StoreDep, agent: int | None = None) -> FiltersOut:
|
||||
"""The tag vocabulary for the feed's filter row, most-used first. The
|
||||
companies `agent` hid (the default agent's without one) are left out
|
||||
unless `show_muted`, matching `/api/roles`."""
|
||||
companies `agent` hid (the default agent's without one) are left out,
|
||||
matching `/api/roles`; with `agent` named, so is every role its fetch
|
||||
gate rejects, so a count is what the feed lists for the tag."""
|
||||
hidden_for = _agent_id(store, agent)
|
||||
return FiltersOut(tags=[TagCount(tag=tag, count=count, unread=unread)
|
||||
for tag, count, unread in store.tag_counts(hidden_for, show_muted)])
|
||||
for tag, count, unread in store.tag_counts(
|
||||
hidden_for, fetch_gate=agent is not None)])
|
||||
|
||||
|
||||
# ------------------------------------------------------------ suggestions
|
||||
@@ -343,14 +345,17 @@ def suggest_companies(
|
||||
folded as places are (`places.fold`), so `orsted` finds Ørsted and
|
||||
`kobenhavn` finds a name with København in it. By open roles, most
|
||||
first. The companies `agent` (the default agent when absent) hid are
|
||||
left out, as the feed leaves them out. An empty `q` suggests nothing."""
|
||||
left out, as the feed leaves them out; with `agent` named, the count is
|
||||
the roles its feed shows, past its fetch gate. An empty `q` suggests
|
||||
nothing."""
|
||||
text = places.fold(q)
|
||||
if not text:
|
||||
return []
|
||||
aliases = store.company_aliases()
|
||||
companies = store.open_companies(_agent_id(store, agent), fetch_gate=agent is not None)
|
||||
found = [CompanySuggestion(key=key, name=names[0] if names else key, count=count,
|
||||
affinity=affinity)
|
||||
for key, names, count, affinity in store.open_companies(_agent_id(store, agent))
|
||||
for key, names, count, affinity in companies
|
||||
if any(_starts_a_word(text, places.fold(name)) for name in names)
|
||||
or any(_starts_a_word(text, fold) for fold in aliases.get(key, ()))]
|
||||
found.sort(key=lambda c: (-c.count, c.name.casefold()))
|
||||
@@ -370,11 +375,13 @@ def suggest_places(
|
||||
open role's place line and every open remote role's area). The count of
|
||||
`remote:any` is the remote roles in the scope of `agent`, the default
|
||||
agent when absent, as the feed counts them, and the other remote tokens
|
||||
are kept to that agent's scope, its `require` gates on country. An empty
|
||||
`q` suggests nothing."""
|
||||
are kept to that agent's scope, its `require` gates on country. With
|
||||
`agent` named, every count is of the roles its feed shows, past its
|
||||
fetch gate. An empty `q` suggests nothing."""
|
||||
if not places.fold(q):
|
||||
return []
|
||||
viewer = _agent_id(store, agent)
|
||||
shown = viewer if agent is not None else None
|
||||
gaz = store.gazetteer()
|
||||
found = store.agent(viewer) if viewer is not None else None
|
||||
scope = {p for g in (found.gates if found is not None else ())
|
||||
@@ -382,10 +389,11 @@ def suggest_places(
|
||||
for v in g.values for p in gaz.lookup(f"country:{v}") or ()}
|
||||
return [PlaceSuggestion(token=s.token, label=s.label, kind=s.kind, count=s.count)
|
||||
for s in places.suggest(
|
||||
q, store.open_places(), limit, remote=store.open_remote_areas(),
|
||||
q, store.open_places(shown), limit, remote=store.open_remote_areas(shown),
|
||||
gaz=gaz, scope=scope,
|
||||
remote_any=lambda: store.count_roles(places=["remote:any"], agent=viewer,
|
||||
by_score=False))]
|
||||
by_score=False,
|
||||
fetch_gate=agent is not None))]
|
||||
|
||||
|
||||
@router.get("/places/labels")
|
||||
@@ -506,9 +514,10 @@ class CompanyOut(BaseModel):
|
||||
`careers_opened_at` are the Watchlist's (`Company`'s), the latter also
|
||||
when a person last looked at the careers page. `by_hand` marks a company
|
||||
with no board shown that is not `from_search`, and `hand_reason` says why
|
||||
it is read by hand (`app.models.HAND_REASONS`). `all_countries` and
|
||||
`all_roles` are the company's two fetch overrides: its big boards read
|
||||
with the agents' wide reach, and with no role-type narrowing. `target` is
|
||||
it is read by hand (`app.models.HAND_REASONS`). `all_countries` (All
|
||||
locations) and `all_roles` (All roles) are the company's two fetch
|
||||
overrides: its big boards read with the agents' wide reach, and with no
|
||||
role-type narrowing. `target` is
|
||||
the attached board in full, `host:slug`, where `board` is its host alone;
|
||||
the page compares it with the board typed to know whether a replacement
|
||||
closes anything. `ats_slug` cannot stand in for it, since the careers site
|
||||
@@ -771,7 +780,7 @@ def set_affinity(key: str, body: AffinityBody, store: StoreDep,
|
||||
"""-1 hidden, 0 neutral, 1 liked, 2 favourite, for `body.agent` or the
|
||||
default agent. A key with roles and no company row gets one first (see
|
||||
`Store.set_affinity`); 404 for a key with neither. A change here is not
|
||||
seen in Agent match until `score` has run, so it queues one
|
||||
seen in "What I'm looking for" until `score` has run, so it queues one
|
||||
(`_score_after_change`) and answers with its id."""
|
||||
if body.affinity not in AFFINITY:
|
||||
raise HTTPException(422, "affinity is -1, 0, 1 or 2")
|
||||
@@ -826,7 +835,8 @@ def _override(store: Store, queue: Any, key: str, on: bool, setter: str) -> Comp
|
||||
@router.post("/companies/{key}/all-countries")
|
||||
def set_all_countries(key: str, body: OverrideBody, store: StoreDep,
|
||||
queue: QueueDep) -> Company:
|
||||
"""All countries on or off: global, apart from affinity. On, the
|
||||
"""All locations on or off (the field and the path keep the name
|
||||
`all_countries`): global, apart from affinity. On, the
|
||||
company's roles skip the place part of every agent's fetch gate (the
|
||||
`any` gates, `jobs.score.failed_gates`), and its big boards are read
|
||||
with the agents' wide reach, Europe today, rather than their places
|
||||
@@ -1241,7 +1251,7 @@ class AgentBody(BaseModel):
|
||||
it is stored as one prompt named after the agent. `fetch_terms` narrow
|
||||
what big boards send (`app/jobs/reach.py`); `[]` narrows nothing, and
|
||||
leaving the key out keeps an existing agent's. `cv_text`,
|
||||
`cv_languages`, `like_text`, `avoid_text` and `must_text` are what `judge` reads a
|
||||
`cv_languages`, `like_text` and `avoid_text` are what `judge` reads a
|
||||
role against (`app/jobs/judge.py`), kept the same way when left out or
|
||||
null."""
|
||||
|
||||
@@ -1257,7 +1267,6 @@ class AgentBody(BaseModel):
|
||||
cv_languages: CvLanguages | None = None
|
||||
like_text: str | None = None
|
||||
avoid_text: str | None = None
|
||||
must_text: str | None = None
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
@@ -1297,8 +1306,7 @@ def save_agent(body: AgentBody, store: StoreDep, queue: QueueDep) -> Agent:
|
||||
body.enabled, agent_id=body.id, limits=body.limits,
|
||||
prompts=body.prompts, fetch_terms=body.fetch_terms,
|
||||
cv_text=body.cv_text, cv_languages=body.cv_languages,
|
||||
like_text=body.like_text, avoid_text=body.avoid_text,
|
||||
must_text=body.must_text)
|
||||
like_text=body.like_text, avoid_text=body.avoid_text)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(409, str(exc)) from exc
|
||||
except KeyError:
|
||||
|
||||
+59
-65
@@ -51,11 +51,10 @@ agent:
|
||||
max_tokens: 8192 # the answer's cap per call
|
||||
cli_path: "" # where the `claude` binary is, if not on PATH
|
||||
|
||||
# Title and description in, gate columns out (app/jobs/extract.py). Off by
|
||||
# default: on the API backend every batch of 20 roles is a paid call. On, a
|
||||
# pull reads only the cut of every enabled agent (its limits, see agents
|
||||
# below), after a first score; a second score then applies the fetch gate and
|
||||
# the cut to what was filled. `enabled` is only the
|
||||
# Title and description in, gate columns out: location, remote area, seniority,
|
||||
# languages and deadline (app/jobs/extract.py). Off by default: on the API
|
||||
# backend every batch of 20 roles is a paid call. On, a pull reads the cut of
|
||||
# every enabled agent (its limits, see agents below). `enabled` is only the
|
||||
# default for a fresh database: the switch under Settings, Sources, Runs (or
|
||||
# `app stages extract on|off`) overrides it and needs no restart.
|
||||
extract:
|
||||
@@ -66,11 +65,11 @@ extract:
|
||||
thinking: off # off | adaptive
|
||||
|
||||
# About 60 words on a role: what it is, the stack, where, the pay if stated
|
||||
# (app/jobs/summarise.py). Off by default. On, each pull summarises the top
|
||||
# of every enabled agent's feed that has none yet, the best summarise_top by
|
||||
# final score (its limits), and every role liked or saved, so the cost stays
|
||||
# bounded however many roles are open. The Summarise button on
|
||||
# a role's page asks for that one role either way. `enabled` is only the
|
||||
# (app/jobs/summarise.py). Off by default. On, each pull summarises the
|
||||
# summarise_top (an agent's limits) best roles of each enabled agent's two
|
||||
# orders, What I'm looking for and What fits me, and every role liked or
|
||||
# saved, when they have no summary yet. The Summarise button on a role's page
|
||||
# asks for that one role either way. `enabled` is only the
|
||||
# default for a fresh database: the switch under Settings, Sources, Runs (or
|
||||
# `app stages summarise on|off`) overrides it and needs no restart.
|
||||
summarise:
|
||||
@@ -83,12 +82,12 @@ summarise:
|
||||
# Only extract reads it.
|
||||
|
||||
# Each search agent's cut read against its CV: skills, experience and career 0
|
||||
# to 100, a total, and 1 to 3 strengths and gaps (app/jobs/judge.py), for the
|
||||
# CV match order on Roles. Off by default. On, each pull judges the cut of
|
||||
# every enabled agent with a CV and a prompt, and only the roles not judged
|
||||
# yet or judged from another CV, prompt, like, avoid or must text, or role
|
||||
# text. The CV rides in every call, so batches are
|
||||
# small. `enabled` is only the default for a fresh database: the switch under
|
||||
# to 10, and 1 to 3 strengths and gaps (app/jobs/judge.py); the CV match, which
|
||||
# orders What fits me on Roles, weights the three. Off by default. On, each pull
|
||||
# judges the cut of every enabled agent with a CV and a prompt, once per change
|
||||
# of its CV, prompts, likes, dislikes or must text, or of the role's text. The
|
||||
# CV rides in every call, so batches are small. `enabled` is only the default
|
||||
# for a fresh database: the switch under
|
||||
# Settings, Sources, Runs (or `app stages judge on|off`) overrides it and needs
|
||||
# no restart. on_ingest is accepted and ignored, as for summarise.
|
||||
judge:
|
||||
@@ -109,25 +108,23 @@ fetch_text:
|
||||
per_run: 1000 # reads per run
|
||||
retry_days: 7 # days before a page that gave no text is read again
|
||||
|
||||
# A role and each agent's prompts as vectors, for the fit term of a score
|
||||
# (app/jobs/embed.py). Needs the `embeddings` extra; without it `embed` posts
|
||||
# a notice and scores leave fit out. Changing the model re-embeds.
|
||||
# multilingual-e5-small, not English-only MiniLM, because the corpus is now
|
||||
# European: measured against the "Copenhagen engineering" agent's intent, it
|
||||
# separates engineering from non-engineering roles better, especially among
|
||||
# Danish-country roles (see docs/decisions.md), for about 3x the CPU time. An
|
||||
# e5 model's "query: " / "passage: " prefixes are added automatically, keyed
|
||||
# off this name.
|
||||
# calibration maps a model's cosines to 0..1 levels, as [lo, hi] for fit and
|
||||
# for taste: lo is 0, hi is 1, clipped outside. multilingual-e5-small has
|
||||
# built-in constants (app/config.py, CALIBRATION); an entry here replaces
|
||||
# them for that model. A model with none is min-max scaled per run instead,
|
||||
# so its fit is a rank, not a level, and the log says so.
|
||||
# A role, each agent's prompts and its CV as vectors, for prompt closeness and
|
||||
# CV closeness (app/jobs/embed.py). Needs the `embeddings` extra; without it
|
||||
# `embed` posts a notice and scores leave closeness out. Changing the model
|
||||
# re-embeds. multilingual-e5-small, not English-only MiniLM, because the corpus
|
||||
# is European (see docs/decisions.md). An e5 model's "query: " / "passage: "
|
||||
# prefixes are added automatically, keyed off this name.
|
||||
# calibration maps a model's cosines to 0..1 levels, as [lo, hi] for fit (prompt
|
||||
# closeness), taste (job like) and cv (CV closeness, fit's band when left out):
|
||||
# lo is 0, hi is 1, clipped outside. multilingual-e5-small has built-in
|
||||
# constants (app/config.py, CALIBRATION); an entry here replaces them for that
|
||||
# model. A model with none is min-max scaled per run instead, so its levels are
|
||||
# ranks, and the log says so.
|
||||
embed:
|
||||
model: intfloat/multilingual-e5-small
|
||||
per_run: 50000
|
||||
calibration: {}
|
||||
# intfloat/multilingual-e5-small: {fit: [0.75, 0.88], taste: [-0.14, 0.19]}
|
||||
# intfloat/multilingual-e5-small: {fit: [0.75, 0.88], taste: [-0.14, 0.19], cv: [0.77, 0.90]}
|
||||
|
||||
# One block per plugin, keyed by the plugin's name. The ats and search plugins
|
||||
# read their boards and searches from the subscriptions table, so they need
|
||||
@@ -140,39 +137,36 @@ sources: {}
|
||||
# search:
|
||||
# adzuna: {app_id: "...", app_key: "..."}
|
||||
|
||||
# Starter search agents, imported once into an empty `agents` table. The gates
|
||||
# are the agent's fetch gate, each {field, values, mode: require|any|exclude}:
|
||||
# a require gate is the region, the any gates are one group a role must match
|
||||
# one of ("in Denmark, or remote"), the place part, and an exclude gate leaves
|
||||
# out the roles that match one of its values (title: the titles to leave
|
||||
# out). A gate rejects only a role whose column is filled, since extract fills
|
||||
# the rest for the roles in the cut. A company set to All countries skips the
|
||||
# any group, one set to All roles the exclude gates on title. Fields are
|
||||
# place, country, remote, remote_from, seniority, function, salary_min,
|
||||
# currency, permit, languages, company and title. weights are the final
|
||||
# score's, the feed's order, default {fit: 0.6, recency: 0.2, affinity: 0.2,
|
||||
# taste: 0.2}; taste is how close a role is to the ones the agent liked minus
|
||||
# the ones it passed. A country is an ISO code, filled from the board's
|
||||
# location line; EU is a role open anywhere in Europe.
|
||||
# limits is the agent's cut, the roles that get fetch text, extract and
|
||||
# judge: the best cut_fit_top by fit level at cut_fit_min or more, then the
|
||||
# best cut_cv_top by CV closeness at cut_cv_min or more that the first list
|
||||
# leaves out, then every role liked or saved. Both levels are calibrated, so
|
||||
# a minimum means the same in every run, and a liked company's roles sit
|
||||
# about 0.05 higher, a favourite's about 0.1. summarise_top is how many of
|
||||
# the best final scores summarise reads. The defaults are {cut_fit_top: 500,
|
||||
# cut_fit_min: 0.6, cut_cv_top: 150, cut_cv_min: 0.6, summarise_top: 50}. For
|
||||
# a broad agent the count stops a list, for a narrow one the minimum. An
|
||||
# agent without a prompt that has text has no fit, so nothing of it is in
|
||||
# the cut by fit.
|
||||
# prompts is a list of {name, text}, each what the agent looks for; a role's
|
||||
# fit is its best prompt's, so one agent can hunt several kinds of role
|
||||
# without blending them into one paragraph. Votes, gates and limits stay the
|
||||
# agent's, shared by its prompts. intent, as in config.yaml, is one prompt
|
||||
# named after the agent.
|
||||
# enabled (default true) leaves an agent out of the scores and the pull when
|
||||
# false. The Search agents sub-tab lists and edits them; the order menu on
|
||||
# Roles picks one by name. Two more, beside the starter in config.yaml:
|
||||
# Starter search agents, imported once into an empty `agents` table. The Search
|
||||
# agents page lists and edits them afterwards, and the Viewing as select on
|
||||
# Roles picks one by name. An agent has:
|
||||
# - name, and enabled (default true; false leaves it out of the scores and the
|
||||
# pull).
|
||||
# - prompts, a list of {name, text}, each what the agent looks for; a role is
|
||||
# ranked by the prompt it is closest to. intent, as in config.yaml, is one
|
||||
# prompt named after the agent.
|
||||
# - gates, its fetch gate, each {field, values, mode: require|any|exclude}: a
|
||||
# require gate is the region (the world), the any gates are one group a role
|
||||
# must match one of ("in Denmark, or remote"; the location), and an exclude
|
||||
# gate leaves out the roles that match one of its values (title: the titles
|
||||
# to leave out). A gate rejects only a role whose column is filled, since
|
||||
# extract fills the rest for the roles in the cut. A company set to All
|
||||
# locations skips the any group, one set to All roles the exclude gates on
|
||||
# title. Fields are place, country, remote, remote_from, seniority,
|
||||
# function, salary_min, currency, permit, languages, company and title. A
|
||||
# country is an ISO code, filled from the board's location line; EU is a
|
||||
# role open anywhere in Europe.
|
||||
# - weights, the want feed score's, which orders What I'm looking for: {fit:
|
||||
# 0.6, recency: 0.2, affinity: 0.2, taste: 0.2}, that is prompt closeness,
|
||||
# newness, company like and job like; taste is how close a role is to the
|
||||
# ones the agent liked minus the ones it passed. cv_skills, cv_experience and
|
||||
# cv_career {30, 25, 30} weight the judge's three numbers in the CV match.
|
||||
# - limits, the analytics filter and the summarise size: {cut_fit_top: 500,
|
||||
# cut_fit_min: 0.6, cut_cv_top: 150, cut_cv_min: 0.6, summarise_top: 30}, the
|
||||
# count and the minimum by prompt closeness and by CV closeness, and how deep
|
||||
# into each order summarise reads. See documentation/ADR Fetch prep and
|
||||
# score.md.
|
||||
# Two more, beside the starter in config.yaml:
|
||||
agents: []
|
||||
# - name: Remote from Europe
|
||||
# prompts:
|
||||
@@ -191,7 +185,7 @@ agents: []
|
||||
# LV, LT, LU, MT, NL, PL, PT, RO, SK, SI, ES, SE, IS, LI, NO,
|
||||
# GB, CH, EU], mode: require}
|
||||
# limits: {cut_fit_top: 1000, cut_fit_min: 0.5, cut_cv_top: 150, cut_cv_min: 0.6,
|
||||
# summarise_top: 50}
|
||||
# summarise_top: 30}
|
||||
# - name: Europe hybrid
|
||||
# intent: >-
|
||||
# Senior backend or platform engineer in a European tech hub, hybrid or
|
||||
|
||||
+4
-4
@@ -27,10 +27,10 @@ embed:
|
||||
model: intfloat/multilingual-e5-small
|
||||
sources: {}
|
||||
# The starter ranking, imported once into an empty table. Edit it under
|
||||
# Settings, Ranking afterwards; this list is not read again. EU in a country
|
||||
# gate is a role open anywhere in Europe ("Remote - Europe", "EMEA"). Which
|
||||
# roles get fetch text, extract and judge is each agent's cut, its `limits`, see
|
||||
# config.example.yaml, which has two more agents.
|
||||
# Settings, Search agents afterwards; this list is not read again. EU in a
|
||||
# country gate is a role open anywhere in Europe ("Remote - Europe", "EMEA").
|
||||
# Which roles get fetch text, extract and judge is each agent's cut, its
|
||||
# `limits`, see config.example.yaml, which has two more agents.
|
||||
agents:
|
||||
- name: Copenhagen engineering
|
||||
intent: >-
|
||||
|
||||
+130
-174
@@ -1,319 +1,275 @@
|
||||
# Key decisions
|
||||
|
||||
What was decided for roledesk and the one reason for each. The longer reasoning is in `../news/docs/JobBoardPlan/roles-build-plan.md` and `jobhunt-hq-assessment.md`.
|
||||
What was decided for roledesk and the reason for each. The target design is the two ADRs in `documentation/`; where a paragraph names an ADR section, the rule is there and this file adds only the reasoning. Host details are in `plugins/README.md`.
|
||||
|
||||
## Shape
|
||||
|
||||
**Built on the boilerplate, with `desk` copied from news.** `pipeline/` (queue and scheduler, pipeline 2.0), `auth/` and `deploy/` come from `../boilerplate` and `desk/` from `../news`, all four as downstream copies refreshed with `git archive`. A fix lands upstream first, or the next refresh reverts it. The package is `app`, per the boilerplate rule, since the deploy kit and the `APP_*` variables assume it.
|
||||
|
||||
**The domain noun is `role`.** `jobs/` keeps meaning the work the app queues, so "job" never has two senses in one sentence.
|
||||
|
||||
**Own database, queue and port.** `roledesk.db`, `config.yaml`, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently.
|
||||
|
||||
**Storage is written, not imported.** `desk` has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`.
|
||||
**Storage is written, not imported.** `desk` has no storage package on purpose, so the connection plumbing is copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`.
|
||||
|
||||
**Layers are `models, places, seniority, config, storage, jobs, web, main, cli`.** `places` and `seniority` read what a board wrote and know no database, so they sit under `config` and storage may use them. `tests/test_architecture.py` holds the direction and the doors within `app`; `tests/test_coupling.py` holds that `pipeline`, `auth` and `desk` import nothing from `app` or each other, except `desk` from `pipeline`, and that a plugin imports `desk` only.
|
||||
**Layers are `models, places, dupes, seniority, config, storage, prep, jobs, web, main, cli`.** `places`, `dupes` and `seniority` read what a board wrote and know no database, so they sit under `config` and storage may use them. `tests/test_architecture.py` holds the direction and the doors within `app`; `tests/test_coupling.py` holds that `pipeline`, `auth` and `desk` import nothing from `app` or each other, except `desk` from `pipeline`, and that a plugin imports `desk` only.
|
||||
|
||||
## Data
|
||||
|
||||
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one role under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
|
||||
**Copies are matched on towns, not on the stored place key** (`app/dupes.py`; the rule is ADR Fetch prep and score 1.4). Host does not matter, since a search provider and the company's board list the same role, and one board may list one role under several ids. The stored `(company_key, title_key, place_key)` finds none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each.
|
||||
|
||||
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, a row with no country matches Japan and Malaysia, and joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member; a vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own. A known miss: `Remote - USA NH` and `Remote - USA WA` merge, since a state code is not a town.
|
||||
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, and joining on one match merged both on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member.
|
||||
|
||||
**A copy is stored as `roles.dup_of` and rebuilt after every pull.** The feed, its counts and facets, the tag and company counts, scoring, embedding, extract and summarise all add `dup_of IS NULL`, which keeps `idx_roles_posted` usable where a window function over the filtered rows would scan them all. The kept role is a board over a search provider, then one with a description, then the earliest `posted_at`, then the smallest uid. `close_boards` rebuilds from scratch in 0.3 s after its closes, and so do `set_fetch` and `remove_search` when they close roles, so a closed kept role hands its group to the next copy. Where a group changed, the kept role takes the earliest `read_at` and each agent's latest vote from its copies and the role it replaces; an unchanged group is left alone, so marking a role unread again sticks. The per-board counts on the Sources page still count copies, since they report what a board holds. A feed row shows `×N` for N postings and the reader lists them under "Also posted", newsdesk's "Also covered by"; a copy opened by its link says which role the feed lists instead.
|
||||
**One posting per place is folded too, on every host.** A host that lists one job once per office gives rows of the same company and title whose towns never meet; `dupes.fold` merges those of one host when their texts share at least `SAME_TEXT` (30%) of their words, and the kept role keeps each copy's place and link. It applies to every host, not only to readers that declare `places="copies"`, because most such rows came from hosts that do not declare it. It keeps to one country, as ADR 1.4 has it, since the same rule across countries merged different openings whose texts happened to be alike (a federal sales role in the US behind one in Australia); a job posted once per country is then one role per country, which costs a few rows for a pan-European posting and hides nothing. A remote listing and an office listing stay two roles: the kept role takes a copy's places and not its remote, and the fetch gate reads a remote role by its area, so a remote listing folded behind an office elsewhere was rejected with it, and an office listing behind a remote role elsewhere likewise.
|
||||
|
||||
**`place`, `country` and `remote` are separate columns.** `location` already means where an `Item`'s original lives. `remote` alone over-matches, because boards write "UK | Remote", so a gate on remote also reads `country`.
|
||||
**A copy is stored as `roles.dup_of` and rebuilt from scratch after every pull.** Every reader of open roles adds `dup_of IS NULL`, which keeps `idx_roles_posted` usable where a window function over the filtered rows would scan them all. Where a group changed, the kept role takes the earliest `read_at` and each agent's latest vote from its copies; an unchanged group is left alone, so marking a role unread again sticks. The per-board counts on Sources still count copies, since they report what a board holds.
|
||||
|
||||
**`raw` is transport, columns are storage.** No `Item.facets`. A plugin puts structured fields in `raw` under the column's name and `upsert_role` unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what `extract` found.
|
||||
**`place`, `country` and `remote` are separate columns.** desk's `Item.location` already means where the original lives.
|
||||
|
||||
**`extract` fills, the board wins.** `set_extracted` writes a model's value only into an empty column. A board's structured field is first-party, and `upsert_role` would put it back on the next pull anyway, so letting the model replace it would make a column flip between two answers. `place` brings `place_key` with it, which only ever moves a role from the empty place key to a real one.
|
||||
**`raw` is transport, columns are storage.** A plugin puts structured fields in `raw` under the column's name and `upsert_role` unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what `extract` found.
|
||||
|
||||
**`function` is the one column where the model wins, because the board never filled it.** `classify(department, title)` in `plugins/ats.py` is `ats`'s own keyword guess, not a board field, so "the board wins" does not apply to it: `set_extracted` writes a non-empty `function` over the guess outright, rather than only into an empty column, and `upsert_role` then leaves `function` alone for any role with `extracted_at` set, whatever the classifier says on the next pull. A role not yet extracted still gets a fresh guess every pull, the same as any other raw field. One rule, so `function` cannot flip between the classifier's answer and the model's on alternating pulls, which is exactly what "the board wins" above was written to prevent for a real board field.
|
||||
**A board's field wins over extract's, so a column cannot flip** (ADR Fetch prep and score 4.1). `set_extracted` writes a model's value only into an empty column. A board's field is first-party and `upsert_role` would put it back on the next pull, so letting the model replace it would make the column flip between two answers. Seniority and remote are the exceptions, below.
|
||||
|
||||
**`classify` matches whole words, not substrings, and checks the title before the department.** The ported version matched substrings, so `" it"` (meant to catch an "IT" department) matched inside "Italy" and "Italia", filing European sales roles ("Commercial Italy") as engineering; it now requires a word boundary, doubled on both sides for the handful of short tokens ("it", "seo", "qa", "ux", ...) that are themselves the start of an unrelated word ("Seoul", "Qatar"), while the rest keep a leading boundary only, since most are deliberate stems ("manag", "engineer", "recruit") meant to also match "management", "engineering", "recruiting". A department is often a generic umbrella ("Operations", "Engineering & Product") that says less than the title does, so the title is checked first and the department is the fallback; a hand-checked sample of 60 roles where the two orders disagreed found title-first right about twice as often. Nothing matching returns `''`, not `"other"`, which claimed a knowledge the classifier does not have. A modest set of German, Dutch, Danish, Norwegian, Swedish and French keywords (developer/engineer, sales) covers the seed's other languages; it is deliberately not complete, since `extract` is the rest.
|
||||
**`function` is the one column where the model wins outright, because no board fills it.** `classify(department, title)` in `plugins/ats.py` is a keyword guess, not a board field: `set_extracted` writes a non-empty `function` over it, and `upsert_role` leaves `function` alone once `extracted_at` is set, so it cannot flip between the classifier and the model on alternating pulls.
|
||||
|
||||
**Structured pay is unpacked the same as any other board field, except that Greenhouse and Ashby need an extra query parameter to get it.** Ashby sends it with `?includeCompensation=true`, Lever's `salaryRange` and Recruitee's `salary` are already on the plain list response, and Greenhouse needs `pay_transparency=true` on the same request it already makes (no per-role fetch); Personio's XML rarely carries a `salaryInformation` block, but reads the same way when it does. Workable and SmartRecruiters give neither, checked live against their real boards (2026-09-29). Every host converts to yearly `salary_min`/`salary_max`/`currency`: a monthly figure is multiplied by 12, an hourly or a daily one is left out rather than annualised into a number nobody stated, and a range with no currency is ignored. Greenhouse can give several ranges for one role (one per office or pay zone); with more than one there is no way to tell which office the role in hand is at, so all of them are skipped rather than one being picked at random.
|
||||
**`classify` matches whole words and checks the title before the department.** Substring matching filed "Commercial Italy" as engineering through `" it"`. Short tokens that start unrelated words ("it", "seo", "qa") need a boundary on both sides, while the rest keep a leading boundary only, since they are stems ("manag", "engineer"). A department is often an umbrella ("Operations") that says less than the title; on a hand-checked sample of 60 roles where the two orders disagreed, title first was right about twice as often. No match returns `''`, not `"other"`, which would claim a knowledge the classifier does not have.
|
||||
|
||||
**Pay that does not normalise is not dropped, only kept out of the gate columns.** `roles.salary_text` (migration 12) is one human-readable line per parser, written whether or not the board's pay conformed (an hourly or a daily rate, several Greenhouse pay zones, a range with no currency), so a person can read it and do their own sums; it is filled through `raw` like the other columns, but replaced whole on every pull, including with `''` when a board stops stating pay, since the usual keep-filled rule (`_RAW_TEXT`) would otherwise leave a stale line behind. `salary_state` (`app.models.Role`) is derived, not stored: 'normalised' when `salary_min` or `salary_max` is set, 'stated' when only `salary_text` is, `''` otherwise; it is what the feed's Pay filter and badge read, and gates and scoring keep reading only `salary_min`/`salary_max`/`currency`, unchanged.
|
||||
**Several pay ranges for one role give no figures.** Greenhouse can state one range per office or pay zone, and with more than one there is no way to tell which office the role in hand is at, so none is picked.
|
||||
|
||||
**`extract` reads the board's location too.** The model gets title, the board's `place` text, and the first 3000 characters of the description. Greenhouse's location line ("Berlin, Germany") is often the only place a country is stated, and it is part of the posting, so the "only what the posting states" contract still holds.
|
||||
**Pay that does not normalise is kept as a line, out of the gate columns.** `roles.salary_text` is one readable line written whether or not the pay conformed (an hourly rate, several pay zones, no currency), so a person can do their own sums. It is replaced whole on every pull, `''` included, since the usual keep-filled rule would leave a stale line when a board stops stating pay. `salary_state` is derived, not stored, and is what the Pay filter and badge read; gates and scoring read only `salary_min`, `salary_max` and `currency`.
|
||||
|
||||
**`country` is filled from `place` by a lookup, before `extract`.** `country_of` maps the location line to an ISO code from every ISO country by its English name (and its own-language name for a European one) and the cities the boards name most, where `extract` would be a paid call per role for the same answer. On the live data (2026-09-29) 7.0% of open roles resolve to nothing, down from 16.8%, and most of those write only "Hybrid", "Remote" or a region such as "Americas". The line is folded by `_place_fold`, which also maps ł, đ, ø and æ since NFKD leaves them whole; `_fold` stays as it is because it builds `company_key`. Within one location (the line split at `;`, `|` and `/`) the widest unit named wins: a country beats a US state or a Canadian province, which beats a city, so "London, Ontario" is CA and "Tilburg, NB, Netherlands" is NL. A state code that is also a country's (DE, CO, MA, IN) counts as the state only when the line also names a US place, so "Bogota, CO" is CO and "Paderborn, DE" is ''. A line naming several countries is `EU` when all of them are European (the UN's Europe, plus Cyprus, Turkey and the Council of Europe's Caucasus members) and '' otherwise: "Dublin, London" is EU, "Remote, EU or US" is ''. `EU`, ISO's reserved code, also stands for "Remote - Europe" and "EMEA", so a Europe gate is a list of codes that includes it; a country beats the region it is in ("London, EMEA" is GB). `backfill_from_place` only fills an empty column, so a fix that changes answers comes with a migration: v12 recomputes `country` for every role `extract` has not read and whose board sent no `country` in `raw`, which changed 1,705 open roles, 48 of them from one code to another or to '' (25 Hong Kong roles moved from CN to HK).
|
||||
**`extract` reads the board's location line beside the text.** It gets the title, the board's `place` and the opening 2,500 and closing 1,500 characters of the description (`HEAD` and `TAIL` in `app/jobs/extract.py`). Greenhouse's location line ("Berlin, Germany") is often the only place a country is stated, and it is part of the posting, so the "only what the posting states" contract holds. Salary, place and remote sit in the opening lines, languages and the deadline in the closing ones.
|
||||
|
||||
**`permit` and `languages` are columns, and so is `extract_tries`.** `permit` answers the plan's open item 1 (visa sponsorship as a gate field) while nothing depends on it yet. newsdesk keeps give-up counts in `raw`; here `upsert_role` rewrites `raw` on every pull, which would reset the count, so the counter is its own column.
|
||||
**`country` comes from the places by lookup, not from `extract`.** A lookup gives the answer a paid call per role would. prep takes the one country the resolved places lie in, `EU` when they lie in several European countries, and the host's stated country only as a fallback (`app/prep/country.py`).
|
||||
|
||||
**Postings are persisted.** Storing every posting is what gives `first_seen`, when the app itself first stored the role, and `closed_at`, set by `mark_closed` when a complete fetch of a board no longer lists a role. A role listed again is open again.
|
||||
**`extract_tries` is a column.** newsdesk keeps give-up counts in `raw`; here `upsert_role` rewrites `raw` on every pull, which would reset the count.
|
||||
|
||||
**`posted_at` (migration 10) is the earlier of `first_seen` and a trusted host's `published_at`; `first_seen` is left alone, as when the app found the role.** `first_seen` being the only honest posted date was true of Greenhouse's `updated_at`, which resets on every edit; `plugins.ats` already reads `first_published` instead, and that plus the other hosts' dates (Lever's `createdAt`, Ashby's `publishedAt`, SmartRecruiters' `releasedDate`, Recruitee's `published_at`/`created_at`, Personio's `createdAt`, Workable's `published`, and the two `search` providers' (Jobindex's RSS `pubDate`, Adzuna's `created`)) are creation dates, checked against live boards and the live database (`app.models.TRUSTED_POSTED_HOSTS`, with the evidence in its comment). A date in the future or before 2015 is not trusted either, host or no host. Without this, a newly added board's whole backlog scored as freshly posted for two weeks: on the live database, 10,223 of 15,415 board roles were posted more than 30 days before they were first seen (average gap in days: Ashby 78, Greenhouse 143, Lever 226, Personio 333), and open roles with `recency` (`app.jobs.score.recency`) above 0.5 fell from all 15,558 of them to 3,089 (19.9%) once it read `posted_at` instead. `upsert_role` keeps the earliest `posted_at` a role has ever had: an edit on the board can only move its own date later, never earlier, so it can only move `posted_at` earlier too, when the board finally reports the true one. `TRUSTED_POSTED_HOSTS` is gone: each reader declares `Fields(posted="trusted")`, read through `app/models.py`, and there are three search providers, Platsbanken's `publication_date` trusted beside the two named.
|
||||
**Postings are persisted.** Storing every posting gives `first_seen`, when the app first stored the role, and `closed_at`; a role listed again is open again.
|
||||
|
||||
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with status in `saved, applied, screen, technical, onsite, offer, rejected, withdrawn`, `next_action`, `next_date`, and an append-only `application_events` written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.
|
||||
**`posted_at` is the earlier of `first_seen` and a trusted host's date, and only moves earlier** (ADR Fetch prep and score 5.1.1). Without it a newly added board's backlog counted as freshly posted for two weeks: on a copy of the live data two thirds of board roles were posted more than 30 days before the app first saw them. Greenhouse's `updated_at` resets on every edit, so its reader reads `first_published`. A date in the future or before 2015 is trusted from no host. `upsert_role` keeps the earliest `posted_at` a role has had, since an edit on the board can only move its own date later. `first_seen` stays when the app found the role.
|
||||
|
||||
**`subscriptions` names its labels `company` and `board`.** newsdesk's `outlet` and `section` are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.
|
||||
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with its statuses, `next_action` and `next_date`, and an append-only `application_events` written on every status entered, which is what days to first screen and conversion by source are computed from.
|
||||
|
||||
## Ingestion
|
||||
|
||||
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** The `ats` plugin stamps each board it parses without raising with the run's marker, the ids it saw and whether they are the whole board, and the pull's settle closes a board only when its stamp matches the marker its own batches carried and the read was complete. So a batch that raises part way still closes the boards it finished, and a board no batch reached stays open instead of being closed on an empty id list.
|
||||
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** Each plugin stamps a board it parsed with the run's marker, the ids it saw and whether the read was complete, and the settle closes a board only when its stamp carries the marker of its own batches and the read was complete. A batch that raises part way still closes the boards it finished, and a board no batch reached stays open instead of being closed on an empty id list. A failing board's record is plugin state for the same reason: the plugin sees only `self.state`.
|
||||
|
||||
**One pull runs at a time, claimed in the database.** The schedule, the Run button and `app job pull`, which drains a queue of its own in another process, could each start a pull while one was fetching; the second overwrote the stored run marker, so the first run's boards failed their stamp check and stayed open, or a settle closed on the other run's id lists, and with extract on two cascades could send the same roles to the model twice. `claim_run` sets `pull:run` to a new marker in one conditional statement, only while it equals `pull:settled` or is older than an hour, so of two starters in two processes one wins; the settle writes `pull:settled` when it ends. A refused `pull` job queues no stage and says "a pull is already running" in its result and a notice, and the CLI prints the same and exits 1. The marker travels in every batch's payload, the plugins stamp with the marker they were handed, and the settle reads its marker off its own batches' payloads, which a batch that raised still has. The hour is the escape for a run that died: a killed process or a killed cascade never settles, and blocking every pull until someone clears a key would be worse than one duplicated fetch after an hour. The guard lives in `app` because `pipeline`'s queue dedups a job against a live job of its kind, not a cascade against a live cascade, and `solo=True` on the `pull` job would not help, since that job ends in milliseconds once it has asked for the chain.
|
||||
**One pull runs at a time, claimed in the database.** The schedule, the Run button and `app job pull` in another process could each start a pull while one was fetching; the second overwrote the run marker, so the first run's boards failed their stamp check or a settle closed on another run's id lists, and with extract on two cascades could send the same roles to the model twice. `claim_run` sets `pull:run` in one conditional statement, only while it equals `pull:settled` or is older than an hour. The hour is the escape for a run that died: blocking every pull until someone clears a key would be worse than one duplicated fetch. The guard lives in `app` because `pipeline`'s queue dedups a job against a live job of its kind, not a cascade against a live cascade, and `solo=True` on `pull` would not help, since that job ends in milliseconds once it has asked for the chain.
|
||||
|
||||
**Boards follow companies; searches find them.** `ats` only reads companies someone listed, so it cannot surface a role at a company nobody knew of, and the seed list came from a US-centred project: of 29,000 open roles, 31 were in Denmark. The `search` plugin runs saved queries on boards that span companies, as `search` subscriptions whose `params` are the query, and pulls them in the same task as the boards.
|
||||
**Boards follow companies; searches find them.** `ats` reads only companies someone listed, so it cannot surface a role at a company nobody knew of, and the first seed came from a US-centred project with almost no Danish roles. The `search` plugin runs saved queries on sites that span companies, as `search` subscriptions whose `params` are the query, in the same pull as the boards.
|
||||
|
||||
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list. A re-run attaches the board an entry names to a row that never had one, so a seed update that gains boards reaches an existing database; it never replaces a board, including a detached one, and never touches affinity, notes or status. Its boards go through `set_board` without the fetch, since they were read when the seed was written and `app companies check` finds the ones that died since. The count of 175 is out of date: the seed has 286 boards, as the paragraph on the seed's gap below says.
|
||||
**The seed is a list of companies, not of boards.** A company whose careers site the app cannot read yet stays in the list with the host its site runs on, so "cannot read yet" is a filter rather than a company never written down, and the next reader to write is whichever host that filter shows most. A re-run attaches the board an entry names to a row that never had one, so a seed update reaches an existing database; it never replaces a board, a detached one included, and never touches affinity, notes or status. Seed boards are attached without the check request (`set_board(check=False)`), since they were read when the seed was written.
|
||||
|
||||
**`discover` reads the site before it guesses.** A careers page that links its board is the authority, so the home page, its own careers links, then the usual paths are read for a link into a known host, and a page's own "open positions" link one level down. Only when no page names a host are Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio probed with the company's slug, because a board drawn by a script leaves no link. A probe hit on a common name can be another tenant (`recruitee:google` exists, and is not Google), which is why the seed's boards were fetched and read before they landed, and why `discover --add` reports the roles it found. For the same reason a page that names only an unreadable host ends discovery with that host: probing anyway, as the first version did, could replace a known Workday site with a stranger's Greenhouse board.
|
||||
**`discover` reads the site before it guesses.** A careers page that links its board is the authority, so the home page, its careers links and the usual paths are read for a link into a known host. Only when no page names a host are the hosts with a cheap per-slug endpoint probed with the company's slug, because a board drawn by a script leaves no link. A probe hit on a common name can be another tenant (`recruitee:google` is not Google), so a page that names only an unreadable host ends discovery with that host: probing anyway could replace a known Workday site with a stranger's Greenhouse board.
|
||||
|
||||
**A company row carries `ats`, `ats_slug`, `country` and `city`.** `ats` is what discovery found whether or not the app reads it; a company with an `ats` and no board subscription is one it cannot read yet. `ats_slug` is the name that host knows the company by (the `acme` of `acme.teamtailor.com`), so the parser written later has its targets without a second discovery run; with a board it is the board's slug. `country` and `city` come from the seed for now and are the first of the company facts the feed will filter by; funding, size and industry come from a source that has them, not from a column filled by hand.
|
||||
**Discovery that finds nothing readable does not detach a board.** A board that answers is stronger evidence than a page that names no board; `set-board` or Detach is the explicit step.
|
||||
|
||||
**An employer known only from a search gets a company row when it is first judged, and the Companies list derives that it is one.** Jobindex and Adzuna roles carry a `company_key` with no row in `companies` (67 keys with open roles on the live data of 2026-09-29), so they could not be hidden or liked. `Store.set_affinity` now creates a row for a key that has roles, named by the name its roles give most often, with nothing else set, and the reader draws the control for any role with a `company_key`. Such a row has no board, `ats` or `careers_url`, which the Companies list read as "board not found" and put under "Not readable yet". The rule: a company with no `ats` subscription (not even a detached one) and no `ats`, `careers_url` or `domain` is from job boards (`from_search` on `/api/companies`), labelled so and left out of that filter. Derived rather than a column, because every way of following a company writes one of those (the seed and discover a domain, Follow and `set-board` a board, which detaching hides but keeps), and a search-only company that later gets a domain or a board is followed from then on without a flag to flip.
|
||||
**A company row carries `ats` and `ats_slug`.** `ats` is what discovery found whether or not the app reads it, so a company with an `ats` and no board is one the app cannot read yet. `ats_slug` is the name that host knows the company by, so a reader written later has its targets without a second discovery run.
|
||||
|
||||
**Jobindex and Adzuna first, not Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API. Jobnet was left out at first because its search answered 401; see the next paragraph.
|
||||
**An employer known only from a search gets a company row when it is first judged, and that it came from a search is derived.** Search roles carry a `company_key` with no row, so they could not be hidden or liked; `Store.set_affinity` creates a row for such a key with only its name. A company with no `ats` subscription (not even a detached one) and no `ats`, `careers_url`, `domain` or `hand_reason` is from job boards (`from_search`). Derived rather than a column, because every way of following a company writes one of those, so a search-only company that later gets a domain or a board is followed from then on without a flag to flip.
|
||||
|
||||
**freehire, Jobnet and Akademikernes Jobbank are the fourth to sixth search providers (2026-10-03).** They came from comparing roledesk with github.com/MadsLorentzen/ai-job-search, which reads them. freehire.me is a public JSON API over about 50 hosts (no key, 300 requests a minute, robots.txt invites API use) whose search carries the full text; most of its Danish rows are employers roledesk does not follow, and a copy of a board role folds into the board's by the dupes rule, which now treats every `readers.PROVIDERS` name as a search host. Its query is an OR of words over title, company, text and place, so a search is narrowed by country. Jobnet's `/bff/FindJob/Search` answers 401 without the header `x-csrf: 1` and 200 with it, no login; it has no robots.txt and carries the full text. Jobbank is an RSS feed per search with the text on each ad's page as JSON-LD, so it is the first search reader with `detail`: fetch text and the Fetch button read a search role's page through the board path (`plugins.ats.reader_of`), with the same bar, budgets and per-role state, and `Store.enrich_waiting` hands over search roles too. Its robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. On a copy of the live data, a first page of each gave 266 roles, of which 20 were copies of roles already read. Jobdanmark is left out: about a quarter of its ads are on Jobnet, freehire carries the rest, and its listing has no text. LinkedIn's guest endpoints are left out: robots.txt disallows `/` for every agent and the terms forbid automated access.
|
||||
**Jobindex ads are read from the feed, and a posting is fetched only on demand.** Following each item's link would count as a click in Jobindex's statistics and cost a request per role, so the pull never follows it, and discovering a company's board from a search result is left for later. `POST /api/roles/{uid}/text` (`app/jobs/posting.py`) follows the "Se jobbet" link for a role somebody opened, since one click for a posting somebody opened is fair, and keeps it as `roles.posting_url`, which the reader's link-out then prefers.
|
||||
|
||||
**Platsbanken is the third search provider, for Sweden.** The JobTech API (`jobsearch.api.jobtechdev.se/search`) needs no key, states a total, so completeness follows Adzuna's rule, and filters by the taxonomy id of a län (`platsbanken.REGIONS`; Skåne is `CaRE_1nn_cSU`). It refuses an offset above 2000, which the page limit never reaches. The ads carry no usable remote field, so `remote` is read off the headline. Free text ranks by all words rather than requiring them; a quoted phrase narrows it.
|
||||
**A search's read never closes by its list** (ADR Fetch prep and score 1.3). A search that read as many results as its provider stated used to close every role its query no longer returned, liked and saved ones included, although a result list is never a company's whole list. Search roles close by age alone.
|
||||
|
||||
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. An Adzuna search listed everything when it read as many distinct results as the `count` it states; a Jobindex search, whose feed states no count, when a page within its page limit came back with fewer than 20 items, counted before the parser drops any, so one malformed item does not pass for the end. Such a search closes like a board; the rest close once no search has listed them for 21 days. A Jobindex answer with no RSS channel raises instead of reading as an empty, complete search. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists. Jobindex has since been declared `complete="never"` (its feed states no total), so a Jobindex search closes its roles only by the 21 days and the short-page rule above no longer applies to it.
|
||||
**A read that stalls is incomplete, and a sweep's cursor may be a pair.** A reader that sees the host send a page again hands it over empty with a cursor, and the runner ends the read incomplete; a cursor of None there read as the last page, and on a host that states no total as a complete read that closed what the pages had not listed. With a stated total a page of ids already read is not a stall, since a resumed sweep starts on one when the list grew at the top. A cursor is kept across pulls when it survives JSON apart from a tuple coming back as a list, so Workday's and SuccessFactors' sweeps resume.
|
||||
|
||||
**Jobindex ads are read from the feed, not followed to the employer.** Each item links to Jobindex's own page, and its `/c?t=` redirect leads to the employer's site or ATS. Following it per role would count as a click in Jobindex's statistics and costs a request per role, so discovering a company's ATS board from a search result is left for later. The on-demand fetch (A search role's posting text is fetched on demand, below) does follow the "Se jobbet" link to the posting, for a role somebody opened; the pull still never does.
|
||||
**A host that cannot filter on remote is asked for the agents' world where it takes the world in one read** (ADR Fetch prep and score 1.2). Asked for "DK, SE" alone such a host never listed a remote role filed under Germany, and 18 boards fetched no new remote role outside those two countries. Phenom and the Workday tenants without a remote facet take a list of countries, so Europe is one request and the gate sorts the rest out. SmartRecruiters, SuccessFactors and Eightfold take one country a request, where Europe is the whole board (Bosch 49 pages, a SuccessFactors site 10 roles a page), so there the location read stays and the row says remote was not asked.
|
||||
|
||||
**Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch already in flight when the switch happens reopens them again, so the pull's settle closes every switched-off target once more.
|
||||
**A role whose text says remote while no field does is possibly remote, read in prep with no model** (ADR 1.1 and 1.5). A role outside the agent's locations whose remote is unstated could never "turn out to be remote": extract reads only the cut, and the cut needs the gate. 5,016 open roles were rejected that way, 805 of them saying "remote" in their text. A phrase table ("fully remote", "work from home", the Danish and Swedish words) less its denials ("no remote", "remote sensing") marks `remote_text`; the gate's remote member takes the mark as a match while `remote` is empty, and once extract has answered, prep composes `remote` from the model and the mark no longer counts. A false mark costs at most a cut slot.
|
||||
|
||||
**A destructive switch asks first; a reopening one queues a pull instead of waiting for the schedule.** Switching a company's board or a job board search off, replacing a company's board (the Follow form and the row's Set board/Change editor, both `set_board`) and detaching one each close open roles in a tap; the page asks first when there are roles to lose, naming the count, a replace also naming the old and the new board, the same as removing a search and deleting an agent already do. Switching a target back on, following a board, setting a board, tagging a company All roles and saving a search used to answer with nothing until the schedule's next pull, up to six hours later; each now queues `pull` right after, in the API route (`/api/sources/{id}/switch`, `/api/companies/{key}/board`, `/api/companies/{key}/all-roles`, `/api/boards`, `/api/searches`) rather than in the page's script, so a script or `app job pull` calling the same endpoint gets what a click gets. `pull`'s own refusal while one is already running (`claim_run`) is not surfaced as an error: the change is read by the run already in flight, or by the very next one either way.
|
||||
**Age is the floor under every board's closing** (ADR Fetch prep and score 1.3). An incomplete read (a ceiling, a page cap without a sweep) closed nothing ever, and a filtered read spared the roles acted on from its list and from age alike, so such roles stayed open for good. Now what no read has listed for 21 days closes under an incomplete read, and what a filtered read spares closes by age too; a role acted on is spared from the list close, which is guesswork, not from three weeks of absence. The alternative was to strike the sentence from the ADR, which would have left "may stay open for good" in the design.
|
||||
|
||||
**The Sources tab and the CLI share `app.jobs.sources`.** Adding a board or a search and switching one are one function each, called by `app companies`, `app searches` and `/api/sources*`, so the two cannot disagree about what "off" does. A search or board is run once before it is saved, and refused if it does not answer. A search that finds nothing is refused too, since that is usually a typo in the query; a board that lists nothing is kept, since a company between openings is still one to follow.
|
||||
**Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch in flight when the switch happens reopens them, so the settle closes every switched-off target once more.
|
||||
|
||||
**Every way a company gets a board is `app.jobs.sources.set_board`.** `app companies add|set-board|discover --add|seed`, `/api/boards` and `/api/companies/{key}/board` all call it, so attaching a board to an existing row, replacing one and creating a company are one code path; the seed importer moved out of `app/cli.py` for this. The company row and its subscription change in one transaction (`Store.set_company_board`), so a board another company already has leaves no company row behind. Its host and slug become the row's `ats` and `ats_slug`; other facts only fill empty columns, so a person's or the seed's values survive.
|
||||
**A destructive switch asks first; a reopening one queues a pull.** Switching a board or search off, replacing a board and detaching one each close roles in a tap, so the page asks first when there are roles to lose, naming the count. Switching a target on, following or setting a board, setting All locations or All roles on and saving a search each queue `pull` at once instead of waiting up to six hours for the schedule. The pull is queued in the API route, not in the page's script, so a script calling the endpoint gets what a click gets; `claim_run`'s refusal is not an error, since the run in flight or the next one reads the change.
|
||||
|
||||
**A company has at most one `ats` subscription, rewritten in place.** Replacing a board changes the row's target and closes the old board's open roles, as switching a board off does; detaching switches it off and hides it (`show` 0), which is how a removed search is kept. Deleting the row instead would leave a batch already in flight writing roles against a subscription id that no longer exists, and adding a second row would clash on the label, which is the company's name. A detached board is left out of the Companies and Sources lists, and the next `set_board` reuses the row.
|
||||
**The Sources tab and the CLI share `app.jobs.sources`.** Adding and switching a board or a search are one function each, called by the CLI and `/api/sources*`, so the two cannot disagree about what "off" does. A search or board is run once before it is saved and refused if it does not answer. A search that finds nothing is refused too, since that is usually a typo; a board that lists nothing is kept, since a company between openings is still one to follow.
|
||||
|
||||
**Discovery that finds nothing readable does not detach a board.** `discover --add` on a company that is read from a board and whose site now names only, say, Workday changes nothing and says so. A board that answers is stronger evidence than a page that names no board; `set-board` or Detach is the explicit step.
|
||||
**Every way a company gets a board is `app.jobs.sources.set_board`.** The CLI, the seed, `/api/boards` and `/api/companies/{key}/board` all call it, so attaching, replacing and creating a company are one code path. Row and subscription change in one transaction (`Store.set_company_board`), so a board another company already has leaves no company row behind. Other facts only fill empty columns, so a person's or the seed's values survive.
|
||||
|
||||
**SmartRecruiters' board list has no description, so its text is read per role by fetch text.** Its `/postings` endpoint gives title, location and department; the text is only on `/postings/{id}`, whose `jobAd.sections` hold `jobDescription`, `qualifications` and `additionalInformation` (joined) and `companyDescription` (left out, the same blurb on every role). Reading it for every role would multiply a board's requests by its roles, so the reader declares `text="detail"` and fetch text reads only the roles in the cut (2026-09-30; 209 open roles had no text before).
|
||||
**A company has at most one `ats` subscription, rewritten in place.** Replacing a board changes the row's target and closes the old board's roles; detaching switches it off and hides it (`show` 0). Deleting the row instead would leave a batch in flight writing roles against a subscription id that no longer exists, and a second row would clash on the label, which is the company's name. The next `set_board` reuses the row.
|
||||
|
||||
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing. The seven hosts became 22 modules, each with its own allow-list of host names; Greenhouse is still the only reader with a fallback feed.
|
||||
**A company listed inside its owner's board is read by hand.** No field separates its roles reliably from the owner's, so it is on the Watchlist with reason `group`.
|
||||
|
||||
**A board request is streamed, with a 30 s read timeout and a 60 s deadline.** `_get` counts the body as it arrives and refuses it at `MAX_BYTES`, where reading `response.content` first held the whole body in memory before the check. httpx's timeouts are per phase, so a board that keeps sending a byte now and then never timed out; the deadline covers the whole request, redirects and body included, and is checked between chunks. openings.go's 10 s read timeout failed `greenhouse:mongodb` and `greenhouse:feedzai` in a pull of 2026-09-28. Measured on 2026-09-29, three runs each of those two and the five Greenhouse boards with the most open roles, with and without `pay_transparency=true` (84 requests, every one a CloudFront miss): a body of up to 10.4 MB arrives in under a second once it starts, and the first byte waited up to 16.6 s, with the three slowest at 16.6, 14.4 and 10.9 s. The flag made no measurable difference; the slow first byte is the origin's, with or without it. A read may wait 30 s, about twice the worst seen, and the deadline is 60 s, which leaves a large board room on a slower link.
|
||||
**No reader is built on a key lifted from a site's script.** Such a key can change without notice and marks an endpoint that is not open; those companies are on the Watchlist with reason `key`.
|
||||
|
||||
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published role object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The role page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by fetch text for the roles in the cut as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id.
|
||||
**The body cap is 96 MiB** (ADR Plugins 3.4). openings.go's 8 MB cap was written against arbitrary URLs, while the readers fetch only their hosts' allow-lists, and Greenhouse boards with `?content=true` measured between 9 and 41 MB. Greenhouse retries once without the text, trading the description for the listing.
|
||||
|
||||
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse. Since then there are 22 host modules, not seven; a board over 100 pages is swept across pulls and closes by two finished sweeps (below), and `_paged` is `_runner.read_listing`.
|
||||
**A request is streamed, with a deadline over the whole request** (ADR Plugins 3.4). The body is counted as it arrives and refused at `MAX_BYTES`, where reading it first held the whole body in memory before the check. httpx's timeouts are per phase, so a host that keeps sending a byte now and then never timed out; the deadline covers redirects and body. On Greenhouse's largest boards a body of up to 10 MB arrived in under a second once it started, while the first byte waited up to 16.6 s, so a read may wait 30 s, about twice the worst seen, and the deadline is 60 s. openings.go's 10 s read timeout had failed two of those boards.
|
||||
|
||||
**Teamtailor is read from the career site's `jobs.rss`, and a slug with a dot is the site's own host.** The official API needs a key per company; the feed needs none and carries title, link, guid, `pubDate`, the description, `remoteStatus` and locations, department and role in the `tt:` namespace. `teamtailor:acme` reads `acme.teamtailor.com` and `teamtailor:careers.acme.com` that host, since most career sites run on the company's domain. `_allowed_host` allows exactly that host, redirects included, and a slug must be a DNS name whose last label is not a number; beyond that a target may name any host that resolves, the same trust as any board the owner adds. The feed states no total, so it is read by `per_page=50` and `offset` to an empty page, and a page with nothing new ends the read as incomplete. 23 of the 32 seed companies answered on their `careers_url` host on 2026-09-29; the rest list a marketing site there.
|
||||
**A paged read counts distinct ids of the raw results** (ADR Plugins 2.1). Counting raw results means a row the reader drops does not block closing, and counting distinct ids means a role repeated by a shifted offset does not fake completeness.
|
||||
|
||||
**A board that fails on every fetch for 21 days closes its roles.** A board whose fetch raised was only logged, so a company that moved ATS kept its roles open for good behind a 404. The `ats` plugin now records `ats:failing:<target>` in `state` with the time of the first failure in a row and the latest reason, and a fetch that answers removes it. Once the first failure is `STALE_DAYS` old, the same window a search role gets, the settle closes the board's roles; the next answer reopens what it lists through `upsert_role`, as for a board switched back on. The record is plugin state rather than a column because the plugin only sees `self.state`. `/api/companies` returns it as `failing_since` and `failing_reason`, the Companies list shows "not answering since", and `app companies check` prints it beside its live result.
|
||||
**Teamtailor is read from the career site's feed, and a slug with a dot is the site's own host.** The official API needs a key per company; the feed needs none. Most career sites run on the company's domain, so `teamtailor:careers.acme.com` reads that host. A target may name any host that resolves, the same trust as any board the owner adds, as long as it is a DNS name whose last label is not a number.
|
||||
|
||||
**A host is a module in `plugins/readers/`, and registers by existing.** `plugins/ats.py` held eight hosts in six parallel tables and `app/jobs/discover.py` three more, so two people adding a host edited the same lines. Each host is now one file that states its name, label, URL patterns, slug check, allowed host names and either a feed and a parser or a pager; `ats.py` and `discover.py` build their tables from the package, ordered by the module's `RANK` so the select and the probes kept their order. The package sits beside `ats.py` and is not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module. The contract is the docstring of `plugins/readers/__init__.py`, and a module that breaks it fails the import by name.
|
||||
**A host is a module in `plugins/readers/`, beside `ats.py`.** `plugins/ats.py` held eight hosts in six parallel tables and discovery three more, so two people adding a host edited the same lines. The package sits beside `ats.py` and is not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module.
|
||||
|
||||
**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
|
||||
**A host is handed the fetch.** A reader receives `get`, which `ats.py` binds to its `_get` at call time, so the allow-list, the redirect rule, the byte cap and the deadline stay in one place and a test that patches `ats._get` reaches every host. The fetch refuses an IP address, a local host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
|
||||
|
||||
**Per-role text is read by fetch text, for the roles in the cut.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. The pull reads listings only. After scoring, the `fetch_text` stage (called `enrich` until 2026-10-04, renamed because it reads data and calls no model) reads the role pages of roles without text that are in the cut of an enabled agent, 200 per board and 1,000 per run, and only the roles it filled are embedded and scored again. It follows every pull, a saved agent, an affinity change and an All countries or All roles switch, not a vote. A read that brought no text is tried again after `retry_days` (7). `detail` may fill `DETAIL_FIELDS` only, the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing. Completeness is decided by the listing alone. The pull used to read 200 pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered. Eightfold, SmartRecruiters and Workable list their roles without text too and are read the same way.
|
||||
**One runner calls every reader, and acts on what each declares.** Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `plugins/readers/_runner.py` owns the guarded fetch, paging, series, completeness, the fallback feed, 429s and failures; `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` continues from its cursor on the next pull without reading page 1 again, so a new role on page 1 waits one sweep; the owner accepted that over reading the head every pull.
|
||||
|
||||
**robots.txt is decided when a reader is built, not checked at run time.** The builder reads the host's robots.txt, the owner decides per reader whether to read it anyway, and `READS.robots` records the decision and the date. The runtime matcher went with the readers rework. Jobindex is read in full although its robots.txt disallows `sort=`, `page=` and `geoareaid=`; Adzuna's API host disallows `/` for every agent; the owner decided to read its documented API with the account's key anyway, since the rule is aimed at crawlers. It needs a key in the config and has not run live yet. HR Manager's portal disallows everything but the application form; no attached company uses it. The companies once kept off the automatic route by robots.txt (Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy, Umbraco) stay on the Watchlist with reason `robots` until each is decided. The SuccessFactors reader uses `/search/` and `/job/` only. Since then `hrmanager:teknologisk` (the Danish Technological Institute) has been attached and has 7 open roles, so an attached company does use HR Manager; and Novonesis, Darktrace and Criteo are Workday boards, no longer on the Watchlist.
|
||||
**Fetch text reads only the roles in the cut, after scoring** (ADR Fetch prep and score 3). The pull used to read role pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered. The stage was called `enrich` until 2026-10-04 and was renamed because it reads data and calls no model.
|
||||
|
||||
**Every source is a reader module that declares what it can do.** The 22 hosts and the 3 search providers are modules in `plugins/readers/`, each with `READS = Reads(...)`: board or search, whole or paged listing, where the text is (listing, excerpt, detail, none), how completeness is known (stated, end, whole, never), page size and the host's ceiling, the filters it takes per dimension (none, one value a request, many, host ids through `facets`), whether it states one place, several, or one posting per place, which fields it states and whether its date is trusted, a key, a provisional title, whether an unknown slug answers an empty board, and the robots decision. Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `problems()` checks the declaration against the functions at import, and `tests/test_readers_contract.py` runs every module against fixtures, so a new reader that drops a place or a teaser fails there. The contract is the docstring of `plugins/readers/__init__.py`; the types are `_contract.py`.
|
||||
**robots.txt is decided when a reader is built** (ADR Plugins 3.7). Adzuna's API host disallows `/` for every agent; the owner reads its documented API with the account's key anyway, since the rule is aimed at crawlers.
|
||||
|
||||
**One runner calls every reader.** `plugins/readers/_runner.py` owns the guarded fetch, paging, one request series per filter value unioned by id, completeness, the fallback feed, 429s and failures. `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` saves its cursor and continues on the next pull, without reading page 1 again, so a new role on page 1 waits one sweep; the owner accepted that over reading the head every pull. A host ceiling (Workday 2,000) is not swept; a filter has to get under it.
|
||||
**The company switches are global, not per agent, and separate from affinity** (ADR Fetch prep and score 1.1, overrides). They decide what is fetched, and one fetch serves every agent. All locations exists because, for a company the owner cares about, a role in Freiburg may be open to remote or worth asking about for Denmark, while Microsoft Germany is not wanted. The wide reach of an All locations company includes the agents' own reach, so the switch never reads less than the board would without it. A board's own `params` override both the agents' reach and All locations (`app/jobs/reach.py`). A board under 300 roles is read whole, since filtering it saves little and risks closing roles.
|
||||
|
||||
**A board swept across pulls closes a role only when two finished sweeps in a row missed it.** Roles closing above the saved offset between pulls shift later roles back past it, so a finished sweep misses roles that are still open, and the stated-total check can still pass because `seen` holds ids that closed meanwhile. A complete read that finished a sweep is stamped `ats:swept:<target>`; the settle closes the open roles missing now that are also in `ats:missing:<target>`, the ids the previous finished sweep missed, and keeps the rest there. A complete read that was not swept closes as before and clears the record. A role the host closes takes up to two sweeps to close.
|
||||
**A board's size is set when it is added** (ADR Fetch prep and score 1.2). A new board without a size used to be read whole on its first pull, worldwide, until the page cap; so `set_board` sizes a new board, and one without a size is sized in its batch before its reach is chosen.
|
||||
|
||||
**The agents set the reach of a pull.** `app/jobs/reach.py` computes it once per pull: the union of the enabled agents' `any` gates on country, town (as its country) and remote, or everything when one agent has none. Fetching has two filters, countries and role types, and a company has an override for each. A company set to **All countries** (`companies.all_countries`, migration 24) is read with the wide reach: the union of the enabled agents' `require` reach, Europe today, and the union of the `any` gates, so the override never reads less than the board would without it. An agent with no `require` gate on place makes the wide reach everywhere. For a company the owner cares about, a role in Freiburg may be open to remote, or worth asking about for Denmark, while Microsoft Germany is not wanted. The overrides are separate from favourite and liked, and global rather than per agent, because they decide what is fetched and fetching is shared. They matter only for a board of 300 roles or more on a host that filters, so a small company is affected once it grows past that. The Companies list states both halves per board, e.g. "follows agents: DK, SE, remote · all roles" or "all countries (Europe) · follows agents' terms: engineer, product, data", or why the board is read whole. Favourite (affinity 2) affects scoring only, where it skips the `any` gates; it used to widen the fetch to its agent's `require` reach, and migration 21 tagged every company that was a favourite of any agent so nothing read before is lost. That tag, first named All roles in `companies.all_roles`, also replaced a board's `{"reach": "all"}` param, which read everywhere: migration 21 tagged those companies and logged each such board, since no everywhere level is left. Migration 24 renamed the tag to All countries and gave its companies (Mistral AI and OpenAI on live) All roles as well, since the owner wanted their every role. A board's `params` override the agents' reach and All countries. Boards under 300 roles, and hosts with a one-country filter over more than 8 countries, are read whole. A board's size (`ats:size:<target>`, with `ats:sizechecked:<target>`) is set once, not by every read: `set_board` sizes a new board with `ats.board_size` (a stated total from one unfiltered page 1, else a whole read), a board with no size is sized in its batch before its reach is chosen, and Recheck on the Companies row, `app companies size <key>` and `--missing` size it again. Before, a new board had no size and was read whole on its first pull: Infineon read 999 roles worldwide and cut at page 100. A board whose reader cannot filter is sized by its whole read instead of a second request. On 2026-09-30 `--missing` sized 38 boards the old open-role fallback had sized, among them Bosch 4,830, Verisure 3,447, Microsoft 2,345 and NVIDIA at Workday's 2,000. For a Workday or Oracle site, which filters by its own location ids, the Companies line names a country the site does not list and a missing remote option ("follows agents: DK, remote · SE not on this site"), and "nothing asked" when it lists none, where it used to name what was meant to be asked. SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday and Oracle filter at the source; countries and remote go as separate series, because the hosts intersect two facets. A complete filtered read closes roles outside the filter, except roles with a vote or an application; a filtered read, or one from a host whose unknown slug answers empty, that lists under 10% of the last read with the same filter counts as failed.
|
||||
**A role has a list of places, resolved against a gazetteer.** Readers kept one place of several, joined them, or the host posted a copy per place. A place text resolves by its own words (`Gazetteer.resolve`), never by the role's stated `country`, which names one country of a multi-place line: `San Francisco; London, UK` with `US` is in Europe through London. The gates and the Place filter read every place of a kept role, its folded copies' included, so OpenAI's San Francisco role with a Dublin copy meets a Europe gate.
|
||||
|
||||
**A role has a list of places, resolved against a gazetteer.** Readers kept one place of several, joined them, or the host posted a copy per place. A reader now returns every place and never multiplies a row. `places` (city, region, country, continent, with `place_includes` for Øresund across two countries) and `place_aliases` are seeded from the dictionaries that were in code; `role_places` links each place of a role, and an unknown text is listed by `unresolved_places` until an alias links it. The Place filter and the gates join over the tree, which halved their time and dropped the `EU` fallback. Copies of one posting per place, same host, company and title and texts at least 30% alike, fold into one role that keeps each copy's place and link; the rule applies to every host, since the 5,024 such roles came mostly from hosts that do not declare copies. A place text resolves by its own words (`Gazetteer.resolve`), never by the role's stated `country`, which names one country of a multi-place line: `San Francisco; London, UK` with `US` is in Europe through London. The gates and the Place filter read every place of a kept role, a folded copy's included, so OpenAI's San Francisco role with a Dublin copy meets a Europe gate although its `place` and `country` are San Francisco's; its row read "San Francisco +1" until a row led with the place nearest the agent (below). On 2026-10-03 this kept 45 open roles in agent 2's feed that its own places alone would gate.
|
||||
**A role in several cities leads with the one nearest the viewing agent** (`places.Nearness`). Mistral's Forward Deployed role, kept as its Montréal posting with a Paris copy, read "Montréal +3" and opened the Montréal page, while for the owner it is a Paris role. The gazetteer holds no coordinates, so within a tier the agent's home (the first value of the first `any` gate on country or place) stands in for distance. Only the row and the reader follow the order; the gates and the copies rule do not read it.
|
||||
|
||||
**A role in several cities leads with the one nearest the viewing agent** (`places.Nearness`). Mistral's Forward Deployed role, kept as its Montréal posting with Abu Dhabi, New York and Paris copies, read "Montréal +3" and opened the Montréal page, while for the owner it is a Paris role. A role's places, its copies' included, are ordered in three tiers read off the agent's gates: a place that meets the `any` group (under one of its `country` or `place` values, or a remote part whose area is open to it through `remote_from`, or that meets the scope with a plain `remote` any gate), then a place inside the `require` scope on country, then the rest, a remote part with no stated area among them. The gazetteer holds no coordinates, so within a tier the home stands in for distance: the first value of the first `any` gate on country or place. A place in one of the group's own towns inside the home leads, then one elsewhere in the home, then the host's order; for agent 2 Copenhagen leads "Denmark", Billund and Stockholm. `GET /api/roles` and `GET /api/roles/{uid}` compute it for the agent the feed is viewed as and only for the roles served: the row carries `place_texts`, `lead_place` and `lead_url` (the leading copy's link, '' when the role's own place leads), and `place` is the lead and "+N" as before. The reader's Places list follows the order, and its posting links and the `o` key open `lead_url`. Without an agent the host's order stands. The gates and the dupes rule do not read the order. On the live copy of 2026-10-04 agent 2 sees the Mistral role as "Paris +3" linking the Paris copy, OpenAI's Applied AI Architect as "Dublin, Ireland +1", and 277 of the 279 kept roles with Copenhagen among several places lead with it; the other two lead with a joined line that names København.
|
||||
**A remote role has a remote area: the places a remote hire may live in** (`role_remote`, `app/prep/area.py`). The Place field listed "USA (Remote)", "Remote US" and "United States (Remote)" as three towns, and no filter said which remote roles someone in Denmark could take. "Remote" alone names no place, so such a role is remote with no stated area. Anywhere is a standalone place with nothing under it, so the Place filter never reaches every role through it.
|
||||
|
||||
**A remote role has a remote area: the places a remote hire may live in** (migration 27, `role_remote`, `app/prep/area.py`). On 2026-09-30 the Place field listed "USA (Remote)", "Remote US" and "United States (Remote)" as three towns, and no filter said which of 3,177 remote roles someone in Denmark could take. Those lines came from a few whole-read Greenhouse and Ashby boards (Grafana Labs, Cloudflare, Monzo, Talkdesk), not from an ingest filter. Danish employers barely post remote roles (11 of 1,395 open Danish roles). The remote roles open to Denmark are the roughly 300 scoped to Europe or EMEA and the 210 scoped to anywhere. The area is the first of these that names a place: the model's `remote_where`, the parts of the place texts that say remote ("Germany (Remote)", "Home Based - EMEA"; "Worldwide" or "Global" in such a part is Anywhere), and the role's resolved places lifted to their countries. "Remote" alone names none, so such a role is remote with no stated area. Anywhere is a standalone place with no aliases and nothing under it, so the Place filter and `under` never reach every role through it, and it never counts as a place for a country gate's empty check. Remote lines are no longer suggested as towns.
|
||||
**A remote chip matches by overlap, and Anywhere is in every one.** `remote:country:DK` ("Remote · Denmark") finds a remote role whose area is Denmark or in it, a place Denmark lies in (Europe, Øresund), or Anywhere: the roles a person living in Denmark can take. Danish employers barely post remote roles, so the remote roles open to Denmark are mostly those scoped to Europe or to anywhere, and the owner wanted Remote · Denmark to hold them rather than have a separate "Remote · anywhere" chip; `remote:anywhere` is a valid token that is never suggested. A remote role with no stated area matches no `remote:<place>` chip.
|
||||
|
||||
**A remote chip matches by overlap, and Anywhere is in every one.** `remote:country:DK` ("Remote · Denmark") finds a remote role whose area is Denmark or in it, a place Denmark lies in (Europe, Øresund), or Anywhere: the roles a person living in Denmark can take. `remote:country:EU` finds every European remote role, "Germany (Remote)" included. The owner wanted Remote · Denmark to hold the Europe-wide and worldwide roles rather than have a separate "Remote · anywhere" chip, so `remote:anywhere` is a valid token that is never suggested. A remote role with no stated area matches no `remote:<place>` chip. On the live data of 2026-09-30, Remote · Denmark found 499 roles, Sweden 531, Germany 760 and Europe 1,395, and 122 remote roles had no stated area.
|
||||
**Plain Remote is every remote role within the world gate.** The token `remote:any` ("Remote") and the plain `remote` line of the location gate both mean every remote role inside the agent's `require` gates on country. Such a gate reads a remote role that has an area through the area alone, so "Home Based - Worldwide", "Germany (Remote)" and "Remote - Europe" meet Europe and "Remote US" does not. The form's Remote line offers no, anywhere within the world gate, or only when open to someone in a list of places (`remote_from`, matched as the chips are). Remote suggestions are kept to the asking agent's world gate, so an agent kept to Europe is not offered Remote · United States.
|
||||
|
||||
**An agent's scope is the first line of the Where block, and plain Remote is every remote role within it.** "Only roles in: Europe / these countries / everywhere" reads and writes the agent's `require` gate on country, which was the implicit scope before, and which a company set to All countries is already read within. The token `remote:any` ("Remote", suggested first for "remote") and the agent's `remote: [remote]` any gate both mean every remote role in that scope: a `require` gate on country reads a remote role that has an area through the area alone, so "Home Based - Worldwide", "Germany (Remote)" and "Remote - Europe" meet Europe, and "Remote US" does not. `any` and `prefer` gates on country keep the place reading. The Where block's Remote line is a choice of no, anywhere within the scope (the plain gate, so the live agent loads and saves unchanged), or only when open to someone in a list of countries and regions (`remote_from`, which matches as the chips do). The remote suggestions are kept to the asking agent's scope, so an agent kept to Europe is not offered Remote · United States. This supersedes the "remote within Europe or anywhere" select of the Where block below.
|
||||
**For remote, the model wins, as it does for seniority** (ADR Fetch prep and score 4.1). Black Forest Labs' Freiburg roles are `OnSite` on Ashby while the text offers remote work with an office week a month, and the board's word kept them onsite. The prompt says remote when the hire works from home most of the time, even with a regular office week, and hybrid when there are set office days every week. "Remote sensing", "remote control" and "remote monitoring" in a title are not remote (`app/prep/words.py`).
|
||||
|
||||
**For remote, the model wins, as it does for seniority.** `remote` is `remote_model`, else the host's word, else the place line (prep `VERSION` 3). Black Forest Labs' Freiburg roles are `OnSite` on Ashby while the text offers remote work with an office week a month, and "the board wins" kept them onsite. The prompt says remote when the hire works from home most of the time, even with a regular office week, and hybrid when there are set office days each week. Extract also answers `remote_where`, the names of where a remote hire may live, and a role waits while `remote_where_model` is NULL, so the roles read before are read once more within the agents' existing limits. Migration 27 moved a board role's stored `remote` into `remote_host` where that was empty, since composing `remote` outright would have wiped 1,280 values that an older reader had mapped. "Remote sensing", "remote control" and "remote monitoring" in a title are not remote.
|
||||
**A company has aliases.** `company_aliases` maps a folded name to a company, so "Orsted" and "Ørsted" are one company without changing `company_key`, the primary key of `companies`. The enrichment columns (sector, stage, size band, founded, last funding) stay empty until a reader states them rather than being filled by hand.
|
||||
|
||||
**A company has aliases.** `company_aliases` maps a folded name (marks stripped, ø to o, legal form dropped) to a company, so "Orsted" and "Ørsted" are one company without changing `company_key`. `merge_companies` moves roles, subscriptions and ratings. Empty enrichment columns (sector, stage, size band, founded, last funding, source and date) wait for the startupmap.one and The Hub readers.
|
||||
**A description keeps its shape as marked lines.** `_shared.html_text` turns HTML into a line per paragraph, `• ` before a list item and `## ` before a heading. Plain text with marks, not sanitised HTML, is stored, because summarise, extract and embed read the same column, and the reader escapes each line before drawing it. Titles and teasers stay one line.
|
||||
|
||||
**A list teaser is kept.** A short description in a listing (Phenom, Oracle, Jobylon, Jobindex, Adzuna) is `roles.teaser`, shown and embedded until the full text arrives, when the vector is dropped and the text replaces it.
|
||||
**Readers scrape; `prep` decides** (ADR Plugins 1 and 2.5). Mapping lived in 25 readers, each spelling its tables its own way, so changing what "Experienced" means touched a plugin. `upsert_role` stores the host's words in `*_host` columns, and when a host table changes, its fingerprint makes the next prep map every open role again from the stored words, with no pull. A host's `remote: False` stays unknown rather than onsite, because SmartRecruiters, Ashby, Recruitee and Workable send it for hybrid roles too.
|
||||
|
||||
**A description keeps its shape as marked lines.** `_shared.html_text` turns a host's HTML into a line per paragraph, `• ` before a list item and `## ` before a heading (an `<h1>`..`<h6>`, or a paragraph of at most 80 characters set wholly in bold). `strip_html` had flattened a posting into one line, so sections and bullets ran together in the reader. Plain text with marks, not sanitised HTML, is stored because summarise, extract and embed read the same column, and the reader pane escapes each line before drawing it as `<h3>`, `<li>` or `<p>`; a row's excerpt drops the marks. Titles and teasers stay one line through `strip_html`. List readers overwrite the description on every pull, so their roles take the new shape on the next pull; a detail reader reads a role's page once, so its roles read before 2026-09-30 keep the flat text.
|
||||
**Unmapped places and words are fixed in code, from a list on Sources.** Settings › Sources › Unmapped lists the place texts no place resolves and the host words no table maps, with no editor: the owner tells Claude what a text is, and the `unmapped` skill says which dictionary or host table takes it. Code rather than `add_place_alias` rows, because a fix then reaches a new database and is reviewed like any other change. The prep fingerprint covers `places.seed()`, so the next prep adds the new spellings and maps every open role again.
|
||||
|
||||
**Readers scrape; `prep` decides.** A reader hands on the host's own words (`seniority`, `employment`, `remote`, and `country` only when the host states one per role) and does light cleanup only: text from HTML, dates, the places of a role as a list, pay figures. Mapping lived in 25 readers, each spelling its tables its own way, so changing what "Experienced" means touched a plugin. The `prep` stage (`app/prep/`) maps centrally after every pull and every fetch text run. Seniority, employment and remote go through the host's table in `app/prep/hosts/<reader>.py` (whole words, or `MATCH` fragments by substring or token), then a shared word map. Country comes from the resolved places (several European countries give `EU`), and the host's country is only a fallback. Places and company aliases are resolved, and copies and duplicates are folded over the whole database. `upsert_role` stores the host words (`*_host` columns, migration 22) and marks the role `prep_due`. When a table changes, its fingerprint makes the next prep map every open role again from the stored words, with no pull. A word no table knows is listed in `unmapped` (`app prep unmapped`, the Sources page). A host's `remote: False` stays unknown rather than onsite, because SmartRecruiters, Ashby, Recruitee and Workable send it for hybrid roles too.
|
||||
**Seniority has three sources, and the model wins** (ADR Fetch prep and score 4.1). `seniority_stated` is the host's word through its table, `seniority_title` is `seniority.from_title` and `seniority_model` is extract's; `roles.seniority` is the first of model, stated, title. The model gets the other two as hints it may overrule, since host vocabularies differ and most roles state no level. `app seniority` shows per host how often the stated level agrees with the model.
|
||||
|
||||
**Unmapped places and words are fixed in code, from a list on Sources.** Settings › Sources › Unmapped lists the place texts of open roles no place resolves (`app prep unlinked`, with the countries and companies of their roles) and the host words no table maps. It has no editor: the owner tells Claude what a text is, and the `unmapped` skill in `.claude/skills` says which dictionary or host table takes it. Code rather than `add_place_alias` rows, because a fix then reaches a new database and is reviewed like any other change. The prep fingerprint covers `places.seed()`, so the next prep adds the new places and spellings to the database and maps every open role again. Copy the list puts both lists on the clipboard, or selects them on the page where there is no clipboard.
|
||||
**`pull` is a job that starts the cascade, on a default schedule of every 6 h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs.
|
||||
|
||||
**Fetch terms narrow what every big board sends.** An agent's `fetch_terms` (migration 23) are words a board of 300 or more roles must match, on a host that takes one query per request (SmartRecruiters, SuccessFactors, Workday, Phenom, Eightfold, Oracle), on the narrow and the wide reach alike (2026-10-04; until then only the wide reach of an All countries company took them). A company set to **All roles** (`companies.all_roles`, migration 24) takes no terms on any of its boards, and its roles skip the titles to leave out in scoring. The terms of a pull are the union over the enabled agents; one enabled agent without terms switches them off. The survey of 2026-09-30 found that every host's keyword matches the whole text, so a term cuts less than a title filter would ("engineer" left 1,713 of Bosch's 4,830). One series per term is unioned by id. Over 8 series a wide read drops the place filter and keeps the terms; a narrow read keeps the places, which the agent's fetch gate needs, and drops the terms. A term-filtered read is never the whole board: it writes no size, prunes no per-role state and closes roles by age only, which is accepted because ranking relies on recency, not on closing. A town in a gate is still fetched as its country.
|
||||
**Every whole read is guarded, not only filtered ones** (ADR Plugins 3.3). A Greenhouse or Lever body whose shape changed is HTTP 200 and zero rows, which the settle took for a complete read and closed every role on. `_runner.read` raises `Suspect` on any read that is not a sweep and lists under `SUSPECT_SHARE` of the same read's last count, once that count is at least `SUSPECT_MIN` (10) for a plain read and 1 for a filtered, narrowed or `empty="suspect"` read. A board that really shrank costs a Recheck, which resets the count (`keep_size`), or 21 days of the same answer.
|
||||
|
||||
**Seniority has three sources.** `seniority_stated` is the host's word mapped by that host's table in `app/prep/hosts/`, `seniority_title` is `seniority.from_title`, and `seniority_model` is `extract`'s verdict; `roles.seniority` is the first of model, stated, title. `extract` reads roles without a model value, every agent's first pick before anyone's second, with the other two as hints it may overrule, since host vocabularies differ and 57% of roles state nothing. `app seniority` shows per host how often the stated level agrees with the model.
|
||||
**A read that could not ask for remote keeps the remote roles it has.** On a host with no remote filter a read for "DK, SE, remote" goes out as "DK, SE" and counts as filtered, so its settle closed every remote role outside those countries. The listing now records `remote_unasked`, those remote roles close by age only, and the Companies line says "remote not asked". New remote roles on such a board are still not read; reading it whole instead is with the owner (`docs/open.md`).
|
||||
|
||||
**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key the page's JavaScript carries, which can change without notice and marks an endpoint that is not open. They are on the Watchlist with reason `key`.
|
||||
**A read that asks nothing is not a read.** When an `ids` dimension maps to no host id, the read asked nothing, and counting it as complete and empty would close the board; the listing is incomplete and the Companies line ends "not closing".
|
||||
|
||||
**A company has one board, and a company inside its owner's board is read by hand.** DFDS has eleven Teamtailor boards by country and is read from the Danish one. Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio and Native Instruments list their roles in Entrust's, Visa's, Delivery Hero's, NICE's, Scrive's, SAP's and inMusic's boards, where no field separates them reliably; they carry reason `group`.
|
||||
**Every target is guarded in its batch.** A row that failed to store used to fail the whole `pull_batch`: that board had neither a stamp nor a failing record, and the boards after it in the batch, the same ones every pull since batches are ordered, were never read, closed or aged. `run_batch` wraps each target, records the failure under the source's failing key and moves on; the failing key is removed only after the stamp.
|
||||
|
||||
**The seed's gap was mostly discovery, not parsers.** A survey of the 155 companies without a board (2026-09-29, every careers site opened) found 49 on a host the app already read: the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain or on Lever's EU instance (`lever:eu/<site>`), or the careers page refuses a plain request. 43 were attached by hand from the survey, 46 more with the fourteen hosts added that day, and 286 of 362 companies have a board. Discovery keeps a slug's case for a module with `KEEP_CASE`, since Workday's site names, Gem's, Deel's and Rippling's slugs are case-sensitive; SmartRecruiters and Ashby are not, checked live.
|
||||
|
||||
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs.
|
||||
|
||||
**Every whole read is guarded.** The 10% guard held only filtered, narrowed and `empty="suspect"` reads, so 206 of 309 boards, holding two thirds of the open board roles, could close every role on one answer that parsed to nothing: a Greenhouse or Lever body whose shape changed is HTTP 200 and zero rows, and the settle took it for a complete read. Now `_runner.read` raises `Suspect` on any read that is not a sweep and lists under `SUSPECT_SHARE` of what the same read listed last time, once that count is at least `SUSPECT_MIN` (10) for a plain read; filtered, narrowed and suspect reads keep their floor of one. The board is then failing with that reason, its roles stay open, and the Companies row says so. A board that really shrank costs a Recheck, which resets the count through `keep_size`, or `STALE_DAYS` of the same answer, after which the read is accepted and the count reset, since by then its roles have closed anyway.
|
||||
|
||||
**A read that could not ask remote keeps the remote roles it has.** `ask_for` keeps only the dimensions a reader declares, so on SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" went out as "DK, SE", the read counted as filtered, and the settle closed every remote role outside those countries: 72 on 2026-09-29 and 30 (Philips, Bosch, Coolblue, Delivery Hero), with the line saying "follows agents: DK, SE". Now the listing records `remote_unasked` (`ats:remote_unasked:<target>`), `_close_unlisted` and `_close_swept` spare roles whose `remote` is `remote`, those close when the board has not listed them for `STALE_DAYS`, and the line says "remote not asked on SmartRecruiters". New remote roles on such a board are still not read; reading it whole instead is the owner's call (`open.md`).
|
||||
|
||||
**A read that asks nothing is not a read.** When an `ids` dimension maps to no host id, `series` used to return nothing, the listing was complete and empty, and the settle closed the board while the line read "site lists none of DK, SE: nothing asked", a fact about the site. The listing is incomplete now, logged at warning, and the line ends "not closing".
|
||||
|
||||
**Every target is guarded in its batch, and the failing record goes after the stamp.** `ats._board` dropped `ats:failing` before the rows were stored, and a row that failed in `item_of` or `upsert_role` failed the whole `pull_batch` job: that board had neither a stamp nor a record, and the boards after it in the batch, the same ones every pull since batches are ordered, were not read, never closed and never aged. `run_batch` now wraps each target, records the failure under the source's failing key and moves on; a Kill still propagates. The failing key is removed after `stamp`.
|
||||
|
||||
**Searches share the runner's bookkeeping.** `search.py` logged a failure and moved on, wrote its own `search:ok/complete/ids`, and read through `read_listing` rather than `read`, so it had no guard: a Platsbanken answer with `hits` renamed read as complete and empty and closed the search at once. It now reads through `_runner.read` (no sweep), records `search:failing:<target>` through `record_failure`, stamps through `stamp`, and the search row shows "not answering since" as a board row does; `failing_boards` reads both prefixes and a search failing for `STALE_DAYS` closes its roles. A removed search forgets its record.
|
||||
|
||||
**The settle speaks.** Its counts went into the job result, which nothing renders: the settle of 2026-09-30 10:06 closed 12,588 roles with one target unanswered and one incomplete, and the Problems log stayed empty. `close_boards` now raises one `pull.settled` notice when something needs a look: every target unanswered (with its failing reason) or read incomplete (boards only, and not one mid-sweep, or the notice would fire every pull), every board that closed more than `CLOSE_NOTICE` (50) roles or more than half of at least `CLOSE_SHARE_MIN` (10) open ones, and the totals. Its result key `failed` is `unanswered`. `run_pull` and `_boards` release the run claim when building the chain raises, so a finder that fails no longer refuses every pull for an hour with "a pull is already running".
|
||||
|
||||
**Narrowed boards close by age.** A read by fetch terms closed nothing, so a role that left such a board stayed open for good. A complete narrowed read now closes the board's roles it did not list and that were last seen more than `STALE_DAYS` ago, the rule searches have; no agent has terms yet, so this is latent.
|
||||
|
||||
**Declarations are checked against fixtures.** Of the `READS` traits only `posted`, `filters`, `complete`, `ceiling`, `empty`, `key` and `scope` were read at run time; `pay`, `remote`, `country`, `seniority`, `employment`, `teaser`, `places` and `text` were checked by nothing, so Platsbanken declared `remote="none"` while its rows set it and HR Manager declared `places="one"` and returned none. `tests/test_readers_contract.py` now parses each reader's listing and detail fixtures and asserts every declared field, a missing fixture fails instead of skipping, `places` has `none`, a search module needs `PARAMS` and `clean_params` and its rows `company`, every prep table names a reader and every reader stating words has a table, and `robots` must carry a date and name the paths the reader requests: seven records named a page the reader never asked for. Every host's robots.txt was read again on 2026-09-30; `api.smartrecruiters.com` disallows `/` for every agent but LinkedInBot, which is with the owner. The paths for modules without `READS`, `robots_page` and `ROBOTS_ANY_CASE` are gone, and `discover._UNREADABLE` lost the ten hosts that have modules.
|
||||
**The settle says what needs a look.** Its counts went into the job result, which nothing renders, so a settle could close twelve thousand roles with the Problems log empty. `close_boards` raises one `pull.settled` notice naming every target unanswered or read incomplete (not one mid-sweep, or it would fire every pull) and every board that closed more than `CLOSE_NOTICE` (50) roles or more than half of at least `CLOSE_SHARE_MIN` (10) open ones.
|
||||
|
||||
## Ranking
|
||||
|
||||
**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the `any` group, `-1` mutes.
|
||||
**Scoring is per search agent.** A role is scored as `(uid, agent)`, because a role is only good for a given search. A gated role keeps its row with the reason, which Show gated on Roles lists.
|
||||
|
||||
**Three decisions per agent: fetch gate, analysis score, final score (2026-10-04).** The one blended score used to pick what the models read and order the feed, and the owner could not tell which rule did what. Now the fetch gate (region, place, role) is a yes or no applied at the host, in scoring, again after extract and in the feed; the analysis score picks the cut the models read; the final score, `(0.6 fit + 0.2 newness + 0.2 affinity + 0.2 taste) / sum`, orders the feed. Everything else, such as seniority, location and pay, is a filter. `docs/architecture.md` is the target design this follows.
|
||||
**Three decisions per agent: the fetch gate, the cut and the feed scores** (ADR Fetch prep and score). One blended score used to pick what the models read and order the feed, and the owner could not tell which rule did what. Newest first used to show roles the agent rejected, so a change of order changed which roles were shown; every order now hides them. Wishes such as hybrid are feed filters, not score terms (migration 32 dropped the `prefer` gates), so a role is never judged twice, by the score and by a filter.
|
||||
|
||||
**The feed shows every role the agent's fetch gate passes, in every order.** Newest first used to show roles the agent rejected, so a change of order changed which roles were shown. With an agent named, every order joins `scores` and hides gated roles unless Show gated is on.
|
||||
**Saved filters are per agent and stored on the server** (`saved_filters`). They are per agent because an agent's filters change with what it hunts, and on the server so they follow the owner across devices.
|
||||
|
||||
**Saved filters are per agent, stored on the server (`saved_filters`, migration 32).** A saved filter is the feed's query string under a name: applied with one click, deleted, never edited, since a change is a new one. They are per agent because an agent's filters change with what it hunts, and on the server so they follow the owner across devices.
|
||||
**An agent is the group: it holds several prompts, and prompt closeness is the best prompt's.** A hunt spans several kinds of role (AI and data leadership, customer-facing technical leadership, AI research), and one blended paragraph ranks a role by its distance to the blend. Each prompt in `agent_prompts` is embedded on its own and cached under the sha1 of model and text, so an edit re-embeds one prompt; `scores.prompt` names the winner. Votes, liked companies, gates, weights and limits stay per agent, so a vote counts for every prompt; a market watch is a second agent with its own. The embedder separates kinds of role but not seniority: an individual contributor's posting on the same subject ranks beside a head's, so seniority comes from extract, the Seniority filter or votes.
|
||||
|
||||
**An agent is the group: it holds several prompts, and `fit` is the best prompt's.** A hunt spans several kinds of role (AI and data leadership, customer-facing technical leadership, AI research), and one blended paragraph ranks a role by its distance to the blend. Each prompt is a named text in `agent_prompts` (migration 14, which moved every non-empty `agents.intent` into a prompt named after the agent and dropped the intent columns), embedded on its own and cached under sha1 of model and text, so an edit re-embeds one prompt. A role's fit is the maximum over the prompts of the calibrated cosine, the first prompt winning a tie, and `scores.prompt` names the winner. Votes, liked companies, gates, weights, limits and `taste` stay per agent, so a vote counts for every prompt; a market watch is a second agent with its own. No third level above the agent was added. `POST /api/agents` still takes `intent` when `prompts` is absent. On the live data of 2026-09-29 the embedder separates the three kinds of role but not seniority: an individual contributor's posting on the same subject ranks beside a head's, so seniority has to come from `extract` and a prefer gate, or from votes.
|
||||
**Closeness is a cosine mapped to a level by constants calibrated per embedding model: `clip((cos - lo) / (hi - lo), 0, 1)`.** `(cos + 1) / 2` spanned only about 0.87 to 0.95 on multilingual-e5-small, so a term weighted 0.2 outweighed all of prompt closeness. Min-max across a run's roles fixed the spread but made it a rank: the best role of a poor set got 1, a level's meaning drifted from run to run, and the cut's minimum compared an absolute number with a relative one. The constants are `CALIBRATION` in `app/config.py`, keyed by model name so a change of `embed.model` cannot carry another geometry's constants; `embed.calibration` in the config replaces a model's entry. A model with none falls back to min-max per run. Job like (`taste`) is mapped the same way with its own pair.
|
||||
|
||||
**`fit` is the cosine between a role and a prompt of the agent, mapped to a level by constants calibrated per embedding model: `clip((cos - lo) / (hi - lo), 0, 1)`.** `(cos + 1) / 2` spanned only about 0.87 to 0.95 on multilingual-e5-small, so a prefer gate at 0.2 outweighed all of fit. Min-max across the roles an agent scored in the run fixed the spread but made fit a rank: the best role of a poor set got 1, a role's fit and the 0.9 favourite floor's meaning drifted from run to run, and the `limits` bars that bound the paid stages compared an absolute number with a relative one. The constants are `CALIBRATION` in `app/config.py`, keyed by model name, and an entry under `embed.calibration` replaces a model's built-in one; keyed by model, a change of `embed.model` cannot carry another geometry's constants along. A model with none falls back to min-max per run, with gated roles left out of the range, and the log warns once per run. `taste` is mapped the same way with its own pair. The starter agents' prefer weights are 0.05.
|
||||
**multilingual-e5-small's bands are `fit` 0.75 to 0.88, `taste` -0.14 to 0.19 and `cv` 0.77 to 0.90, measured on a copy of the live database.** Over 16,841 open roles and five intents the raw prompt cosine pooled to p1 0.753, p99.5 0.860 and p99.9 0.871; `hi` sits above every intent's p99.9, so only a handful of an agent's roles clip to 1, where 0.86 would clip about 75 and lose their order where it matters most. Taste came from synthetic votes (30 liked and 30 passed for each of three intents) and should be re-measured once real votes exist. A CV chunk is longer than a prompt and sits closer to every posting: the prompt band clipped 88 of one agent's 7,270 roles to 1, so the 50 closest tied on posting date, while (0.77, 0.90) clips one.
|
||||
|
||||
**multilingual-e5-small's constants are fit 0.75 to 0.88 and taste -0.14 to 0.19, measured on a copy of the live database.** Over 16,841 open roles and five different intents the raw fit cosine pooled to p1 0.7528, p50 0.7989, p99 0.8528, p99.5 0.8595, p99.9 0.8711 and max 0.9031; per intent p1 ranged 0.7456 to 0.7700 and p99.9 0.8413 to 0.8773. `hi` is above every intent's p99.9, so about 5 of the "Copenhagen engineering" agent's 4,156 ungated roles clip to 1; at 0.86 (p99.5) 75 would, and lose their order where it matters most. The span 0.13 is close to the 0.145 min-max span it replaces, so a prefer weight keeps about its old size against fit. The copy had no votes, so taste comes from synthetic ones, 30 liked and 30 passed roles for each of three intents (pooled p1 -0.1404, p50 0.0429, p99.9 0.1924), and should be re-measured once real votes exist.
|
||||
**The embedding backend is effectively required, and it is `intfloat/multilingual-e5-small`.** Prompts and a CV are paragraphs, and TF-IDF cannot relate "Kubernetes" to "container orchestration". roledesk started with the English-only `all-MiniLM-L6-v2`, but the corpus turned European: on a sample of the live roles against an engineering prompt, MiniLM separated engineering from other roles with an AUC of 0.68 overall and 0.74 on Danish-country roles, multilingual-e5-small with 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. `paraphrase-multilingual-MiniLM-L12-v2` measured worse than MiniLM (AUC 0.60). e5 needs its "query: " and "passage: " prefixes, which `app/jobs/embed.py` adds by matching the model name rather than by a setting.
|
||||
|
||||
**The cut's defaults are 500 by fit and 150 by CV closeness, each at least 0.6 (2026-10-04).** The old bars, 0.55 for extract and 0.65 for summarise, were measured on the blended score. Migration 32 reset every agent's limits, since a bar on the blended score means nothing on plain fit. Among agent 2's 7,263 roles passing the fetch gate, 2,086 reach fit 0.6, 1,580 reach CV closeness 0.6 and 2,437 either, so for a broad agent the counts stop the lists; the minimum is for a narrow agent, so roles that do not fit are not read to fill the count.
|
||||
**desk's own summary and tagging are off.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`; roledesk's `extract` fills the gate columns and its `summarise` writes the summary and tags, as `ModelTask`s of its own.
|
||||
|
||||
**An agent without a prompt that has text has no fit list.** Its cut is the CV list and its liked and saved roles; it feeds no summarise top, and each score run raises one `score.no_intent` notice naming the enabled agents without an intent. The agent still ranks its own list.
|
||||
**A gate rejects on evidence, not on absence, and the location lines are one group** (ADR Fetch prep and score 1.1). A gate used to reject a role whose column was empty, and extract, which fills that column, reads only the cut, so every role from a board that writes "Remote" or "Hybrid" as its location was out of reach for good. The location lines are `any` gates that form one group a role must match one of, so "in Denmark, or remote" is two `any` gates where `require` alone could only say "and". Three cases count as an unknown location rather than as evidence. A place text the gazetteer cannot resolve is one: "Storkøbenhavn" and "Headquarters" were read as "not Malmö" and rejected about 650 open roles, most of them in Greater Copenhagen. A role whose country and place are both unknown passes whatever its remote says, since a board that writes only "Hybrid" may sit in one of the agent's towns. A place that holds the agent's places, Øresund or Europe for Denmark, is no evidence against either, but only when the role names no place elsewhere: `London; Europe` is in London. Such a role enters the cut on its level, extract reads it and the gate is applied again on what it found.
|
||||
|
||||
**The embedding backend is effectively required.** The intent is a CV or a paragraph per agent, and TF-IDF cannot relate "Kubernetes" to "container orchestration". The model is `intfloat/multilingual-e5-small`, not the English-only `all-MiniLM-L6-v2` roledesk started with: the corpus turned European, and on a sample of the live roles (all open Danish-country roles plus a random sample of the rest, language labelled by a stop-word count) against the "Copenhagen engineering" agent's intent, MiniLM separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles, where a Danish-country engineering role beats the rest's non-engineering median only 68% of the time; multilingual-e5-small reaches an AUC of 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. A second multilingual candidate, `paraphrase-multilingual-MiniLM-L12-v2`, measured *worse* than plain MiniLM (AUC 0.60) and was dropped. e5 needs its own "query: " / "passage: " prefixes, which `app/jobs/embed.py` adds by matching the model name, not a separate setting.
|
||||
**A favourite only raises the level; the company switches do the widening** (ADR Fetch prep and score 2.1). Affinity used to skip gates, which made it both a widening and a ranking; widening now belongs to All locations and All roles, which are the same for every agent because fetching is shared.
|
||||
|
||||
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty. Summaries and tags came back as the `summarise` stage and the Summarise button (below); what stays off is desk's own `SummaryPolicy` and `TaggingPolicy`.
|
||||
**Gates match loosely where the column is free text, and each gate is one SQL column.** `country` and `place` match over the gazetteer as the Place filter does, `company` compares company keys, `title` matches whole words of `title_key`, `languages` passes when every language the role requires is one of the agent's, `salary_min` is at least the smallest value, and the rest are case-insensitive equality (`_gate_sql` in `app/storage/store.py`). Each gate is one SQL `CASE` column, so the gated row keeps every failed gate as its reason.
|
||||
|
||||
**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency`, `affinity` and `taste` weights form one denominator, so a score stays in 0..1 whatever the weights are. Prefer gates were removed on 2026-10-04 (migration 32 dropped them): hybrid and similar wishes are feed filters, so a role is never judged twice by the score and by a filter.
|
||||
**Role exclusion matches title phrases, not `function`.** Sales titles (Account Executive, Client Partner) and individual data engineering ranked high, since their postings read like the sales engineering and data leadership the prompts describe. `function = sales` could not remove them: the keyword classifier files "Solutions Engineer" and "pre-sales" under sales, so it would also have gated Field CTO, Staff Solutions Engineer and every AI Deployment Strategist near the agent's top. A value matches as whole words of `title_key`, so `data engineer` gates "Senior MLOps/Data Engineer" and not "Data Engineering Manager", and the reason names the value that matched ("title account executive"). Bare `sales` and `vice president` stay off the owner's list because they would take Sales Engineer and VP roles, and `account manager` because it would take Technical Account Manager.
|
||||
|
||||
**A gate rejects on evidence, not on absence, and `any` gates are one group.** A require gate used to gate a role whose column was empty ("country unknown"), and extract, which fills that column, is bounded to the roles that rank, so 2,281 of 15,556 open roles were out of reach for good: every board that writes "Remote", "Hybrid" or "Distributed" as its location. Now an empty column neither passes nor fails; the role ranks, extract reads it, and the rescore gates it on what it found. `any` is the third mode: the `any` gates form one group a role must match one of, so "in Denmark, or remote" is two `any` gates, where require alone could only say "and". `remote` is also read off the location line, as `country` is, since Greenhouse puts "Hybrid" there and nowhere else.
|
||||
**A role that gains text is extracted again.** Extract reads a role once, and a role read from its title alone (fetch text capped or failing, or the role entering the cut late) kept title-only place, remote and level for good. The pull that brings a better text now resets `extracted_at` beside dropping the vector, so both stages read the text.
|
||||
|
||||
**A favourite only raises the level; All countries skips the places (2026-10-04).** A favourite used to skip every gate and score at least 0.9, then (2026-09-29) to skip the `any` group, which made affinity both a widening and a ranking. Since 2026-10-04 the widening belongs to the company switches, the same for every agent: All countries skips the place part of the fetch gate (the `any` group) and All roles the role part (fetch terms and `exclude` gates on title). The `require` gates (the region) apply to every role. Affinity adds about 0.05 (liked) or 0.1 (favourite) to both levels of the cut and its weighted share to the final score.
|
||||
**A role with no vector is scored without prompt closeness and job like.** Their weights leave the denominator, so a role `embed` has not reached ranks on newness and affinity instead of sinking to the bottom. It is gated as "no vector" only when `fit` is the only weighted term, and as "no prompt vector" when no prompt could be embedded.
|
||||
|
||||
**Gates match loosely where the column is free text.** `place` takes the Place filter's tokens on whole folded words, so `Berlin` matches "Berlin, Germany" and `Lund` no longer matches "Billund"; `company` compares company keys, `languages` passes when every required language is a value, `salary_min` is at least the smallest value, and the rest are case-insensitive equality. Each gate is one SQL `CASE` column, so a require gate filters while the gated row keeps every failed gate as its reason.
|
||||
**Each run replaces an agent's scores.** `replace_scores` deletes and inserts in one transaction, so a role closed or hidden since the last run drops out rather than keeping a stale score, and a reader sees one list or the other.
|
||||
|
||||
**`exclude` gates titles out by phrase, not by `function`.** Sales titles (Account Executive, Account Director, Client Partner) and individual data engineering ranked high for the owner's agent, since their postings read like the sales engineering and data leadership the prompts describe. `function = sales` could not remove them: the keyword classifier files "Solutions Engineer" and "pre-sales" under sales, so on the data of 2026-09-30 it would also have gated Field CTO, Director Field Engineering, Staff Solutions Engineer and every AI Deployment Strategist in the agent's top 400. The fourth mode, `exclude`, gates a role that matches any of its values, and the gate's `CASE` column returns the position of the first value that matched so the reason names it ("title account executive"). It applies to favourites, which skip only the `any` group. The new `title` field matches a value as whole words of `title_key`, `LIKE '% phrase %'` over the padded key, so `data engineer` gates "Senior MLOps/Data Engineer" and not "Data Engineering Manager", and Forward Deployed Engineer is untouched. The form has a Leave out titles line under Where. Bare `sales` and `vice president` stay off the owner's list, because they would take Sales Engineer and VP roles, and `account manager` stays off because it would take Technical Account Manager. Measured on the agent's 5,089 ranked roles, the list the owner took gated 283, 13 of them in the top 100.
|
||||
|
||||
**A role with no vector is scored without `fit`.** The fit weight leaves the denominator, so a role `embed` has not reached yet ranks on recency and affinity instead of sinking to the bottom. It is gated as "no vector" only when fit is the only weighted term, and as "no intent vector" when the intent could not be embedded.
|
||||
|
||||
**Each run replaces an agent's scores.** `replace_scores` deletes and inserts in one transaction, so a role closed or muted since the last run drops out rather than keeping a stale score, and a reader sees one list or the other.
|
||||
|
||||
**Starter agents are imported once, by a state marker.** newsdesk seeds whenever its table is empty. Here `agents:seeded` in `state` is set after the import, so deleting every agent survives a restart.
|
||||
**Starter agents are imported once, by a state marker.** newsdesk seeds whenever its table is empty; here `agents:seeded` in `state` is set after the import, so deleting every agent survives a restart.
|
||||
|
||||
**`embed` and `score` run in the server process.** newsdesk isolates `embed` in a child process with a timeout. roledesk's run is one bounded pass over open roles, and in-process keeps the embedder replaceable in tests. The cost is torch's memory staying in the server after the first run; isolate it if that becomes the problem.
|
||||
|
||||
**The pull cascade is `pull → prep → embed → score → fetch_text → prep → embed → score → extract → score → judge → summarise`.** Every stage is always in the chain and a stage that is off (the switch under Runs, else config.yaml) finds nothing, as in newsdesk. The free stages run first: embed and score cost no tokens, and score writes the cut that bounds everything after it. Extract reads the cut, then score runs again so the fetch gate and the cut see the real location and remote area before judge reads anything. Summarise runs last, on the final order. A stage is addressed by its position in the tuple, since `score` is in it three times. Extract reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly. `CASCADES` in `app/jobs/cascade.py` is the source.
|
||||
**Every stage is always in the pull cascade** (`CASCADES` in `app/jobs/cascade.py`). A stage that is off finds nothing, as in newsdesk. The free stages run first, since embed and score cost no tokens and score writes the cut that bounds everything after it. A stage is addressed by its position in the tuple, since `score` is in it three times.
|
||||
|
||||
**The cut is the one selection for fetch text, extract and judge (2026-10-04).** Per agent, over the roles passing its fetch gate, the cut is the best `cut_fit_top` by fit level at or above `cut_fit_min`, then the best `cut_cv_top` by CV level at or above `cut_cv_min` among the rest, then every role it liked and every saved role (`scores.cut` is `fit`, `cv` or `kept`). A level is the plain fit or CV closeness plus the affinity bonus. Newness and taste are out: a role is read once and pulls run every few hours, and votes come after the read. The two lists are not merged into one number, since they answer different questions (what I want, what I am) and a merged number lets the stronger side crowd the other out. One depth: a second bar inside the cut would defeat its purpose. Without the Where lines only 118 of agent 2's 500 best by fit were roles it could take, so the cut keeps the fetch gate. The cost is mostly a one-off catch-up (about 16 minutes of extract and an hour of judge for 650 roles on the subscription backend); a change to the CV, a prompt or the like texts re-judges the whole cut, which is accepted. A batch dealt named uids finishes them whatever their rank.
|
||||
**The cut is one selection with one depth** (ADR Fetch prep and score 2.2). Fetch text, extract and judge read the same cut, and a second bar inside it would defeat its purpose. The cut keeps the fetch gate because without it only 118 of one agent's 500 best by prompt closeness were roles it could take.
|
||||
|
||||
**`extract_batch` and `summarise_batch` are `isolate=True`, with a timeout.** `context.check()` only runs between the calls `modeltask.one` makes, not inside one, so a call that hangs on either backend was otherwise unkillable short of restarting the server; a process boundary is what lets a Kill, or the timeout itself, actually end it. One call is already bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for summarise): a batch still running after that is stuck, not slow. The cost: an isolated job's context has no working `stop`, `task` or `fan_out` (see `pipeline/child.py`'s `context_for`), so a refusal or an account-level failure inside a batch can no longer stop the rest of that task's queued siblings, which run on to the same failure instead of being spared it.
|
||||
**`extract_batch`, `judge_batch` and `summarise_batch` run isolated, with a timeout.** `context.check()` runs only between the calls `modeltask.one` makes, not inside one, so a call that hangs was unkillable short of restarting the server; a process boundary lets a Kill, or the timeout, end it. One call is bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for judge and summarise): a batch still running after that is stuck, not slow. The cost is that an isolated job's context has no working `stop`, `task` or `fan_out` (`context_for` in `pipeline/child.py`), so a refusal or an account-level failure inside a batch cannot spare the task's queued siblings.
|
||||
|
||||
**A third order, CV match, from a model reading each agent's best roles against its CV (2026-10-03).** Taken from the `/rank` of github.com/MadsLorentzen/ai-job-search, which has a model score a posting against a written profile on five dimensions. Fit compares topics; it cannot tell a step down, years of experience or a stated deal-breaker, which a model reading both texts can. An agent may hold a CV (pasted, or a `.md`/`.txt` file the browser reads into the box), its languages with a level (suggested from the CV by one model call, then edited), and a soft profile of two free texts, "What I like" (`like_text`) and "What I don't like" (`avoid_text`, the column keeps its name), each covering the work, the workplace and situations, with no split into tasks versus environment; the prompts stay what the owner wants. Both texts feed the career score, which the rubric judges against what the person wants (the prompts), what they like and what they do not like, and whether the tasks and the setting would energise them. A deal-breaker is a hard rule and lives in the agent's gates, so the form has no must-haves box; `must_text` stays a column that only the API sets, and the verdicts and the cap below apply whenever it is non-empty. The stage `judge` (after extract and its rescore in the pull, switched under Runs like extract, Haiku by default; since 2026-10-04 it reads the agent's whole cut instead of its own limits) reads the pairs of an enabled agent with a CV, once per `input_key` (a hash of the CV, prompts, both texts and the role's text), and stores skills, experience and career (0 to 100, their rubric bands verbatim), 1 to 3 strengths and gaps in `role_judgements`, which rescoring and closing never touch. The CV match is the weighted mean of the three, with the weights per agent (`cv_skills`, `cv_experience`, `cv_career` in `agents.weights`, default 30, 25, 30) applied when a role is served, so changing them needs no model call. Culture is left out (guessed from ad text) and location too (the gates do it): a must-haves text (set through the API) that names a place makes the model mark location down, so places belong in the Where rules. The model reports facts and code applies the rules: the must-haves text, when set, is read as one must-have per line (leading bullets and numbers dropped), the judge's profile (`<likes>`, `<dislikes>` and, when set, `<must>`) numbers them M1..Mn, and for each judgement the model answers `musts`, a verdict per must-have, "met", "broken" only when the posting text clearly contradicts it, or "unclear" when the text does not say (never inferred from silence; a must-have the model leaves out counts as unclear). It scores career on its rubric as before and the career is stored as given. The cap is code's: when a role is served, any "broken" verdict makes career count as `min(career, BROKEN_CAP)` with `BROKEN_CAP = 39` (`app.models`), in `cv_score`, in `Store.judgement` (which serves the model's career, `career_capped` and the must-haves with their text) and in `store._cv_sql`, which reads a `broken` count stored beside the verdicts (`role_judgements.musts` and `.broken`, migration 29), so CV match orders by the capped value too. The verdicts store each line's text, so the reader still shows the must-haves a judgement was made against after the text is edited. The judge's `input_key` carries a format version (`judge.FORMAT`, 3) and the likes text (`agents.like_text`, migration 30), so judgements made before the verdicts or the likes existed are made again; until then they serve with the verdicts they have, or none and no cap. The reader shows each must-have with ✓ met, ✗ broken or ? unclear, and career as "39 (capped: breaks a must-have; model said 55)". CV closeness (2026-10-03) is a free, local signal beside the judge, never in its place: the score run cuts the CV into chunks (`embed.cv_chunks`: paragraphs, a longer one cut by line, then sentence, then word, neighbours joined again up to `CV_CHUNK_CHARS` = 1,200 characters, inside e5's 512 tokens), embeds each as a prompt is ("query: " for an e5 model), caches them in `agent_cv_vectors` under sha1 of the model, the chunking version and the CV text, and stores each ungated role's best cosine over the chunks, scaled to 0..1, as `scores.cv_fit` (migration 31). It is no term of the Agent match score. It has its own band, `Calibration.cv`, fit's when a model has none: a CV chunk is longer than a prompt and sits closer to every posting, so on the copy of 2026-10-03 fit's band (0.75, 0.88) clipped 88 of agent 2's 7,270 ungated roles to 1 and the 50 closest tied on posting date; (0.77, 0.90) clips one. The judge then also read the 50 ungated roles closest to the CV that its score list left out; this became the cut's CV list (2026-10-04). On that copy 43 of agent 2's 50 closest roles were outside its 150 best by score, ranked 153 to 2,632 by score (median 831), 20 of them under `judge_min`. CV match puts the judged roles first by `cv_scoLine truncated
|
||||
**The profile is the CV, the languages, the prompts, the likes and the dislikes; must-haves are gone** (ADR 4.2). The must-have text with a verdict per line and a career cap at 4 was the harder form of what likes and dislikes now say softly; it left the form on 2026-10-03 and its machinery followed on 2026-10-06 (migration 34). The judge reads the languages with their levels, since a role that asks for fluent Danish against conversational Danish is a skills gap the CV text alone did not show; a change to the languages re-judges the cut like a change to the CV.
|
||||
|
||||
**A posting's languages and deadline, and a flag rather than a score.** Extract reads `languages_required` as objects (language, level on the ladder basic, conversational, professional, fluent, native, and required or plus) and a deadline, from the opening and the closing lines of the text, since both usually sit at the end. Jobnet and Jobbank state a deadline, kept as `deadline_host`; prep composes `deadline` host first. The language flag is computed when a role is served from the posting's languages and the agent's: ⚠ when a required one is missing or its level is above the agent's, ★ when a plus one is the agent's; it never changes a score. Closing soon is the deadline from today to today plus four days, in UTC.
|
||||
**Judge reads a role against a written profile, after the `/rank` of github.com/MadsLorentzen/ai-job-search.** Prompt closeness compares topics; it cannot tell a step down, years of experience or a deal-breaker, which a model reading both texts can. The profile is the CV, its languages with a level, and two soft texts, "What I like" (`like_text`) and "What I don't like" (`avoid_text`), each covering the work, the workplace and situations with no split into tasks versus environment, since the prompts already say what the owner wants. A deal-breaker is a hard rule and belongs in the fetch gate, so the form has no must-haves box; `must_text` is set only through the API. Culture is left out because it is guessed from ad text, and location because the fetch gate does it.
|
||||
|
||||
**The model reports facts and code applies the rules.** For each must-have line the model answers "met", "broken" only when the posting clearly contradicts it, or "unclear" when the text does not say, never inferred from silence; a must-have it leaves out counts as unclear. The cap is code's: a broken must-have makes career count as at most `BROKEN_CAP` (4) when a role is served (`app/models.py`), so the stored career stays the model's and the reader shows both. The CV match weights are applied when a role is served too, so changing them needs no model call. The verdicts store each line's text, so the reader still shows what a judgement was made against after the text is edited. The `input_key` carries `judge.FORMAT` (4) and the texts, so a change to the prompt's contract makes every judgement again.
|
||||
|
||||
**CV closeness is a free, local signal beside the judge, never in its place.** The score run cuts the CV into chunks (`embed.cv_chunks`, up to `CV_CHUNK_CHARS` = 1,200 characters, inside e5's 512 tokens), embeds each as a prompt is, and stores each role's best cosine over the chunks, as a level, in `scores.cv_fit`. It is a list of the cut of its own because on one agent 43 of the 50 roles closest to the CV were outside its 150 best by score.
|
||||
|
||||
**A posting's languages and deadline give a flag, not a score.** Extract reads `languages_required` (language, level, required or plus) and a deadline from the closing lines, where both usually sit; Jobnet and Jobbank state a deadline, kept as `deadline_host`, which prep takes first. The language flag is computed when a role is served: ⚠ when a required language is missing or above the agent's level, ★ when a plus one is the agent's. Closing soon is a deadline from today to four days on, in UTC.
|
||||
|
||||
## The pages
|
||||
|
||||
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models`, `plugins.ats.FUNCTIONS` and `plugins.ats.HOSTS`, so they cannot drift from what the store and the plugin accept). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change. Only Roles, Applications and Watchlist are tabs now: Agents and Companies are settings pages, and Companies is a section of Sources (Two tabs and a cog, Watchlist is a third tab and Settings are three sub-tabs, below).
|
||||
**Each page renders a shell that its own script fills.** The server renders only select options, from `app.models`, `plugins.ats.FUNCTIONS` and `plugins.ats.HOSTS`, so they cannot drift from what the store and the plugin accept. One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change.
|
||||
|
||||
**The feed loads whether or not its enhancements do.** `/api/agents` and `/api/companies` only dim hidden companies, draw the affinity control and pick the viewing agent's score order; the init block used to await both before ever calling `loadFeed`, so a slow or failed one left the feed blank for good behind a silent `catch (err) { return; }`. Now the feed's own request goes out alongside them rather than after, agents and companies fill in and redraw what they touch (the rail, the feed's dimming, the reader's affinity slot) whenever they land, and a feed already fetched with no agent known reloads once the real one is. A failed feed fetch of its own gets an empty state with a retry (`state.feedError`) instead of the same silent blank.
|
||||
**The feed's first request names the agent, and waits only when it cannot.** The server gates nothing for a request without an agent, so a first request sent before the agents were known listed every open role until they arrived, and for good when `/api/agents` failed. The request carries the agent of the address, else the one this browser viewed last; with neither it waits for the agents. Companies, tag counts and saved filters still fill in when they arrive and the feed does not wait on them, since awaiting them once left the feed blank behind a silent catch. When the agents cannot be had, the feed shows its error state with a retry (`state.feedError`).
|
||||
|
||||
**`roles(company=, place=)` match a case-insensitive substring of what the board wrote.** The API's company and place fields are free text a person types, not a slug, so "Trade Republic" finds `trade-republic-bank` without knowing the key. Other pages link with them; the rail no longer types them and shows one as a chip to take off.
|
||||
**A page starts a row earlier for each vote or read since the last one.** The server sorts afresh on every request, and a pass, or a read with Show read off, takes a role out of the part already loaded, so the next page asked from the count of rows on screen skipped one role and showed another twice. The offset the server has read is kept apart from the rows on screen, the next page steps back one row per change and drops the rows already shown, and the end of the list is the server's to say. A rescore during scrolling can still shift the list; only a snapshot of the order on the server would rule that out.
|
||||
|
||||
**A place is matched on whole folded words, by a function registered on the connection.** `roles.place_fold` (migration 17) holds `places.fold(place)`: casefolded, marks stripped, `ø` to `o`, `æ` to `ae`, punctuation to spaces, so `Skane` finds `Skåne` and `Kobenhavn` finds `København`. A filter is a token: `country:DK` is the country column, or a line naming the country or one of its cities while the column is empty or `EU`, which finds 1,193 roles against 1,132 on the column alone; `region:skane` is any name or town of a region in `app/places.py`; `place:malmo` is a town with its aliases. Several tokens are joined by OR. A region is up to 161 phrases, and as `LIKE` clauses over 34,538 roles `region:oresund` took 400 ms, twice per feed request. `place_match(place_fold, country, token)`, which `Store._connect` registers, takes 10 to 30 ms.
|
||||
**The rail's counts cover what the feed shows.** Tag, company and place counts ignored the fetch gate, so a chip promised more roles than the list held (Novo Nordisk 134 against 102). With an agent named, each count keeps to the roles that agent's gate passes.
|
||||
|
||||
**The regions are a hand-written list.** Skåne, the Capital Region of Denmark, Zealand, Øresund (the two capital regions), Jutland and Funen, each with its names and towns. On the data of 2026-09-29 no Danish or Skåne town with more than one role was missing. A region elsewhere is added to `REGIONS`.
|
||||
**`company=` and `place=` on `/api/roles` match a case-insensitive substring of what the board wrote.** They are free text a person types, not a slug, so "Trade Republic" finds `trade-republic-bank` without knowing the key, and other pages link with them.
|
||||
|
||||
**`seniority` is read from the title until extract has read the role.** Extract had never run, so the column was empty on all 33,881 open roles and a seniority filter or gate had nothing to read. `seniority.from_title` matches whole words in English, Danish, Swedish, German and French and gives 43% of open roles a level. "X Manager" is a level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor; intern wins over a boss named in the title. Known mistakes: "VP" is always exec, "Field CTO" and "Chief of Staff" get none, "Account Director" is director. Extract overwrites the title's level, a level a board states wins over both. Since the ladder, `roles.seniority` is the first of the model's level, the host's stated level and the title's, in that order (Seniority has three sources, above), so a stated level no longer wins over the model's.
|
||||
**Every filter runs in SQL before `LIMIT`.** Filtering after `LIMIT` showed nothing once a hidden company's freshly pulled roles filled the newest 200 rows. Hidden companies, tags (an `EXISTS` or `COUNT(DISTINCT tag)` clause) and every other filter narrow the rows a page is built from, so a filter matching only older roles is not silently empty on page one.
|
||||
|
||||
**Seniority is a ladder of rungs with two tracks.** The first vocabulary was one line (intern to exec) and put a Principal below a Team Lead, and a Staff engineer below any manager, while the owner counts a Principal Solutions Engineer as very senior. From `staff` up, each rung pairs an individual contributor's level with a people lead's, as big tech firms grade them: `staff` with `lead` (a team lead) and `manager`, `principal` (also Senior Staff) with `senior_manager` (also Associate Director, AVP), `distinguished` (Senior Principal, Distinguished, Technical Fellow) with `director` (also Head of), then `exec`. `LADDER` in `app/models.py` gives each word its rung and track, and `roles.seniority` stays one column of words, so gates, `extract` and the host tables work on words. `lead` changed meaning: "Tech Lead", "Lead Engineer" and "Marketing Lead" are `staff`, and only a team lead, supervisor or shift lead is `lead`. In a title, a people level wins over a grade on its own rung, and a grade above it lifts it one rung ("Principal Engineering Manager" is `senior_manager`); a slash is "or" ("Senior/Staff Engineer" is `staff`). Principal sits one rung above Staff, although at some firms it is the top individual grade and at others a senior manager's peer. Migration 25 cleared the model levels the split made ambiguous (staff, lead, manager, director), and prep `VERSION` 2 mapped every open role again. On 2026-09-30 that gave, over 19,230 open roles without copies: staff 1,756, principal 535, distinguished 23, lead 222, manager 736, senior_manager 312, director 883, exec 101, none 9,565.
|
||||
**`GET /api/roles` serves a row, not a role.** The whole `Role`, description, `raw` and article text included, went to a feed that shows a title, a few badges and an excerpt. `Store.roles_page` selects only what a row draws (`RoleRow`), with the excerpt cut in SQL, which made a page about a tenth of the size; the reader fetches the full role by uid when it opens one.
|
||||
|
||||
**A filter that most roles cannot answer keeps the ones that do not say.** The Seniority filter keeps roles with no level unless its box is unticked, since 57% have none. The Pay filter leaves out roles that state no pay unless its box is ticked, since 87% state none and the filter would otherwise do nothing. `GET /api/roles/facets` counts each level and pay state under the other filters, so the rail shows what a click would add or lose.
|
||||
**Seniority is a ladder of rungs with two tracks.** One line from intern to exec put a Principal below a Team Lead and a Staff engineer below any manager, while the owner counts a Principal Solutions Engineer as very senior. From `staff` up, each rung pairs an individual contributor's level with a people lead's, as big tech firms grade them (`LADDER` in `app/models.py`), and `roles.seniority` stays one column of words, so gates, extract and the host tables work on words. "X Manager" is a people level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor. Principal sits one rung above Staff, although at some firms it is the top individual grade and at others a senior manager's peer.
|
||||
|
||||
**The Seniority filter is a range of rungs and a track, not a button per level.** "At least Staff" is a range, and "manager roles only" is the Staff rung on the people track, so From, To and Track (Both, Individual, People) cover what the buttons did. Only a scattered pick such as junior and director is lost. `GET /api/roles` and `/facets` take `level_from`, `level_to` (names in `RUNGS`) and `track`, which `levels_between` turns into the list of words the store filters on. The rungs below Staff have no track and pass either. The facets still count per word; the rail sums them per rung for the track chosen, and names a rung after the track ("Staff / Manager" with both). The page carries `RUNGS` and `LADDER` as JSON, so the words live in `models.py` only. An old `?seniority=` address is ignored.
|
||||
**A filter that most roles cannot answer keeps the ones that do not say.** The Seniority filter keeps roles with no level unless its box is unticked, since most roles state none. The Pay filter leaves out roles that state no pay unless its box is ticked, since most state none and the filter would otherwise do nothing. `GET /api/roles/facets` counts each level and pay state under the other filters, so the rail shows what a click would add or lose.
|
||||
|
||||
**Company and Place are chips picked from suggestions.** Only a suggestion becomes a chip, so a filter names something the database holds: a company key, a place token. `GET /api/suggest/companies` and `/api/suggest/places` fold what is typed and match word prefixes, computed per request over 743 companies and 5,600 place lines. `app/static/chips.js` is the input, with no dependency; its tests run with `node --test tests/js/*.test.mjs`.
|
||||
**The Seniority filter is a range of rungs and a track, not a button per level.** "At least Staff" is a range, and "manager roles only" is the Staff rung on the people track, so From, To and Track cover what the buttons did; only a scattered pick such as junior and director is lost. The page carries `RUNGS` and `LADDER` as JSON, so the words live in `models.py` only.
|
||||
|
||||
**A filter is a history entry, text is not.** Filters used to replace the entry, so back left the page. Now each filter set pushes one and back takes it off: a tag, the period, a chip, a level, a company. The search box and typed text replace, or back would step through a search a letter at a time, and so do the agent and the order. Every pushed entry carries the address it was pushed over, so closing the reader or taking a company off is `history.back()` when the entry under it is that list, and a new entry otherwise.
|
||||
**Company and Place are chips picked from suggestions.** Only a suggestion becomes a chip, so a filter names something the database holds: a company key or a place token. `app/static/chips.js` has no dependency; its tests run with `node --test tests/js/*.test.mjs`.
|
||||
|
||||
**A company's name is a link that adds the company to the filters on screen.** In a row and in the reader. With one company filtered a strip over the feed names it, counts its roles, links its careers page and carries the affinity control, which is where a company is starred from. After a change the strip waits for the `score_job` the route answers with and reloads the feed.
|
||||
**A filter is a history entry, text is not.** Each filter set pushes an entry and back takes it off. The search box and typed text replace the entry, or back would step through a search a letter at a time, and so do the agent and the order. Every pushed entry carries the address it was pushed over, so closing the reader or taking a company off is `history.back()` when the entry under it is that list.
|
||||
|
||||
**A muted company is excluded in SQL, before `LIMIT`.** Filtering after `LIMIT` shipped first and showed nothing once Databricks' 884 freshly pulled roles filled the newest 200 rows and Databricks was muted.
|
||||
|
||||
**`GET /api/roles` serves a row, not a role.** It used to answer `roles.*`, the whole `Role` (`description`, `feed_text`, `raw` and `article_text` included, about 5.5 KB a role on average) for a feed that only ever shows a title, a few badges and an excerpt; a 1000-row restore was several megabytes of text a phone never renders. `Store.roles_page` selects only the fields a row draws (`RoleRow`), with `excerpt` built by `app.models.excerpt` off a `SUBSTR` of `summary` or `description`, never the column whole. Measured on a copy of the live database: a 60-row page went from 392 KiB to 39 KiB and a 1000-row restore from 6.6 MB to 631 KiB, both about a tenth. The reader is unaffected: it already fetched the whole role by uid when it opened one (`GET /api/roles/{uid}`, still the full `Role`); what changed is that the list row it painted from first, before that fetch lands, no longer carries a body to paint with, so the reader now waits on the fetch for the posting text and the original link the same way it always waited for the summary.
|
||||
|
||||
**A board's limits are words on its row, and a filter.** The readers declare what they cannot deliver and the runner records how a board was read, and none of it reached the Companies list beyond the reach line and "not answering since". `app/jobs/limits.py` turns both into words per board, in a fixed order: `failing`, `read incomplete`, `swept`, `filtered by reach`, `remote not asked`, `narrowed by terms`, `nothing asked`, `size 0`, `over the ceiling`, `text per role` or `no text` or `excerpt only`, `no pay`, `date not trusted`, `without text: N`, `text failed: M`, `read whole`. The row shows them after the size, the Companies filter offers each as an option under Limits, and `board_open` beside the company's count says how many open roles the board itself has, since the company's count includes search roles. The readers list on Sources shows every trait a reader declares.
|
||||
**A board's limits are words on its row, and a filter.** The readers declare what they cannot deliver and the runner records how a board was read, and none of it reached the Companies list. `app/jobs/limits.py` turns both into phrases per board (`WORDS`), shown after the size and offered under the Companies filter's Limits; `board_open` says how many open roles the board itself has, since the company's count includes search roles.
|
||||
|
||||
## The phone
|
||||
|
||||
**Two tabs and a cog.** `TABS` in `app/web/pages.py` becomes Roles and Applications; the other seven pages (Agents, Companies, Sources, Jobs, Schedules, Messages, Account) move behind a settings cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and six of them were setup or system pages. The Watchlist became a third tab and the settings pages became three sub-tabs, both below.
|
||||
**Three tabs and a cog.** `TABS` in `app/web/pages.py` is Roles, Applications and Watchlist, and the settings pages sit behind a cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and most were setup or system pages. Settings are three pages, Search agents, Sources and Account, since six were three setup and three system pages for two things, what the app ranks by and where roles come from.
|
||||
|
||||
**Watchlist is a third tab: the companies the app cannot read, rated yes or no.** The owner goes through the careers pages of such companies by hand and wanted that list one click from Roles rather than behind the cog, so `TABS` has three entries and the phone bar's tabs are tighter. A company is on it when it has no board subscription and is not known from a search only, or when its board is failing: the rule of the Company boards filter "Not readable yet", read from `GET /api/companies`. The rating is `companies.watch` (migration 15: NULL unrated, 1 on the list, 0 not), global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. A row rated under Unrated leaves the list at once. On a phone the search field of Roles is left a few letters wide by the third tab, so while it has the focus the tabs and the queue chip step aside.
|
||||
**Watchlist is a tab: the companies the app cannot read, rated yes or no.** The owner goes through their careers pages by hand and wanted that list one click from Roles. A company is on it when it has no board and is not known from a search only, or when its board is failing. The rating is `companies.watch`, global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. On a phone the third tab leaves the Roles search field a few letters wide, so while it has the focus the tabs and the queue chip step aside.
|
||||
|
||||
**A company on the Watchlist says why it is there, and can be added, edited and removed on the tab.** `companies.hand_reason` (migration 16) is one of `unread` (readable, no module yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone` and `added` (by hand or by an import, not looked at); attaching a board clears it. `POST`, `PATCH` and `DELETE /api/companies` and `app companies add`, `edit`, `checked`, `remove` and `by-hand` are what the LinkedIn import will write through. A company with a reason is never "known from a search only", so one added with a name alone is listed. No second table and no `checked_at`: a company has one careers link, and `careers_opened_at` already says when somebody last looked. A careers link somebody typed is kept: discovery and the seed fill it only when empty. A name is changed only while the company has no board, since the subscription is found by the key the name folds to.
|
||||
**A Watchlist company says why it is there** (`companies.hand_reason`, `HAND_REASONS`). Attaching a board clears it. A company with a reason is never "known from a search only", so one added with a name alone is listed. No second table and no `checked_at`: a company has one careers link, and `careers_opened_at` already says when somebody last looked. A careers link somebody typed is kept, since discovery and the seed fill it only when empty. A name changes only while the company has no board, since the subscription is found by the key the name folds to.
|
||||
|
||||
**Settings are three sub-tabs: Search agents, Sources, Account.** Six sub-tabs were three setup pages and three system pages for two things, what the app ranks by and where roles come from, so each is one page now and Account stays its own.
|
||||
**The old paths' 301s are gone.** A browser caches a 301 indefinitely, so a path a later page reused would be unreachable; a one-person app with no outstanding bookmarks does not need to carry that risk.
|
||||
|
||||
**The old paths' 301s are gone.** `MOVED` answered 301 for the twelve paths above, kept for the few days between the move and this cut, on the assumption that a bookmark should still land. A browser caches a 301 indefinitely, so a path a future page reused would be unreachable through no fault of that page; a one-person app with no outstanding bookmarks does not need to carry that risk.
|
||||
**Sources is one page in sections: Job boards, Company boards, Runs and Unmapped.** Job boards are the `search` plugin's queries and company boards are `ats`'s. The UI says "job board" for what the API and CLI call a search, since "search agent" already means something else.
|
||||
|
||||
**Sources is one page, one section per plugin.** Job boards are the `search` plugin's queries, company boards are `ats`'s, and the runs that read both end the page with the message log. A source's section says which plugin it is, and following a company, hiding it and switching its board off are one list. The UI says "job board" for what the API and CLI call a search, since "search agent" already means something else.
|
||||
**Settings pages sit beside a rail, and Sources shows one section at a time, chosen by the URL fragment.** The company boards list runs long, so Runs at the end of a stacked page was out of reach and jump links still meant scrolling back up. The rail (`settings/_rail.html`, from `SETTINGS` and `SECTIONS`) is sticky beside the content, and a strip under the header on a narrow screen. A fragment rather than a path per section, because a hidden section stays in the DOM, so each section's script runs unchanged.
|
||||
|
||||
**Search agents are back in the UI as a list.** One person may run several: their own hunt, and one that monitors roles for someone else, each with its own intent and gates. The list has New and Delete; a delete takes the agent's votes, liked companies and scores with it.
|
||||
**Runs is the queue's own elements plus a Problems log.** `<desk-jobs>` and `<desk-schedules>` draw what is running and when the pull runs next; the app adds "Run the pull now", disabled while a pull cascade is live, and the stage switches. "Run one stage" went because the cascade is the unit of work: a stage run alone is a partial pull whose result the next stage never sees. `app job <stage>` and `/api/jobs` keep the per-stage entry points.
|
||||
|
||||
**Viewing as on Roles is a select, also with one agent.** One enabled agent used to show as its name, since one is not a choice, but the owner looked for the select to switch agent and found it gone after the second agent was switched off. The select lists the enabled agents; an agent that is off has no fresh scores and is left out.
|
||||
**Search agents are cards, edited in place.** One person may run several agents: their own hunt and one that watches roles for someone else. An agent is defined by what it looks for, so its prompts are the card's body and the weights a small line under it. Edit turns that card into the form and New puts a blank one on top, because a form at the bottom of the page was out of sight of the card it changed. Job boards and Watchlist companies are edited the same way, one open at a time.
|
||||
|
||||
**Settings pages sit beside a rail, and Sources shows one section at a time, chosen by the URL fragment.** The company boards list runs to 50 or 60 companies, so Runs at the end of a stacked page was out of reach and jump links still meant scrolling back up. The rail (`settings/_rail.html`, from `SETTINGS` and `SECTIONS` in `app/web/pages.py`) is sticky beside the content, and a strip under the header below 60rem. The fragment rather than a path per section, because a hidden section stays in the DOM, so each section's script runs unchanged.
|
||||
**The form's gates are the four parts of the fetch gate, and nothing else** (ADR Fetch prep and score 5.2 and Settings). Free rows of field, mode and values over twelve fields sat under Advanced, first to keep a rule an older form had saved; a row on seniority or pay was a permanent cut on fields half the roles leave empty, which the rail's filters do with a click and undo. The rows are gone; a stored gate the form cannot say is named under the fetch gate and kept as it is on save, so nothing is lost silently.
|
||||
|
||||
**Runs is the queue's own elements plus a Problems log.** `<desk-jobs>` and `<desk-schedules>` draw what is running and when the pull runs next, the app adds only "Run the pull now", disabled while a pull cascade is live, and the message log is the section's last block rather than a section of its own; `#messages` still lands on it. "Run one stage" went because the cascade is the unit of work: a stage run alone is a partial pull whose result the next stage never sees. `app job <stage>` and `POST /api/jobs` keep the per-stage entry points, which is what "Save and score" uses.
|
||||
**Viewing as on Roles is a select, also with one agent.** The owner looked for the select to switch agent and found it gone after the second agent was switched off. It lists the enabled agents, since an agent that is off has no fresh scores.
|
||||
|
||||
**Search agents are cards, edited in place.** An agent is defined by what it looks for, so the intent is the card's body and the weights a small line under it; a list of names and weights hid the part that matters. Edit turns that card into the form and New puts a blank one on top, because the new agent sorts first by id and a form at the bottom of the page was out of sight of the card it changed.
|
||||
**Affinity and votes are per agent.** A company hidden for one hunt may be a favourite for another, and a vote says what one agent should learn from, so both carry the agent id. Without an agent id the API uses the default agent, the enabled one with the lowest id, so the CLI and older callers keep working.
|
||||
|
||||
**Job boards are a list with New, edited in place like the agent cards.** The form used to sit open above the list and Edit filled it, which read as a stray form. It is a `<template>` now: Edit turns the row into the form, New puts a blank one on top, Save and Cancel close it, and one is open at a time.
|
||||
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because nearly every board role carries its full text, and a search role's teaser is worth reading in place before fetching the rest.
|
||||
|
||||
**Where is one question, and a role passes when it matches any of its lines; the other rules sit under Advanced.** The first Where block showed the two levels of the gates as they are stored, a required set of countries and then a one-of set, with ISO codes, and the owner could not read it. It asks "Where can the role be?" now: in these countries (by name), in these towns, remote within Europe or anywhere, and a sentence below says what they add up to, the same sentence the card shows. The stored gates did not change: the lines are `any` gates on `country`, `place` and `remote`, and "within Europe" is a require gate on `country` holding the Europe list plus the countries line. A require gate on `country` the block cannot say, such as Denmark and Sweden only, is a row under Other rules, which moved under Advanced above the read-only JSON and opens when the agent has one. Prefer hybrid ranks rather than filters, so it stands apart. Spellings of a town that differ by accents only share one chip, since `place` matches without case but with accents, and both are stored. A gate the block cannot express whole lands in the rows, so a load and save without edits returns the same gates.
|
||||
**Every link built from someone else's data goes through `safeUrl`** (`app.js`). A board, a search result, a careers page and an application's `url` are free text from outside the app; an `http(s)` check before it lands in an `href` keeps a `javascript:` link or a bare string from being clickable.
|
||||
|
||||
**Affinity and votes are per agent.** A company hidden for one hunt may be a favourite for another, and a vote says what one agent should learn from, so both carry the agent id, and the Roles tab and the company list ask for one agent's. Without an agent id the API uses the default agent, the enabled one with the lowest id, so the CLI and older callers keep working.
|
||||
**A vote moves the rank when the feed is served and never hides** (ADR Fetch prep and score 5.1.3). Applied at read time rather than by a rescore, a vote shows at once and needs no job; never hiding means a stray tap stays visible and one more tap undoes it.
|
||||
|
||||
**`taste` is the liked-minus-passed centroid, as in newsdesk.** An agent's votes become one vector, the mean of the liked roles' embeddings minus the mean of the passed ones', and a role's taste term is its cosine to that vector, calibrated like fit, weighted beside fit, recency and affinity. It is the one term an agent learns rather than is told, and it costs no model call.
|
||||
**Read state is a column on `roles`, and a vote stamps it too.** `read_at` is on `roles`, as newsdesk's `items.read_at`, so the feed filters and counts by it without joining a side table. A vote stamps it only when it is NULL, since voting is reading; clearing a vote leaves it, because taking back an opinion is not un-reading the posting.
|
||||
|
||||
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest. The figure is of 2026-09-28: hosts that list a role without its text now leave 1,974 of 24,003 open roles without a description on 2026-09-30, until `enrich` reads them.
|
||||
**A role's buttons run ▼ ● ▲ ★, and saving a role likes it.** The order is low to high, so ▲ sits beside ★. A role worth saving is a liked one, so `judge.js` casts ▲ for the viewing agent after a save, unless ▲ is cast already; unsaving leaves the vote alone. This is done in the page, not in `POST /api/applications`, because an application belongs to no agent and a vote does.
|
||||
|
||||
**Every link built from someone else's data goes through `safeUrl`.** A board, a search result, a discovered careers page and a saved application's `url` are all free text from outside the app; an `http(s)` check before it lands in an `href` is what keeps a stray `javascript:` link or a bare string typed into a field from being clickable. The reader needed this first, for `original_url`, so `safeUrl` moved to `app.js` beside the app's other shared helpers rather than staying roles.js's own; companies.js (`careers_url`), sources.js (a search sample's `url`) and applications.js (`application.url`) now use the same one.
|
||||
**Tags are a table, written only by `summarise`.** `role_tags(uid, tag)` follows newsdesk's `tags` table rather than a JSON array on `roles`, so tag counts and the any/all filter are ordinary SQL. `summarise` already reads the whole posting once per role, so writing 3 to 8 tags beside the summary costs no extra call.
|
||||
|
||||
**A vote moves the rank at read time and never hides.** `Store.roles` multiplies the served score by the vote (`0.5` passed, `1.2` liked, capped at 1) instead of triggering a rescore, so a vote's effect on Agent match shows at once and needs no job. A vote never removes a role from the list either, because a stray tap must stay visible and a single further tap must be able to undo it.
|
||||
**A company affinity change or an agent save queues a `score` that will see it.** `score` is `solo`, so the queue dedups a request into the live job; a queued one sees the change, but a running one read the old state, and deduping into it reported "Saved and scored" for a score that did not include the save. So a request during a run queues one follow-up with `depends_on` the running job, keyed `score:after:<id>`, which makes ten clicks during one run one extra pass. The routes answer with that job's id and the page waits on it.
|
||||
|
||||
**Read state is a column on `roles`, and a vote stamps it too.** `read_at` sits beside `vote`, the same shape as newsdesk's `items.read_at`, so the feed filters and counts by it directly rather than joining a side table on every page load. `set_vote` stamps it only when it is still NULL, since voting is reading in newsdesk as well; clearing a vote leaves it alone, because taking back an opinion is not un-reading the posting.
|
||||
**Summarise is off unless switched on, and saving or liking a role summarises it at once** (ADR Fetch prep and score 6). Saving or liking queues a `summarise_batch` for that one uid, under the key the Summarise button uses so a double tap is one job, when the stage is on, the role has text and no summary and a model is reachable, rather than waiting for the next pull.
|
||||
|
||||
**A role's buttons run ▼ ● ▲ ★, and saving a role likes it.** The order is low to high, so ▲ sits beside ★. A role worth saving is a liked one, so `judge.js` casts ▲ for the viewing agent after a save that succeeded, unless ▲ is cast already. Unsaving leaves the vote alone, as clearing a vote leaves the read mark. This is done in the page, not in `POST /api/applications`, because an application belongs to no agent and a vote does. The company control follows the same order, ⊘ ▲ ★.
|
||||
|
||||
**Tags are a table, written only by `summarise`.** `role_tags(uid, tag)` follows newsdesk's `tags` table rather than a JSON array on `roles`, so `tag_counts` and the any/all filter are ordinary SQL (`GROUP BY`, `EXISTS`) instead of scanning every row's array. `summarise` already reads the whole posting once per role, so writing 3 to 8 tags alongside the summary costs no extra model call.
|
||||
|
||||
**Tag filtering runs in SQL before `LIMIT`, the same as `hide_muted`.** An `EXISTS` or `COUNT(DISTINCT tag)` clause in `_roles_from` keeps the guarantee the muted-company exclusion already established: a filter narrows the rows a page is built from, not the page after it is already built, so a tag matching only older roles is not silently empty on page one.
|
||||
|
||||
**"Hidden" is the word the UI uses for affinity -1; the code keeps `mute` and `hide_muted`.** Renaming the columns would touch every reader of `agents` and `companies` for no behavioural gain, so only the toggle's title, the filter label and the dimmed row say "hidden" in the UI.
|
||||
|
||||
**A company affinity change or an agent save queues a `score` that will see it.** `POST /api/companies/{key}/affinity` and `POST /api/score`, which the Search agents page calls after a save, answer with the id of that job, and the page waits on it. `score` is `solo`, so the queue dedupes a plain request into the live job keyed `score`; a queued one has not read anything yet and sees the change, but a running one read the old state, and deduping into it reported "Saved and scored" for a score that did not include the save. So a request during a run queues one follow-up with `depends_on` the running job, `after="done"` and the key `score:after:<id>`, which makes ten clicks during one run one extra pass, and a queued follow-up is the answer until it starts. Without a score after it, an affinity change is invisible in Agent match until the next 6-hourly pull, and an unmuted company's roles have no score rows until then.
|
||||
|
||||
**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` (`app/jobs/posting.py`) fetches it only when the read page asks, because following Jobindex's link for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair. A Jobindex role's `vis-job` page is a short ad with a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's `jobannonce`); the fetch keeps that link as `roles.posting_url` (migration 26, never written by a pull), reads the text from it, and falls back to the `vis-job` page's own text. The reader's link-out prefers `posting_url`, so it leads to the full posting once the text was fetched.
|
||||
|
||||
**Summarise runs after the ranking, on the top of the feed, and is off unless switched on.** It is no part of any score. The pull's last stage considers roles with no summary yet that are in the best `summarise_top` (default 50) by final score of an enabled agent with a prompt, so the cost stays flat whatever the backlog is. Whether the stage runs is the switch under Runs, or `summarise.enabled` in config.yaml until the switch is used; the read page's Summarise button asks for one role's summary either way. The roles the owner marked are summarised whatever their rank (2026-10-03): every open role with an application (the star) or an ▲ from an enabled agent joins the waiting set, once each and under the same give-up rule. Saving or liking such a role queues a `summarise_batch` for that one uid at once, under the key the button uses so a double tap is one job, when the stage is on, the role has text and no summary and a model is reachable.
|
||||
|
||||
**`extract` and `summarise` are switched on the page, and a stored setting beats config.yaml.** `extract.enabled` and `summarise.enabled` were read at startup and both false, and nothing in the UI said so: an agent's card read "reads 500 at 0.55 · summarises 30 at 0.65" while neither stage had ever run. The `state` keys `stage:extract` and `stage:summarise` ("on" or "off") override the file, `stages.stage_on` in `app/jobs/stages.py` is the one function that decides, and the finders of both tasks ask it, so the stage in a pull, `app job extract` and `app job summarise` agree; with no key the file applies, so config.yaml is the default for a fresh database and an existing deployment behaves as before until someone uses the switch. It is a stored setting rather than a rewrite of config.yaml because rewriting config.yaml from the app would lose its comments and a restart is what the switch is meant to avoid. The switch is under Runs (`GET` and `PUT /api/stages`, `app stages`), beside "Run the pull now": turning a stage on starts nothing, the next pull runs it. It works with no model reachable and says who would answer, since a pull passes a model stage quietly then. A search agent's card says "extract is off" and links to the switch; its limits stay editable either way, since they are what the stage reads once on. The Summarise button queues `summarise_batch` for one uid and never asked the switch.
|
||||
|
||||
**The schedules element's narrow layout is fixed upstream, in `../boilerplate`.** `<desk-schedules>` is a shadow root the app cannot lay out from outside, and `pipeline/` is a downstream copy refreshed with `git archive`, so a fix landing only in roledesk would be reverted by the next refresh.
|
||||
|
||||
**The phone CSS is copied from health and news, not written from scratch.** Both apps have already been through several rounds of phone fixes, and the comments in their `app.css` say what each rule avoids, so adapting their fragments keeps those fixes rather than rediscovering the same edge cases, such as iOS zooming on a 16px input or a card losing width to a side gutter.
|
||||
**The model stages and fetch text are switched on the page, and a stored setting beats config.yaml.** `extract.enabled` and `summarise.enabled` were read at startup and both false, and nothing on the page said so. The `state` keys `stage:extract`, `stage:summarise`, `stage:judge` and `stage:fetch_text` override the file, and `stages.stage_on` in `app/jobs/stages.py` is the one function that decides, so a stage in a pull and `app job <stage>` agree; with no key the file applies, so an existing deployment behaves as before until someone uses the switch. A stored setting rather than a rewrite of config.yaml, because rewriting it from the app would lose its comments and a restart is what the switch avoids. Turning a stage on starts nothing; the next pull runs it.
|
||||
|
||||
## Outside the app
|
||||
|
||||
**Unposted roles and the network stay outside.** Their state is inside: an application row with no `role_uid`, a `contact`, `next_action` and `next_date`, and a company `status` such as researching or contacted.
|
||||
|
||||
**LinkedIn enters only as alert mail over IMAP.** Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.
|
||||
**LinkedIn enters only as alert mail over IMAP** (not built yet). Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.
|
||||
+2
-92
@@ -1,6 +1,6 @@
|
||||
# Survey: which filters the hosts take at the source
|
||||
|
||||
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. It covers the nine readers that declare a filter (SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken, Adzuna) and Workable, Teamtailor, Jibe and Lever, which declare none. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed. Sections 2 and 3 began as the plan for the fetch terms, which it called F2; F2 has landed (`decisions.md`, Fetch terms), and those sections now say what was recommended and what became of it.
|
||||
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. Twelve readers declare a filter in `READS.filters`: SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken and Adzuna were surveyed; freehire, Jobnet and Jobbank came later and were not, and their parameters are in the reader's docstring. Workable, Teamtailor, Jibe and Lever declare none and were surveyed. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed.
|
||||
|
||||
Marks used in the table:
|
||||
|
||||
@@ -12,7 +12,7 @@ Marks used in the table:
|
||||
|
||||
All request counts are exact: a host with a "live" mark on a claim spent its two requests on it, so anything else about that host is `code`, `front end` or `untested`.
|
||||
|
||||
## 1. Filters per host
|
||||
## Filters per host
|
||||
|
||||
Country and remote are left out; they are in the readers already. "Combine" is how a keyword and a country (or another facet) meet; "unknown" is what an unknown value does.
|
||||
|
||||
@@ -149,93 +149,3 @@ The environment has no key, so nothing was requested. From the public API docume
|
||||
| category | `occupation-field=apaJ_2ja_LuF` with `q=python` answered 497, the first hit `Data/IT` | untested | | untested | live |
|
||||
| place | `region=<län id>` (reader); `municipality=AvNB_uwa_6n6` returned a Stockholm hit | untested | | reader raises on an id outside `REGIONS` | live |
|
||||
| level | `experience=false` was in the same request as `municipality` and `occupation-field`; the answer was 1 hit, so it is not separable | | | | live, inconclusive |
|
||||
|
||||
## 2. Towns
|
||||
|
||||
**How a town reaches the fetch today.** An agent's `any` gate on `place` holds names ("Malmö", "Lund", "Helsingborg", "copenhagen", "odense" in agent 2). In `reach.gate_reach`, each name goes through `gaz.lookup` and `_countries_of`, which turns it into the country the place lies in; a name the gazetteer does not know makes the whole reach everywhere. The reach is a set of ISO codes and a remote flag; it holds no town. `ask_for` narrows it to the reader's declared dimensions, so `Ask.countries` gets `SE` and `DK` and `Ask.places` is empty for every board host, since none declares `place`. The gates then keep only the towns after the pull. A town gate therefore fetches its whole country.
|
||||
|
||||
For agent 2 this costs nothing: its `any` group has DK and SE beside the towns, so the towns are already covered. It costs where the group has towns and no country, and the country is large (Munich as DE).
|
||||
|
||||
**Which hosts could take a town instead.**
|
||||
|
||||
- Could, by what was seen: Eightfold (`location=`, its facet values are town strings; untested), Workable (`location` objects with `city`; untested body), Lever (`location=Stockholm`, live: exact text, matches any of a role's places), Adzuna (`where`, `location0..7`), Jobindex (`geoareaid`), Platsbanken (`municipality`).
|
||||
- Could, with facets: Workday if a tenant has a city facet (none seen: LEGO's is countries only, and `locationMainGroup` holds a country facet), SuccessFactors (`locationsearch=`, untested), Phenom (`location` and `locationData` in the body, untested), Jibe (a city facet is listed; the parameter is not known).
|
||||
- Cannot: SmartRecruiters has `city=` and `region=` in its documentation but it is untested; Oracle lists ten countries and sites only; Teamtailor takes ids.
|
||||
- Only the search providers do so today (`Filters(place="one")` on Adzuna, Jobindex and Platsbanken).
|
||||
|
||||
**Recommendation.** Leave towns as their country for the board hosts, and build `place` only where it saves a lot and the town is the agent's only reach:
|
||||
|
||||
1. The saving is small in the live data. For agent 2, the DK and SE reach already leaves at most 149 roles on any big board (Ericsson), so a town would cut a country read of one or two requests.
|
||||
2. Town strings are the host's own, unlike ISO codes: each host needs a name-to-string table or a facet lookup, and a town the host spells differently returns nothing and, on the closing guard, wrongly closes roles.
|
||||
3. Hosts that state their places per role (all the big ones) leave the exact match to the gates and the gazetteer, which is already right.
|
||||
|
||||
So: the plan was to carry `places` beside `countries` in the reach for a group that has towns and no country, and to pass them only to readers that declare `place`. That was not built. `Reach` (`app/jobs/reach.py`) carries countries and remote, a town in a gate is fetched as its country (`decisions.md`, Fetch terms), and the three search providers take their `place` from the saved params (`PARAMS`), never from a reach. Eightfold and Lever wait for a run that shows the town strings work.
|
||||
|
||||
## 3. What was recommended, and what landed
|
||||
|
||||
Landed: `query` on SmartRecruiters, SuccessFactors, Workday, Phenom, Eightfold and Oracle (`Filters(query="one")`). Not built: the rows below marked "no" or "later", and the rows for Workable, Teamtailor and Jibe, which declare no `query` yet.
|
||||
|
||||
### Declarations per host
|
||||
|
||||
| Host | `query` | `category` | `level` | `place` |
|
||||
|---|---|---|---|---|
|
||||
| SmartRecruiters | yes: `q`, one request per term (multi-word behaviour untested) | no: the parameter is not verified | no | no |
|
||||
| Eightfold | yes: `query` | later: `filter_function`, ids learnt from `filterDef` | no: no tenant filter seen | later: `location=<town string>` after a test |
|
||||
| SuccessFactors | yes: `q` | no: `department` gave a wrong answer for an unknown value and its values need script | no: the "Work Experience" facet is a tenant label | no |
|
||||
| Phenom | yes: `keywords` | later: `selected_fields.category` needs the value list | no: `positionLevel` is empty | no |
|
||||
| Workday | yes: `searchText` | later: `jobFamily` ids from the existing `facets` call | no | no |
|
||||
| Oracle | yes: `keyword` | later: `selectedCategoriesFacet` ids from `categoriesFacet` | no | no |
|
||||
| Workable | not built: `query` answered in the survey | no: the body is not settled (400) | no | no |
|
||||
| Teamtailor | not built: needs a baseline request that confirms the feed filters | no | no | no |
|
||||
| Jibe | not built: `keywords` | later: `categories` | no | no |
|
||||
| Lever | no: no keyword parameter; boards are small and read whole | no | no | later, as above |
|
||||
| Jobindex, Platsbanken, Adzuna | already `query` and `place`; `subid`, `occupation-field` and `category` are available for `category` | | | |
|
||||
|
||||
`level` is not worth declaring on any host: only SuccessFactors and Jibe have a facet for it, and on the tenants seen it is a tenant label or empty (Lever documents `level=` and SmartRecruiters states `experienceLevel` on each role; neither is filtered on). Seniority stays a prep matter (title rules, host word).
|
||||
|
||||
### What F2 built, and why in this order
|
||||
|
||||
The table shows how much terms can narrow big boards, from the live database of 2026-09-30:
|
||||
|
||||
| Board | Host | Open | In Europe | In DK or SE (today's reach) | Roles with a term in the title, Europe |
|
||||
|---|---|---|---|---|---|
|
||||
| Bosch | SmartRecruiters | 4,830 | 1,564 | 15 | 149 |
|
||||
| Delivery Hero | SmartRecruiters | 975 | 314 | 11 | 143 |
|
||||
| Coolblue | SmartRecruiters | 477 | 477 | 0 | 30 |
|
||||
| Wise | SmartRecruiters | 417 | 279 | 0 | 160 |
|
||||
| SAP | SuccessFactors | 807 | 365 | 6 | 128 |
|
||||
| Scania | SuccessFactors | 750 | 532 | 58 | 28 |
|
||||
| Vestas | SuccessFactors | 668 | 327 | 61 | 56 |
|
||||
| Novo Nordisk | SuccessFactors | 428 | 138 | 93 | 45 |
|
||||
| Philips | Phenom | 868 | 194 | 3 | 65 |
|
||||
| Zeiss | Workday | 848 | 264 (295 with no country) | 1 | 59 |
|
||||
| LEGO | Workday | 455 | 175 (45 with none) | 83 | 60 |
|
||||
| Ericsson | Eightfold | 618 | 210 | 149 | 64 |
|
||||
| Nokia | Oracle | 554 | 87 (54 with none) | 0 | 24 |
|
||||
|
||||
"Term in the title" is a count of the words head, director, VP, chief, CTO, CDO, data, AI, machine learning, forward deployed, solutions engineer or architect, professional services, research, innovation, principal, lead and manager in the title. It is a proxy for `fetch_terms`, and a lower bound on what a text filter returns: where the scope could be told, the keyword matched text and not only titles (SmartRecruiters, Eightfold and Oracle by the examples here; on SuccessFactors, Workday, Workable and Jibe it is not determined) (Ericsson: `software` matched a thesis title without it; Nokia: `engineer` matched 468 of 554; Bosch: 1,713 of 4,830). Expect a keyword read to return several times these counts, and a broad term like `engineer` to cut little.
|
||||
|
||||
Findings:
|
||||
|
||||
1. **Narrow reach needs no terms.** At DK and SE every big board is already at most 149 roles (Ericsson), and Bosch is two requests. Terms save requests only on the wide reach, which is the All countries tag; a company set to All roles takes no terms, and a favourite no longer widens the fetch (`decisions.md`).
|
||||
2. **The wide reach is where terms pay.** SmartRecruiters reads Europe (33 countries) as the whole board, because `MAX_SERIES` is 8 for a `one` host: Bosch is 49 requests of 100 for 1,564 European roles. One `q` request per term (about 35% of the board for `engineer`, far less for `director` or `data`) replaces it.
|
||||
3. **The order followed the size of the whole read that terms replace**, then how sure the parameter was:
|
||||
1. SmartRecruiters `q`: Bosch, Delivery Hero, Wise and Coolblue, 6,700 roles in the live DB, one live-verified parameter, AND with country.
|
||||
2. SuccessFactors `q`: SAP, Scania, Vestas and Novo Nordisk, 2,650 roles; verified. The reader already reads a country a request, so a term multiplies the series.
|
||||
3. Workday `searchText` (Zeiss 848, LEGO 455), Phenom `keywords` (Philips 868), Eightfold `query` (Ericsson 618) and Oracle `keyword` (Nokia 554): each verified once, each a small change to the request body or URL.
|
||||
4. **Category comes next, and only for Workday, Oracle, Jibe and Eightfold** (Phenom's needs its value list first). They state their own category ids in a response the reader already reads or can read (`facets`, `categoriesFacet`, `filter.categories`, `filterDef`), and the union across values is verified on Workday and Oracle. But an agent's term (`engineer`) is not a category name, so it needs a mapping from term to category per tenant that nobody has: not before terms have run on real boards.
|
||||
5. **Level is not filtered at the source.** Place is filtered by the search providers; Eightfold and Lever wait for a test that proves a town string, and the rest stay on country.
|
||||
|
||||
### Guards the fetch terms keep
|
||||
|
||||
- A term filter needs the closing guards to hold: a keyword read is not a whole board, so it does not write `ats:size` (a pull no longer rewrites a size at all) and roles outside it are not closed as gone (`narrowed` in `plugins/readers/_runner.py`, `close_boards` in `app/jobs/pull.py`).
|
||||
- Unknown-value behaviour differs: Lever returns nothing, Oracle returns everything, SuccessFactors returned something unrelated, Workable answered 400 to a body whose refused key is not known. The runner treats a facet value as known only when the host's own facet list stated it, as `series` does for ids.
|
||||
- Keywords are text matches everywhere except Adzuna's `title_only`; the agent's own scoring still decides relevance.
|
||||
|
||||
### Open, and cheap to settle with one request each
|
||||
|
||||
- SmartRecruiters: does `q="head of data"` mean all words or any, and are `department`, `function`, `experienceLevel` accepted (one request each)?
|
||||
- Teamtailor: a baseline request to confirm the feed's `query` filter.
|
||||
- Workable: which key of the survey's body was refused (`department` or `location`).
|
||||
- Eightfold: a `location` with a town string, and repeated `filter_function` values.
|
||||
- Adzuna: everything, once the key exists.
|
||||
+43
-44
@@ -1,55 +1,54 @@
|
||||
# Open
|
||||
|
||||
What is known and not yet fixed, checked against the code on 2026-09-30. What landed is in `decisions.md`; the unbuilt readers are planned in `plan-hosts-wave-2.md`.
|
||||
What is known and not yet fixed, checked against the code on 2026-10-06, after the architecture review of 2026-10-05. What was decided is in `decisions.md`, the target design in `documentation/`, the unbuilt readers in `plan-hosts-wave-2.md`. Counts are of the live database on that day.
|
||||
|
||||
## For the owner to decide
|
||||
|
||||
- **Remote on hosts that take one country a request.** SmartRecruiters, SuccessFactors and Eightfold (14 boards, about 15,000 roles) are still asked for DK, SE alone when the agents want remote, since Europe there is the whole board. Reading them whole is the only way to their remote roles elsewhere.
|
||||
- **Remote listings across countries.** A job posted remote once per country is now one role per country (the fold keeps to one country), which lists a pan-European remote posting several times.
|
||||
- **A search cut at its page cap leaves no record.** In the last pull 4 of 26 searches stopped at a cap (Jobbank and freehire at 1,000 rows, two Jobindex searches at 200). Nothing on a page says so, and Jobbank's feed is not sorted by date, so new ads can lie past the cap.
|
||||
|
||||
## What the ADRs still do not say
|
||||
|
||||
`ADR Fetch prep and score` was brought up to the code on 2026-10-06; left for the owner's read as a whole.
|
||||
|
||||
`ADR Plugins` was rewritten on 2026-10-06 and names a target the code has not reached: a broken plugin left out instead of stopping the app; providers declaring their params with label, kind and choices, so form, CLI and API follow from the module; detail state kept for a search's roles as for a board's; a national provider declaring its implied country; page markers for hosts on a company's own domain; a contract test that names no reader.
|
||||
|
||||
## Structure
|
||||
|
||||
From the review of the plugin layer and the inbound flow.
|
||||
|
||||
- **Per-target state is string keys written and read in six files** (`_runner.stamp`, `ats.py`, `pull.py`, `limits.py`, `reach.py`, `api.py`). One typed accessor in the runner would own them.
|
||||
- **Detail handling is spread over five places** (`ats._board`, `_runner.read_details`, `ats.read_board_details`, `fetch_text.plan`, and none of it in `search.py`), which is why a search role's detail places are not kept across pulls.
|
||||
- **`plugins/ats.py` holds three things:** the source, compatibility names only tests use (`_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused`, `DETAIL_FIELDS`), and the app's function classifier (`FUNCTIONS`, `classify`), which search roles never pass through.
|
||||
- **`plugins` is not in the layering test's trees,** so `app` imports the private `_runner` in four files and single readers in two, and `models.py`, `config.py` and `dupes.py` reach the registry through a path hack.
|
||||
- **The two sources build an item each their own way** (`ats.py`, `search.py`) and already differ.
|
||||
- **The contract test names readers** (`POSTED_BY_DETAIL`, `SEARCH_PARAMS`), so a new reader that dates by detail edits the shared test.
|
||||
- **`has_key` re-implements `resolve_key`,** and `search_params` raises `KeyError` for a provider param other than q, area, country, where, region or remote.
|
||||
- **`pipeline/` only logs when a cascade's next stage fails to start** (`queue.py`); the fix lands in the boilerplate first. `pipeline/`, `auth/`, `deploy/` and `desk/` lag their upstreams.
|
||||
|
||||
## Text
|
||||
|
||||
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only `summarise` reads. `embed` and `extract` read `description or teaser`, so a Jobindex role keeps being ranked on its teaser. The fix is one text rule, `article_text or description or teaser`, in embed and extract, and queuing embed and score after a fetch.
|
||||
- **An edited posting is never embedded again.** The vector is dropped only when the text ranks better than before (a title-only role gaining a teaser, or a teaser replaced by a description). A hash of the embedded text on `vectors` would re-embed on any change.
|
||||
- **Extract marks a title-only role done for good.** `upsert_role` does not reset `extracted_at` when text arrives. Extract is on, so this bites. `_EXTRACT_WAITING` re-reads roles whose `seniority_model` is empty, which covers level and not pay.
|
||||
- **Lever keeps only the intro paragraph.** `lever.py` reads `description` and drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way.
|
||||
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 17.9% of open roles. Extract can return `salary_min` and `salary_max` but has read 554 roles so far.
|
||||
- **A title-only role has no mark in the feed.** 1,180 open roles have no description, teaser or fetched posting: 677 of them Workday, 256 SmartRecruiters, 150 Workable, 57 SuccessFactors and 26 Eightfold. A mark would show which fits are guesses.
|
||||
|
||||
## Ranking
|
||||
|
||||
- **The owner's agent has no rule on seniority.** Fit compares subject, not level. The ladder and the rail's From/To/Track filter exist (`LADDER` in `app/models.py`), and the level is filled for about half of open roles. A gate still names words (`principal`, `senior_manager`, `director`, `exec`); a gate on "at least a rung" would read as the rail's filter does.
|
||||
- **Votes train `taste` only.** The liked-minus-passed centroid weighs 0.2 on both agents. Feeding votes into the prompt vectors is worth doing once there are a few hundred.
|
||||
- **A starred company's roles crowd the top of Agent match.** With the companies weight at 0.2 a favourite's roles gain about 0.13; after starring Databricks it held 33 of the top 60 rows. Lower the weight on the agent's card, or star fewer companies.
|
||||
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only summarise and the judge read.
|
||||
- **An edited posting is not embedded again** unless its text ranks better than before (title, teaser, description). A hash of the embedded text would catch any change.
|
||||
- **Lever keeps only the intro paragraph** (`lists[]` and `additional` are dropped), and Recruitee drops `requirements`.
|
||||
- **Pay stated only in the text is not in the fields** unless extract has read the role.
|
||||
- **A role ranked on its title alone has no mark in the feed.**
|
||||
|
||||
## Sources
|
||||
|
||||
- **The boilerplate moved on.** `pipeline/` equals boilerplate `a1c6f0d` once line endings are ignored: 87 files of the working tree have CRLF although the index has LF, so a plain `diff -r` shows every file as different (use `--strip-trailing-cr`). `auth/` lacks `c6a90ff`, the phone commit (`auth/README.md`, `auth/static/account.js`, `auth/templates/login.html`). `deploy/` is at `1679e88` and lacks the WSL scripts of `e1b3746` (`wsl-setup.sh`, `wsl-setup.ps1`), and `docs/release.md` lacks its `wsl.md` sentence. `desk/` is news `a275c38` less the prune and lags news by four commits (`b0e1f6f`, `f01a6bd`, `3bb6775`, `7b9c8a2`). The three languages of boilerplate `7bb7da2` are starter code in `app/`, not part of the copies. Refresh with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
|
||||
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
|
||||
- **Six hosts' dates are not trusted.** Eightfold, Jibe, Gem, Deel, HiBob and HR Manager declare `posted="listed"`, so their backlog counts as posted when first seen. Eightfold now maps `creationTs`, Jibe's `posted_date` equals the page's `datePosted`, and HR Manager's `pubDate` rises with the id; each needs one check on a live board before it declares `posted="trusted"`. Platsbanken's local time is read as UTC.
|
||||
- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board.** All three answer an empty listing for an unknown id and declare `empty="suspect"`. The 10% guard keeps a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
|
||||
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the runner waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
|
||||
- **A SuccessFactors site on a company's own domain is not discovered.** Its pages link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
|
||||
- **HR Manager is attached although its robots.txt disallows reading.** `hrmanager:teknologisk` (the Danish Technological Institute) has 7 open roles, while `READS.robots` says "disallowed by robots.txt; owner to decide". Seaborg stays unattached until the owner decides whether the feed counts as allowed.
|
||||
- **Whether SmartRecruiters' identifier is case-sensitive is unsettled.** The module docstring says it is (`BoschGroup`), `decisions.md` says it is not, checked live; the live boards hold the identifiers with case. A live request with a lowercased identifier settles it.
|
||||
- **The seed lags the live database.** The seed has 286 boards and the database 310: it lacks Microsoft, Infineon, Novonesis, Darktrace, Criteo, NVIDIA, HR Manager and seventeen more, and names five of them as `unread` or `robots`. `app companies seed` never replaces a board, so the seed is brought in line by hand.
|
||||
- **Stripe's places carry a "Locations" suffix.** 316 open roles (285 without copies) have place text like "Singapore Locations" or "Dublin; Ireland Locations".
|
||||
- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board**, and nine boards have never listed a role. `set_board` keeps a board that lists nothing.
|
||||
- **Six hosts' dates are not trusted** (Eightfold, Jibe, Gem, Deel, HiBob, HR Manager declare `posted="listed"`); each needs one check on a live board. Greenhouse declares `trusted` and falls back to `updated_at`.
|
||||
- **The hosts added on 2026-09-29 were read from this machine only;** read one Workday and one SuccessFactors board from the deployed server.
|
||||
- **robots.txt decisions wait for the owner:** SmartRecruiters' API host disallows `/` for every agent but LinkedInBot and is read meanwhile; HR Manager is attached although its record says "owner to decide". Whether SmartRecruiters' identifier is case-sensitive is unsettled.
|
||||
- **`Crawl-delay` is not honoured.**
|
||||
- **A SuccessFactors site on a company's own domain is not discovered.**
|
||||
- **freehire states seniority and employment and the reader declares neither.** Stripe's places carry a "Locations" suffix.
|
||||
- **The 10% guard compares only reads with the same filter,** so the first read after an agent's locations change is unguarded.
|
||||
|
||||
## The pages
|
||||
|
||||
- **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite.
|
||||
- **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it.
|
||||
- **`company_key` keeps `ø`.** Aliases make "Orsted" and "Ørsted" one company; folding the key itself would change the primary key of `companies`.
|
||||
|
||||
## From the review of 2026-09-30
|
||||
|
||||
- **Remote roles are not read from hosts that cannot filter on remote.** On SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" is asked as "DK, SE"; since 2026-09-30 the filtered read keeps the remote roles it has and the line says "remote not asked", but new remote roles on those 21 boards are never fetched, and the 72 closed on 2026-09-29 and 30 (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3) reopen only if a read lists them. Reading such boards whole is the owner's call; Bosch is about 70 requests a pull.
|
||||
- **SmartRecruiters' API host disallows `/` for every agent but LinkedInBot** (read 2026-09-30), which covers the listing and the detail requests; `READS.robots` says "owner to decide" and the eleven boards are read meanwhile. Jobylon allows `*` but disallows named job crawlers (JobrapidoBot, JoobleBot, AdzunaBot); ours is not one of them. HR Manager's record says the owner decided on 2026-09-29 to read the API host; confirm it.
|
||||
- **`has_key` re-implements `resolve_key`**, and the form hard-codes the labels of Jobindex, Adzuna and Platsbanken; since 2026-10-03 the CLI, dupes, `search_params` and the other providers in the form come from `readers.PROVIDERS`.
|
||||
- **A search role's detail places are not kept across pulls.** `plugins/search.py` has no `_KEPT_FROM_DETAIL`, so the next pull may overwrite the town Jobbank's `detail` added; the text is kept. Per-role detail state of a search is never pruned, since a search never lists everything.
|
||||
- **Jobbank's JSON-LD `addressLocality` is sometimes not a town** ("Ørsted" on an Ørsted ad), and the feed's regions (`Øresundsregionen`, `Udlandet (øvrige)`) will show under Unmapped.
|
||||
- **freehire states seniority and employment type, but the reader declares neither**, since a stated word needs a table in `app/prep/hosts/`. Jobnet keeps `utm_*` on employer links; dupes match on company, title and town, so it does not matter there.
|
||||
- **`ats.py` keeps compatibility names that only tests reach for:** `_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused` and `DETAIL_FIELDS`.
|
||||
- **`title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`.** Wave 2's WP9 must own `_runner.py` (`plan-hosts-wave-2.md`).
|
||||
- **Some `posted="trusted"` declarations rest on the detail read** (Workday, BambooHR, Rippling) **or on a fallback the host moves** (Greenhouse `updated_at`, Phenom `postedDate`).
|
||||
- **Dupes fold same-host roles with the same title and town without comparing text.** 99 open copies share under 30% of their words with the kept role.
|
||||
- **Fetch text never reads a role outside the cut.** A role without text is ranked on its title until it enters the cut.
|
||||
- **A pull never re-measures a board's size.** Recheck is the only way, so a board that grows past 300 roles stays read whole until someone rechecks it.
|
||||
- **Fetch terms are configured on no agent**, so no board is read with terms today.
|
||||
- **`SMALL` (300), `STALE_DAYS` (21), the guard's 10% and `DETAIL_CAP` (200) are code constants**, and the pages hard-code 300 in tooltips.
|
||||
- **`The todo list.md` holds items the code has done** (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.
|
||||
- **★ means two things on Roles:** on a row it saves the role, in the company strip it marks a favourite.
|
||||
- **All locations and All roles answer without a score job id,** so the company view is stale until the rescore has run and the page is reloaded.
|
||||
- **Place suggestions split every place line on each request.**
|
||||
- **`company_key` keeps `ø`;** aliases make "Orsted" and "Ørsted" one company.
|
||||
@@ -16,7 +16,7 @@ In the seed (`seed/companies.json`, 362 companies, 286 with a board), 76 compani
|
||||
| A host that is read, with a gap | Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
|
||||
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
|
||||
|
||||
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures; `own-1.md` and `own-2.md` name `out-own-1.json` and `out-own-2.json`, which do not exist, and `survey-own.json` is the whole record.
|
||||
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures, and `survey-own.json` is the whole record.
|
||||
|
||||
## Decisions to take first
|
||||
|
||||
@@ -30,14 +30,14 @@ Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tes
|
||||
|
||||
| WP | Model | Builds | Needs |
|
||||
|---|---|---|---|
|
||||
| 9 Provisional title | Opus | `title` in `DETAIL_FIELDS`, `upsert_role` keeping a stored title over a provisional one, and their tests; owns `plugins/readers/_runner.py`, `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | `plan-remote.md` WP-A |
|
||||
| 9 Provisional title | Opus | `title` in `DETAIL_FIELDS`, `upsert_role` keeping a stored title over a provisional one, and their tests; owns `plugins/readers/_runner.py`, `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | |
|
||||
| 10 Sitemap reader | Sonnet | `sitemap`: role URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
|
||||
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
|
||||
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
|
||||
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
|
||||
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
|
||||
|
||||
WP9 owns files that `plan-remote.md` WP-A also owns (`app/storage/`, `plugins/`, `tests/`) and `plan-review.md` R1 edits (`_runner.py`, `ats.py`), so it waits until those have merged. WP12 to WP15 own new modules only and can start at once and in parallel. WP10 waits for WP9.
|
||||
WP12 to WP15 own new modules only and can start at once and in parallel. WP10 waits for WP9.
|
||||
|
||||
### WP10 notes: Mambu and Arm
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Own-site batch 1: how each career site can be read
|
||||
|
||||
31 companies, all classified from live responses (2026-09-29). Full data is in `out-own-1.json`.
|
||||
31 companies, all classified from live responses (2026-09-29).
|
||||
|
||||
## Counts per class
|
||||
|
||||
@@ -57,4 +57,4 @@ Discovery misses worth fixing in the app: Lever EU (`api.eu.lever.co`), custom d
|
||||
|
||||
## Method notes
|
||||
|
||||
One request per second per host, `curl -m 20 -L`, browser User-Agent, no login. The Algolia call for new-work used a Referer header matching the site that ships the key. Scratch files are in `wo1/` next to this note.
|
||||
One request per second per host, `curl -m 20 -L`, browser User-Agent, no login. The Algolia call for new-work used a Referer header matching the site that ships the key.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Own-site career pages, batch 2 (30 companies)
|
||||
|
||||
Every request was a live curl on 2026-09-29. Machine-readable results are in out-own-2.json.
|
||||
Every request was a live curl on 2026-09-29.
|
||||
|
||||
## Counts per class
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Architecture
|
||||
# ADR Fetch prep and score
|
||||
|
||||
The target design of roledesk's ranking: how a role gets from a board to the Roles feed for one search agent, and the key decisions behind it. What gets built is tested against this document.
|
||||
|
||||
- **Fetch gate**: the agent's hard rules, a yes or no per role, in four parts: world, location, role exclusion and role fetch.
|
||||
- **Analytics score** and **analytics filter**: what decides the **cut**, the roles that pass the analytics filter and get fetch text and the model pass.
|
||||
- **Feed scores**: the two orders of the feed, the want feed score and the match feed score.
|
||||
- **Input analytics score** and **input analytics filter**: what decides the **cut**, the roles that pass the analytics filter and get fetch text and the `model analytics score`, which is the `CV match` (4.2.1).
|
||||
- **Feed scores**: the two scores that order the feed, the want feed score and the match feed score; Newest is the third order.
|
||||
- **Fetch text**: the second HTTP read of a role's own page, without a model.
|
||||
- **Extract**, **judge**, **summarise**: the model calls (Haiku).
|
||||
- **Affinity**: an agent's setting for a company: hidden, neutral, liked or favourite.
|
||||
@@ -17,18 +17,19 @@ The target design of roledesk's ranking: how a role gets from a board to the Rol
|
||||
world · location · role exclusion · role fetch, with All locations and All roles as overrides
|
||||
company boards: whole under 300 roles, else filtered at the host where it can
|
||||
search providers: their own saved query
|
||||
then prep, no tokens: host words, places, remote, country; the copies folded
|
||||
│
|
||||
▼
|
||||
2 Analytics score, no tokens ─ prompt closeness, CV closeness, affinity → the cut
|
||||
2 Input analytics score, no tokens ─ prompt closeness, CV closeness, affinity → the cut
|
||||
│
|
||||
▼
|
||||
3 Fetch text ─ the role's own page, for cut roles without text
|
||||
│
|
||||
▼
|
||||
4 Model pass ─ extract the cut → fetch gate and cut again → judge the cut
|
||||
4 Model analytics score ─ extract the cut → fetch gate and cut again → judge the cut
|
||||
│
|
||||
▼
|
||||
5 Feed ─ every role that passes the fetch gate, in two orders;
|
||||
5 Feed ─ every role that passes the fetch gate, in three orders;
|
||||
filters and saved filters narrow it
|
||||
│
|
||||
▼
|
||||
@@ -51,7 +52,7 @@ The stricter location filter. A role passes the `location fetch gate` when it ma
|
||||
|
||||
#### 1.1.3 Role exclusion gate
|
||||
|
||||
A list of titles to leave out: phrases that reject a role by its title. No host can filter by exclusion, so this gate is applied to every role after fetch.
|
||||
A list of titles to leave out: phrases that reject a role by its title, matched on whole words. No host can filter by exclusion, so this gate is applied to every role after fetch.
|
||||
|
||||
#### 1.1.4 Role fetch gate
|
||||
|
||||
@@ -64,83 +65,101 @@ The fetch terms: words a big board's host must match before it sends a role. The
|
||||
- A search provider is read by its own saved query, which replaces the `role fetch gate` (1.1.4).
|
||||
- A hidden company never passes.
|
||||
|
||||
The gate rejects on evidence only. A role with an unknown location passes until extract has read it.
|
||||
The gate rejects on evidence only. A role with an unknown location passes until extract has read it. A location is unknown when no place of the role resolved and no country is stated, an unresolved place text included. A place that holds the agent's places (Øresund or Europe for Denmark) is no evidence against the role, unless the role also names a place elsewhere. A role in several places passes when one is inside. A hybrid or onsite role at an unknown location passes. A role whose text says remote while its fields do not (1.5) passes the location gate until extract has read it. The world gate reads a remote role by its remote area.
|
||||
|
||||
### 1.2 Reading a board
|
||||
|
||||
- A company board with fewer than 300 roles is read whole.
|
||||
- A company board with 300 or more roles is asked for the fetch gate at the host: the world and location gates, and the fetch terms, where the host can filter on them. A host that cannot filter on a part sends the board whole for that part.
|
||||
- A board's size is measured once, when it is added or rechecked, not on every read.
|
||||
- One fetch serves all enabled agents. The host is asked for the union of their world and location gates and the union of their fetch terms; one enabled agent without fetch terms means no terms. Each agent then applies its own gate to what arrived.
|
||||
- One fetch serves all enabled agents. The host is asked for, per agent, its location gate, or its world gate when it sets no location, united over the agents, and for the union of their fetch terms; one enabled agent without fetch terms means no terms. Each agent then applies its own gate to what arrived.
|
||||
- A host that filters on countries but not on remote is asked for the world gate's countries when an agent wants remote roles, so a remote role listed under any country in the world gate still arrives; the read says that remote was not asked. This holds where the host takes the world in one read; on a host that takes one country a request, the world is the whole board, so the location gate is asked and remote roles elsewhere are not fetched.
|
||||
- A board's own saved params win over the agents' gates on the dimensions they set. With no enabled agent every board is read whole.
|
||||
|
||||
### 1.3 Closing
|
||||
|
||||
- A role closes when a complete read of its board no longer lists it.
|
||||
- A read that is not complete closes nothing: one narrowed by fetch terms, a search provider's, a partial or failed read. Roles on such boards close by age. This is accepted because ranking relies on recency, not on closing.
|
||||
- A read that is not complete closes nothing: one narrowed by fetch terms, a search provider's, a partial or failed read. Roles on such boards close by age. This is accepted because ranking relies on recency, not on closing. A read filtered by location is complete for what it asked: it closes the roles inside its filter that it no longer lists.
|
||||
- Switching a board or search off closes its roles. A closed role that a read lists again reopens. A search is read 5 pages deep, 10 at most.
|
||||
- A role that no read has listed for 21 days closes. A board that has failed on every read for 21 days closes its roles.
|
||||
- A liked or saved role is never closed by a filtered read.
|
||||
|
||||
### 1.4 Copies
|
||||
|
||||
- One posting listed several times (by a search provider and by the company's board, or by one board under several ids) is one role. Two listings are one posting when they share company, title and country, and their towns overlap; a listing without a town matches any town.
|
||||
- The kept copy is the company board's over a search provider's, then the one with text, then the earliest posted. Only the kept copy is scored, read by a model and shown; it lists the others.
|
||||
- The kept copy is the company board's over a search provider's, then the one with text, then the earliest posted. Only the kept copy is scored, read by a model and shown; it lists the others. Votes and read state move to the kept copy.
|
||||
- A host that posts one job once per office gives listings of one title whose towns never meet. Those of one host are one role when their texts share 30% of their words, within one country, remote and office kept apart; the kept copy carries every office's place.
|
||||
|
||||
## 2 Analytics score and filter
|
||||
### 1.5 Prep
|
||||
|
||||
After fetch, without a model: a host's words for seniority, employment and remote map through a table per host; each place text resolves against a gazetteer of towns, regions, countries and continents, where a region may lie in two countries; remote and the remote area are read from the place words; the country comes from the resolved places; a role whose title or text says remote ("fully remote", "work from home", not "no remote") is marked as possibly remote for the gate (1.1). A text nothing maps is listed under Unmapped for the owner to add, and until then the role's location counts as unknown.
|
||||
|
||||
## 2 Input analytics score and filter
|
||||
|
||||
#### 2.1 Closeness
|
||||
|
||||
Each role is embedded and scored semantically:
|
||||
|
||||
- **Prompt closeness** is its level against the best of the agent's prompts.
|
||||
- **CV closeness** is its level against the closest part of the CV.
|
||||
|
||||
Both are calibrated levels from 0 to 1, the same in every run. A liked company adds 0.05 to both levels, a favourite 0.1. The bonus counts before the count and the minimum apply.
|
||||
Both are calibrated levels from 0 to 1. A company's affinity adds to both levels the share it has in the `want feed score`: the company like weight times the company like's rise above neutral (5.1.1), over the sum of all four want feed score weights, also when one of them has left the sum. On the start weights a liked company adds about 0.056, a favourite about 0.111. The bonus counts before the count and the minimum apply.
|
||||
|
||||
The levels drive the `analytics filter`. A role passes it when it passes the fetch gate (1.1.1 to 1.1.3) and:
|
||||
The embedding is `intfloat/multilingual-e5-small`, with its query and passage prefixes, over the title and the first 2,000 characters of the text, else the teaser. The CV is cut into chunks of at most 1,200 characters and the closest chunk counts. A level is a linear clip of the cosine between two bounds measured per model: prompt closeness 0.75 to 0.88, CV closeness 0.77 to 0.90, job like -0.14 to 0.19. With several prompts each is scaled on its own and the best counts.
|
||||
|
||||
- is among the best N by prompt closeness, each at least X (500 and 0.6 to start), or
|
||||
- is among the best M by CV closeness that the first list leaves out, each at least Y (150 and 0.6 to start), or
|
||||
- is liked or saved, whatever its levels.
|
||||
#### 2.2 Input analytics filter
|
||||
|
||||
The `closeness`es are the main input for the `input analytics filter`, which determines which roles get `fetch text` and the `model analytics score`. A role passes the `input analytics filter` when it:
|
||||
|
||||
- passes the `fetch gate` (1.1.1 to 1.1.3), and
|
||||
- is among the best N by `prompt closeness`, each at least X (500 and 0.6 to start), or
|
||||
- is among the best M by `CV closeness` that the first list leaves out, each at least Y (150 and 0.6 to start), or
|
||||
- is liked by this agent or saved under any agent, whatever its `closeness`.
|
||||
|
||||
The two lists are kept apart because they answer different questions (what I want, what I am), and a merged number lets the stronger side crowd the other out. For a broad agent the count stops each list; for a narrow agent the minimum stops it, so roles that do not fit are not read just to fill the count.
|
||||
|
||||
Newness and job likes are not part of the `analytics score`. A role is read once, pulls run every few hours, and votes come after the read.
|
||||
Newness and job likes are not part of the `input analytics score`. A role is read once, pulls run every few hours, and votes come after the read.
|
||||
|
||||
An agent without a CV has no CV list and no judge. An agent without prompts has no prompt list and no prompt closeness in the want feed score.
|
||||
|
||||
The `analytics filter` determines which roles get `fetch text` and the model pass.
|
||||
|
||||
## 3 Fetch text
|
||||
|
||||
Some job boards have one API for the listings and another for the job specs. `fetch text` reads the role's own page for every role in the cut that came without text. Once the full text is fetched, the role is embedded again and its levels recomputed. A role that enters the cut after this recompute gets its text on the next pull.
|
||||
Some job boards have one API for the listings and another for the job specs. `fetch text` reads the role's own page for every role in the cut that came without text, on hosts that have a role page, at most 200 a board and 1,000 a run, a failed read retried after 7 days; Jobindex is never read in a pull, since each read counts as a click there. Once the full text is fetched, the role is embedded again, its levels recomputed, and extract reads it again if it had read the title alone. A role that enters the cut after this recompute gets its text on the next pull.
|
||||
|
||||
## 4 Model pass
|
||||
## 4 Model analytics
|
||||
|
||||
All roles in the cut go through the detailed analysis and metadata extraction. Extract and judge are two calls: `extract` needs no CV, it is faster, and it is the final step of the ingestion, so it decides what finally passes the `analytics filter`.
|
||||
All roles in the cut go through the detailed analysis and metadata extraction. Extract and judge are two calls: `extract` needs no CV, it is faster, and it is the final step of the ingestion, so it decides what finally passes the `analytics filter`. All three calls go to Haiku (`claude-haiku-4-5`), extract 20 roles a call, judge 5, summarise 10. A stage that is switched off, or has no model to ask, is skipped, and the Runs page says so. A cut role without text is neither judged nor summarised.
|
||||
|
||||
### 4.1 Extract
|
||||
|
||||
Often information is only in the free text, e.g. that a role in a specific location is actually remote. So every role has raw and processed fields for e.g. salary, location, remote and seniority. Extract reads what is hidden in the free text. A field the board states wins over what extract reads, except seniority and remote, where extract wins.
|
||||
|
||||
**Extract** reads the cut first. It fills location, remote area, seniority, languages and deadline.
|
||||
**Extract** reads the cut first, the first 2,500 and last 1,500 characters of each role. It fills location, remote and remote area, seniority, languages, deadline, pay and currency, function and work permit. A role the model leaves unanswered three times is given up.
|
||||
|
||||
The fetch gate and the cut are then applied again with what extract filled. A role can drop out, such as one that turns out to be outside the agent's locations, or enter, such as one that turns out to be remote. A role that enters is extracted on the next pull.
|
||||
The fetch gate and the cut are then applied again with what extract filled. A role can drop out, such as one that turns out to be outside the agent's locations, or enter, such as one that turns out to be remote. A role that enters is judged in this pull and extracted on the next.
|
||||
|
||||
### 4.2 Judge
|
||||
|
||||
Once extract has completed the ingestion, `judge` reads each role in the cut against the agent's CV, languages, likes and dislikes. It outputs:
|
||||
Once extract has completed the ingestion, `judge` reads each role in the cut, up to 8,000 characters of it, against the agent's CV, languages, prompts, likes and dislikes. It outputs:
|
||||
|
||||
- the `CV match`
|
||||
- one to three strengths and gaps
|
||||
- a verdict on each must-have
|
||||
|
||||
Each role is extracted once. A judgement is redone only when its input changes: a change to the CV, a prompt, or the like or dislike text re-judges the whole cut (about an hour for 650 roles); a changed posting re-judges that role only.
|
||||
Each role is judged once. A judgement is redone only when its input changes: a change to the CV, the languages, a prompt, or the like or dislike text re-judges the whole cut (about an hour for 650 roles); a changed posting re-judges that role only.
|
||||
|
||||
#### 4.2.1 CV match
|
||||
|
||||
Judge rates how closely the job spec matches the experience in the CV on three dimensions, skills, experience and career, each from 0 to 5. The **CV match** is the three numbers weighted as given in the agent settings (30/25/30 to start). A must-have the role breaks caps career at 2. Roles with the same CV match are ordered by CV closeness.
|
||||
Judge rates how closely the job spec matches the experience in the CV on three dimensions:
|
||||
|
||||
- skills, where a language the posting requires above the level spoken counts as a missing skill,
|
||||
- experience,
|
||||
- career, rated against the prompts, likes and dislikes rather than the CV,
|
||||
|
||||
each from 0 to 10. A model's rating is an estimate, not a measurement, so a coarse scale serves better than 0 to 100. The **CV match** is the three numbers weighted as given in the agent settings (30/25/30 to start), times 10, so 0 to 100. Roles with the same CV match are ordered by CV closeness.
|
||||
|
||||
## 5 Feed
|
||||
|
||||
The feed shows every role that passes the `world fetch gate`, the `location fetch gate` and the `role exclusion gate` (1.1.1 to 1.1.3), loaded as you scroll, in every order, newest first included. No order shows a role the fetch gate rejects. A hidden company's roles are never shown.
|
||||
The feed shows every open role, kept copies only, that passes the `world fetch gate`, the `location fetch gate` and the `role exclusion gate` (1.1.1 to 1.1.3), loaded as you scroll, 60 at a time, in every order, newest first included. No order shows a role the fetch gate rejects. A hidden company's roles are never shown. Opening a role or voting on it marks it read; Show read, on by default, hides the read ones when switched off. Saving a role (★) also likes it.
|
||||
|
||||
### 5.1 Feed scores
|
||||
|
||||
@@ -151,10 +170,11 @@ The `want feed score` consists of `prompt closeness`, newness, company likes and
|
||||
- `newness` counts from the posting date, the earlier of when the app first saw the role and the date the board states (only for boards whose dates are reliable), and halves every 14 days.
|
||||
- `company like` is 1/3 for a neutral company, 2/3 for a liked one and 1 for a favourite.
|
||||
- `job like` is described in 5.1.3.
|
||||
- A role without an embedding yet is scored without `prompt closeness` and `job like`; their weights leave the sum.
|
||||
|
||||
#### 5.1.2 Match feed score, "What fits me"
|
||||
|
||||
The `match feed score` orders the judged roles by `CV match`, then the rest by `CV closeness` (embedding only, as they had no model pass).
|
||||
The `match feed score` orders the judged roles by `CV match`, then the rest by `CV closeness` (embedding only, as they had no `model analytics score`).
|
||||
|
||||
#### 5.1.3 Job likes
|
||||
|
||||
@@ -162,6 +182,7 @@ The user likes or passes on roles in the feed, per agent. Each role has an embed
|
||||
|
||||
- With only likes, the direction is the mean of the liked roles; with only passes, away from the mean of the passed roles.
|
||||
- With no votes, `job like` drops out and its weight leaves the sum.
|
||||
- A role's own vote also counts directly when the feed is shown: a liked role's want feed score is multiplied by 1.2 (at most 1), a passed role's by 0.5. A vote never hides a role.
|
||||
|
||||
### 5.2 UI
|
||||
|
||||
@@ -174,16 +195,17 @@ The user likes or passes on roles in the feed, per agent. Each role has an embed
|
||||
|
||||
- The Roles page says that the feed is already filtered by the agent's fetch gate, location included.
|
||||
- The company view says what is filtered: which parts of the fetch gate apply to this company, for example "every role in Europe, not filtered by country" for an All locations company. Looking up a company never silently hides roles elsewhere.
|
||||
- Show gated lists the roles the fetch gate rejects, each with the reason, so what is kept out can be checked. It is the one way a rejected role is shown.
|
||||
|
||||
## 6 After the ranking
|
||||
|
||||
Summarise is not part of any score. It runs after the feed scores, on the top N of the `want feed score` and the top N of the `match feed score` (N set per agent, 30 to start, so at most 60 roles), and on every role liked or saved.
|
||||
Summarise is not part of any score. It runs after the feed scores, on the top N of the `want feed score` and the top N of the `match feed score` (N set per agent, 30 to start, so at most 60 roles), and on every role liked or saved. It writes about 60 words and 3 to 8 tags, once per role, and needs the role's text.
|
||||
|
||||
## Settings
|
||||
|
||||
Per agent, on the form:
|
||||
|
||||
- name and prompts
|
||||
- name, an On/Off switch, and prompts
|
||||
- the fetch gate: world (the outer countries), location (countries, towns, remote), role exclusion (titles to leave out), role fetch (fetch terms)
|
||||
- CV, languages, likes and dislikes
|
||||
- the analytics filter: count and minimum by prompt closeness, count and minimum by CV closeness
|
||||
@@ -191,3 +213,5 @@ Per agent, on the form:
|
||||
- summarise size
|
||||
|
||||
Per agent, per company: affinity (hidden, neutral, liked, favourite). Per company, for every agent: All locations and All roles.
|
||||
|
||||
The pull runs every 6 hours.
|
||||
@@ -0,0 +1,282 @@
|
||||
# ADR Plugins
|
||||
|
||||
What a plugin has to deliver so that the flow in `ADR Fetch prep and score` gets every role a source lists, and what is deliberately left to the rest of the app. A new plugin is written, and an existing one tested, against this document. The API at the bottom is enough to write one.
|
||||
|
||||
## Terms
|
||||
|
||||
- **Source**: a site that lists roles.
|
||||
- **Plugin**: one module for one kind of source. There are two kinds:
|
||||
- a **host**, which reads a company's own list on an ATS (Workday, Greenhouse, ...). That list is a **board**, attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
|
||||
- a **provider**, which runs a **search**, a saved query on a site that lists roles across companies (Jobindex, Jobnet, ...).
|
||||
- **Runner**: the shared code that calls a plugin. It asks, pages, retries, guards, validates and stamps, so no plugin repeats that. It has one driver per kind, for boards and for searches, which differ only in what they store.
|
||||
- **Row**: one role as a plugin hands it over, in the host's own words.
|
||||
- **Declaration**: what a plugin states it can do (`READS`). The runner acts on the declaration, never on guesses.
|
||||
- **Read**: one pull's requests to one target, and what they yielded. A read is **complete** when it listed every role the target has, else **incomplete**; the runner stamps which.
|
||||
|
||||
## 1 What a plugin delivers
|
||||
|
||||
A plugin turns one source into rows. It:
|
||||
|
||||
- hands over every role the source lists for what it was asked, one row per listing, as the host lists them;
|
||||
- asks the host only for the filters it was handed, by building them into the host's query;
|
||||
- keeps the host's own words for places, remote, seniority, employment and country;
|
||||
- declares truthfully what the source can do: how the list arrives, where the text is, how completeness is known, what it can filter and which fields it states;
|
||||
- raises on anything it cannot read: an error page, an answer of another shape, a page that is not the page it asked for. An empty list is a valid answer only when the host says the list is empty.
|
||||
|
||||
A plugin never:
|
||||
|
||||
- filters, drops or reorders rows; a listing it cannot make a row of (no id, no title or no link) it hands over as far as it can, and the runner drops and counts it (3.8);
|
||||
- maps a word to the app's vocabulary;
|
||||
- merges, splits or multiplies rows;
|
||||
- decides closing, retries or failure;
|
||||
- reads robots.txt at run time;
|
||||
- logs in, runs a page's script, or uses a key taken from a site's script.
|
||||
|
||||
## 2 The five cases
|
||||
|
||||
A source can present roles in five ways that need handling. Three are the plugin's job and two are the app's, after the plugin.
|
||||
|
||||
| # | Case | Who handles it |
|
||||
|---|---|---|
|
||||
| 1 | The list arrives in pages | Plugin, with the runner |
|
||||
| 2 | The list has no text; each role's text is on its own page | Plugin, with the fetch text stage |
|
||||
| 3 | One role states several places | Plugin |
|
||||
| 4 | One job is listed several times | The app |
|
||||
| 5 | Facts such as remote are only in the free text | The app |
|
||||
|
||||
### 2.1 Pagination (plugin, with the runner)
|
||||
|
||||
The plugin says how to ask for one page and what the page holds. The runner does the rest.
|
||||
|
||||
- The plugin declares `listing="paged"`, the page size and the host's **ceiling**, the offset past which the host answers nothing whatever is asked (Workday 2,000).
|
||||
- `page` asks for one page from a cursor and returns the page's rows, the raw results with their ids, the total the host states if any, and the next cursor, or none after the last page. The first call gets cursor none. A cursor is any JSON value; a tuple comes back as a list when the runner hands it over again (a sweep), and a plugin reads it by position.
|
||||
- The runner reads the pages one after another, 0.5 s apart, at most 100 pages a pull. A board above 100 pages is **swept**: each pull continues from the saved cursor, series and ids, and the sweep is finished when the read reaches the end.
|
||||
- **Completeness**: a read is complete when the distinct ids read equal the stated total (`complete="stated"`), or when the plugin hands over the last page with no cursor (`complete="end"`: an empty page, or a short one where the host has no empty page). A read that reaches the ceiling is incomplete, is not swept and closes nothing; only a filter brings such a board under the ceiling.
|
||||
- **Stalls**: a page that brings no new id while the plugin hands over a cursor is a stall when the host states no total, or when the page came empty: the host ignores the cursor, or sent a page again. A plugin that sees a page come again (the same first id as the page before) hands it over with no results and a cursor that is not none, never none, since none would read as the last page. A stalled read is incomplete. With a stated total, a page of ids already read is not a stall, since a resumed sweep starts on one when roles were added at the top since.
|
||||
- A role of a swept board closes only when two finished sweeps in a row missed it, since roles closing between pulls shift later roles back past the cursor.
|
||||
- A 429 after the first page ends the read. What was read is kept, the read is incomplete, and a board continues from there in the next pull as a sweep.
|
||||
- A plugin whose list comes in one answer declares `listing="whole"` and `complete="whole"`; a whole answer that parses is complete. One whose list comes in one answer that states a total it does not fill raises.
|
||||
|
||||
### 2.2 List without text (plugin, with the fetch text stage)
|
||||
|
||||
Some hosts list title, place and date only, and state the text on each role's own page.
|
||||
|
||||
- The plugin declares `text="detail"` and supplies `detail`, which reads one role's page and returns the fields it found. A list that carries a short excerpt hands it over as `teaser` and declares `teaser=True`.
|
||||
- The plugin does not decide which roles are read. The fetch text stage calls `detail` after scoring, only for roles in the cut (`ADR Fetch prep and score`, section 3), at most 200 per board and 1,000 per run, 0.5 s apart.
|
||||
- `detail` may fill only description, location, places, teaser, remote, country, employment, pay figures, currency and the posting date. Title, link, department and the pay line come from the list, which states them on every pull.
|
||||
- A role read in detail is sent by later pulls without the list's place and remote, so what its own page said stays. This holds for a search's roles as for a board's.
|
||||
- A 404 or 410 counts as read. Another failure is retried on a later run, after the roles never tried. A read that gave no text is retried after 7 days. A 429 ends the reads of that target for the run.
|
||||
- Completeness, and so closing, is decided by the list alone.
|
||||
|
||||
### 2.3 One role, several places (plugin)
|
||||
|
||||
- A row carries `places`: every place the host states for the role, in the host's order and the host's words, each marked remote, office or unsaid as the host marks that place. The host's own word for how the role is worked ("Hybrid", "Remote", "On-site") goes on the row as `remote`, unmapped.
|
||||
- A plugin never makes one row per place, never drops a place and never joins places into one text.
|
||||
- A host that states no place declares `places="none"` and hands over an empty list.
|
||||
- Resolving a place text to a town, region or country is the app's job.
|
||||
|
||||
### 2.4 One job listed several times (the app)
|
||||
|
||||
One job can come as several rows: one per office, each with its own id; several requisition ids for the same job; or a search provider and the company's board listing the same role.
|
||||
|
||||
- The plugin hands every listing over as its own row, with its own id and link. A host that publishes one posting per place declares `places="copies"`.
|
||||
- Folding them into one role, and choosing the copy that is kept, happens in the app (`ADR Fetch prep and score`, section 1.4).
|
||||
- Two listings under one id are one row; a host that lists one id once per place hands over one row with all the places.
|
||||
|
||||
### 2.5 Facts in the free text (the app)
|
||||
|
||||
- The plugin passes on only what the host states in a field: remote, country, seniority, employment, pay and date, in the host's words, and declares each as `stated` or `none`. A provider that covers one country states that country on every row and declares it `stated`.
|
||||
- Mapping the host's words to the app's vocabulary happens in the app, through a table per host.
|
||||
- Facts found only in the prose, e.g. a Copenhagen role that is actually remote, are read in the app: from the place words first, then by extract (`ADR Fetch prep and score`, section 4.1).
|
||||
|
||||
## 3 Further rules
|
||||
|
||||
### 3.1 Filtering at the host
|
||||
|
||||
- The plugin declares per dimension (country, place, remote, query) whether the host filters it: `none`, `one` (one value a request), `many` (a list in one request) or `ids` (the host's own ids, looked up with `facets`).
|
||||
- The runner hands the plugin an `Ask` that holds only declared dimensions. With `one`, it runs one series of requests per value and unions the rows by id. A read costs at most 8 series. Above that the runner narrows less: more than 8 countries on a `one` host means a read without countries; when terms times the location series pass 8, a read for the location gate keeps the places and drops the terms, and a read of an All locations company keeps the terms and drops the place part; more than 8 terms alone are dropped.
|
||||
- "In these countries, or remote" cannot be asked in one request, because two facets intersect. The runner makes remote a series of its own, so a plugin never gets remote and a country in the same `Ask`. A host that cannot filter on remote is asked for the countries alone, and the read is stamped as not having asked for remote, which the app shows and spares the remote roles it holds from closing.
|
||||
- A host with `query="one"` also takes the fetch terms, one term a series, handed over as `query`.
|
||||
- A host's facet ids are looked up at most once a day per board. A value the host's facets do not know is dropped; a read for which nothing remains asks nothing, lists nothing and closes nothing.
|
||||
- A board's own saved params win over the agents' reach on the dimensions they set.
|
||||
- Which boards are filtered at all, and by what, is decided in the app (`ADR Fetch prep and score`, section 1.2).
|
||||
|
||||
### 3.2 Completeness and closing
|
||||
|
||||
The plugin declares how completeness is known (`stated`, `end`, `whole` for a one-answer list, `never`). The runner stamps each read as complete or not, as filtered (a location filter), as narrowed (fetch terms or a provider's query), and with the ids it listed. Closing follows `ADR Fetch prep and score`, section 1.3: a search never closes a role by its list, whatever it declares, so a provider's `stated` only says how far a read got.
|
||||
|
||||
### 3.3 Failures
|
||||
|
||||
- A plugin lets every error through: HTTP errors, timeouts, parse errors, an answer of another shape. The runner logs, records and decides. An error on a later page fails the whole read; what a 429 read is kept (2.1).
|
||||
- A read that lists under 10% of what the same read listed last time counts as failed. For a whole read of a board this holds once the last count was at least 10 roles; a filtered or narrowed read, and a read of a host that answers an empty list for an unknown slug (`empty="suspect"`), is held from a count of 1. "The same read" is one with the same filters: a changed filter starts the count again.
|
||||
- A board that fails on every read for 21 days closes its roles, and a suspect answer that has stood that long is accepted.
|
||||
- A failing target is named on its company's row and in the pull's notice, with the reason.
|
||||
|
||||
### 3.4 Fetching
|
||||
|
||||
Every request goes through the runner's `get`:
|
||||
|
||||
- https on port 443 only, never to an IP address or a local name;
|
||||
- only to the host names the plugin allows for that slug;
|
||||
- at most 3 redirects, each checked against the same allow-list;
|
||||
- 10 s to connect, 30 s between bytes, 60 s for the whole request;
|
||||
- at most 96 MiB a response. A plugin may name a smaller fallback feed, read once when the main one is over the cap.
|
||||
|
||||
The figures in this document are the defaults as they stand; the rules are what counts.
|
||||
|
||||
### 3.5 Searches
|
||||
|
||||
A provider has saved params instead of a board. It declares each param: the filter dimension it fills, its label, its kind (text, a choice from a table, a list of country codes) and the table of choices where there is one, so the form, the command line and the API offer a provider's params from the declaration alone. It cleans a param it would not accept with a reason a person can read, and receives the saved params in `Ask.params`. A search row carries `company`, the employer's name. A provider has no slug patterns, probes or careers page, and takes no fetch terms: its query replaces them. A provider that needs an API key declares it, and the runner hands it the key; a search reads 5 pages by default and 10 at most.
|
||||
|
||||
### 3.6 Discovery
|
||||
|
||||
A host can be found from a careers page link: its slug patterns are regexes over a URL, with the slug in group 1, or several groups joined by `/`. A host may offer a cheap probe URL that answers 2xx only for a real board, and a check of the probe's body where 2xx is not enough. The slug is lowercased unless the host's paths are case-sensitive. A host whose boards live on a company's own domain names the page markers that identify it, and the path discovery confirms with.
|
||||
|
||||
### 3.7 robots.txt
|
||||
|
||||
Whoever builds a plugin reads the robots.txt of every host it requests, for the API paths it requests, and the owner decides whether the host is read. The declaration records what the file says, the date it was read and the decision. A record without a date is refused. Nothing reads robots.txt at run time.
|
||||
|
||||
### 3.8 Rows
|
||||
|
||||
- `source_id` is the host's id, unique on the board or search. `title` and `url` (http or https) are required. The runner, not the plugin, drops a row that lacks one of them; the read's record counts the rows dropped, and a board whose dropped rows pass 10% of its list counts as failed, since a changed page shape looks like that.
|
||||
- `description` and `teaser` are plain text, never HTML.
|
||||
- `posted` is an aware datetime or none. `posted="trusted"` declares it the posting date; `listed` declares a date the host moves, e.g. on an edit. A plugin whose only date comes from the role's page or from a field the host moves declares `listed`.
|
||||
- Pay figures are yearly integers with an ISO 4217 currency. A monthly figure is multiplied by 12 where the host states the period. An hourly, daily or weekly figure is kept as a pay line only.
|
||||
- `country` is ISO 3166 alpha-2, and only where the host states it (or the provider covers that one country, 2.5).
|
||||
- Any other key the host states is kept as raw, so a new field needs no change in the runner.
|
||||
|
||||
### 3.9 What the runner keeps per target
|
||||
|
||||
So that a plugin keeps nothing: the last read's marker, whether it was complete, filtered, narrowed and asked for remote, the ids it listed, the sweep in progress, the roles read and failed in detail, the facet ids and when they were read, the board's size and when it was measured, the last count the guard compares with, and the failure record with its first date and reason. All of it is shown or summarised on the board's row.
|
||||
|
||||
---
|
||||
|
||||
## API
|
||||
|
||||
A plugin is one module, named after its source, importing only the shared plugin helpers, the standard library, `httpx`, `pydantic` and the shared model base. It registers by existing: a module that breaks the contract is refused at import, every problem named; the other plugins load, and the Sources page names the refused one.
|
||||
|
||||
### Module attributes
|
||||
|
||||
```python
|
||||
NAME = "workday" # the host of a target `workday:<slug>`; equals the module name
|
||||
LABEL = "Workday" # shown in the source select
|
||||
PATTERNS = (r"...",) # host only: discovery regexes over a URL, IGNORECASE
|
||||
RANK = 100 # optional: order in the select and in probes, lower first
|
||||
KEEP_CASE = False # optional: True keeps a discovered slug's case
|
||||
MARKERS = ("...",) # optional, host on a company's own domain: page markers
|
||||
PARAMS = { # provider only: saved param -> what it is
|
||||
"q": Param(fills="query", label="Query", kind="text"),
|
||||
"region": Param(fills="place", label="Region", kind="choice", choices=REGIONS),
|
||||
}
|
||||
|
||||
READS = Reads(
|
||||
scope="board", # "board" | "search"
|
||||
listing="paged", # "whole" | "paged"
|
||||
page_size=20, # paged only
|
||||
ceiling=2000, # paged only: offset past which the host answers nothing
|
||||
text="detail", # "listing" | "excerpt" | "detail" | "none"
|
||||
teaser=False, # with "detail": the list also carries a teaser
|
||||
complete="stated", # "stated" | "end" | "whole" | "never"
|
||||
places="many", # "one" | "many" | "copies" | "none"
|
||||
filters=Filters(country="ids", place="none", remote="ids", query="one"),
|
||||
# each "none" | "one" | "many" | "ids"
|
||||
fields=Fields(pay="none", posted="trusted", remote="stated",
|
||||
country="none", seniority="none", employment="stated"),
|
||||
# pay "figures" | "line" | "none"; posted "trusted" | "listed" | "none";
|
||||
# the rest "stated" | "none"
|
||||
key=None, # Key(name, fields, env) when an API key is needed
|
||||
empty="trusted", # "suspect" when an unknown slug answers an empty list
|
||||
robots="read: ... (checked 2026-09-30)", # required, with the date
|
||||
)
|
||||
```
|
||||
|
||||
### Functions
|
||||
|
||||
```python
|
||||
# Every plugin
|
||||
def allowed(slug: str) -> Callable[[str], bool] # the host names a request may reach
|
||||
|
||||
# Every host
|
||||
def slug_ok(slug: str) -> bool # full match; refuses anything that adds a path,
|
||||
# query, fragment, port or user to a URL
|
||||
def careers_url(slug: str) -> str # the page a person opens
|
||||
|
||||
# listing="whole"
|
||||
def feed_url(slug: str, ask: Ask) -> str # the one request
|
||||
def parse(body: bytes, slug: str) -> list[Row]
|
||||
def fallback_url(slug: str) -> str # optional: a smaller feed when over 96 MiB
|
||||
|
||||
# listing="paged"
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page # first call: cursor None
|
||||
def rows(decoded: Any, slug: str) -> list[Row]
|
||||
|
||||
# text="detail"
|
||||
def detail(get: Get, slug: str, row: Row) -> dict[str, Any] # fields from the role's own page
|
||||
|
||||
# any filter declared "ids"
|
||||
def facets(get: Get, slug: str) -> dict[str, dict[str, str]]
|
||||
# {"country": {"DE": "<host id>"}, "place": {...},
|
||||
# "remote": {...}}; a missing dimension is read
|
||||
# unfiltered
|
||||
|
||||
# Every provider
|
||||
def clean_params(params: dict[str, Any]) -> dict[str, Any] # raises ValueError with a reason
|
||||
|
||||
# Discovery, optional
|
||||
def probe_url(slug: str) -> str # 2xx only for a real board
|
||||
def probe_ok(body: bytes) -> bool # when 2xx is not enough
|
||||
def probe_slugs(guess: str) -> tuple[str, ...] # slugs a guessed company name is tried as
|
||||
```
|
||||
|
||||
### Types
|
||||
|
||||
```python
|
||||
get(url, *, method="GET", payload=None, headers=None) -> bytes
|
||||
# POST sends payload as JSON. Raises httpx.HTTPStatusError on non-2xx,
|
||||
# httpx.HTTPError on timeout or connection failure, and the runner's own
|
||||
# errors on a refused host, an oversized body or a broken redirect. A
|
||||
# plugin lets these through.
|
||||
|
||||
Ask(countries=(), places=(), remote=None, remote_ids=(), query="", terms=(), params={})
|
||||
# only declared dimensions are set; with "one" a series carries one value;
|
||||
# with "ids" countries and places hold host ids; never remote together with
|
||||
# a country or place; terms are handed over one a series as query, so a
|
||||
# plugin reads query only; params holds a search's saved params and a key under "key"
|
||||
|
||||
Page = (decoded, results, total, cursor)
|
||||
# decoded: handed to rows; results: raw dicts with an "id", counted by
|
||||
# distinct id; total: stated by the host, or None; cursor: next page, or
|
||||
# None after the last; any JSON value, a tuple returning as a list
|
||||
|
||||
Place = {"text": str, "remote": bool | None}
|
||||
# the host's words; True remote, False office, None unsaid
|
||||
|
||||
Row = {
|
||||
"source_id": str, "title": str, "url": str, "places": list[Place], # required
|
||||
"location": str, "posted": datetime | None, "department": str,
|
||||
"description": str, "teaser": str, "remote": bool | str | None,
|
||||
"salary_text": str, "salary_min": int, "salary_max": int, "currency": str,
|
||||
"country": str, "seniority": str, "employment": str,
|
||||
"company": str, # search rows only
|
||||
}
|
||||
|
||||
Param(fills, label, kind, choices=None)
|
||||
# fills: "query" | "country" | "place" | "remote" | "pages" | "" (the host's own);
|
||||
# kind: "text" | "choice" | "countries" | "number"; choices: {value: label}
|
||||
```
|
||||
|
||||
### Helpers the plugins share
|
||||
|
||||
- `place(text, remote=None)` makes a `Place`, remote when the text says remote; `add_place(found, entry)` appends one unless it is empty or already there.
|
||||
- `html_text(html)` turns HTML into plain text, one line per paragraph, with list items and headings marked; `parse_dt(value)` makes an aware datetime; `yearly_amount(...)` makes a yearly figure and `pay_line(...)` a pay line; `is_web_url(url)` checks a link.
|
||||
- `path_slug_ok(slug)` and `label_slug_ok(slug)` check a slug that is one URL path segment or one DNS label; `only(host)` is the usual `allowed`.
|
||||
- A table of the country names hosts write, both ways (`code_of`, `name_of`).
|
||||
|
||||
### Adding a plugin
|
||||
|
||||
1. Write the module with its declaration and functions.
|
||||
2. Add a table of the host's words for seniority, employment and remote, when the host states them in its own words.
|
||||
3. Add fixtures from real answers, and tests. A contract test, shared by every plugin and naming none, checks each declared field against the fixtures: a field declared `stated` must show on some row, a field declared `none` on none; required row keys, plain text, aware dates, ISO codes and integer pay; that results carry ids and cursors survive JSON; that a declared filter reaches the request.
|
||||
4. Record the robots.txt decision in the declaration.
|
||||
|
||||
Nothing else changes: the list of plugins, the source select, discovery, a provider's form, the filters a board is read with and what its row says all follow from the module.
|
||||
+20
-196
@@ -1,127 +1,12 @@
|
||||
# Where roles come from
|
||||
|
||||
This file lists every source roledesk reads roles from, how each is read and what it cannot deliver, as of the code on main and the live database of 2026-09-30. The contract for writing a reader (a host module or a search provider) is the docstring of `plugins/readers/__init__.py`, with its types and the declaration check in `plugins/readers/_contract.py` and the conformance test in `tests/test_readers_contract.py`, not this file.
|
||||
This file lists every source roledesk reads roles from, how each is read and what it cannot deliver. The contract for writing a reader (a host module or a search provider) is `documentation/ADR Plugins.md`, the owner's draft, and in code the docstring of `plugins/readers/__init__.py`, with its types and the declaration check in `plugins/readers/_contract.py` and the conformance test in `tests/test_readers_contract.py`, not this file.
|
||||
|
||||
The code has two plugins: `ats` (`plugins/ats.py`) reads the board of a company someone listed, with one module per host in `plugins/readers/`; `search` (`plugins/search.py`) runs saved queries on sites that list roles across companies. Below, a module of `ats` is a "host" and a module of `search` is a "provider". A company's list of roles on a host is its "board"; a board is attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
|
||||
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the ADRs state the ones that bear on a reader.
|
||||
|
||||
## Flow
|
||||
|
||||
The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section says what each step leaves behind.
|
||||
|
||||
1. A reader declares what it does (`READS`) and how to ask and parse (`feed_url` and `parse`, or `page` and `rows`, and optionally `detail` and `facets`).
|
||||
2. `pull` queues batches of 10 targets (`app/jobs/pull.py`). In each batch `ats` or `search` calls `_runner.read`: it asks the reader for the facets a filter needs, builds the series of requests from the reach, reads the listing whole or in pages, continues or starts a sweep, and applies the 10% guard.
|
||||
3. The plugin stores every row with `upsert_role` and then stamps the target's state (`self.state`, the `state` table). A search stamps only `ok`, `complete` and `ids`. Keys are `ats:<key>:<target>`:
|
||||
|
||||
| Key | Written by | Read by |
|
||||
|---|---|---|
|
||||
| `ok` | `_runner.stamp`, the run marker, after every row was stored | `close_boards`: a target whose marker is not this run's did not answer |
|
||||
| `complete` | `stamp` | `close_boards`: only a complete read closes |
|
||||
| `ids` | `stamp`, the compound ids the read listed | `close_boards`: closes open roles not in it |
|
||||
| `filtered` | `stamp`, a read by country, place or remote | `close_boards`: keeps roles with a vote or an application |
|
||||
| `remote_unasked` | `stamp`, a filtered read while the agents wanted remote and the host has no remote filter | `close_boards`: keeps remote roles, closes them by age; the Companies line says "remote not asked" |
|
||||
| `narrowed` | `stamp`, a read by fetch terms | `close_boards`: closes only roles not listed for `STALE_DAYS` |
|
||||
| `swept` | `stamp`, a complete read that finished a sweep begun on an earlier pull | `close_boards`: applies the two-strikes rule |
|
||||
| `sweep` | `stamp`, the cursor of a read cut at `MAX_PAGES` | `_runner.read` on the next pull |
|
||||
| `count` | `stamp`, the roles the last read listed; reset by Recheck | `_runner.read`, the 10% guard |
|
||||
| `missing` | `close_boards` (`_close_swept`), the ids the last finished sweep missed | the next finished sweep |
|
||||
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `fetch_text`, the guard's escape after 21 days |
|
||||
| `facets` | `_runner.facets_for`, kept one day | `reach.site_facets`, the Companies line |
|
||||
| `size`, `sizechecked` | `ats.keep_size`: `set_board`, Recheck, `app companies size`, or the first read of an unsized board | `reach.sizes`, which decides whether the board is filtered |
|
||||
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`fetch_text`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `fetch_text.plan` through `detail_due` |
|
||||
|
||||
4. `close_boards`, the settle of the pull, closes what a complete read did not list, closes a board or search that failed for `STALE_DAYS`, releases the run, and raises one `pull.settled` notice in the Problems log when a target did not answer or was read incomplete, or a board closed more than 50 roles or more than half of its open ones.
|
||||
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score` (which writes each agent's cut), `fetch_text` (reads the role pages of the roles in the cut without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `judge`, `summarise`. Extract, judge and summarise are switches under Runs; fetch text is on by default.
|
||||
|
||||
## The three ways a source is read
|
||||
|
||||
**Whole board.** One request returns every role, with its text for most hosts. The read is complete whenever the answer parses, so the pull can close every role the answer no longer lists.
|
||||
|
||||
**Paged.** The list arrives in pages of a fixed size (`READS.page_size`), read one after another `PAGE_DELAY` (0.5 s) apart, at most `MAX_PAGES` (100) pages a pull. The read is complete when the number of distinct roles read equals the total the host states (`complete="stated"`), or, for a host that states none, when the read reaches an empty page (`complete="end"`); a Jobindex search is never complete (`complete="never"`). A board of more than 100 pages is swept: each pull stores what it read and the next continues from the saved cursor, and the read is complete when a sweep reaches the end. Roles closing between pulls shift an offset-paged host's later roles back past the cursor, so a role of a swept board closes only when two finished sweeps in a row missed it. A host with a `ceiling` (Workday 2,000) answers nothing past it: a read that reaches it is incomplete, is not swept and closes nothing, so a filter must bring the board under it.
|
||||
|
||||
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The fetch text stage reads those pages after scoring (`app/jobs/fetch_text.py`):
|
||||
|
||||
- It takes open roles without text in the cut of an enabled agent (the best by fit and by CV closeness, and the liked and saved, see `app/jobs/score.py`), best final score first.
|
||||
- It reads at most `fetch_text.per_board` (200) roles per board and `fetch_text.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
|
||||
- A read that gave no text is tried again after `fetch_text.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
|
||||
- It runs after every pull, after a saved agent, an affinity change or an override switched, not after a vote, and it can be switched off under Runs.
|
||||
|
||||
A role in no agent's cut is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles in the cut do. Four consequences follow from the code:
|
||||
|
||||
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow fetch text encode it again.
|
||||
- A role page may fill only `_runner.DETAIL_FIELDS`: description, location and places, teaser, remote, country, employment, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
|
||||
- Completeness, and so closing, is decided by the list alone. A board of 800 roles closes its roles from the first pull.
|
||||
- SmartRecruiters and Workable list their roles in pages and belong here too, since their list has no text.
|
||||
|
||||
## Overview
|
||||
|
||||
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles fetch text has read.
|
||||
|
||||
| Source | Boards | Open roles | How it is read | Roles per request | Description | Pay | Limit that matters most |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Greenhouse | 87 | 10,457 | whole board | all | in the list | figures | a board over 96 MiB loses its text and pay |
|
||||
| Ashby | 74 | 4,964 | whole board | all | in the list | figures | an unknown board name answers an empty board |
|
||||
| Workday | 17 | 2,041 | list, then each role | 20 | per role (1,280 read) | none | 2,000 roles per board; country arrives per role |
|
||||
| Lever | 26 (25 read) | 1,347 | whole board | all | in the list, the intro paragraph only | figures | text is the opening paragraph |
|
||||
| Oracle Recruiting Cloud | 5 | 937 | list, then each role | 200 | per role (603 read), teaser in the list | none | no pay field |
|
||||
| SuccessFactors | 14 | 749 | list, then each role | the site's own | per role (681 read) | none | default locale only |
|
||||
| Teamtailor | 38 | 611 | paged | 50 | in the list | none | no pay; nine companies without a found site |
|
||||
| Jobindex (search) | 12 saved (8 on) | 562 | paged | 20 | teaser | none | teaser only; at most 200 results per search; never complete |
|
||||
| Platsbanken (search) | 6 saved | 439 | paged | 100 | in the list | none | at most 1,000 results per search |
|
||||
| Phenom | 5 | 389 | list, then each role | 100 | per role (142 read), teaser in the list | none | locale spelling known from three sites |
|
||||
| SmartRecruiters | 11 | 343 | list, then each role | 100 | per role (1 read) | none | one country a request; remote is not filtered |
|
||||
| Personio | 3 | 320 | whole board | all | in the list | figures | no country field |
|
||||
| Eightfold | 3 | 185 | list, then each role | 10 | per role (158 read) | none | 10 per request, so 1,000 roles a pull |
|
||||
| Recruitee | 8 | 176 | whole board | all | in the list | figures | an empty board looks like no board to discovery |
|
||||
| Workable | 8 | 157 | list, then each role | 10 | per role (1 read) | none | answers 429 to bursts |
|
||||
| Jibe | 1 | 137 | paged | 100 | in the list | none | Booking Holdings' roles read with Booking.com's |
|
||||
| Deel | 1 | 96 | whole board | all | in the list | line only | pay has no period |
|
||||
| Gem | 2 | 45 | list, then each role | all | per role (44 read) | none | undocumented query, can break |
|
||||
| Jobylon | 2 | 33 | list, then each role | all | per role (33 read), teaser in the list | none | scraped from the careers widget |
|
||||
| HR Manager | 1 | 7 | paged | 100 | in the list | none | the portal's robots.txt disallows reading; the owner has not decided |
|
||||
| Rippling | 1 | 2 | list, then each role | all | per role | figures | text, date and pay cost one request per role |
|
||||
| Pinpoint | 1 | 2 | whole board | all | in the list | figures | no date |
|
||||
| HiBob | 1 | 2 | whole board | all | in the list | figures | pay never seen filled |
|
||||
| BambooHR | 1 | 2 | list, then each role | all | per role | none | paging of a large board unverified |
|
||||
| Adzuna (search) | 0 | 0 | paged | 50 | teaser | figures | never run live; needs a key |
|
||||
| freehire (search) | 0 | 0 | paged | 100 | in the list | figures | 10,000 results per search; the query is noisy |
|
||||
| Jobnet (search) | 0 | 0 | paged | 100 | in the list | none | 10,000 results per search; needs the header `x-csrf: 1` |
|
||||
| Akademikernes Jobbank (search) | 0 | 0 | paged | 100 | per role | none | not sorted by date, and no total, so never complete |
|
||||
|
||||
"Figures" means yearly `salary_min`, `salary_max` and `currency`, plus a pay line; "line only" means a pay line (`salary_text`) and no figures. Figures are yearly: a monthly figure is multiplied by 12 where the host states the period, and an hourly, daily or weekly one is kept as a line only.
|
||||
|
||||
## Rules every source shares
|
||||
|
||||
| Limit | Value | What it means |
|
||||
|---|---|---|
|
||||
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); fetch text is on by default. |
|
||||
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. Enrich reads one batch per board on the same pool. One pull runs at a time. |
|
||||
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
|
||||
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
|
||||
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request | The same for boards, role pages and searches. A slow board fails for this pull and is asked again next pull. |
|
||||
| Pages per board | 100 a pull | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
|
||||
| Wait | 0.5 s before every page after the first, for boards and searches alike, and before every role page | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
|
||||
| Per-role reads | `fetch_text`, after scoring: 200 per board and 1,000 per run, the roles in the agents' cut | See "List, then each role". Roles in no cut are not read. |
|
||||
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing, and so does a read that asked nothing. A role listed again reopens. A board switched off closes its roles at once. A filtered read keeps roles with a vote or an application, and remote roles when the host could not be asked for remote; a read by fetch terms closes only roles not listed for 21 days. |
|
||||
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. A Jobindex search is never complete, so its roles close by the 21 days only. A search failing for 21 days closes its roles. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, a row that would not store, and a read that lists under 10% of what the same read listed last time (`SUSPECT_SHARE`) once that was at least `SUSPECT_MIN`, 10. A search that fails is recorded the same way and its row says so. |
|
||||
| A board that fails for 21 days | all its open roles close at the next pull, and a suspect answer that has stood that long is accepted | The next answer reopens what it lists. `STALE_DAYS` is the one 21. |
|
||||
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next fetch text run | Nothing is closed on a list cut short by a 429. |
|
||||
|
||||
How far a board is read: a pull asks each board of 300 roles or more for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of once per country (33 codes today), unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application. A board's size is set once, when it is added, on Recheck or by `app companies size`, and a pull does not rewrite it; a board with no size yet is sized first in its batch and then read as its size says, and a board whose reader cannot narrow the reach is read whole and sized by that read. A host that cannot filter a dimension ignores it: SmartRecruiters, Phenom, SuccessFactors and Eightfold have no remote filter, so their reach asks for countries only.
|
||||
|
||||
Fetch terms narrow a wide read by words. Each agent has a list of them (`fetch_terms`, set in the agent editor); the terms of a pull are the union over the agents that are on, and none when one of those agents has none. They go only to a board read with the wide reach: a company set to All countries, or a board whose reach is more countries than a one-country host is asked for (`MAX_SERIES`, 8). The narrow reach (Denmark, Sweden) leaves at most about 150 roles on any big board, so it takes no terms, and nor does a board under 300 roles or one with its own reach. A company set to All roles takes none on any of its boards. A host that takes a keyword gets one series per term, unioned by id, each combined with the country series; the survey shows that as an AND on SmartRecruiters, Eightfold, SuccessFactors, Phenom and Workday. When terms times the country series exceed 8, the place part of the reach goes first, whole, and the terms stay; more than 8 terms read the board unfiltered. The survey found the keyword matching the ad text and not only the title where it could tell, and could not tell on SuccessFactors, Workday, Workable and Jibe (`docs/filter-survey.md`), so a term like `engineer` cuts less than it looks. A read with terms is never the whole board: it records no size, closes no role it did not list, and the 10% guard holds it only to a read with the same terms and places. The search providers take no terms: each saved search has its own query.
|
||||
|
||||
What no source does:
|
||||
|
||||
- Filter on the host's side for a host other than SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom and Eightfold. Greenhouse, Ashby, Recruitee and the rest are read whole. The search providers filter by their own query, place and country.
|
||||
- Filter by category or level at the source. Workday, Oracle, Jibe and Eightfold state category ids, but nothing maps an agent's terms to them yet (`docs/filter-survey.md`).
|
||||
- Read more than one board per company. DFDS is read from its Danish Teamtailor board only.
|
||||
- Use a key taken from a site's script. Maersk, New Work and Welcome to the Jungle are on the Watchlist for that reason.
|
||||
- Read a page that needs a browser to run its script.
|
||||
- Log in anywhere.
|
||||
No reader logs in, runs a page's script or uses a key taken from a site's script. A company that needs one is on the Watchlist tab (`app companies by-hand` lists them), and the readers planned for the rest are in `docs/plan-hosts-wave-2.md`.
|
||||
|
||||
## Hosts read as a whole board
|
||||
|
||||
@@ -130,25 +15,21 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board token, e.g. `greenhouse:trustpilot` |
|
||||
| Companies | 87 boards: Trustpilot, Cognite, Wolt |
|
||||
| List | `boards-api.greenhouse.io/v1/boards/<token>/jobs` with `content=true` and `pay_transparency=true`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1; 2 when the first answer is over 96 MiB |
|
||||
| Description | in the list, full text (`content`) |
|
||||
| Date | `first_published`, else `updated_at`, which the host moves on an edit; trusted as the posting date |
|
||||
| Place and country | every `offices` entry; `location.name` when the role has no office or the name says more, with placeholder names such as "Hybrid" dropped. The host states no country; it is resolved from the places |
|
||||
| Remote | none from the host; prep reads remote, hybrid or onsite from the place words |
|
||||
| Pay | figures from `pay_input_ranges` when there is exactly one range; a range whose title says hour or day is a line only; any other range is taken as yearly. Several ranges give one line per range and no figures |
|
||||
| Found by discovery | yes, from a link to `boards.greenhouse.io`, `job-boards.greenhouse.io` or the embed script; a guessed name is probed first of all hosts |
|
||||
| Limitations | Over 96 MiB the board is read again without `content=true`: titles land, text and pay of the whole board are lost. An `eu.greenhouse.io` link is read from the main API. The pay flag must be sent, or no range comes back. |
|
||||
| Limitations | Over 96 MiB the board is read again without `content=true`: titles land, text and pay of the whole board are lost. It is the only host with a `fallback_url`. An `eu.greenhouse.io` link is read from the main API. The pay flag must be sent, or no range comes back. |
|
||||
|
||||
### Lever
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the site name, or `eu/<site>` on Lever's EU instance, e.g. `lever:spotify`, `lever:eu/tomtom` |
|
||||
| Companies | 26 boards, one switched off: Spotify, TomTom, Pipedrive |
|
||||
| List | `api.lever.co/v0/postings/<site>?mode=json` (`api.eu.lever.co` for `eu/`), the whole board in one answer; no total, no paging parameter sent |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, `description` (else `descriptionPlain`); the module reads no `lists` or `additional` sections, so it is the opening text |
|
||||
| Date | `createdAt`; trusted |
|
||||
| Place and country | `categories.location` and every `allLocations` entry; the host's `country` is handed on when the role has one place, otherwise the country is resolved from the places |
|
||||
@@ -163,9 +44,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board name, which may carry a dot or be a uuid, e.g. `ashby:pleo`, `ashby:jimdo.com` |
|
||||
| Companies | 74 boards: Pleo, Too Good To Go, Lovable |
|
||||
| List | `api.ashbyhq.com/posting-api/job-board/<name>?includeCompensation=true`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text (`descriptionHtml`) |
|
||||
| Date | `publishedAt`; trusted |
|
||||
| Place and country | the primary `location` and every `secondaryLocations` entry; country resolved from the places |
|
||||
@@ -180,9 +59,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `recruitee:bunq` |
|
||||
| Companies | 8 boards: bunq, idealo, Climeworks |
|
||||
| List | `https://<sub>.recruitee.com/api/offers/`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, `description` only (`requirements` is not read) |
|
||||
| Date | `published_at`, else `created_at`; trusted |
|
||||
| Place and country | every `locations` entry as "city, state, country"; `location` or `city` when there are none; a remote offer with no place gets the place "Remote" |
|
||||
@@ -197,9 +74,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `personio:penneo` |
|
||||
| Companies | 3 boards: Penneo, tado, 1KOMMA5° |
|
||||
| List | the XML feed `https://<sub>.jobs.personio.de/xml`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, every `jobDescription` value joined |
|
||||
| Date | `createdAt`; trusted |
|
||||
| Place and country | `office` and every `additionalOffices` office; the feed has no country field, so it is resolved from the places |
|
||||
@@ -214,9 +89,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `pinpoint:xeneta` |
|
||||
| Companies | 1 board: Xeneta |
|
||||
| List | `https://<sub>.pinpointhq.com/postings.json`, the whole board in one answer; no total, no paging keys |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, skills and benefits sections |
|
||||
| Date | none: the list states none and the role page is not read for it; the role counts as posted when first seen |
|
||||
| Place and country | `location.name` (else `city`) plus `province`, one place; country resolved from that line |
|
||||
@@ -231,9 +104,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board slug on `jobs.deel.com`, with case, e.g. `deel:klarna` |
|
||||
| Companies | 1 board: Klarna |
|
||||
| List | the HTML page `https://jobs.deel.com/<slug>`, from which the Next.js server payload is read; the whole board in one answer; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text (`richtextDescription`) |
|
||||
| Date | the publication's `createdAt`, else the posting's; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every `jobLocations` name as a place, repeated names dropped, each carrying the role's remote flag; country resolved from the places |
|
||||
@@ -248,9 +119,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain of `careers.hibob.com`, e.g. `hibob:dixa` |
|
||||
| Companies | 1 board: Dixa |
|
||||
| List | `https://<sub>.careers.hibob.com/api/job-ad` with the header `companyIdentifier: <sub>`, the whole board in one answer; no total. Declared as paged with `complete="end"` because `page` may send headers; it reads one page |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, requirements, benefits |
|
||||
| Date | `publishedAt`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `site` plus `country`. A company that hires at two sites posts one ad per site with its own id (`places="copies"`); `app/dupes.py` folds them into one role |
|
||||
@@ -267,25 +136,21 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain of `teamtailor.com`, or the whole host of a career site on the company's domain, e.g. `teamtailor:lunar`, `teamtailor:careers.trackunit.com` |
|
||||
| Companies | 38 boards: Lunar, Trackunit, DFDS |
|
||||
| List | the RSS feed `https://<site>/jobs.rss`, 50 per request by offset; no total, so the read goes on to an empty page |
|
||||
| Requests per pull | roles / 50, rounded up, plus 1 for the empty page; the largest board attached has 75 roles, so 3 |
|
||||
| Description | in the list, full text |
|
||||
| Date | `pubDate`; trusted |
|
||||
| Place and country | every `tt:location` as a place "city, country", each remote when `remoteStatus` says so; country resolved from the places |
|
||||
| Remote | `remoteStatus`, handed on as the host's word: fully is remote, hybrid, onsite; `none` and `temporary` stay unknown |
|
||||
| Pay | none: the feed states no pay |
|
||||
| Found by discovery | yes, from a `<sub>.teamtailor.com` link; a page that loads Teamtailor's CDN names the page's own host as the board; a guessed name is probed |
|
||||
| Limitations | A board of more than 100 pages is swept across pulls and completes on the empty page. A page with nothing new ends the read incomplete. Nine seed companies have no board because their careers link is a marketing site, not the career host. |
|
||||
| Limitations | A board of more than 100 pages is swept across pulls and completes on the empty page. A page with nothing new ends the read incomplete. Nine seed companies have no board because their careers link is a marketing site, not the career host. A company has one board, so DFDS, which has a Teamtailor site per country, is read from its Danish one only. |
|
||||
|
||||
### Jibe
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the career site's host, e.g. `jibe:jobs.booking.com` |
|
||||
| Companies | 1 board: Booking.com |
|
||||
| List | `https://<host>/api/jobs?limit=100&page=<n>`, 100 per request, the most the host serves; states `totalCount` |
|
||||
| Requests per pull | roles / 100, rounded up; 2 for Booking.com |
|
||||
| Description | in the list, full text: description and qualifications |
|
||||
| Date | `posted_date`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `full_location` split at ";" into places; the host's `country_code` is handed on when the role has one place |
|
||||
@@ -300,9 +165,7 @@ What no source does:
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the portal's customer name, e.g. `hrmanager:teknologisk` |
|
||||
| Companies | 1 board: Danish Technological Institute (`hrmanager:teknologisk`, 7 open roles). Seaborg, the one seed company on the portal, is unattached and on the Watchlist as `robots` |
|
||||
| List | the RSS feed `api.hr-manager.net/JobPortal.svc/<customer>/PositionList/rss/?incads=true`, 100 per request by `skip`; no total, so the read goes on to an empty page |
|
||||
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page |
|
||||
| Description | in the list, the whole advertisement |
|
||||
| Date | `pubDate`; not trusted |
|
||||
| Place and country | none: the feed names no place |
|
||||
@@ -313,37 +176,33 @@ What no source does:
|
||||
|
||||
## Hosts read as list, then each role
|
||||
|
||||
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page (`documentation/ADR Plugins.md`, 2.2). "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
|
||||
### SmartRecruiters
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the company identifier, e.g. `smartrecruiters:BoschGroup` |
|
||||
| Companies | 11 boards: Bosch (about 4,800 roles on the site, 13 open in reach), Delivery Hero, Wise |
|
||||
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
|
||||
| Requests per pull | one series per country of the reach (two for Bosch: DK and SE), each roles / 100, rounded up; a board read whole is roles / 100 |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only: 1 of 343 open roles has text |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only |
|
||||
| Date | `releasedDate`; trusted |
|
||||
| Place and country | `fullLocation`, else city and country, with empty parts removed; the host's `country` is handed on |
|
||||
| Remote | `location.remote`, true or false, as the host's word |
|
||||
| Seniority, employment | `experienceLevel` and `typeOfEmployment`, the host's ids |
|
||||
| Pay | none: the list states no pay |
|
||||
| Found by discovery | yes, from a `jobs.`, `careers.` or `api.smartrecruiters.com` link; a guessed name is probed |
|
||||
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
|
||||
| Limitations | 100 pages a pull; a larger board is swept across pulls. Whether the identifier is case-sensitive is unsettled: the module docstring says it is, `docs/decisions.md` says it is not (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
|
||||
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so it cannot be asked for. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text |
|
||||
| Limitations | 100 pages a pull; a larger board is swept across pulls. The identifier is stored with the case it was attached with; whether the host is case-sensitive is unsettled (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
|
||||
|
||||
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until fetch text has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
The reader's Fetch button reads one posting page on demand (see "The search providers"); whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
|
||||
### Workable
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the account name, e.g. `workable:huggingface` |
|
||||
| Companies | 8 boards: Hugging Face, Kahoot!, Starling Bank |
|
||||
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
|
||||
| Requests per pull | roles / 10, rounded up, 0.5 s apart; the largest board has 50 roles, so 5 |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only: 1 of 157 open roles has text |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only |
|
||||
| Date | `published`; trusted |
|
||||
| Place and country | every `locations` entry as "city, region, country", each remote when the role is; `location` when there are none |
|
||||
| Remote | `workplace` (remote, hybrid, on_site) as the host's word, else the `remote` flag |
|
||||
@@ -357,16 +216,14 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the career site's host, or host plus a brand path, e.g. `successfactors:careers.vestas.com`, `successfactors:jobs.teradyne.com/MiR` |
|
||||
| Companies | 14 boards: Novo Nordisk, Vestas, Carlsberg |
|
||||
| List | the site's HTML search page `/search/?startrow=<n>`, parsed in its two known layouts; page size is the site's own (10, 15, 25 or 100) and cannot be set; a total where the pagination label states one |
|
||||
| Requests per pull | roles / the site's page size, rounded up |
|
||||
| Description | per role, the `itemprop="description"` block of the role page |
|
||||
| Date | the list's date column where it parses; the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's location, with a two-letter country code turned into the name; the role page's microdata replaces it |
|
||||
| Remote | none from the host |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from a link to a role, `/go/` page or search on the site's own host; a link to the SAP back end names the host only; not probed. A site on the company's own domain whose pages link only the back end is set by hand |
|
||||
| Filter at the source | `optionsFacetsDD_country=DK`, one country a request; no remote facet. Vestas DK: 64 of 668 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as the form's `q=engineer`, one a request, AND with the country (Vestas DK: 40 of 64) |
|
||||
| Filter at the source | `optionsFacetsDD_country=DK`, one country a request; no remote facet. Fetch terms as the form's `q=engineer`, one a request, AND with the country; whether it matches title or text is not determined (`docs/filter-survey.md`) |
|
||||
| Limitations | Only the default locale's roles are listed: a role that exists only in another locale (Carlsberg's French brand, 37 of 105) is not read. Only `/search/` and `/job/` are read, never `/services/` or the `successfactors.eu` back end, so a company with only the back end (Bang & Olufsen) cannot be read. 100 pages a pull; a larger site is swept across pulls. A third layout or a bot wall that answers 200 fails the board with "not a SuccessFactors search page". Read from this machine only; not yet from the deployed server. |
|
||||
|
||||
### Workday
|
||||
@@ -374,9 +231,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | `<tenant>.<wdN>/<Site>`, site with case, e.g. `workday:lego.wd103/LEGO_External` |
|
||||
| Companies | 17 boards: LEGO Group, Zendesk, Equinor, and since 2026-09-30 Criteo, Darktrace, Novonesis and NVIDIA |
|
||||
| List | a POST to `/wday/cxs/<tenant>/<Site>/jobs`, 20 per request by offset (21 or more answers 400); the first page states the total |
|
||||
| Requests per pull | roles / 20, rounded up; 50 for a board of 1,000 roles |
|
||||
| Description | per role, `jobDescription` from the role's JSON |
|
||||
| Date | the list's "Posted N Days Ago" as a day, possibly one day off; "Posted 30+ Days Ago" and other languages give none; the role's `startDate` replaces it; trusted |
|
||||
| Place and country | the list has `locationsText` ("Copenhagen" or "2 Locations"); the per-role read adds every location and the country, so the country arrives with the role page |
|
||||
@@ -384,7 +239,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| Employment | `timeType`, the host's own word |
|
||||
| Pay | none: no pay field; pay appears only inside the text |
|
||||
| Found by discovery | yes, from a `myworkdayjobs.com` link with a site; not probed |
|
||||
| Filter at the source | the tenant's own facets in `appliedFacets`, ids read by `facets` once a day: the facet whose values are countries, and any value that reads as remote. Two facets intersect, so remote is a series of its own. LEGO DK: 97 of 455 (2026-09-30). A tenant with no country facet is read whole. Which countries is the agents' reach, see above. Fetch terms as `searchText` in the body, one a request, AND with the facets (LEGO: 106 for `engineer`) |
|
||||
| Filter at the source | the tenant's own facets in `appliedFacets`, ids read by `facets` once a day: the facet whose values are countries, and any value that reads as remote. Two facets intersect, so remote is a series of its own. A tenant with no country facet is read whole. Fetch terms as `searchText` in the body, one a request, AND with the facets; whether it matches title or text is not determined (`docs/filter-survey.md`) |
|
||||
| Limitations | 2,000 roles per board (100 pages of 20), the `ceiling`. A tenant has several sites and each is its own board. Every tenant is behind Cloudflare and was read from this machine only. The slug's case is kept (`KEEP_CASE`). |
|
||||
|
||||
### Oracle Recruiting Cloud
|
||||
@@ -392,16 +247,14 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | `<pod host>/<siteNumber>`, e.g. `oracle:fa-evmr-saasfaprod1.fa.ocs.oraclecloud.com/CX_1` |
|
||||
| Companies | 5 boards: Danske Bank, Nokia, Milestone Systems, Sinch, Pioneer Centre for AI |
|
||||
| List | the REST resource `recruitingCEJobRequisitions`, 200 per request by offset, with secondary places expanded; states `TotalJobsCount` |
|
||||
| Requests per pull | roles / 200, rounded up; 3 for Nokia's roughly 600 |
|
||||
| Description | per role: external description, responsibilities and qualifications joined. The list's one-paragraph `ShortDescriptionStr` is stored as the teaser (`READS.teaser=True`), which is shown and embedded until the description arrives |
|
||||
| Date | `PostedDate`; the role's `ExternalPostedStartDate` replaces it when earlier; trusted |
|
||||
| Place and country | the primary place plus every secondary one, each a place of the role |
|
||||
| Remote | `WorkplaceTypeCode` as the host's word (e.g. `ORA_HYBRID`); prep maps it |
|
||||
| Pay | none: the resource states no pay |
|
||||
| Found by discovery | yes, from a candidate site or role link on the pod, or a REST call naming `siteNumber`; not probed |
|
||||
| Filter at the source | `selectedLocationsFacet` with the pod's geography ids, and `selectedWorkplaceTypesFacet=ORA_REMOTE` as a series of its own. `locationsFacet` lists the ten biggest countries only, so a country missing from a full list is read unfiltered: fa-evmr (Nokia) has neither DK nor SE listed and is read whole. Which countries is the agents' reach, see above. Fetch terms as `keyword=` in the finder, one a request; it matches the text (Nokia: 468 of 554 for `engineer`) |
|
||||
| Filter at the source | `selectedLocationsFacet` with the pod's geography ids, and `selectedWorkplaceTypesFacet=ORA_REMOTE` as a series of its own. `locationsFacet` lists the ten biggest countries only, so a country missing from a full list is read unfiltered. Fetch terms as `keyword=` in the finder, one a request; it matches the text |
|
||||
| Limitations | The host is the tenant's Fusion pod, not the company's domain (`jobs.nokia.com` redirects to it). Links are built for locale `en`. The department fields were empty on every role of the first four boards; Pioneer's were not checked. |
|
||||
|
||||
### Phenom
|
||||
@@ -409,16 +262,14 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the career site's host plus its locale root, e.g. `phenom:www.careers.philips.com/global/en` |
|
||||
| Companies | 5 boards: King, Just Eat Takeaway.com, Philips, BCG, Verisure |
|
||||
| List | a POST to `https://<host>/widgets`, 100 per request by offset; states `totalHits` |
|
||||
| Requests per pull | roles / 100, rounded up; 9 for Philips' roughly 880 |
|
||||
| Description | per role, the JSON-LD `JobPosting` description of the role page. The list's `descriptionTeaser` is stored as the teaser (`READS.teaser=True`) |
|
||||
| Date | the list's `dateCreated`, else `postedDate`, which the site moves on a refresh; the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | every place of `multi_location`, each a place of the role |
|
||||
| Remote | per role only, from the page's `checkRemote` |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from a `/<country>/<lang>/search-results` link; a link to Phenom itself names the host only; not probed |
|
||||
| Filter at the source | `selected_fields.country`, a list of English names in one request; no remote. Philips NL and DE: 111 of 873 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `keywords` in the body, one a request, AND with the countries; it matches the text, not the title |
|
||||
| Filter at the source | `selected_fields.country`, a list of English names in one request; no remote. Fetch terms as `keywords` in the body, one a request, AND with the countries; it matches the text, not the title |
|
||||
| Limitations | The locale is built as `<lang>_<country>`, the spelling of the three sites first read; another site may spell it differently. 100 pages a pull; a larger board is swept across pulls. A role page without a description fails the read and is retried on a later run. |
|
||||
|
||||
### Eightfold
|
||||
@@ -426,26 +277,22 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | `<tenant host>/<company domain>`, e.g. `eightfold:jobs.ericsson.com/ericsson.com` |
|
||||
| Companies | 3 boards: Ericsson, Microsoft, Infineon |
|
||||
| List | `https://<host>/api/pcsx/search?domain=<domain>&sort_by=timestamp`, newest first, 10 per request whatever `num` asks; states `count` |
|
||||
| Requests per pull | roles / 10, rounded up, at most 100 |
|
||||
| Description | per role, `jobDescription` from `position_details` |
|
||||
| Date | `creationTs`, else `postedTs`, which the host refreshes in bulk; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location, repeated names dropped, each a place of the role |
|
||||
| Remote | `workLocationOption`: onsite, hybrid, remote |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from an `eightfold.ai` link that carries `domain=`; a bare `eightfold.ai` link names the host only; not probed |
|
||||
| Filter at the source | `location=<country name>`, one country a request; no remote. Ericsson SE: 154 of 618 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `query=`, one a request, AND with the location; it matches title, text and skills (Ericsson SE: 102 of 154 for `software`) |
|
||||
| Limitations | With 10 per request and 100 pages a pull, a board over 1,000 roles is swept across pulls; what the host answers past offset 1,000 was not checked. The country filter brings big boards under the cap: Microsoft (size 2,345) has 29 open roles in reach, and Infineon (999 stored) has none in DK and SE. Verified on three tenants only. |
|
||||
| Filter at the source | `location=<country name>`, one country a request; no remote. Fetch terms as `query=`, one a request, AND with the location; it matches title, text and skills |
|
||||
| Limitations | With 10 per request and 100 pages a pull, a board over 1,000 roles is swept across pulls; what the host answers past offset 1,000 was not checked. The country filter brings big boards under the cap. Verified on three tenants only. |
|
||||
|
||||
### Gem
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board name on `jobs.gem.com`, with case, e.g. `gem:productboard` |
|
||||
| Companies | 2 boards: sennder, Productboard |
|
||||
| List | a GraphQL POST to `jobs.gem.com/api/public/graphql`, the whole board in one answer; no total. Declared as paged with `complete="end"`; it reads one page |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, `descriptionHtml`, one GraphQL POST per role |
|
||||
| Date | per role only, `firstPublishedTsSec`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location as a place (the city, else the location name); `isoCountry` is fetched and not used |
|
||||
@@ -460,9 +307,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the numeric company id, e.g. `jobylon:3143` |
|
||||
| Companies | 2 boards: Einride, Vipps MobilePay |
|
||||
| List | the careers widget's embed page `cdn.jobylon.com/jobs/companies/<id>/embed/v2/`, whose script literal holds the whole board; no total. The list carries a `summary`, stored as the teaser |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, the JSON-LD description of the role page on `emp.jobylon.com` |
|
||||
| Date | the list's day ("Sept. 25, 2026"); the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's place names, which can be cut short ("Houston +3 more"); the per-role read fills every place |
|
||||
@@ -477,9 +322,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the board slug on `ats.rippling.com`, with case, e.g. `rippling:prisma-careers` |
|
||||
| Companies | 1 board: Prisma |
|
||||
| List | `api.rippling.com/platform/api/ats/v1/board/<slug>/jobs`, the whole board in one array, one entry per role and place, merged by role; no total |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, `description.role`, one role record per role |
|
||||
| Date | per role only, `createdOn`; trusted |
|
||||
| Place and country | the list's place labels as places; the per-role read replaces them with every work location |
|
||||
@@ -494,9 +337,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| | |
|
||||
|---|---|
|
||||
| Target | the subdomain, e.g. `bamboohr:configit` |
|
||||
| Companies | 1 board: Configit |
|
||||
| List | `https://<sub>.bamboohr.com/careers/list`, the whole board in one answer; a list shorter than its `totalCount` fails the board rather than being read as whole |
|
||||
| Requests per pull | 1 |
|
||||
| Description | per role, `/careers/<id>/detail` |
|
||||
| Date | per role only, `datePosted`; trusted |
|
||||
| Place and country | the list's city and state as one place; the country arrives with the role page |
|
||||
@@ -508,7 +349,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
|
||||
## The search providers
|
||||
|
||||
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads up to 5 pages by default and at most 10. A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; `embed` and `extract` read the description, else the teaser. The reader shows "Fetch the posting" for any role with no `article_text` and a description under 500 characters; it fetches the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not. For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text (`app/jobs/posting.py`). A search that fails is skipped for the pull and leaves no record. No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
|
||||
A saved search is a query with its parameters, stored as a `search` subscription `provider:name`. It reads 5 pages by default and at most 10 (`DEFAULT_PAGES` and `MAX_PAGES` in `plugins/search.py`). A search that finds nothing is not saved. A provider that sends only an excerpt (Jobindex, Adzuna) stores it as the role's `teaser`, and a provider that sends the text (Platsbanken, freehire, Jobnet) stores it as the description; Jobbank states the text only on a role's own page, which fetch text reads. The reader's Fetch button shows for a role with a link, no `article_text` and a description under 500 characters. It reads the posting page once, on demand, into `article_text`, which the reader and `summarise` use and `embed` and `extract` do not (`app/jobs/posting.py`). For a Jobindex role the fetch follows the "Se jobbet" link of Jobindex's page to the posting itself, keeps the link as `roles.posting_url` and falls back to Jobindex's own text; for a Jobbank role it calls the reader's `detail`. A search that fails is recorded as `search:failing:<target>` and skipped for the pull (`plugins/search.py`). No provider reads robots.txt at run time; the decisions for Jobindex (which is read although its robots.txt disallows `sort=`, `page=` and `geoareaid=`) and Adzuna (whose API host disallows `/` for every agent) are in `READS.robots` and `docs/decisions.md`.
|
||||
|
||||
### Jobindex
|
||||
|
||||
@@ -517,7 +358,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| Saved search | `q` in Jobindex's syntax (words are OR, `"a b"` a phrase, `+word` required) and optionally `area`, Jobindex's region id (1 Region Hovedstaden, 8 Aarhus, ...) |
|
||||
| List | the RSS twin of the search page, `jobindex.dk/jobsoegning.rss`, newest first, 20 per page; no total. `READS.complete="never"`: a search is never complete, so its roles close by the 21 days only |
|
||||
| Pages read | 5 by default (100 results), 10 at most (200) |
|
||||
| Text | an excerpt of a couple of hundred characters from the feed, stored as the teaser; 561 of 562 open roles have some text. The full text only through "Fetch the posting", on demand, since every fetch of Jobindex's link counts as a click in its statistics |
|
||||
| Text | an excerpt of a couple of hundred characters from the feed, stored as the teaser. The full text only through "Fetch the posting", on demand, since every fetch of Jobindex's link counts as a click in its statistics |
|
||||
| Date | `pubDate`; trusted |
|
||||
| Place | the item's area, else the search's area; country always Denmark |
|
||||
| Remote | the reader states none; prep reads place and title words |
|
||||
@@ -575,7 +416,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| | |
|
||||
|---|---|
|
||||
| Saved search | `q` and optionally `region`, one of Jobnet's five region ids (`REGIONS` in the module); empty for all of Denmark |
|
||||
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login; this supersedes the earlier note that its search answers 401 outside its own session |
|
||||
| List | `jobnet.dk/bff/FindJob/Search`, newest first, 100 per page; states `totalJobAdCount`. The site's own backend answers 401 without the header `x-csrf: 1` and 200 with it, with no login |
|
||||
| Pages read | 5 by default (500 results), 10 at most (1,000); page 101 at 100 a page answers empty |
|
||||
| Text | the whole text, as HTML in the answer |
|
||||
| Date | `publicationDate`; trusted |
|
||||
@@ -601,20 +442,3 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| Limitations | Academic and research roles, plus some corporate ads. robots.txt allows `/job/` and disallows `/fwd/webjob/`, which is never followed. Cloudflare may answer a challenge instead of the page, which is an error for that role and not an empty text. |
|
||||
|
||||
Jooble was left out: it gives a key only on a manual application and answered a Cloudflare challenge. Jobdanmark was left out too: about a quarter of its newest ads are on Jobnet, freehire carries the rest as source `jobdanmark`, and its list has no text. LinkedIn's guest pages are out because its robots.txt disallows `/` and its terms forbid automated access; LinkedIn enters only as alert mail. Indeed has no public search API.
|
||||
|
||||
## Sources that cannot be read, and why
|
||||
|
||||
The table is the seed's: 76 of the 362 companies in `seed/companies.json` have no board there and carry a careers link the survey opened on 2026-09-29. The live database has moved on, and its Watchlist tab and `app companies by-hand` are the current list. Since the seed was written, Infineon and Microsoft (Eightfold) and Novonesis, Darktrace and Criteo (Workday) have been attached, so they are struck from the company lists below; the seed still names them.
|
||||
|
||||
| Reason | Seed | Live 2026-09-30 | What it means | Companies |
|
||||
|---|---|---|---|---|
|
||||
| `unread` | 35 | 35 | readable by a program; no module reads it yet | Zalando, Ørsted, Siemens, Google DeepMind, Amazon, Apple, ASML, Netcompany, Tryg, DNB |
|
||||
| `robots` | 10 | 7 | the site's robots.txt disallows the page a board would be read from | Bang & Olufsen (SuccessFactors), Pandora, Terma, Zenegy, Umbraco, Snyk, Seaborg (HR Manager) |
|
||||
| `group` | 8 | 8 | the roles are inside another company's board, with no field that separates them | Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio, Native Instruments, Visma |
|
||||
| `blocked` | 7 | 8 | the site answers a program with a challenge | Tradeshift, Personio, Adjust, Revolut, Silverfin, HashiCorp, Meta |
|
||||
| `gone` | 4 | 5 | no careers page | wefox, WeTransfer, Featurespace, Unbabel |
|
||||
| `key` | 3 | 3 | the only listing answers with a key the page's script carries | Maersk, New Work SE, Yousign (Welcome to the Jungle) |
|
||||
| `added` | – | 14 | added by hand or by an import and not looked at yet | Danish Centre for AI Innovation, 2021.AI, McKinsey, AMD, Ingka Group and nine more |
|
||||
| no reason | 9 | 9 | on Teamtailor, but the careers link is a marketing site and the career host is not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial |
|
||||
|
||||
The live counts are companies with no `ats` board subscription; the company lists are the seed's, less the five attached companies. What is planned for the `unread` companies and for the gaps in hosts already read (SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, Avature for Siemens) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
|
||||
+14
-6
@@ -17,8 +17,9 @@ A target is `host:slug`, e.g. `greenhouse:databricks`, `ashby:deepl`,
|
||||
`lever:eu/tomtom`, `teamtailor:careers.acme.com`; each host module says what
|
||||
its slug is. Every board is read independently; one that fails is logged,
|
||||
recorded as failing under `ats:failing:<target>`, and skipped, and so is one
|
||||
whose read lists under a tenth of what it listed last time (`_runner.read`'s
|
||||
guard), until it has failed that way for the pull's `stale_days`. A board's
|
||||
whose read lists under a tenth of what it listed last time, or drops more
|
||||
than a tenth of the rows its reader handed over (`_runner.read`'s guards),
|
||||
until it has failed that way for the pull's `stale_days`. A board's
|
||||
filter is the `reach` option the pull hands it (the agents', see
|
||||
`app/jobs/reach.py`), over which its subscription `params` (`{"countries":
|
||||
["DK"], "remote": true}`) win, narrowed to what its reader declares
|
||||
@@ -343,8 +344,10 @@ SIZE_PAGES = 100
|
||||
def board_size(target: str, *, client: httpx.Client | None = None) -> int:
|
||||
"""How many roles board `target` lists, unfiltered and without terms: a
|
||||
whole host's listing, the total a `complete="stated"` host gives on page
|
||||
1, else the listing read to the end, at most `SIZE_PAGES` pages. Raises
|
||||
as `_fetch_rows` does, so a board that does not answer is refused."""
|
||||
1, else the listing read to the end, at most `SIZE_PAGES` pages. A read
|
||||
counts the rows the runner dropped as well (`_listed`), as a stated total
|
||||
does. Raises as `_fetch_rows` does, so a board that does not answer is
|
||||
refused."""
|
||||
if not is_known_target(target):
|
||||
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
|
||||
if client is None:
|
||||
@@ -358,7 +361,12 @@ def board_size(target: str, *, client: httpx.Client | None = None) -> int:
|
||||
if isinstance(total, int):
|
||||
return total
|
||||
limits = _runner.Limits(max_pages=SIZE_PAGES, page_delay=PAGE_DELAY)
|
||||
return len(_runner.read_listing(module, slug, get, limits=limits).ids)
|
||||
return _listed(_runner.read_listing(module, slug, get, limits=limits))
|
||||
|
||||
|
||||
def _listed(listing: _runner.Listing) -> int:
|
||||
"""The listings a read found: the ids it kept and the rows it dropped."""
|
||||
return len(listing.ids) + listing.dropped
|
||||
|
||||
|
||||
def keep_size(state: Any, target: str, size: int, *, today: str = "",
|
||||
@@ -462,7 +470,7 @@ class ATSSource(Source):
|
||||
[compound(i) for i in listing.ids])
|
||||
self.state.pop(failing, None)
|
||||
if unsized and listing.complete and not (listing.filtered or listing.narrowed):
|
||||
keep_size(self.state, target, len(listing.ids), reset=False)
|
||||
keep_size(self.state, target, _listed(listing), reset=False)
|
||||
if detail_sets is not None:
|
||||
# A complete listing drops the ids it no longer lists.
|
||||
for key, ids in zip(detail_keys(target), detail_sets, strict=False):
|
||||
|
||||
@@ -167,15 +167,19 @@ shared code logs, records and decides.
|
||||
handed to `rows`, its raw results as dicts with an `id` (counted, by distinct
|
||||
id, to decide completeness), the total the host states or None, and the
|
||||
cursor of the next page or None after the last. The first call gets cursor
|
||||
None. A board with a total is complete when as many distinct ids were read;
|
||||
one without, when the read ended on an empty page. A 429 after the first page
|
||||
None, and a later one may get its cursor back from JSON, a tuple as a list,
|
||||
when a sweep continues. A board with a total is complete when as many
|
||||
distinct ids were read; one without, when the read ended on an empty page.
|
||||
A reader that sees a page come again hands it over with no results and a
|
||||
cursor that is not None, which ends the read incomplete; None there would
|
||||
read as the last page. A 429 after the first page
|
||||
ends the read, incomplete; `_runner.MAX_PAGES` pages a pull at most,
|
||||
`_runner.PAGE_DELAY` apart, and a board cut there is swept across pulls
|
||||
(`_runner.read_listing`). A paged module may also have `parse(body, slug)` for one page's body.
|
||||
|
||||
**Rows.** A `Row` (`_contract.Row`) is a dict with `source_id` (the host's
|
||||
own id for the role, unique on the board), `title`, `url` (http or https,
|
||||
checked with `_shared.is_web_url`) and `places`; `location` (the host's
|
||||
which the runner checks with `_shared.is_web_url`) and `places`; `location` (the host's
|
||||
one-line place, kept while storage reads it), `posted` (an aware datetime
|
||||
or None, see `_shared.parse_dt`), `department`, `description` (plain text,
|
||||
see `_shared.html_text`), `remote` (True, False, a word or None) and
|
||||
@@ -183,7 +187,12 @@ see `_shared.html_text`), `remote` (True, False, a word or None) and
|
||||
(yearly integers) and `currency` (ISO 4217), see `_shared.yearly_amount`;
|
||||
where it states them, `country` (ISO 3166 alpha-2), `seniority` and
|
||||
`employment`; `teaser`; and, on a search row, `company`. The runner keeps
|
||||
every stated key in `raw`, so a new field needs no change there.
|
||||
every stated key in `raw`, so a new field needs no change there. A reader
|
||||
never drops a listing: one it cannot make a whole row of (no id, no title or
|
||||
no link) it hands over with what it has, an empty string for what is
|
||||
missing, and the runner drops it and counts it (`_runner.valid`). A reader
|
||||
leaves out only what is not a listing of the board: an entry that is not an
|
||||
object, a second copy of an id, a role of another board.
|
||||
|
||||
**Places.** A role has a list of places: `places` is `[Place(text=...,
|
||||
remote=...)]`, one entry per place the host states, in the host's order, in
|
||||
|
||||
@@ -205,6 +205,8 @@ class Row(TypedDict, total=False):
|
||||
`company` is the employer's name on a search row, which lists many
|
||||
(`plugins/search.py` stores it); a board row carries none, the board is
|
||||
the company. Any other stated key is kept in `raw` by the runner.
|
||||
`source_id`, `title` and `url` come empty from a listing that lacks
|
||||
them: the runner drops such a row and counts it (`_runner.valid`).
|
||||
"""
|
||||
|
||||
source_id: Required[str]
|
||||
|
||||
+132
-31
@@ -4,10 +4,12 @@ the reader declares in `READS`.
|
||||
A reader says how its host answers; this module reads it: the guarded fetch
|
||||
(`fetch`, bound to one target by `getter`), the `Ask` a subscription resolves
|
||||
to (`ask_for`) and its series (`series`), the listing whole or paged, one
|
||||
series per `one`-filter value unioned by id (`read_listing`), completeness,
|
||||
the sweep of an oversized paged board across pulls, the 10% guard on every
|
||||
read that is not a sweep (`read`), the per-role detail reads
|
||||
(`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
|
||||
series per `one`-filter value unioned by id (`read_listing`), the rows it
|
||||
keeps (`valid`: a row without an id, a title or a link is dropped and
|
||||
counted here, never by the reader), completeness, the sweep of an oversized
|
||||
paged board across pulls, the two 10% guards (`read`: on what a read that
|
||||
is not a sweep listed, and on what any read dropped), the per-role detail
|
||||
reads (`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
|
||||
bookkeeping (`record_failure`, `failure_reason`). `plugins/ats.py` and
|
||||
`plugins/search.py` are thin `Source` classes over it: they pass their own
|
||||
limits (`Limits`) and state prefix (`ats`, `search`), so their state keys stay
|
||||
@@ -30,6 +32,8 @@ State keys, under the Source's prefix and the target (`ats:sweep:<target>`):
|
||||
drifts when roles close between pulls, so `app.jobs.pull` closes
|
||||
a role of such a board only when two finished sweeps in a row
|
||||
missed it (it keeps the ids the last one missed in `missing`)
|
||||
dropped the rows the last stamped read dropped (`Listing.dropped`), as a
|
||||
number, "" for none; `app.jobs.limits` names it on the board's row
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -52,7 +56,7 @@ import httpx
|
||||
from desk.sources import USER_AGENT
|
||||
|
||||
from ._contract import Ask, Key, Place, Reads
|
||||
from ._shared import Get, Row
|
||||
from ._shared import Get, Row, is_web_url
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -96,7 +100,8 @@ SUSPECT_SHARE = 0.1
|
||||
#: The last count from which the guard holds every read. A read that is
|
||||
#: filtered, narrowed by terms or of an `empty="suspect"` reader is held from
|
||||
#: a count of 1, as it was before every read was; a whole read of a board
|
||||
#: that listed a handful of roles can drop to none by itself.
|
||||
#: that listed a handful of roles can drop to none by itself. It is also the
|
||||
#: rows a read must hand over before the share it dropped can fail it.
|
||||
SUSPECT_MIN = 10
|
||||
|
||||
#: The keys of a row that an `Item` carries in its own fields, not in `raw`.
|
||||
@@ -113,8 +118,8 @@ class TooLarge(ValueError):
|
||||
|
||||
|
||||
class Suspect(Exception):
|
||||
"""A read that listed under `SUSPECT_SHARE` of what it listed last time
|
||||
(see `read`)."""
|
||||
"""A read that listed under `SUSPECT_SHARE` of what it listed last time,
|
||||
or dropped more than that share of its rows (see `read`)."""
|
||||
|
||||
|
||||
class MissingKey(ValueError):
|
||||
@@ -147,7 +152,9 @@ class Listing:
|
||||
and roles closing meanwhile may have shifted an offset-paged host's later
|
||||
roles back past the cursor, so `ids` can miss a role that is still open.
|
||||
`remote_unasked` says a filtered read was meant to ask for remote roles
|
||||
as well and could not, so its `ids` say nothing of the remote roles."""
|
||||
as well and could not, so its `ids` say nothing of the remote roles.
|
||||
`handed` counts the rows the reader handed over this pull, `dropped`
|
||||
those of them `valid` refused, which are in neither `rows` nor `ids`."""
|
||||
|
||||
rows: list[Row]
|
||||
complete: bool
|
||||
@@ -158,6 +165,8 @@ class Listing:
|
||||
narrowed: bool = False
|
||||
swept: bool = False
|
||||
remote_unasked: bool = False
|
||||
handed: int = 0
|
||||
dropped: int = 0
|
||||
|
||||
|
||||
# --------------------------------------------------------------- modules
|
||||
@@ -397,7 +406,7 @@ def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None =
|
||||
its reach (see `app/jobs/reach.py`), and a board whose params name a query
|
||||
of their own takes none. Terms times the reach's series (`_cost`) may cost
|
||||
at most `MAX_SERIES`. Over it, a wide read (`reach["wide"]`, a company set
|
||||
to All countries, or a reach this rule just dropped as too many
|
||||
to All locations, or a reach this rule just dropped as too many
|
||||
countries) drops the place part, whole, since remote without its
|
||||
countries would read only the remote roles, and keeps the terms. A narrow
|
||||
read keeps the places, which the agent's fetch gate needs, and drops the
|
||||
@@ -524,8 +533,42 @@ def _result_ids(results: Iterable[dict[str, Any]]) -> set[str]:
|
||||
return {str(r.get("id") or r.get("shortcode") or "") for r in results}
|
||||
|
||||
|
||||
def valid(row: Mapping[str, Any]) -> bool:
|
||||
"""Whether the runner keeps a row: a non-empty `source_id`, a non-empty
|
||||
`title` and a web URL (`is_web_url`). A reader hands over every listing
|
||||
as far as it can make it out; a row without one of the three cannot be
|
||||
stored or opened, and the runner drops it and counts it."""
|
||||
try:
|
||||
link = is_web_url(str(row.get("url") or ""))
|
||||
except (httpx.InvalidURL, ValueError): # a string httpx cannot read as a URL
|
||||
link = False
|
||||
return (link and bool(str(row.get("source_id") or "").strip())
|
||||
and bool(str(row.get("title") or "").strip()))
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _Tally:
|
||||
"""The rows a read was handed and dropped, across its series and pages."""
|
||||
|
||||
handed: int = 0
|
||||
dropped: int = 0
|
||||
|
||||
|
||||
def _keep(found: Iterable[Row], rows: dict[str, Row], tally: _Tally) -> None:
|
||||
"""The rows a reader's `parse` or `rows` returned, into `rows` by
|
||||
`source_id`, the first of an id kept; those `valid` refuses are counted
|
||||
in `tally` and left out. The one place a row is checked, for the whole
|
||||
and the paged listing alike."""
|
||||
for row in found:
|
||||
tally.handed += 1
|
||||
if not valid(row):
|
||||
tally.dropped += 1
|
||||
continue
|
||||
rows.setdefault(row["source_id"], row)
|
||||
|
||||
|
||||
def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
|
||||
limits: Limits) -> dict[str, Row]:
|
||||
limits: Limits, tally: _Tally) -> dict[str, Row]:
|
||||
"""A whole listing, one request a series, unioned by `source_id`. A body
|
||||
over the cap gets one retry at the reader's `fallback_url`."""
|
||||
rows: dict[str, Row] = {}
|
||||
@@ -541,8 +584,7 @@ def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
|
||||
log.warning("readers: %s:%s: body over the cap; reading the smaller feed",
|
||||
module.NAME, slug)
|
||||
body = get(module.fallback_url(slug))
|
||||
for row in module.parse(body, slug):
|
||||
rows.setdefault(row["source_id"], row)
|
||||
_keep(module.parse(body, slug), rows, tally)
|
||||
return rows
|
||||
|
||||
|
||||
@@ -556,17 +598,21 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
|
||||
A whole listing is complete whenever it parses. A paged one reads each
|
||||
series' pages, `limits.page_delay` apart, until the host has no next
|
||||
page, the stated total is reached, a page brings nothing new (a host
|
||||
that ignores the cursor, incomplete) or `limits.max_pages` pages in all.
|
||||
A series is complete when it read as many distinct result ids as its
|
||||
stated total, or, without a total, when it ended on the host's last
|
||||
page; never for `complete="never"`. The read is complete when every
|
||||
series is. A 429 after the first page ends the read, incomplete.
|
||||
page, the stated total is reached, the read stalls or `limits.max_pages`
|
||||
pages in all. A stall is a page that brings no new id while the reader
|
||||
still hands a cursor: any such page on a host that states no total (it
|
||||
ignores the cursor), and with a total one the reader hands over empty
|
||||
(it saw the page come again). A series is complete when it read as many distinct result
|
||||
ids as its stated total, or, without a total, when it ended on the
|
||||
host's last page; never after a stall, and never for
|
||||
`complete="never"`. The read is complete when every series is. A 429
|
||||
after the first page ends the read, incomplete.
|
||||
|
||||
With `can_sweep` (boards), a read cut at `max_pages` or by a 429 returns
|
||||
the `sweep` to keep, and the next pull, handed it back, continues from
|
||||
its series and cursor; the sweep is complete when it reaches the end,
|
||||
and `ids` then hold every id it listed, and the listing is `swept`. A
|
||||
and `ids` then hold every id it listed, and the listing is `swept`. The
|
||||
cursor comes back as JSON gives it, a tuple as a list (`_storable`). A
|
||||
read at the reader's `ceiling` is not swept: the host answers nothing
|
||||
past it.
|
||||
|
||||
@@ -576,7 +622,29 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
only: a search's places are what the search is, as its query is.
|
||||
`remote` says the reach or the params wanted remote roles; when no series
|
||||
asked for them (a reader without a remote filter, or one whose facets
|
||||
state none beside places) the listing is `remote_unasked`."""
|
||||
state none beside places) the listing is `remote_unasked`.
|
||||
|
||||
Every row `rows` or `parse` returns goes through `valid`: one without a
|
||||
`source_id`, a title or a web URL is dropped here, counted in
|
||||
`Listing.dropped` and logged once for the read. Completeness counts the
|
||||
distinct result ids `page` returns beside its rows, not the rows kept, so
|
||||
a dropped row still counts as read; a whole listing is complete whenever
|
||||
it parses, whatever it dropped."""
|
||||
tally = _Tally()
|
||||
listing = _listing(module, slug, get, ask, limits=limits, sweep=sweep,
|
||||
can_sweep=can_sweep, facets=facets, remote=remote, tally=tally)
|
||||
listing.handed, listing.dropped = tally.handed, tally.dropped
|
||||
if tally.dropped:
|
||||
log.warning("readers: %s:%s: dropped %d of %d listings: no id, title or link",
|
||||
module.NAME, slug, tally.dropped, tally.handed)
|
||||
return listing
|
||||
|
||||
|
||||
def _listing(module: ModuleType, slug: str, get: Get, ask: Ask | None, *,
|
||||
limits: Limits | None, sweep: Mapping[str, Any] | None, can_sweep: bool,
|
||||
facets: Mapping[str, Mapping[str, str]] | None, remote: bool,
|
||||
tally: _Tally) -> Listing:
|
||||
"""`read_listing` before its count of dropped rows, which `tally` keeps."""
|
||||
limits = limits or Limits()
|
||||
ask = ask or Ask()
|
||||
reads: Reads = module.READS
|
||||
@@ -591,7 +659,7 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
"nothing asked, not closing", module.NAME, slug)
|
||||
return Listing([], False, [], True, None, key, by_words, remote_unasked=unasked)
|
||||
if not hasattr(module, "page"):
|
||||
whole_rows = _whole(module, slug, get, asks, limits)
|
||||
whole_rows = _whole(module, slug, get, asks, limits, tally)
|
||||
return Listing(list(whole_rows.values()), True, list(whole_rows), is_filtered, None,
|
||||
key, by_words, remote_unasked=unasked)
|
||||
|
||||
@@ -628,12 +696,17 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
pages += 1
|
||||
before = len(seen)
|
||||
seen |= _result_ids(results)
|
||||
for row in module.rows(data, slug):
|
||||
rows.setdefault(row["source_id"], row)
|
||||
_keep(module.rows(data, slug), rows, tally)
|
||||
if cursor is None or (isinstance(total, int) and len(seen) >= total):
|
||||
ended = True
|
||||
break
|
||||
if total is None and len(seen) == before:
|
||||
# Nothing new while the reader says more follow. Without a total
|
||||
# the host ignores the cursor, and the read would never end. With
|
||||
# one, only a page the reader hands over empty is a stall (it saw
|
||||
# the page come again, as SuccessFactors does): a page of ids
|
||||
# already read is what a resumed sweep starts on when the list
|
||||
# grew at the top since, and the read goes on from there.
|
||||
if len(seen) == before and (total is None or not results):
|
||||
stalled = True
|
||||
break
|
||||
if stop:
|
||||
@@ -663,13 +736,26 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
|
||||
|
||||
def _storable(cursor: Any) -> bool:
|
||||
"""Whether a cursor survives a trip through JSON in state."""
|
||||
"""Whether a cursor survives a trip through JSON in state. A tuple comes
|
||||
back as a list (Workday's `(offset, total)`, SuccessFactors' `(start
|
||||
row, first id)`), which counts as surviving: the next pull hands the
|
||||
reader its cursor as JSON gives it back, and every paged reader takes
|
||||
the list as it took the tuple."""
|
||||
try:
|
||||
return json.loads(json.dumps(cursor)) == cursor
|
||||
return json.loads(json.dumps(cursor)) == _as_json(cursor)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _as_json(value: Any) -> Any:
|
||||
"""`value` with every tuple in it a list, as JSON hands it back."""
|
||||
if isinstance(value, tuple | list):
|
||||
return [_as_json(item) for item in value]
|
||||
if isinstance(value, dict):
|
||||
return {key: _as_json(item) for key, item in value.items()}
|
||||
return value
|
||||
|
||||
|
||||
# ------------------------------------------------------------ with state
|
||||
|
||||
def _load(state: Mapping[str, str], key: str) -> dict[str, Any]:
|
||||
@@ -724,13 +810,26 @@ def read(state: MutableMapping[str, str], prefix: str, target: str, module: Modu
|
||||
that really shrank is accepted once the target has been failing for
|
||||
`limits.stale_days` (its `<prefix>:failing:<target>` record), and its
|
||||
stamp then resets the count; a Recheck resets it at once
|
||||
(`plugins.ats.keep_size`)."""
|
||||
(`plugins.ats.keep_size`).
|
||||
|
||||
A read, a sweep's included, that dropped more than `SUSPECT_SHARE` of
|
||||
the rows its reader handed over (`valid`), once it was handed at least
|
||||
`SUSPECT_MIN`, raises `Suspect` the same way: a page whose shape changed
|
||||
parses to rows without a title or a link, and the roles it hid would
|
||||
close. It gives way after `limits.stale_days` of failing as well."""
|
||||
limits = limits or Limits()
|
||||
ask = ask or Ask()
|
||||
facets = facets_for(module, slug, get, ask, state, f"{prefix}:facets:{target}")
|
||||
listing = read_listing(module, slug, get, ask, limits=limits, can_sweep=can_sweep,
|
||||
sweep=_load(state, f"{prefix}:sweep:{target}"), facets=facets,
|
||||
remote=remote)
|
||||
if listing.handed >= SUSPECT_MIN and listing.dropped > SUSPECT_SHARE * listing.handed:
|
||||
why = (f"dropped {listing.dropped} of {listing.handed} listings, over "
|
||||
f"{SUSPECT_SHARE:.0%}: no id, title or link")
|
||||
if not _failing_for(state, f"{prefix}:failing:{target}", limits.stale_days):
|
||||
raise Suspect(why)
|
||||
log.warning("readers: %s:%s: %s, accepted after %d days of failing",
|
||||
module.NAME, slug, why, limits.stale_days)
|
||||
held = listing.filtered or listing.narrowed or module.READS.empty == "suspect"
|
||||
last = _load(state, f"{prefix}:count:{target}")
|
||||
count = int(last.get("count") or 0) if last.get("asks") == listing.asks else 0
|
||||
@@ -765,9 +864,10 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
|
||||
`complete`, `ids` (the compound ids, what the pull closes against),
|
||||
`filtered`, `narrowed` (a read by terms, which the pull never closes
|
||||
against), whether it finished a sweep from earlier pulls (`swept`),
|
||||
whether a filtered read left remote unasked (`remote_unasked`), the
|
||||
sweep to continue or none, and the count for the 10% guard once a read
|
||||
is not a sweep. A read that asked nothing keeps the count it had."""
|
||||
whether a filtered read left remote unasked (`remote_unasked`), the rows
|
||||
it dropped (`dropped`, "" for none), the sweep to continue or none, and
|
||||
the count for the 10% guard once a read is not a sweep. A read that
|
||||
asked nothing keeps the count it had."""
|
||||
state[f"{prefix}:ok:{target}"] = run
|
||||
state[f"{prefix}:complete:{target}"] = "1" if listing.complete else ""
|
||||
state[f"{prefix}:swept:{target}"] = "1" if listing.swept else ""
|
||||
@@ -775,6 +875,7 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
|
||||
state[f"{prefix}:filtered:{target}"] = "1" if listing.filtered else ""
|
||||
state[f"{prefix}:narrowed:{target}"] = "1" if listing.narrowed else ""
|
||||
state[f"{prefix}:remote_unasked:{target}"] = "1" if listing.remote_unasked else ""
|
||||
state[f"{prefix}:dropped:{target}"] = str(listing.dropped) if listing.dropped else ""
|
||||
if listing.sweep is not None:
|
||||
state[f"{prefix}:sweep:{target}"] = json.dumps(listing.sweep)
|
||||
return
|
||||
|
||||
@@ -24,7 +24,7 @@ from urllib.parse import urlencode
|
||||
from desk.sources import strip_html
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Key, Place, Reads
|
||||
from ._shared import Get, Page, Row, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, only, parse_dt
|
||||
|
||||
NAME = "adzuna"
|
||||
LABEL = "Adzuna"
|
||||
@@ -111,8 +111,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
for job in decoded["results"]:
|
||||
title = _text(job.get("title") or "")
|
||||
url = (job.get("redirect_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
predicted = str(job.get("salary_is_predicted") or "0") == "1"
|
||||
town = ((job.get("location") or {}).get("display_name") or "").strip()
|
||||
places: list[Place] = [{"text": town, "remote": None}] if town else []
|
||||
|
||||
@@ -14,7 +14,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads, Row
|
||||
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -126,8 +126,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data.get("jobs") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("jobUrl") or j.get("applyUrl") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
description = j.get("descriptionHtml") or j.get("descriptionPlain") or ""
|
||||
compensation = j.get("compensation") or {}
|
||||
salary_min, salary_max, currency = _salary(compensation.get("summaryComponents") or [])
|
||||
|
||||
@@ -106,8 +106,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in listed:
|
||||
title = (j.get("jobOpeningName") or "").strip()
|
||||
source_id = str(j.get("id") or "")
|
||||
if not title or not source_id:
|
||||
continue
|
||||
places = _places(j)
|
||||
row: Row = {
|
||||
"source_id": source_id,
|
||||
|
||||
@@ -132,8 +132,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for p in postings:
|
||||
title = (p.get("title") or "").strip()
|
||||
post_id = str(p.get("id") or "")
|
||||
if not title or not post_id:
|
||||
continue
|
||||
job = p.get("job") or {}
|
||||
places = [name for loc in job.get("jobLocations") or []
|
||||
if (name := ((loc.get("location") or {}).get("name") or "").strip())]
|
||||
|
||||
@@ -47,7 +47,7 @@ from urllib.parse import quote
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads, Row
|
||||
from ._countries import name_of
|
||||
from ._shared import Get, Page, add_place, html_text, is_web_url, only, parse_dt, place
|
||||
from ._shared import Get, Page, add_place, html_text, only, parse_dt, place
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=10, text="detail", complete="stated",
|
||||
@@ -151,8 +151,6 @@ def rows(positions: list[dict[str, Any]], slug: str) -> list[Row]:
|
||||
source_id = str(p.get("id") or "")
|
||||
path = str(p.get("positionUrl") or "").strip()
|
||||
url = f"https://{host}{path}" if path.startswith("/") and path[1:2] != "/" else ""
|
||||
if not title or not source_id or not is_web_url(url):
|
||||
continue
|
||||
places = _places(p.get("locations") or p.get("standardizedLocations"))
|
||||
out.append({
|
||||
"source_id": source_id,
|
||||
|
||||
@@ -37,7 +37,7 @@ from urllib.parse import urlencode, urlsplit, urlunsplit
|
||||
from desk.sources import USER_AGENT
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import (Get, Page, Row, html_text, is_web_url, num, only, parse_dt, pay_line,
|
||||
from ._shared import (Get, Page, Row, html_text, num, only, parse_dt, pay_line,
|
||||
yearly_amount)
|
||||
|
||||
NAME = "freehire"
|
||||
@@ -170,8 +170,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
title = " ".join(html.unescape(str(hit.get("title") or "")).split())
|
||||
source_id = str(hit.get("public_slug") or "").strip()
|
||||
url = _url(str(hit.get("url") or ""))
|
||||
if not title or not source_id or not is_web_url(url):
|
||||
continue
|
||||
mode = str(hit.get("work_mode") or "").strip()
|
||||
places = _places(hit, mode)
|
||||
countries = [str(c) for c in hit.get("countries") or [] if c]
|
||||
|
||||
@@ -116,8 +116,6 @@ def rows(postings: list[dict[str, Any]], slug: str) -> list[Row]:
|
||||
for p in postings:
|
||||
title = (p.get("title") or "").strip()
|
||||
ext_id = str(p.get("extId") or "")
|
||||
if not title or not ext_id:
|
||||
continue
|
||||
locations = p.get("locations") or []
|
||||
places: list[Place] = []
|
||||
for loc in locations:
|
||||
|
||||
@@ -13,7 +13,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads, Row
|
||||
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place
|
||||
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place
|
||||
|
||||
NAME = "greenhouse"
|
||||
LABEL = "Greenhouse"
|
||||
@@ -149,8 +149,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data.get("jobs") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("absolute_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
departments = j.get("departments") or []
|
||||
ranges = j.get("pay_input_ranges") or []
|
||||
salary_min, salary_max, currency = _salary(ranges)
|
||||
|
||||
@@ -90,8 +90,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
for ad in data.get("jobAdDetails") or []:
|
||||
title = (ad.get("title") or "").strip()
|
||||
ad_id = str(ad.get("id") or "").strip()
|
||||
if not title or not ad_id:
|
||||
continue
|
||||
site = str(ad.get("site") or "").strip()
|
||||
country = str(ad.get("country") or "").strip()
|
||||
location = f"{site}, {country}" if site and country and country.lower() not in site.lower() \
|
||||
|
||||
@@ -30,7 +30,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Reads, Row
|
||||
from ._shared import Get, Page, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, html_text, only, parse_dt
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=100, text="listing", complete="end",
|
||||
@@ -106,8 +106,6 @@ def rows(items: list[ET.Element], slug: str) -> list[Row]:
|
||||
for item in items:
|
||||
title = (item.findtext("title") or "").strip()
|
||||
url = (item.findtext("link") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
out.append({
|
||||
"source_id": (item.findtext("guid") or url).strip(),
|
||||
"title": title,
|
||||
|
||||
@@ -127,8 +127,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
j = job.get("data") or {}
|
||||
title = (j.get("title") or "").strip()
|
||||
role = str(j.get("slug") or j.get("req_id") or "")
|
||||
if not title or not role:
|
||||
continue
|
||||
categories = j.get("categories") or []
|
||||
text = "\n".join(p for p in (html_text(j.get("description")),
|
||||
html_text(j.get("qualifications"))) if p)
|
||||
|
||||
@@ -36,7 +36,7 @@ from typing import Any
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt, place
|
||||
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt, place
|
||||
|
||||
NAME = "jobbank"
|
||||
LABEL = "Akademikernes Jobbank (Denmark)"
|
||||
@@ -132,8 +132,8 @@ def _query(ask: Ask, number: int) -> tuple[str, str]:
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
|
||||
"""One page of a search; the cursor is the page number. The next page
|
||||
follows a full page (counted before any item is dropped, so a dropped item
|
||||
does not pass for the end) while the ceiling is not reached."""
|
||||
follows a full page (counted before the runner drops an item, so a
|
||||
dropped item does not pass for the end) while the ceiling is not reached."""
|
||||
number = int(cursor or 1)
|
||||
url, remote = _query(ask, number)
|
||||
try:
|
||||
@@ -184,8 +184,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
out: list[Row] = []
|
||||
for item in decoded["items"]:
|
||||
link, title = item["link"], item["title"]
|
||||
if not title or not item["id"] or not is_web_url(link):
|
||||
continue
|
||||
line = _LINE.fullmatch(item["description"])
|
||||
kind = line.group("kind").strip() if line else ""
|
||||
company, regions, cut = _regions(line.group("rest")) if line else ("", [], False)
|
||||
|
||||
@@ -117,9 +117,9 @@ def _query(ask: Ask, number: int) -> str:
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
|
||||
"""One page of a search, newest first; the cursor is the page number.
|
||||
The next page follows a full page (counted before any item is dropped, so
|
||||
a dropped item does not pass for the end) while the 200-result ceiling is
|
||||
not reached."""
|
||||
The next page follows a full page (counted before the runner drops an
|
||||
item, so a dropped item does not pass for the end) while the 200-result
|
||||
ceiling is not reached."""
|
||||
number = int(cursor or 1)
|
||||
root = ET.fromstring(get(_query(ask, number)))
|
||||
channel = root.find("channel")
|
||||
@@ -143,8 +143,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
out: list[Row] = []
|
||||
for item in decoded["items"]:
|
||||
link, full_title = item["link"], item["title"]
|
||||
if not full_title or not is_web_url(link):
|
||||
continue
|
||||
description = item["description"]
|
||||
h4 = _H4.search(description)
|
||||
title = _text(h4.group(1)) if h4 else full_title
|
||||
|
||||
@@ -138,8 +138,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
for ad in decoded.get("jobAds") or []:
|
||||
ident = str(ad.get("jobAdId") or "").strip()
|
||||
title = _text(ad.get("title"))
|
||||
if not ident or not title:
|
||||
continue
|
||||
link = str(ad.get("jobAdUrl") or "").strip()
|
||||
town = _text(ad.get("municipality") or ad.get("postalDistrictName"))
|
||||
places: list[Place] = [place(town)] if town else []
|
||||
|
||||
@@ -176,8 +176,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
title = (j.get("title") or "").strip()
|
||||
job_id = str(j.get("id") or "").strip()
|
||||
path = str(j.get("url") or "").strip()
|
||||
if not title or not job_id or not _ROLE_PATH.fullmatch(path):
|
||||
continue
|
||||
function = str(j.get("function") or "").strip()
|
||||
places = [place(name) for name in dict.fromkeys(
|
||||
str(name).strip() for name in j.get("locations") or [] if str(name).strip())]
|
||||
@@ -186,7 +184,7 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
"title": title,
|
||||
"places": places,
|
||||
"location": "; ".join(pl["text"] for pl in places),
|
||||
"url": f"https://{_EMP}{path}",
|
||||
"url": f"https://{_EMP}{path}" if _ROLE_PATH.fullmatch(path) else "",
|
||||
"posted": _published(j.get("published_date")),
|
||||
"department": "" if function == "All functions" else function,
|
||||
"description": "",
|
||||
|
||||
@@ -22,7 +22,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads, Row
|
||||
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -112,15 +112,17 @@ def _places(categories: dict[str, Any]) -> list[Place]:
|
||||
|
||||
|
||||
def parse(body: bytes, slug: str) -> list[Row]:
|
||||
"""The board's postings, a JSON list. Any other answer (an error
|
||||
document sent with a 200) raises, so the read fails and closes nothing
|
||||
rather than reading as a board with no roles."""
|
||||
data = json.loads(body)
|
||||
if not isinstance(data, list):
|
||||
return []
|
||||
raise ValueError(f"Lever's answer is a JSON {type(data).__name__}, "
|
||||
"not a list of postings")
|
||||
rows = []
|
||||
for j in data:
|
||||
title = (j.get("text") or "").strip()
|
||||
url = (j.get("hostedUrl") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
categories = j.get("categories") or {}
|
||||
description = j.get("description") or j.get("descriptionPlain") or ""
|
||||
salary_range = j.get("salaryRange") or {}
|
||||
|
||||
@@ -110,20 +110,31 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
`items[0]`. Without the `expand` only the facets come back; asking for
|
||||
`requisitionList.secondaryLocations` gives the roles and each one's other
|
||||
locations, which the plain `requisitionList` leaves out. Past the last role
|
||||
the list is empty."""
|
||||
the list is empty; an answer without the wrapper raises (`_search`)."""
|
||||
offset = int(cursor or 0)
|
||||
site = _parts(slug)[1]
|
||||
data = json.loads(get(
|
||||
f"{_api(slug)}/recruitingCEJobRequisitions?onlyData=true"
|
||||
f"&expand=requisitionList.secondaryLocations"
|
||||
f"&finder=findReqs;siteNumber={site},{_selected(ask)}limit={PAGE_SIZE},offset={offset}"))
|
||||
search = (data.get("items") or [{}])[0]
|
||||
search = _search(data)
|
||||
roles = search.get("requisitionList") or []
|
||||
# `_paged` counts `id`, which Oracle spells `Id`.
|
||||
results = [{"id": str(role.get("Id") or "")} for role in roles]
|
||||
return search, results, search.get("TotalJobsCount"), offset + len(roles) if roles else None
|
||||
|
||||
|
||||
def _search(data: Any) -> dict[str, Any]:
|
||||
"""The search wrapper of a listing's answer, `items[0]`. Every answer has
|
||||
one, past the last role too; one without (an error document sent with a
|
||||
200) raises, so the read fails and closes nothing rather than reading as
|
||||
an empty last page with no total, which is a complete read."""
|
||||
items = data.get("items") if isinstance(data, dict) else None
|
||||
if not isinstance(items, list) or not items or not isinstance(items[0], dict):
|
||||
raise ValueError("Oracle's answer has no search wrapper in items")
|
||||
return items[0]
|
||||
|
||||
|
||||
#: How many countries `locationsFacet` lists at most; a list this long may
|
||||
#: leave some out.
|
||||
FACET_LIMIT = 10
|
||||
@@ -162,7 +173,7 @@ def facets(get: Get, slug: str) -> dict[str, dict[str, str]]:
|
||||
|
||||
|
||||
def parse(body: bytes, slug: str) -> list[Row]:
|
||||
return rows((json.loads(body).get("items") or [{}])[0], slug)
|
||||
return rows(_search(json.loads(body)), slug)
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
@@ -187,9 +198,9 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
|
||||
for role in search.get("requisitionList") or []:
|
||||
role_id = str(role.get("Id") or "").strip()
|
||||
title = _text(role.get("Title"))
|
||||
# The id goes into a URL, and a listing is external content.
|
||||
if not title or not re.fullmatch(r"[A-Za-z0-9_-]+", role_id):
|
||||
continue
|
||||
# The id goes into a URL, and a listing is external content: an id
|
||||
# of other characters gets no link, and the runner drops the row.
|
||||
safe = bool(re.fullmatch(r"[A-Za-z0-9_-]+", role_id))
|
||||
remote = _text(role.get("WorkplaceTypeCode")) or None
|
||||
at_office = (remote or "").lower() in ("ora_hybrid", "ora_on_site")
|
||||
places: list[Place] = []
|
||||
@@ -203,7 +214,7 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
|
||||
"title": title,
|
||||
"places": places,
|
||||
"location": "; ".join(p["text"] for p in places),
|
||||
"url": f"{_base(slug)}/job/{role_id}",
|
||||
"url": f"{_base(slug)}/job/{role_id}" if safe else "",
|
||||
"posted": parse_dt(role.get("PostedDate")),
|
||||
"department": (_text(role.get("JobFamily")) or _text(role.get("JobFunction"))
|
||||
or _text(role.get("Department")) or _text(role.get("Organization"))),
|
||||
|
||||
@@ -104,8 +104,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for position in root.findall("position"):
|
||||
job_id = (position.findtext("id") or "").strip()
|
||||
title = (position.findtext("name") or "").strip()
|
||||
if not job_id or not title:
|
||||
continue
|
||||
description = " ".join(
|
||||
(value.text or "") for value in position.findall("jobDescriptions/jobDescription/value"))
|
||||
salary_info = position.find("salaryInformation")
|
||||
|
||||
@@ -168,8 +168,6 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
|
||||
for job in search.get("data", {}).get("jobs") or []:
|
||||
key = _key(job)
|
||||
title = _text(job.get("title"))
|
||||
if not title or not key:
|
||||
continue
|
||||
places: list[Place] = []
|
||||
for text in (job.get("location"), *(job.get("multi_location") or [])):
|
||||
if isinstance(text, str):
|
||||
|
||||
@@ -22,7 +22,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Reads
|
||||
from ._shared import HEADING, Row, html_text, is_web_url, label_slug_ok, num, only, pay_line, yearly_amount
|
||||
from ._shared import HEADING, Row, html_text, label_slug_ok, num, only, pay_line, yearly_amount
|
||||
|
||||
NAME = "pinpoint"
|
||||
LABEL = "Pinpoint"
|
||||
@@ -121,8 +121,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("url") or "").strip()
|
||||
source_id = str(j.get("id") or "")
|
||||
if not title or not source_id or not is_web_url(url):
|
||||
continue
|
||||
job = j.get("job") or {}
|
||||
low, high, currency, salary_text = _pay(j)
|
||||
workplace = str(j.get("workplace_type") or "").strip()
|
||||
|
||||
@@ -22,7 +22,7 @@ from urllib.parse import urlencode
|
||||
from desk.sources import strip_html
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import Get, Page, Row, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, html_text, only, parse_dt
|
||||
|
||||
NAME = "platsbanken"
|
||||
LABEL = "Platsbanken"
|
||||
@@ -108,8 +108,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
for hit in decoded.get("hits") or []:
|
||||
title = _text(hit.get("headline") or "")
|
||||
url = (hit.get("webpage_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
addresses = hit.get("workplace_addresses") or [hit.get("workplace_address") or {}]
|
||||
places = [place for address in addresses if isinstance(address, dict)
|
||||
if (place := _place(address))]
|
||||
|
||||
@@ -13,7 +13,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads
|
||||
from ._shared import Row, add_place, html_text, is_web_url, label_slug_ok, num, only, parse_dt, pay_line, yearly_amount
|
||||
from ._shared import Row, add_place, html_text, label_slug_ok, num, only, parse_dt, pay_line, yearly_amount
|
||||
|
||||
NAME = "recruitee"
|
||||
LABEL = "Recruitee"
|
||||
@@ -105,8 +105,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data.get("offers") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("careers_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
salary = j.get("salary") if isinstance(j.get("salary"), dict) else {}
|
||||
salary_min, salary_max, currency = yearly_amount(
|
||||
salary.get("min"), salary.get("max"), salary.get("currency"),
|
||||
|
||||
+14
-10
@@ -82,27 +82,31 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
raise ValueError("not a Rippling board: the answer is not a list")
|
||||
roles: dict[str, Row] = {}
|
||||
places: dict[str, list[Place]] = {}
|
||||
for entry in data:
|
||||
for index, entry in enumerate(data):
|
||||
job_id = str(entry.get("uuid") or "").strip()
|
||||
title = (entry.get("name") or "").strip()
|
||||
if not title or not _UUID.fullmatch(job_id):
|
||||
continue
|
||||
# The id goes into a URL: one of other characters gets no link, and
|
||||
# the runner drops the row. An entry without an id is a role of its
|
||||
# own, not merged with the others that lack one.
|
||||
key = job_id or f"#{index}"
|
||||
place = str(((entry.get("workLocation") or {}).get("label")) or "").strip()
|
||||
places.setdefault(job_id, [])
|
||||
places.setdefault(key, [])
|
||||
if place:
|
||||
add_place(places[job_id], _place(place))
|
||||
roles.setdefault(job_id, {
|
||||
add_place(places[key], _place(place))
|
||||
roles.setdefault(key, {
|
||||
"source_id": job_id,
|
||||
"title": title,
|
||||
"url": f"https://{_SITE}/{slug}/jobs/{job_id}",
|
||||
"url": f"https://{_SITE}/{slug}/jobs/{job_id}" if _UUID.fullmatch(job_id) else "",
|
||||
"posted": None,
|
||||
"department": str(((entry.get("department") or {}).get("label")) or "").strip(),
|
||||
"description": "",
|
||||
"salary_text": "", # the board's list states no pay
|
||||
})
|
||||
for job_id, row in roles.items():
|
||||
row["places"] = places[job_id]
|
||||
row["location"] = "; ".join(p["text"] for p in places[job_id])
|
||||
if title and not roles[key]["title"]:
|
||||
roles[key]["title"] = title # an entry of the role that has one
|
||||
for key, row in roles.items():
|
||||
row["places"] = places[key]
|
||||
row["location"] = "; ".join(p["text"] for p in places[key])
|
||||
return list(roles.values())
|
||||
|
||||
|
||||
|
||||
@@ -115,8 +115,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
for j in data.get("content") or []:
|
||||
title = (j.get("name") or "").strip()
|
||||
job_id = str(j.get("id") or "")
|
||||
if not title or not job_id:
|
||||
continue
|
||||
location = j.get("location") or {}
|
||||
loc_text = (location.get("fullLocation")
|
||||
or ", ".join(p for p in (location.get("city"), location.get("country")) if p))
|
||||
|
||||
@@ -63,7 +63,7 @@ from desk.sources import strip_html
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._countries import NAMES
|
||||
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt
|
||||
|
||||
NAME = "successfactors"
|
||||
LABEL = "SuccessFactors"
|
||||
@@ -247,8 +247,9 @@ def _tile_fields(chunk: str) -> tuple[str, str, str]:
|
||||
def _entries(text: str) -> list[dict[str, Any]]:
|
||||
"""The roles of one search page, in the order shown, whichever layout it is
|
||||
in and before the prefix filter: `id`, `href` (as written, relative),
|
||||
`title`, `place`, `date` and `department`. A role is listed once, and a row
|
||||
without a link to a role is not one."""
|
||||
`title`, `place`, `date` and `department`. A role is listed once. The id
|
||||
and the title are in the role's link, so a row without one is handed over
|
||||
with an empty `id`, `href` and `title`, for the runner to drop and count."""
|
||||
table = _ROW.split(text)[1:]
|
||||
chunks = table or _TILE.split(text)[1:]
|
||||
out: list[dict[str, Any]] = []
|
||||
@@ -256,10 +257,12 @@ def _entries(text: str) -> list[dict[str, Any]]:
|
||||
for chunk in chunks:
|
||||
if table:
|
||||
chunk = chunk.split("</tr>", 1)[0]
|
||||
if not (link := _LINK.search(chunk)):
|
||||
continue
|
||||
href, job_id, title = html.unescape(link.group(1)), link.group(2), _text(link.group(3))
|
||||
if job_id in seen:
|
||||
if link := _LINK.search(chunk):
|
||||
href, job_id, title = (html.unescape(link.group(1)), link.group(2),
|
||||
_text(link.group(3)))
|
||||
else:
|
||||
href = job_id = title = ""
|
||||
if job_id and job_id in seen:
|
||||
continue
|
||||
seen.add(job_id)
|
||||
place, date, department = (_table_fields if table else _tile_fields)(chunk)
|
||||
@@ -269,22 +272,28 @@ def _entries(text: str) -> list[dict[str, Any]]:
|
||||
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
"""One search page, in the one country `ask` names, matching its `query`. The cursor is `(start row, first role of the page
|
||||
before)`: the next start row is what this page returned, and a page that
|
||||
starts with the same role as the last one is the site ignoring `startrow`,
|
||||
which ends the read (incomplete) instead of looping."""
|
||||
"""One search page, in the one country `ask` names, matching its `query`.
|
||||
The cursor is `(start row, first role of the page before)`, a list when
|
||||
a sweep hands it back from state: the next start row is what this page
|
||||
returned. A page that starts with the same role as the last one is the
|
||||
site ignoring `startrow`: it comes back with no results and the cursor
|
||||
it was asked with, which the runner reads as a stall that ends the read
|
||||
incomplete. A cursor of None there would read as the last page, and,
|
||||
on a site that states no total, as a complete read."""
|
||||
start, previous = cursor or (0, None)
|
||||
country = ask.countries[0] if ask.countries else ""
|
||||
text = get(_search_url(slug, start, country, ask.query)).decode("utf-8", "replace")
|
||||
if not _SEARCH_PAGE.search(text):
|
||||
raise ValueError("not a SuccessFactors search page")
|
||||
found = _entries(text)
|
||||
ids = [entry["id"] for entry in found if entry["id"]]
|
||||
total = _total(text)
|
||||
if found and found[0]["id"] == previous:
|
||||
return text, [], total, None
|
||||
first = ids[0] if ids else None
|
||||
if first is not None and first == previous:
|
||||
return text, [], total, (start, previous)
|
||||
more = bool(found) and (total is None or start + len(found) < total)
|
||||
return (text, [{"id": entry["id"]} for entry in found], total,
|
||||
(start + len(found), found[0]["id"]) if more else None)
|
||||
return (text, [{"id": job_id} for job_id in ids], total,
|
||||
(start + len(found), first) if more else None)
|
||||
|
||||
|
||||
def rows(decoded: str, slug: str) -> list[Row]:
|
||||
@@ -292,17 +301,21 @@ def rows(decoded: str, slug: str) -> list[Row]:
|
||||
slug's prefix when it has one; the id in it is the requisition's `source_id`.
|
||||
The table and the tile layout give a title, a location, a date and, where
|
||||
the site has the column, a department; nothing else about a role is on
|
||||
the list, so its description and pay are empty."""
|
||||
the list, so its description and pay are empty.
|
||||
|
||||
A role linked on another host, or outside the prefix, is another board's
|
||||
(another site, or another brand of a shared site), and is left out: it is
|
||||
no role of this company. A row with no link or no title is handed over,
|
||||
and the runner drops it."""
|
||||
host, prefix = _parts(slug)
|
||||
base = f"https://{host}/"
|
||||
out = []
|
||||
for entry in _entries(decoded):
|
||||
url = urljoin(base, entry["href"])
|
||||
if urlsplit(url).hostname != host or not is_web_url(url):
|
||||
url = urljoin(base, entry["href"]) if entry["href"] else ""
|
||||
if url and urlsplit(url).hostname != host:
|
||||
continue
|
||||
if prefix and not urlsplit(url).path.lower().startswith(f"{prefix.lower()}/job/"):
|
||||
continue
|
||||
if not entry["title"]:
|
||||
if url and prefix and not urlsplit(url).path.lower().startswith(
|
||||
f"{prefix.lower()}/job/"):
|
||||
continue
|
||||
out.append({
|
||||
"source_id": entry["id"],
|
||||
|
||||
@@ -15,7 +15,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads
|
||||
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt
|
||||
|
||||
NAME = "teamtailor"
|
||||
LABEL = "Teamtailor"
|
||||
@@ -115,8 +115,6 @@ def rows(items: list[ET.Element], slug: str) -> list[Row]:
|
||||
for item in items:
|
||||
title = (item.findtext("title") or "").strip()
|
||||
url = (item.findtext("link") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
status = (item.findtext("remoteStatus") or "").strip()
|
||||
remote = {"fully": True, "hybrid": False, "onsite": False}.get(status)
|
||||
places: list[Place] = []
|
||||
|
||||
@@ -102,8 +102,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
for j in data.get("results") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
shortcode = (j.get("shortcode") or "").strip()
|
||||
if not title or not shortcode:
|
||||
continue
|
||||
workplace = str(j.get("workplace") or "").strip()
|
||||
remote = bool(j.get("remote")) or workplace == "remote"
|
||||
places = _places(j, remote)
|
||||
|
||||
@@ -124,8 +124,8 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
total = stated if isinstance(stated, int) and not isinstance(stated, bool) else None
|
||||
postings = data.get("jobPostings") or []
|
||||
# Every posting is counted, a path-less one (LEGO lists one that has only
|
||||
# `bulletFields`) by its position, so the parser dropping it does not
|
||||
# leave the board short of its total.
|
||||
# `bulletFields`) by its position, so the runner dropping its row does
|
||||
# not leave the board short of its total.
|
||||
results = [{"id": _posting_id(p, offset + i)} for i, p in enumerate(postings)]
|
||||
following = offset + len(postings)
|
||||
more = bool(postings) and (total is None or following < total)
|
||||
@@ -274,13 +274,13 @@ def rows(decoded: dict[str, Any], slug: str) -> list[Row]:
|
||||
`bulletFields`, which lead with the requisition id, and on some tenants
|
||||
`remoteType`. No description, no date and no country: `detail` has
|
||||
those, and the places of a role that "2 Locations" hides. The id is the requisition id where the path carries it, and the
|
||||
path where not; a posting with no title or no path is skipped."""
|
||||
path where not. A posting with no path has neither an id nor a link, and
|
||||
the runner drops it."""
|
||||
out = []
|
||||
for p in decoded.get("jobPostings") or []:
|
||||
title = (p.get("title") or "").strip()
|
||||
path = (p.get("externalPath") or "").strip()
|
||||
if not title or not _ROLE_PATH.fullmatch(path):
|
||||
continue
|
||||
path = path if _ROLE_PATH.fullmatch(path) else ""
|
||||
remote = _remote(p.get("remoteType"))
|
||||
place = _place(p.get("locationsText"), remote)
|
||||
out.append({
|
||||
@@ -288,7 +288,7 @@ def rows(decoded: dict[str, Any], slug: str) -> list[Row]:
|
||||
"title": title,
|
||||
"places": [place] if place else [],
|
||||
"location": place["text"] if place else "",
|
||||
"url": f"https://{_host(slug)}/{_parts(slug)[2]}{path}", # type: ignore[index]
|
||||
"url": f"https://{_host(slug)}/{_parts(slug)[2]}{path}" if path else "", # type: ignore[index]
|
||||
"posted": _posted(p.get("postedOn") or ""),
|
||||
"department": "",
|
||||
"description": "",
|
||||
|
||||
+6
-5
@@ -13,13 +13,14 @@ and the saved `pages` param bounds the pages read (`DEFAULT_PAGES`, at most
|
||||
`MAX_PAGES`).
|
||||
|
||||
A search result list is never a board's complete list, so a role missing from
|
||||
one run is not closed here. A search that read everything that matches was
|
||||
complete (`complete="stated"`, as many distinct results as the stated total),
|
||||
and the pull closes what it stopped listing; the rest close once no search
|
||||
has seen them for `STALE_DAYS` (see `app/jobs/pull.py`).
|
||||
one run is not closed, also when the search read everything that matches
|
||||
(`complete="stated"`, as many distinct results as the stated total): its
|
||||
roles close once no search has seen them for `STALE_DAYS` (see
|
||||
`app/jobs/pull.py`).
|
||||
|
||||
In a pull a search is read by `_runner.read` under the prefix `search`, as a
|
||||
board is under `ats`: the 10% guard holds it, `_runner.stamp` records it,
|
||||
board is under `ats`: the runner drops a row without an id, a title or a
|
||||
link, the 10% guards hold it, `_runner.stamp` records it,
|
||||
and a search that fails or is suspect is kept as failing under
|
||||
`search:failing:<target>` (`_runner.record_failure`) until it answers, which
|
||||
the settle reads as it reads a failing board's.
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
// static/agents.js without a browser: the gates Save sends from the fetch
|
||||
// gate's four parts, and the stored gates the parts cannot say, which the
|
||||
// form names in a note and sends back as they came.
|
||||
//
|
||||
// node --test tests/js/*.test.mjs
|
||||
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import vm from "node:vm";
|
||||
|
||||
// Only what agents.js touches before DOMContentLoaded.
|
||||
const window = { api: async () => [], count: (n, noun) => `${n} ${noun}` };
|
||||
const document = { addEventListener() {} };
|
||||
Object.assign(globalThis, { window, document });
|
||||
for (const file of ["remote.js", "agents.js"]) {
|
||||
vm.runInThisContext(readFileSync(new URL(`../../app/static/${file}`, import.meta.url), "utf8"));
|
||||
}
|
||||
const rh = window.remoteHelpers;
|
||||
const { formGates, keptNote, rulesLine } = window.agentGates;
|
||||
|
||||
/** The four parts as the form shows them after loading `gates`, and what it keeps. */
|
||||
function load(gates) {
|
||||
const { where, rest } = rh.distribute(gates);
|
||||
const parts = {
|
||||
scope: where.scope.kind, scoped: where.scope.values,
|
||||
within: where.countries ? where.countries.values : [],
|
||||
towns: where.places ? where.places.values : [],
|
||||
remote: where.remote.mode, from: where.remote.values,
|
||||
titles: where.titles ? where.titles.values : [],
|
||||
};
|
||||
return { parts, rest };
|
||||
}
|
||||
|
||||
const europe = { field: "country", values: rh.europeWith(["DK"]), mode: "require" };
|
||||
const dk = { field: "country", values: ["DK"], mode: "any" };
|
||||
const remote = { field: "remote", values: ["remote"], mode: "any" };
|
||||
const towns = { field: "place", values: ["Malmö", "Lund"], mode: "any" };
|
||||
const titles = { field: "title", values: ["account executive", "sales"], mode: "exclude" };
|
||||
const level = { field: "seniority", values: ["senior", "staff"], mode: "require" };
|
||||
const acme = { field: "company", values: ["acme"], mode: "exclude" };
|
||||
|
||||
test("a save sends the four parts back as they were loaded", () => {
|
||||
const agents = [
|
||||
[europe, dk, remote, towns, titles],
|
||||
[{ field: "country", values: ["DK", "SE"], mode: "require" },
|
||||
{ field: "remote_from", values: ["country:DK", "country:EU"], mode: "any" }, titles],
|
||||
[dk],
|
||||
[],
|
||||
];
|
||||
for (const gates of agents) {
|
||||
const { parts, rest } = load(gates);
|
||||
assert.deepEqual(rest, []);
|
||||
assert.deepEqual(formGates(parts, rest), gates);
|
||||
}
|
||||
});
|
||||
|
||||
test("each part sends its own gate, and an empty part none", () => {
|
||||
const parts = { scope: "everywhere", scoped: [], within: [], towns: [], remote: "none", from: [], titles: [] };
|
||||
assert.deepEqual(formGates(parts), []);
|
||||
// A remote line open to no area yet, or chosen countries with none picked, say nothing.
|
||||
assert.deepEqual(formGates({ ...parts, remote: "from", scope: "countries" }), []);
|
||||
assert.deepEqual(formGates({ ...parts, scope: "europe", within: ["US"] }), [
|
||||
{ field: "country", values: rh.europeWith(["US"]), mode: "require" },
|
||||
{ field: "country", values: ["US"], mode: "any" },
|
||||
]);
|
||||
});
|
||||
|
||||
test("a stored gate the parts cannot say is kept across a save, named in the note and on the card", () => {
|
||||
const gates = [europe, dk, remote, titles, level, acme];
|
||||
const { parts, rest } = load(gates);
|
||||
assert.deepEqual(rest, [level, acme]);
|
||||
assert.deepEqual(formGates(parts, rest), gates);
|
||||
assert.equal(keptNote(rest),
|
||||
"The form cannot edit these rules, and Save keeps them as stored: "
|
||||
+ "must: with seniority senior, staff; not: at acme.");
|
||||
const words = rulesLine(gates);
|
||||
assert.match(words, /Only roles with seniority senior, staff\./);
|
||||
assert.match(words, /Leaves out roles at acme\./);
|
||||
});
|
||||
|
||||
test("an agent with nothing to keep has no note", () => {
|
||||
assert.equal(keptNote([]), "");
|
||||
assert.equal(keptNote([level]), "The form cannot edit this rule, and Save keeps it as stored: must: with seniority senior, staff.");
|
||||
});
|
||||
@@ -0,0 +1,189 @@
|
||||
// The feed's paging, static/roles.js `pageAsk` and `pageMerge`, without a
|
||||
// browser: against a fake /api/roles that sorts afresh on every request, as
|
||||
// the store does, with a pass halving a role's score and a like lifting it a
|
||||
// fifth (`_VOTED_SCORE`), and Show read off or Judged taking a role out.
|
||||
//
|
||||
// node --test tests/js/*.test.mjs
|
||||
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import vm from "node:vm";
|
||||
|
||||
// Only what roles.js touches before DOMContentLoaded.
|
||||
const window = {
|
||||
matchMedia: () => ({ matches: false, addEventListener() {} }),
|
||||
addEventListener() {},
|
||||
};
|
||||
const document = { addEventListener() {} };
|
||||
Object.assign(globalThis, { window, document });
|
||||
vm.runInThisContext(readFileSync(new URL("../../app/static/roles.js", import.meta.url), "utf8"));
|
||||
const { pageAsk, pageMerge } = window.feedPaging;
|
||||
|
||||
const PAGE = 60;
|
||||
|
||||
// ------------------------------------------------------------ the server
|
||||
|
||||
/** `n` roles, scores falling evenly from 0.99, all unread and unjudged. */
|
||||
function server(n, { includeRead = true, judged = "" } = {}) {
|
||||
const roles = Array.from({ length: n }, (_, i) => ({
|
||||
uid: `r${String(i).padStart(3, "0")}`, score: 0.99 - (i * 0.5) / n, vote: null, read: false,
|
||||
}));
|
||||
const voted = (r) => Math.min(1, r.score * (r.vote === -1 ? 0.5 : r.vote === 1 ? 1.2 : 1));
|
||||
function list() {
|
||||
return roles
|
||||
.filter((r) => includeRead || !r.read)
|
||||
.filter((r) => judged !== "unjudged" || r.vote === null)
|
||||
.sort((a, b) => voted(b) - voted(a) || a.uid.localeCompare(b.uid));
|
||||
}
|
||||
return {
|
||||
page: ({ offset, limit }) => {
|
||||
const all = list();
|
||||
return { items: all.slice(offset, offset + limit).map((r) => ({ uid: r.uid })), total: all.length };
|
||||
},
|
||||
uids: () => list().map((r) => r.uid),
|
||||
// A vote stamps the role read, as `Store.set_vote` does.
|
||||
vote: (uid, value) => { const r = roles.find((x) => x.uid === uid); r.vote = value; r.read = true; },
|
||||
read: (uid) => { roles.find((x) => x.uid === uid).read = true; },
|
||||
};
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------ the client
|
||||
|
||||
/** loadFeed's bookkeeping: the pages asked with `pageAsk`, merged with
|
||||
`pageMerge`, each change counted for two pages. */
|
||||
function client(api) {
|
||||
const c = { items: [], offset: 0, more: false, moved: 0, movedOnce: 0 };
|
||||
c.load = (append = false) => {
|
||||
const ask = append ? pageAsk(c.offset, c.moved + c.movedOnce) : { offset: 0, limit: PAGE };
|
||||
const answer = api.page(ask);
|
||||
c.movedOnce = c.moved;
|
||||
c.moved = 0;
|
||||
Object.assign(c, pageMerge(append ? c.items : [], ask, answer));
|
||||
};
|
||||
c.toEnd = () => { for (let i = 0; c.more && i < 100; i += 1) c.load(true); };
|
||||
c.uids = () => c.items.map((r) => r.uid);
|
||||
return c;
|
||||
}
|
||||
|
||||
/** Every uid once, and nothing the server lists left out. */
|
||||
function assertWhole(shown, listed) {
|
||||
assert.equal(new Set(shown).size, shown.length, "a role is shown twice");
|
||||
const missing = listed.filter((uid) => !shown.includes(uid));
|
||||
assert.deepEqual(missing, [], "a role is never shown");
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------ pageAsk
|
||||
|
||||
test("with nothing moved the next page starts where the last one ended", () => {
|
||||
assert.deepEqual(pageAsk(120, 0), { offset: 120, limit: PAGE });
|
||||
});
|
||||
|
||||
test("each change starts the page a row earlier and asks for a row more", () => {
|
||||
assert.deepEqual(pageAsk(120, 3), { offset: 117, limit: PAGE + 3 });
|
||||
// Never before the first row, and never more than the route allows.
|
||||
assert.deepEqual(pageAsk(2, 5), { offset: 0, limit: PAGE + 2 });
|
||||
assert.deepEqual(pageAsk(5000, 4000), { offset: 5000 - 940, limit: 1000 });
|
||||
});
|
||||
|
||||
// ------------------------------------------------------------ pageMerge
|
||||
|
||||
test("a row already on screen is dropped and the offset counts what the server sent", () => {
|
||||
const items = [{ uid: "a" }, { uid: "b" }];
|
||||
const merged = pageMerge(items, { offset: 1, limit: 3 },
|
||||
{ items: [{ uid: "b" }, { uid: "c" }, { uid: "d" }], total: 10 });
|
||||
assert.deepEqual(merged.items.map((r) => r.uid), ["a", "b", "c", "d"]);
|
||||
assert.equal(merged.offset, 4);
|
||||
assert.equal(merged.more, true);
|
||||
});
|
||||
|
||||
test("the list ends at the total, or at a short page whatever the total says", () => {
|
||||
const at = (items, total) => pageMerge([], { offset: 0, limit: 2 }, { items, total });
|
||||
assert.equal(at([{ uid: "a" }, { uid: "b" }], 2).more, false);
|
||||
assert.equal(at([{ uid: "a" }], 5).more, false);
|
||||
assert.equal(at([{ uid: "a" }, { uid: "b" }], 5).more, true);
|
||||
});
|
||||
|
||||
// ------------------------------------------------------------ against the server
|
||||
|
||||
test("a page asked from the rows on screen skips a role after a pass (the bug)", () => {
|
||||
const api = server(200);
|
||||
let items = api.page({ offset: 0, limit: PAGE }).items;
|
||||
api.vote(items[2].uid, -1);
|
||||
while (items.length < api.page({ offset: 0, limit: 1 }).total) {
|
||||
items = items.concat(api.page({ offset: items.length, limit: PAGE }).items);
|
||||
}
|
||||
const uids = items.map((r) => r.uid);
|
||||
assert.notEqual(new Set(uids).size, uids.length);
|
||||
assert.ok(api.uids().some((uid) => !uids.includes(uid)));
|
||||
});
|
||||
|
||||
test("a pass on the first page skips nothing and shows nothing twice", () => {
|
||||
const api = server(200);
|
||||
const c = client(api);
|
||||
c.load();
|
||||
api.vote(c.items[2].uid, -1);
|
||||
c.moved += 1;
|
||||
c.toEnd();
|
||||
assertWhole(c.uids(), api.uids());
|
||||
assert.equal(c.items.length, 200);
|
||||
assert.equal(c.more, false);
|
||||
});
|
||||
|
||||
test("likes, passes and undos on several pages skip nothing", () => {
|
||||
const api = server(500);
|
||||
const c = client(api);
|
||||
c.load();
|
||||
let seed = 7;
|
||||
const pick = () => { seed = (seed * 48271) % 2147483647; return seed; };
|
||||
while (c.more) {
|
||||
for (let i = 0; i < 6; i += 1) {
|
||||
const uid = c.items[pick() % c.items.length].uid;
|
||||
api.vote(uid, [-1, 1, null][pick() % 3]);
|
||||
c.moved += 1;
|
||||
}
|
||||
c.load(true);
|
||||
}
|
||||
assertWhole(c.uids(), api.uids());
|
||||
assert.equal(c.items.length, 500);
|
||||
});
|
||||
|
||||
test("with Show read off each role opened costs no role", () => {
|
||||
const api = server(200, { includeRead: false });
|
||||
const c = client(api);
|
||||
c.load();
|
||||
for (const i of [0, 5, 30, 59]) { api.read(c.items[i].uid); c.moved += 1; }
|
||||
c.load(true);
|
||||
api.read(c.items[90].uid);
|
||||
c.moved += 1;
|
||||
c.toEnd();
|
||||
assertWhole(c.uids(), api.uids());
|
||||
// The read ones stay on screen; the server's list no longer has them.
|
||||
assert.equal(c.items.length, 200);
|
||||
assert.equal(api.uids().length, 195);
|
||||
assert.equal(c.more, false);
|
||||
});
|
||||
|
||||
test("under Judged, Unjudged a vote takes the role out and the next page makes up for it", () => {
|
||||
const api = server(150, { judged: "unjudged" });
|
||||
const c = client(api);
|
||||
c.load();
|
||||
for (const i of [1, 2, 3]) { api.vote(c.items[i].uid, 1); c.moved += 1; }
|
||||
c.toEnd();
|
||||
assertWhole(c.uids(), api.uids());
|
||||
assert.equal(c.items.length, 150);
|
||||
});
|
||||
|
||||
test("a change counts for the page after the next too, for a vote that lands late", () => {
|
||||
const api = server(200);
|
||||
const c = client(api);
|
||||
c.load();
|
||||
c.load(true);
|
||||
// Clicked before the third page was asked, landed after the server read it.
|
||||
const late = c.items[10].uid;
|
||||
c.moved += 1;
|
||||
c.load(true);
|
||||
api.vote(late, -1);
|
||||
c.toEnd();
|
||||
assertWhole(c.uids(), api.uids());
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
// static/remote.js without a browser: the badge, the suggestions the agent's
|
||||
// remote line offers, and how gates share out between Where and Other rules.
|
||||
// remote line offers, and how gates share out between the fetch gate and the
|
||||
// rest the form keeps as stored.
|
||||
//
|
||||
// node --test tests/js/*.test.mjs
|
||||
|
||||
@@ -89,14 +90,14 @@ test("any other require gate on country is the scope with its codes as chips", (
|
||||
assert.deepEqual(rest, []);
|
||||
});
|
||||
|
||||
test("a second require gate on country stays a row", () => {
|
||||
test("a second require gate on country is kept as stored", () => {
|
||||
const second = must("NO");
|
||||
const { where, rest } = rh.distribute([europe, second]);
|
||||
assert.equal(where.scope.kind, "europe");
|
||||
assert.deepEqual(rest, [second]);
|
||||
});
|
||||
|
||||
test("the titles to leave out are the exclude gate on title, and any other exclude is a row", () => {
|
||||
test("the titles to leave out are the exclude gate on title, and any other exclude is kept as stored", () => {
|
||||
const titles = { field: "title", values: ["account executive"], mode: "exclude" };
|
||||
const level = { field: "seniority", values: ["junior"], mode: "exclude" };
|
||||
const { where, rest } = rh.distribute([titles, level]);
|
||||
@@ -104,7 +105,7 @@ test("the titles to leave out are the exclude gate on title, and any other exclu
|
||||
assert.deepEqual(rest, [level]);
|
||||
});
|
||||
|
||||
test("a gate with a mode the form does not know stays a row and takes no Where line", () => {
|
||||
test("a gate with a mode the form does not know is kept as stored and takes no Where line", () => {
|
||||
const prefer = { field: "remote", values: ["hybrid"], mode: "prefer" };
|
||||
const { where, rest } = rh.distribute([prefer]);
|
||||
assert.deepEqual(where.remote, { mode: "none", values: [] });
|
||||
@@ -112,7 +113,7 @@ test("a gate with a mode the form does not know stays a row and takes no Where l
|
||||
assert.deepEqual(rest, [prefer]);
|
||||
});
|
||||
|
||||
test("a require gate on another field is a row and leaves the scope everywhere", () => {
|
||||
test("a require gate on another field is kept as stored and leaves the scope everywhere", () => {
|
||||
const level = { field: "seniority", values: ["senior"], mode: "require" };
|
||||
const { where, rest } = rh.distribute([level]);
|
||||
assert.equal(where.scope.kind, "everywhere");
|
||||
@@ -129,7 +130,7 @@ test("remote, three modes: the plain gate is within the scope", () => {
|
||||
assert.deepEqual(named.rest, []);
|
||||
});
|
||||
|
||||
test("a remote_from gate wins over the plain one, which stays a row", () => {
|
||||
test("a remote_from gate wins over the plain one, which is kept as stored", () => {
|
||||
for (const gates of [[older, from("country:DK")], [from("country:DK"), older]]) {
|
||||
const { where, rest } = rh.distribute(gates);
|
||||
assert.deepEqual(where.remote, { mode: "from", values: ["country:DK"] });
|
||||
@@ -137,7 +138,7 @@ test("a remote_from gate wins over the plain one, which stays a row", () => {
|
||||
}
|
||||
});
|
||||
|
||||
test("a second remote_from gate or a must stays a row", () => {
|
||||
test("a second remote_from gate or a must is kept as stored", () => {
|
||||
const second = from("country:SE");
|
||||
const required = { field: "remote_from", values: ["country:DK"], mode: "require" };
|
||||
const { where, rest } = rh.distribute([from("country:DK"), second, required]);
|
||||
|
||||
+15
-8
@@ -24,18 +24,24 @@ from plugins.readers import _runner, smartrecruiters # noqa: E402
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
|
||||
|
||||
def _rows(host: str, slug: str = "acme") -> list[ats.Row]:
|
||||
def _parsed(host: str, slug: str = "acme") -> list[ats.Row]:
|
||||
path = FIXTURES / f"{host}.{'xml' if host == 'personio' else 'json'}"
|
||||
body = path.read_bytes()
|
||||
return ats._PARSERS[host](body, slug)
|
||||
return ats._PARSERS[host](path.read_bytes(), slug)
|
||||
|
||||
|
||||
def _rows(host: str, slug: str = "acme") -> list[ats.Row]:
|
||||
"""The rows of a host's fixture the runner keeps (`_runner.valid`)."""
|
||||
return [row for row in _parsed(host, slug) if _runner.valid(row)]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ hosts
|
||||
|
||||
def test_greenhouse_parses_titles_locations_and_html_description() -> None:
|
||||
# The javascript: and data: rows are handed over and the runner drops
|
||||
# them: a feed is external content, never trusted as a URL.
|
||||
assert [r["source_id"] for r in _parsed("greenhouse")] == [
|
||||
"1001", "1002", "1003", "1004", "1005", "1006"]
|
||||
rows = _rows("greenhouse")
|
||||
# The javascript: and data: rows are refused, same as the four real ones
|
||||
# are kept: a feed is external content, never trusted as a URL.
|
||||
assert [r["source_id"] for r in rows] == ["1001", "1002", "1005", "1006"]
|
||||
first = rows[0]
|
||||
assert first["title"] == "Senior Backend Engineer" # trimmed
|
||||
@@ -269,10 +275,11 @@ TEAMTAILOR = """<?xml version="1.0" encoding="UTF-8"?>
|
||||
|
||||
def test_teamtailor_reads_the_rss_feed_with_its_own_namespace() -> None:
|
||||
rows = ats._parse_teamtailor(TEAMTAILOR, "kry")
|
||||
# the javascript: link is refused like on the other hosts
|
||||
# the javascript: link is handed over and refused by the runner
|
||||
assert [r["source_id"] for r in rows] == [
|
||||
"39ac1fca-a904-4905-b780-18c599e99b07", "9787a509-ff84-400a-ad68-84e22c4cf2fb",
|
||||
"0a0a0a0a-ff84-400a-ad68-84e22c4cf2fb"]
|
||||
"0a0a0a0a-ff84-400a-ad68-84e22c4cf2fb", "bad"]
|
||||
assert [_runner.valid(r) for r in rows] == [True, True, True, False]
|
||||
first = rows[0]
|
||||
assert first["url"] == "https://career.kry.se/jobs/8440833-software-engineer"
|
||||
assert first["department"] == "Engineering"
|
||||
@@ -294,7 +301,7 @@ def test_teamtailor_joins_the_distinct_locations_for_country_of() -> None:
|
||||
|
||||
def test_teamtailor_hands_on_remote_status_and_falls_back_to_role_for_department() -> None:
|
||||
rows = ats._parse_teamtailor(TEAMTAILOR, "kry")
|
||||
assert [r["remote"] for r in rows] == ["hybrid", "fully", "none"] # prep maps them
|
||||
assert [r["remote"] for r in rows[:3]] == ["hybrid", "fully", "none"] # prep maps them
|
||||
assert rows[1]["title"] == "Läkare" # trimmed
|
||||
assert rows[1]["department"] == "Software Development"
|
||||
assert ats.classify(rows[1]["department"], "Sales Lead") == "sales"
|
||||
|
||||
@@ -17,7 +17,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs import discover # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, bamboohr # noqa: E402
|
||||
from plugins.readers import Ask, _runner, bamboohr # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
|
||||
@@ -84,11 +84,16 @@ def test_a_page_that_is_not_a_listing_raises_and_an_empty_board_is_no_rows() ->
|
||||
bamboohr.parse(body, "configit")
|
||||
|
||||
|
||||
def test_a_role_without_a_title_or_an_id_is_left_out() -> None:
|
||||
def test_a_role_without_a_title_or_an_id_reaches_the_runner_and_is_dropped_there() -> None:
|
||||
body = json.loads(_read("bamboohr.json"))
|
||||
body["result"][0]["jobOpeningName"] = " "
|
||||
body["result"][1]["id"] = None
|
||||
assert bamboohr.parse(json.dumps(body).encode(), "configit") == []
|
||||
raw = json.dumps(body).encode()
|
||||
rows = bamboohr.parse(raw, "configit")
|
||||
assert len(rows) == 2 and not any(map(_runner.valid, rows))
|
||||
listing = _runner.read_listing(bamboohr, "configit", lambda url, **kw: raw)
|
||||
assert (listing.rows, listing.ids, listing.complete) == ([], [], True)
|
||||
assert (listing.handed, listing.dropped) == (2, 2)
|
||||
|
||||
|
||||
def test_detail_reads_the_text_the_date_and_the_country() -> None:
|
||||
|
||||
@@ -17,7 +17,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, deel # noqa: E402
|
||||
from plugins.readers import Ask, _runner, deel # noqa: E402
|
||||
|
||||
PAGE = (ROOT / "tests" / "fixtures" / "ats" / "deel.html").read_bytes()
|
||||
URL = "https://jobs.deel.com/klarna"
|
||||
@@ -89,10 +89,12 @@ def test_a_board_without_roles_is_an_empty_board_but_another_page_is_not_a_board
|
||||
deel.parse(b"<html><title>Job Board Not Found</title></html>", "nobody")
|
||||
|
||||
|
||||
def test_a_posting_without_a_title_or_id_is_skipped() -> None:
|
||||
def test_a_posting_without_a_title_or_id_is_handed_over_for_the_runner_to_drop() -> None:
|
||||
body = {"jobPostings": [{"id": "a", "title": " ", "job": {}}, {"title": "x", "job": {}},
|
||||
{"id": "c", "title": "Cook", "job": {}}]}
|
||||
assert [r["source_id"] for r in deel.parse(_page_with(body), "acme")] == ["c"]
|
||||
rows = deel.parse(_page_with(body), "acme")
|
||||
assert [r["source_id"] for r in rows] == ["a", "", "c"]
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == ["c"]
|
||||
|
||||
|
||||
# ------------------------------------------------------------ the fetch
|
||||
|
||||
@@ -18,7 +18,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs import discover # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, eightfold # noqa: E402
|
||||
from plugins.readers import Ask, _runner, eightfold # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
SLUG = "jobs.ericsson.com/ericsson.com"
|
||||
@@ -93,10 +93,12 @@ def test_the_work_location_option_is_the_hosts_word(
|
||||
assert eightfold.rows([_position(1) | {"workLocationOption": option}], SLUG)[0]["remote"] == remote
|
||||
|
||||
|
||||
def test_a_position_without_a_name_an_id_or_a_path_is_left_out() -> None:
|
||||
def test_a_position_without_a_name_an_id_or_a_path_is_handed_over_for_the_runner_to_drop() -> None:
|
||||
bad = [_position(1) | {"name": " "}, _position(2) | {"id": None},
|
||||
_position(3) | {"positionUrl": ""}, _position(4) | {"positionUrl": "//evil.example/x"}]
|
||||
assert eightfold.rows(bad, SLUG) == []
|
||||
rows = eightfold.rows(bad, SLUG)
|
||||
assert len(rows) == 4 and not any(map(_runner.valid, rows))
|
||||
assert [r["url"] for r in rows[2:]] == ["", ""] # no link built off the host
|
||||
|
||||
|
||||
def test_the_standardized_location_stands_in_when_there_is_no_raw_one() -> None:
|
||||
|
||||
@@ -16,7 +16,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs import discover # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, gem # noqa: E402
|
||||
from plugins.readers import Ask, _runner, gem # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
API = "https://jobs.gem.com/api/public/graphql"
|
||||
@@ -79,8 +79,9 @@ def test_remote_is_the_location_type_then_the_locations(
|
||||
assert gem.rows([posting], "acme")[0]["remote"] == remote
|
||||
|
||||
|
||||
def test_a_posting_without_a_title_or_an_ext_id_is_left_out() -> None:
|
||||
assert gem.rows([{"extId": "1", "title": " "}, {"extId": "", "title": "Role"}], "acme") == []
|
||||
def test_a_posting_without_a_title_or_an_ext_id_is_handed_over_for_the_runner_to_drop() -> None:
|
||||
rows = gem.rows([{"extId": "1", "title": " "}, {"extId": "", "title": "Role"}], "acme")
|
||||
assert len(rows) == 2 and not any(map(_runner.valid, rows))
|
||||
|
||||
|
||||
def test_the_board_is_one_post_without_origin_or_referer() -> None:
|
||||
|
||||
@@ -17,7 +17,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, hibob # noqa: E402
|
||||
from plugins.readers import Ask, _runner, hibob # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
BOARD = (FIXTURES / "hibob.json").read_bytes()
|
||||
@@ -87,10 +87,12 @@ def test_the_place_leaves_the_country_off_where_the_site_names_it() -> None:
|
||||
assert hibob.parse(_one(**fields), "acme")[0]["location"] == want
|
||||
|
||||
|
||||
def test_an_ad_without_a_title_or_id_is_skipped_and_an_empty_board_is_empty() -> None:
|
||||
def test_an_ad_without_a_title_or_id_is_handed_over_and_an_empty_board_is_empty() -> None:
|
||||
body = json.dumps({"jobAdDetails": [{"id": "a", "title": " "}, {"title": "x"},
|
||||
{"id": "c", "title": "Cook"}]}).encode()
|
||||
assert [r["source_id"] for r in hibob.parse(body, "acme")] == ["c"]
|
||||
rows = hibob.parse(body, "acme")
|
||||
assert [r["source_id"] for r in rows] == ["a", "", "c"]
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == ["c"]
|
||||
assert hibob.parse(b'{"filterGroups": {}, "jobAdDetails": []}', "acme") == []
|
||||
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, hrmanager # noqa: E402
|
||||
from plugins.readers import Ask, _runner, hrmanager # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
BOARD = (FIXTURES / "hrmanager.xml").read_bytes()
|
||||
@@ -52,12 +52,14 @@ def test_a_channel_with_no_items_is_an_empty_board_and_another_answer_is_not_a_f
|
||||
hrmanager.parse(b"<html><body>404 - File or directory not found.</body></html>", "x")
|
||||
|
||||
|
||||
def test_an_item_without_a_title_or_a_web_link_is_skipped() -> None:
|
||||
def test_an_item_without_a_title_or_a_web_link_is_handed_over_for_the_runner_to_drop() -> None:
|
||||
body = (b"<rss><channel><item><guid>1</guid><title> </title><link>https://a.example/1</link></item>"
|
||||
b"<item><guid>2</guid><title>B</title><link>javascript:alert(1)</link></item>"
|
||||
b"<item><guid>3</guid><title>C</title><link>https://a.example/3</link></item>"
|
||||
b"</channel></rss>")
|
||||
assert [r["source_id"] for r in hrmanager.parse(body, "x")] == ["3"]
|
||||
rows = hrmanager.parse(body, "x")
|
||||
assert [r["source_id"] for r in rows] == ["1", "2", "3"]
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == ["3"]
|
||||
|
||||
|
||||
# ------------------------------------------------------------- paging
|
||||
|
||||
@@ -18,7 +18,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs import discover # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, jibe # noqa: E402
|
||||
from plugins.readers import Ask, _runner, jibe # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
SITE = "jobs.booking.com"
|
||||
@@ -72,9 +72,10 @@ def test_the_location_falls_back_to_city_and_country() -> None:
|
||||
assert jibe.rows({"jobs": [{"data": role}]}, SITE)[0]["location"] == "Riga, Latvia"
|
||||
|
||||
|
||||
def test_a_role_without_a_title_or_a_slug_is_left_out() -> None:
|
||||
def test_a_role_without_a_title_or_a_slug_is_handed_over_for_the_runner_to_drop() -> None:
|
||||
jobs = [{"data": {"slug": "1", "title": " "}}, {"data": {"title": "Role"}}, {}]
|
||||
assert jibe.rows({"jobs": jobs}, SITE) == []
|
||||
rows = jibe.rows({"jobs": jobs}, SITE)
|
||||
assert len(rows) == 3 and not any(map(_runner.valid, rows))
|
||||
|
||||
|
||||
def test_a_page_states_its_total_and_the_cursor_is_the_next_number() -> None:
|
||||
|
||||
@@ -16,7 +16,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import Ask, jobylon # noqa: E402
|
||||
from plugins.readers import Ask, _runner, jobylon # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
BOARD = (FIXTURES / "jobylon.html").read_bytes()
|
||||
@@ -65,12 +65,14 @@ def test_the_literal_reader_takes_quotes_escapes_and_trailing_commas() -> None:
|
||||
assert [r["title"] for r in jobylon.parse(body, "9")] == ["A ] B"]
|
||||
|
||||
|
||||
def test_a_role_whose_link_is_not_a_role_page_is_skipped() -> None:
|
||||
def test_a_role_whose_link_is_not_a_role_page_gets_no_link_and_is_dropped() -> None:
|
||||
body = (b"JBL.embed_v2['jobs'] = [{id: '1', title: 'A', url: 'https://evil.example/jobs/1/'},"
|
||||
b"{id: '2', title: 'B', url: '//evil.example/jobs/2/'},"
|
||||
b"{id: '3', title: 'C', url: '/companies/3/'},"
|
||||
b"{id: '4', title: 'D', url: '/jobs/4-d/'}];\n")
|
||||
assert [r["source_id"] for r in jobylon.parse(body, "9")] == ["4"]
|
||||
rows = jobylon.parse(body, "9")
|
||||
assert [r["url"] for r in rows[:3]] == ["", "", ""]
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == ["4"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- detail
|
||||
|
||||
+26
-1
@@ -1,15 +1,20 @@
|
||||
"""The Lever host module's places, country and employment, against a small
|
||||
saved board. No network. The pay and the plain row are in `test_ats.py`."""
|
||||
saved board, and an answer that is no board. No network. The pay and the
|
||||
plain row are in `test_ats.py`."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from plugins import ats # noqa: E402
|
||||
from plugins.readers import Ask, lever # noqa: E402
|
||||
|
||||
BOARD = (ROOT / "tests" / "fixtures" / "ats" / "lever-places.json").read_bytes()
|
||||
@@ -33,6 +38,26 @@ def test_the_commitment_is_the_hosts_word() -> None:
|
||||
assert several["employment"] == "Full-time" and one["employment"] == "Permanent"
|
||||
|
||||
|
||||
def test_an_answer_that_is_not_a_list_raises_and_an_empty_list_is_no_roles() -> None:
|
||||
with pytest.raises(ValueError, match="JSON dict, not a list"):
|
||||
lever.parse(b'{"ok": false, "error": "Document not found"}', "acme")
|
||||
assert lever.parse(b"[]", "acme") == []
|
||||
|
||||
|
||||
def test_an_error_document_with_a_200_fails_the_read_and_stamps_nothing(
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""It used to read as a complete board with no roles, which the settle
|
||||
closed whole under ten roles, where the 10% guard does not hold."""
|
||||
monkeypatch.setattr(ats, "_get", lambda *a, **kw: b'{"ok": false}')
|
||||
source = ats.ATSSource({"boards": [{"url": "lever:acme", "label": "Acme", "params": {}}],
|
||||
"run": "r"})
|
||||
source.state = {}
|
||||
assert list(source.fetch()) == []
|
||||
assert "ats:ok:lever:acme" not in source.state
|
||||
assert json.loads(source.state["ats:failing:lever:acme"])["reason"].startswith(
|
||||
"ValueError: Lever's answer is a JSON dict")
|
||||
|
||||
|
||||
def test_it_declares_a_whole_listing_of_trusted_dates_and_its_robots_record() -> None:
|
||||
reads = lever.READS
|
||||
assert (reads.listing, reads.text, reads.places) == ("whole", "listing", "many")
|
||||
|
||||
@@ -20,7 +20,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats # noqa: E402
|
||||
from plugins.readers import oracle # noqa: E402
|
||||
from plugins.readers import _runner, oracle # noqa: E402
|
||||
from plugins.readers._shared import is_web_url # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
@@ -84,14 +84,15 @@ def test_a_department_is_taken_where_a_tenant_fills_one() -> None:
|
||||
assert [r["department"] for r in rows[:3]] == ["Supply Chain", "Finance", ""]
|
||||
|
||||
|
||||
def test_a_role_without_a_title_or_with_an_id_that_is_no_id_is_left_out() -> None:
|
||||
def test_a_role_without_a_title_or_with_an_id_that_is_no_id_is_dropped_by_the_runner() -> None:
|
||||
data = _listing()
|
||||
roles = data["items"][0]["requisitionList"]
|
||||
roles[0]["Title"] = " "
|
||||
roles[1]["Id"] = "1/../../x"
|
||||
roles[2]["Id"] = ""
|
||||
assert [r["source_id"] for r in oracle.parse(json.dumps(data).encode(), SLUG)] == \
|
||||
["40975", "31680"]
|
||||
rows = oracle.parse(json.dumps(data).encode(), SLUG)
|
||||
assert rows[1]["url"] == "" # the id builds no link
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == ["40975", "31680"]
|
||||
|
||||
|
||||
def test_a_listing_without_the_expand_has_no_roles() -> None:
|
||||
@@ -165,6 +166,28 @@ def test_an_empty_board_is_complete(monkeypatch: pytest.MonkeyPatch, no_delay: N
|
||||
assert ats._fetch_rows(object(), f"oracle:{SLUG}") == ([], True) # type: ignore[arg-type]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("answer", [
|
||||
{}, {"items": []}, {"items": {}}, {"items": [None]}, [], {"title": "Unauthorized"}])
|
||||
def test_an_answer_without_the_search_wrapper_fails_the_read(
|
||||
monkeypatch: pytest.MonkeyPatch, no_delay: None, answer: Any) -> None:
|
||||
"""It used to read as an empty last page with no total: a complete
|
||||
read of a board with no roles."""
|
||||
monkeypatch.setattr(ats, "_get", lambda *a, **k: json.dumps(answer).encode())
|
||||
with pytest.raises(ValueError, match="no search wrapper in items"):
|
||||
ats._fetch_rows(object(), f"oracle:{SLUG}") # type: ignore[arg-type]
|
||||
|
||||
|
||||
def test_a_board_answering_without_the_wrapper_is_recorded_failing(
|
||||
monkeypatch: pytest.MonkeyPatch, no_delay: None) -> None:
|
||||
monkeypatch.setattr(ats, "_get", lambda *a, **k: b'{"title": "Unauthorized"}')
|
||||
source = ats.ATSSource({"boards": [{"url": f"oracle:{SLUG}", "label": "Nokia",
|
||||
"params": {}}], "run": "r"})
|
||||
source.state = {}
|
||||
assert list(source.fetch()) == []
|
||||
assert f"ats:ok:oracle:{SLUG}" not in source.state
|
||||
assert "no search wrapper" in json.loads(source.state[f"ats:failing:oracle:{SLUG}"])["reason"]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- detail
|
||||
|
||||
def _detail_get(body: bytes, asked: list[str]) -> Any:
|
||||
|
||||
@@ -19,7 +19,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats # noqa: E402
|
||||
from plugins.readers import phenom # noqa: E402
|
||||
from plugins.readers import _runner, phenom # noqa: E402
|
||||
from plugins.readers._shared import is_web_url # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
@@ -101,13 +101,15 @@ def test_a_title_with_nothing_to_make_a_slug_of_has_a_page_by_id_alone() -> None
|
||||
"https://careers.king.com/us/en/job/ACMKDSUSR028168EXTERNALENUS"
|
||||
|
||||
|
||||
def test_a_role_without_title_or_id_is_left_out() -> None:
|
||||
def test_a_role_without_title_or_id_is_dropped_by_the_runner() -> None:
|
||||
data = _answer()
|
||||
jobs = data["refineSearch"]["data"]["jobs"]
|
||||
jobs[0]["title"] = ""
|
||||
for key in ("jobSeqNo", "jobId", "reqId"):
|
||||
jobs[1].pop(key, None)
|
||||
assert [r["source_id"] for r in phenom.parse(json.dumps(data).encode(), SLUG)] == IDS[2:]
|
||||
rows = phenom.parse(json.dumps(data).encode(), SLUG)
|
||||
assert len(rows) == len(IDS)
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == IDS[2:]
|
||||
|
||||
|
||||
# ---------------------------------------------------------- the request
|
||||
|
||||
@@ -15,7 +15,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs import discover # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import pinpoint # noqa: E402
|
||||
from plugins.readers import _runner, pinpoint # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
|
||||
@@ -86,11 +86,12 @@ def test_an_unknown_pay_frequency_keeps_the_text_and_leaves_the_columns() -> Non
|
||||
assert (row["salary_min"], row["salary_max"], row["currency"]) == (None, None, "")
|
||||
|
||||
|
||||
def test_a_posting_without_a_title_or_an_id_or_a_web_url_is_left_out() -> None:
|
||||
def test_a_posting_without_a_title_or_an_id_or_a_web_url_is_dropped_by_the_runner() -> None:
|
||||
body = json.loads(_body())
|
||||
body["data"][0]["title"] = ""
|
||||
body["data"][1]["url"] = "javascript:alert(1)"
|
||||
assert pinpoint.parse(json.dumps(body).encode(), "xeneta") == []
|
||||
rows = pinpoint.parse(json.dumps(body).encode(), "xeneta")
|
||||
assert len(rows) == 2 and not any(map(_runner.valid, rows))
|
||||
|
||||
|
||||
def test_an_empty_board_is_no_rows_and_a_page_that_is_not_a_listing_raises() -> None:
|
||||
|
||||
@@ -18,7 +18,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import rippling # noqa: E402
|
||||
from plugins.readers import _runner, rippling # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
BOARD = (FIXTURES / "rippling.json").read_bytes()
|
||||
@@ -61,11 +61,14 @@ def test_an_empty_board_is_no_roles_and_an_error_answer_is_not_a_board() -> None
|
||||
rippling.parse(b'{"error_code":"RESOURCE_NOT_FOUND"}', "acme")
|
||||
|
||||
|
||||
def test_an_entry_without_a_title_or_a_plain_id_is_skipped() -> None:
|
||||
def test_an_entry_without_a_title_or_a_plain_id_is_dropped_by_the_runner() -> None:
|
||||
body = json.dumps([{"uuid": "abcdefgh-1", "name": " "}, {"name": "B"},
|
||||
{"uuid": "../../x", "name": "C"},
|
||||
{"uuid": "abcdefgh-2", "name": "D"}]).encode()
|
||||
assert [r["source_id"] for r in rippling.parse(body, "acme")] == ["abcdefgh-2"]
|
||||
rows = rippling.parse(body, "acme")
|
||||
assert [r["source_id"] for r in rows] == ["abcdefgh-1", "", "../../x", "abcdefgh-2"]
|
||||
assert rows[2]["url"] == "" # the id builds no link
|
||||
assert [r["source_id"] for r in rows if _runner.valid(r)] == ["abcdefgh-2"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- detail
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
"""The SuccessFactors host: `plugins/readers/successfactors.py` against saved
|
||||
career-site pages. No network. The fixtures are trimmed pages of Vestas and
|
||||
career-site pages, and against written ones through a sweep and a pull with
|
||||
its settle. No network. The fixtures are trimmed pages of Vestas and
|
||||
SAP (table layout), Demant and Scania (tile layout), Teradyne (a shared site
|
||||
with a path prefix), and the job pages of Vestas (`addressCountry`),
|
||||
Systematic (`streetAddress`) and Novo Nordisk (no microdata)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, ClassVar
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
@@ -18,13 +20,28 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from pipeline import Tally # noqa: E402
|
||||
from app.jobs import pull # noqa: E402
|
||||
from app.jobs.discover import boards_in # noqa: E402
|
||||
from app.storage import Store # noqa: E402
|
||||
from plugins import ats, readers # noqa: E402
|
||||
from plugins.readers import successfactors # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
|
||||
|
||||
class FakeConfig:
|
||||
sources: ClassVar[dict[str, object]] = {}
|
||||
searches: ClassVar[list[object]] = []
|
||||
|
||||
|
||||
def _pull(store: Store) -> dict[str, Any]:
|
||||
"""One pull of every board, batches and settle, as `test_sweep_strikes`."""
|
||||
results = [pull.run_batch(FakeConfig(), store, **batch) # type: ignore[arg-type]
|
||||
for batch in pull._boards(FakeConfig(), store)] # type: ignore[arg-type]
|
||||
return pull.close_boards(FakeConfig(), store, Tally(results=tuple(results))) # type: ignore[arg-type]
|
||||
|
||||
|
||||
def _read(name: str) -> str:
|
||||
return (FIXTURES / name).read_text(encoding="utf-8")
|
||||
|
||||
@@ -264,6 +281,70 @@ def test_a_site_that_ignores_startrow_does_not_loop(monkeypatch: pytest.MonkeyPa
|
||||
assert len(asked) == 2
|
||||
|
||||
|
||||
def _untotalled(ids: list[int]) -> str:
|
||||
"""A table page that states no total."""
|
||||
return _table_page(ids, 0).split("</span>", 1)[1]
|
||||
|
||||
|
||||
def test_the_repeated_page_hands_back_a_cursor_with_no_results() -> None:
|
||||
text = _untotalled([1_000_000_001, 1_000_000_002])
|
||||
_decoded, results, total, cursor = successfactors.page(
|
||||
lambda url, **kw: text.encode(), "careers.acme.com", [2, "1000000001"])
|
||||
assert (results, total, cursor) == ([], None, (2, "1000000001"))
|
||||
|
||||
|
||||
def test_a_site_that_ignores_startrow_and_states_no_total_closes_nothing(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
"""The repeated first page used to end the read with no cursor, which a
|
||||
site that states no total made complete, and the settle closed every
|
||||
role past the first page."""
|
||||
store = Store(tmp_path / "roledesk.db")
|
||||
store.add_subscription("ats", "successfactors:careers.acme.com", label="Acme",
|
||||
company="Acme", board="successfactors")
|
||||
ids = [1_000_000_000 + n for n in range(4)]
|
||||
site = {"ignores_startrow": False}
|
||||
|
||||
def fake_get(client, url, host, slug, **kw):
|
||||
start = 0 if site["ignores_startrow"] else int(dict(httpx.URL(url).params)["startrow"])
|
||||
return _untotalled(ids[start:start + 2]).encode()
|
||||
|
||||
monkeypatch.setattr(ats, "_get", fake_get)
|
||||
monkeypatch.setattr(ats, "PAGE_DELAY", 0)
|
||||
_pull(store)
|
||||
assert store.state["ats:complete:successfactors:careers.acme.com"] == "1"
|
||||
assert len(store.roles()) == 4
|
||||
|
||||
site["ignores_startrow"] = True
|
||||
result = _pull(store)
|
||||
assert store.state["ats:complete:successfactors:careers.acme.com"] == ""
|
||||
assert (result["incomplete"], result["closed"], len(store.roles())) == (1, 0, 4)
|
||||
|
||||
|
||||
def test_a_site_over_the_page_cap_is_swept_from_its_cursor_as_json_gives_it_back(
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The cursor `(start row, first id)` is a tuple, kept in state as a
|
||||
list. The sweep is kept, and the next pull reads on from it."""
|
||||
asked = _site(monkeypatch, 5, 2)
|
||||
monkeypatch.setattr(ats, "MAX_PAGES", 2)
|
||||
target = "successfactors:careers.acme.com"
|
||||
state: dict[str, str] = {}
|
||||
|
||||
def pull_once() -> list[Any]:
|
||||
source = ats.ATSSource({"boards": [{"url": target, "label": "Acme", "params": {}}],
|
||||
"run": "r"})
|
||||
source.state = state
|
||||
return list(source.fetch())
|
||||
|
||||
assert len(pull_once()) == 4
|
||||
assert state[f"ats:complete:{target}"] == ""
|
||||
assert json.loads(state[f"ats:sweep:{target}"])["cursor"] == [4, "1000000002"]
|
||||
assert len(pull_once()) == 1
|
||||
assert (state[f"ats:complete:{target}"], state[f"ats:swept:{target}"]) == ("1", "1")
|
||||
assert json.loads(state[f"ats:ids:{target}"]) == [
|
||||
f"{target}:{1_000_000_000 + n}" for n in range(5)]
|
||||
assert [int(u.rsplit("=", 1)[1]) for u in asked] == [0, 2, 4]
|
||||
|
||||
|
||||
def test_a_board_with_no_roles_is_complete_at_zero(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_site(monkeypatch, 0, 10)
|
||||
assert ats._fetch_rows(object(), "successfactors:careers.acme.com") == ([], True) # type: ignore[arg-type]
|
||||
|
||||
@@ -10,7 +10,7 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from plugins.readers import Ask, teamtailor # noqa: E402
|
||||
from plugins.readers import Ask, _runner, teamtailor # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
|
||||
@@ -40,7 +40,8 @@ def test_page_takes_the_ask_and_reads_the_same_url() -> None:
|
||||
|
||||
|
||||
def test_every_distinct_location_is_a_place_with_the_roles_work_mode() -> None:
|
||||
first, second, third = _rows()
|
||||
first, second, third, refused = _rows()
|
||||
assert not _runner.valid(refused) # javascript:, for the runner
|
||||
# "Sverige" (a city that is a country) once, Stockholm's twin once.
|
||||
assert first["places"] == [{"text": "Sverige, Sweden", "remote": False}, # hybrid
|
||||
{"text": "Stockholm, Sweden", "remote": False}]
|
||||
|
||||
@@ -20,7 +20,7 @@ if str(ROOT) not in sys.path:
|
||||
|
||||
from app.jobs import discover # noqa: E402
|
||||
from plugins import ats # noqa: E402
|
||||
from plugins.readers import workday # noqa: E402
|
||||
from plugins.readers import _runner, workday # noqa: E402
|
||||
|
||||
FIXTURES = ROOT / "tests" / "fixtures" / "ats"
|
||||
SLUG = "zendesk.wd1/zendesk"
|
||||
@@ -96,7 +96,7 @@ def test_posted_is_read_only_where_the_text_is_exact(text: str, expected: dateti
|
||||
assert workday._posted(text) == expected
|
||||
|
||||
|
||||
def test_a_posting_without_a_title_or_a_path_is_skipped() -> None:
|
||||
def test_a_posting_without_a_title_or_a_path_reaches_the_runner_and_is_dropped() -> None:
|
||||
page = {"jobPostings": [
|
||||
{"bulletFields": ["0000036369"]}, # LEGO's, no title
|
||||
{"title": "No path", "bulletFields": ["R1"]},
|
||||
@@ -105,7 +105,16 @@ def test_a_posting_without_a_title_or_a_path_is_skipped() -> None:
|
||||
{"title": "Whole", "externalPath": "/job/X/Whole_R4", "locationsText": "X",
|
||||
"postedOn": "Posted Today", "bulletFields": ["R4"]},
|
||||
]}
|
||||
assert [r["title"] for r in workday.rows(page, SLUG)] == ["Whole"]
|
||||
rows = workday.rows(page, SLUG)
|
||||
assert len(rows) == 5
|
||||
assert [(r["source_id"], r["url"]) for r in rows[:2]] == [("", ""), ("", "")]
|
||||
assert [r["title"] for r in rows if _runner.valid(r)] == ["Whole"]
|
||||
# Through the runner: four dropped and counted, and the read complete,
|
||||
# since every posting counts by its position towards the stated total.
|
||||
body = json.dumps(page | {"total": 5}).encode()
|
||||
listing = _runner.read_listing(workday, SLUG, lambda url, **kw: body)
|
||||
assert [r["title"] for r in listing.rows] == ["Whole"]
|
||||
assert (listing.handed, listing.dropped, listing.complete) == (5, 4, True)
|
||||
|
||||
|
||||
def test_the_id_is_the_path_where_the_requisition_id_is_not_in_it() -> None:
|
||||
@@ -251,6 +260,31 @@ def test_a_first_page_stating_no_total_is_read_to_a_page_that_adds_nothing(
|
||||
assert len(rows) == 30 and not complete
|
||||
|
||||
|
||||
def test_a_board_over_the_page_cap_is_swept_from_its_cursor_as_json_gives_it_back(
|
||||
monkeypatch: pytest.MonkeyPatch, no_delay: None) -> None:
|
||||
"""The cursor `(offset, total)` is a tuple, kept in state as a list, and
|
||||
the total later pages state as 0 is carried in it. The sweep is kept,
|
||||
and the next pull reads on from it to the first page's total."""
|
||||
tenant = Tenant(45)
|
||||
monkeypatch.setattr(ats, "_get", tenant)
|
||||
monkeypatch.setattr(ats, "MAX_PAGES", 2)
|
||||
target = f"workday:{SLUG}"
|
||||
state: dict[str, str] = {}
|
||||
|
||||
def pull_once() -> list[Any]:
|
||||
source = ats.ATSSource({"boards": [{"url": target, "label": "Zendesk", "params": {}}],
|
||||
"run": "r"})
|
||||
source.state = state
|
||||
return list(source.fetch())
|
||||
|
||||
assert len(pull_once()) == 40
|
||||
assert json.loads(state[f"ats:sweep:{target}"])["cursor"] == [40, 45]
|
||||
assert len(pull_once()) == 5
|
||||
assert (state[f"ats:complete:{target}"], state[f"ats:swept:{target}"]) == ("1", "1")
|
||||
assert len(json.loads(state[f"ats:ids:{target}"])) == 45
|
||||
assert [c[2]["offset"] for c in tenant.calls] == [0, 20, 40]
|
||||
|
||||
|
||||
def test_a_page_that_is_not_json_fails_the_read(
|
||||
monkeypatch: pytest.MonkeyPatch, no_delay: None) -> None:
|
||||
monkeypatch.setattr(ats, "_get", lambda *a, **k: b"<html>maintenance</html>")
|
||||
|
||||
@@ -148,7 +148,7 @@ def test_migration_20_seeds_an_old_file_and_leaves_collisions(tmp_path: Path) ->
|
||||
"VALUES (?, ?, '', '')", (key, name))
|
||||
conn.execute("PRAGMA user_version = 19")
|
||||
reopened = Store(path)
|
||||
assert reopened.migrations_applied == [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
|
||||
assert reopened.migrations_applied == [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]
|
||||
assert reopened.duplicate_companies() == [("orsted", ["orsted", "ørsted"])]
|
||||
assert len(reopened.companies()) == 3
|
||||
|
||||
@@ -186,7 +186,7 @@ def test_migration_21_tags_the_favourites_and_the_boards_that_read_all_roles(tmp
|
||||
conn.execute("ALTER TABLE companies DROP COLUMN all_countries")
|
||||
conn.execute("PRAGMA user_version = 20")
|
||||
reopened = Store(path)
|
||||
assert reopened.migrations_applied == [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
|
||||
assert reopened.migrations_applied == [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]
|
||||
# The tag of 21 is All countries and All roles since 24.
|
||||
tagged = {"mistral-ai": 1, "acme": 0, "black-forest-labs": 1, "liked": 0}
|
||||
assert {c.key: c.all_roles for c in reopened.companies()} == tagged
|
||||
|
||||
+11
-43
@@ -65,8 +65,8 @@ def setup(store: Store) -> tuple[Any, dict[str, str]]:
|
||||
uid[sid] = posting(sid).uid
|
||||
store.replace_scores(owner.id, [(uid["a"], 0.9, ""), (uid["b"], 0.8, ""),
|
||||
(uid["c"], 0.7, ""), (uid["d"], 0.6, "gated")])
|
||||
store.set_judgement(owner.id, uid["c"], "k", 90, 90, 90, ["Leads data."], ["Small team."])
|
||||
store.set_judgement(owner.id, uid["b"], "k", 60, 60, 60, ["Data."], ["Junior."])
|
||||
store.set_judgement(owner.id, uid["c"], "k", 9, 9, 9, ["Leads data."], ["Small team."])
|
||||
store.set_judgement(owner.id, uid["b"], "k", 6, 6, 6, ["Data."], ["Junior."])
|
||||
with store.transaction() as conn:
|
||||
conn.execute("UPDATE roles SET languages_required = ? WHERE uid = ?", (json.dumps(
|
||||
[{"language": "sv", "level": "fluent", "kind": "required"}]), uid["a"]))
|
||||
@@ -91,42 +91,11 @@ def test_best_cv_match_puts_the_judged_first_by_cv_score(
|
||||
"order": "best"}).status_code == 422
|
||||
|
||||
|
||||
def test_a_broken_must_have_caps_career_in_the_order_and_the_reader(
|
||||
client: TestClient, store: Store) -> None:
|
||||
owner, uid = setup(store)
|
||||
# c: 90 on all three, but one must-have broken: career counts as 39.
|
||||
musts = [{"id": "M1", "text": "Permanent", "verdict": "met"},
|
||||
{"id": "M2", "text": "Remote", "verdict": "broken"}]
|
||||
store.set_judgement(owner.id, uid["c"], "k", 90, 90, 90, [], [], musts)
|
||||
capped = round((30 * 90 + 25 * 90 + 30 * 39) / 85, 1)
|
||||
rows = client.get("/api/roles", params={"agent": owner.id, "order": "cv"}).json()["items"]
|
||||
assert [(r["uid"], r["cv_score"]) for r in rows[:2]] == [(uid["c"], capped), (uid["b"], 60.0)]
|
||||
# With career weighted up the capped value drops under b's 60.
|
||||
store.save_agent("Data", prompts=owner.prompts, agent_id=owner.id, cv_text=owner.cv_text,
|
||||
weights=Weights(cv_skills=0, cv_experience=0, cv_career=1))
|
||||
rows = client.get("/api/roles", params={"agent": owner.id, "order": "cv"}).json()["items"]
|
||||
assert [(r["uid"], r["cv_score"]) for r in rows[:2]] == [(uid["b"], 60.0), (uid["c"], 39.0)]
|
||||
one = client.get(f"/api/roles/{uid['c']}", params={"agent": owner.id}).json()
|
||||
j = one["judgement"]
|
||||
assert (j["career"], j["career_capped"], j["broken"]) == (90, 39, 1)
|
||||
assert j["musts"] == musts and one["cv_score"] == j["cv_score"] == 39.0
|
||||
|
||||
|
||||
def test_a_met_or_unclear_must_have_leaves_career_as_the_model_gave_it(
|
||||
client: TestClient, store: Store) -> None:
|
||||
owner, uid = setup(store)
|
||||
musts = [{"id": "M1", "text": "Permanent", "verdict": "met"},
|
||||
{"id": "M2", "text": "Remote", "verdict": "unclear"}]
|
||||
store.set_judgement(owner.id, uid["c"], "k", 90, 90, 90, [], [], musts)
|
||||
j = client.get(f"/api/roles/{uid['c']}", params={"agent": owner.id}).json()["judgement"]
|
||||
assert (j["career"], j["career_capped"], j["broken"], j["cv_score"]) == (90, 90, 0, 90.0)
|
||||
|
||||
|
||||
def test_the_agents_weights_make_the_cv_score_at_read_time(
|
||||
client: TestClient, store: Store) -> None:
|
||||
owner, uid = setup(store)
|
||||
# c: skills 100, experience 0, career 0; b: 60 on all three.
|
||||
store.set_judgement(owner.id, uid["c"], "k", 100, 0, 0, ["Leads data."], ["Small team."])
|
||||
# c: skills 10, experience 0, career 0; b: 6 on all three.
|
||||
store.set_judgement(owner.id, uid["c"], "k", 10, 0, 0, ["Leads data."], ["Small team."])
|
||||
key_before = input_key(owner, "text")
|
||||
|
||||
def asked() -> tuple[list[tuple[str, float | None]], float | None]:
|
||||
@@ -150,7 +119,7 @@ def test_the_agents_weights_make_the_cv_score_at_read_time(
|
||||
assert [c for _, c in rows] == [None, None, None] and one is None
|
||||
assert [u for u, _ in rows] == [uid["a"], uid["b"], uid["c"]], "by score, as none is judged"
|
||||
assert client.get(f"/api/roles/{uid['c']}",
|
||||
params={"agent": owner.id}).json()["judgement"]["skills"] == 100
|
||||
params={"agent": owner.id}).json()["judgement"]["skills"] == 10
|
||||
refreshed = store.agent(owner.id)
|
||||
assert refreshed is not None and input_key(refreshed, "text") == key_before, \
|
||||
"the weights are no part of what the judge read"
|
||||
@@ -160,9 +129,9 @@ def test_the_weights_are_per_agent_json_with_the_old_values_as_default(store: St
|
||||
owner = store.save_agent("Data", prompts=[Prompt(name="Lead", text="Head of data")])
|
||||
assert (owner.weights.cv_skills, owner.weights.cv_experience,
|
||||
owner.weights.cv_career) == (30, 25, 30)
|
||||
assert cv_score(80, 60, 40, owner.weights) == round((2400 + 1500 + 1200) / 85, 1)
|
||||
assert cv_score(80, 60, 40, Weights(cv_skills=0, cv_experience=1, cv_career=1)) == 50.0
|
||||
assert cv_score(80, 60, 40, Weights(cv_skills=0, cv_experience=0, cv_career=0)) is None
|
||||
assert cv_score(8, 6, 4, owner.weights) == round(10 * (240 + 150 + 120) / 85, 1)
|
||||
assert cv_score(8, 6, 4, Weights(cv_skills=0, cv_experience=1, cv_career=1)) == 50.0
|
||||
assert cv_score(8, 6, 4, Weights(cv_skills=0, cv_experience=0, cv_career=0)) is None
|
||||
|
||||
|
||||
def test_a_row_carries_the_deadline_and_the_language_flag(
|
||||
@@ -199,7 +168,7 @@ def test_the_role_carries_its_judgement_and_languages(
|
||||
owner, uid = setup(store)
|
||||
role = client.get(f"/api/roles/{uid['c']}", params={"agent": owner.id}).json()
|
||||
judgement = role["judgement"]
|
||||
assert (judgement["skills"], judgement["cv_score"]) == (90, 90.0)
|
||||
assert (judgement["skills"], judgement["cv_score"]) == (9, 90.0)
|
||||
assert judgement["strengths"] == ["Leads data."] and judgement["gaps"] == ["Small team."]
|
||||
assert judgement["judged_at"]
|
||||
assert role["cv_score"] == 90.0 and role["lang_flag"] == "plus"
|
||||
@@ -218,15 +187,14 @@ def test_an_agent_without_languages_flags_nothing(client: TestClient, store: Sto
|
||||
|
||||
def test_the_cv_fields_round_trip_and_are_kept_when_left_out(client: TestClient) -> None:
|
||||
body = {"name": "Data", "cv_text": "VP Data.", "like_text": "Teams.", "avoid_text": "Reporting.",
|
||||
"must_text": "Permanent.",
|
||||
"cv_languages": [{"language": "da", "level": "native"},
|
||||
{"language": "EN", "level": "fluent"}]}
|
||||
made = client.post("/api/agents", json=body).json()
|
||||
assert made["cv_languages"] == [{"language": "da", "level": "native"},
|
||||
{"language": "en", "level": "fluent"}]
|
||||
kept = client.post("/api/agents", json={"id": made["id"], "name": "Data"}).json()
|
||||
assert (kept["cv_text"], kept["like_text"], kept["avoid_text"], kept["must_text"]) == \
|
||||
("VP Data.", "Teams.", "Reporting.", "Permanent.")
|
||||
assert (kept["cv_text"], kept["like_text"], kept["avoid_text"]) == \
|
||||
("VP Data.", "Teams.", "Reporting.")
|
||||
assert len(kept["cv_languages"]) == 2
|
||||
nulls = client.post("/api/agents", json={"id": made["id"], "name": "Data",
|
||||
"cv_text": None, "cv_languages": None}).json()
|
||||
|
||||
+83
-37
@@ -16,7 +16,7 @@ from pydantic import ValidationError
|
||||
from app import prep
|
||||
from app.config import Config
|
||||
from app.jobs import extract
|
||||
from app.models import CvLanguage, RoleLanguage, cv_score, iso_day, language_flag, must_lines
|
||||
from app.models import CvLanguage, RoleLanguage, cv_score, iso_day, language_flag
|
||||
from app.storage import Store
|
||||
from desk import modeltask
|
||||
from desk.models import Item
|
||||
@@ -44,24 +44,21 @@ def test_an_agent_keeps_its_cv_fields_when_a_save_leaves_them_out(store: Store)
|
||||
cv_languages=[CvLanguage(language="DA", level="native"),
|
||||
CvLanguage(language="en", level="fluent"),
|
||||
CvLanguage(language="da", level="basic")],
|
||||
like_text=" Building teams. ", avoid_text="Pure reporting.",
|
||||
must_text="Permanent contract.")
|
||||
like_text=" Building teams. ", avoid_text="Pure reporting.")
|
||||
assert made.cv_text == "VP Data."
|
||||
assert [(lang.language, lang.level) for lang in made.cv_languages] == \
|
||||
[("da", "native"), ("en", "fluent")]
|
||||
assert (made.like_text, made.avoid_text, made.must_text) == \
|
||||
("Building teams.", "Pure reporting.", "Permanent contract.")
|
||||
assert (made.like_text, made.avoid_text) == ("Building teams.", "Pure reporting.")
|
||||
|
||||
kept = store.save_agent("Data", "data leadership", agent_id=made.id)
|
||||
assert (kept.cv_text, kept.like_text, kept.avoid_text, kept.must_text) == \
|
||||
("VP Data.", "Building teams.", "Pure reporting.", "Permanent contract.")
|
||||
assert (kept.cv_text, kept.like_text, kept.avoid_text) == \
|
||||
("VP Data.", "Building teams.", "Pure reporting.")
|
||||
assert len(kept.cv_languages) == 2
|
||||
|
||||
cleared = store.save_agent("Data", "data leadership", agent_id=made.id, cv_text="",
|
||||
cv_languages=[], like_text="", avoid_text="",
|
||||
must_text="")
|
||||
assert (cleared.cv_text, cleared.cv_languages, cleared.like_text, cleared.avoid_text,
|
||||
cleared.must_text) == ("", [], "", "", "")
|
||||
cv_languages=[], like_text="", avoid_text="")
|
||||
assert (cleared.cv_text, cleared.cv_languages, cleared.like_text, cleared.avoid_text) == \
|
||||
("", [], "", "")
|
||||
|
||||
|
||||
def test_a_cv_language_is_a_code_and_a_level_of_the_ladder() -> None:
|
||||
@@ -75,7 +72,7 @@ def test_new_agents_have_the_cut_by_default(store: Store) -> None:
|
||||
made = store.save_agent("Data")
|
||||
assert made.limits.model_dump() == {"cut_fit_top": 500, "cut_fit_min": 0.6,
|
||||
"cut_cv_top": 150, "cut_cv_min": 0.6,
|
||||
"summarise_top": 50}
|
||||
"summarise_top": 30}
|
||||
|
||||
|
||||
# --------------------------------------------------------------- migration
|
||||
@@ -92,25 +89,24 @@ def test_v28_adds_the_cv_columns_and_the_host_deadline_and_runs_twice(tmp_path:
|
||||
conn.execute("DROP TABLE role_judgements")
|
||||
for column in ("languages_required", "deadline_host", "deadline_model", "deadline"):
|
||||
conn.execute(f"ALTER TABLE roles DROP COLUMN {column}")
|
||||
for column in ("cv_text", "cv_languages", "avoid_text", "must_text"):
|
||||
for column in ("cv_text", "cv_languages", "avoid_text"):
|
||||
conn.execute(f"ALTER TABLE agents DROP COLUMN {column}")
|
||||
conn.execute("PRAGMA user_version = 27")
|
||||
|
||||
migrated = Store(path)
|
||||
|
||||
assert migrated.migrations_applied == [28, 29, 30, 31, 32]
|
||||
assert migrated.migrations_applied == [28, 29, 30, 31, 32, 33, 34, 35]
|
||||
rows = {r["source_id"]: r for r in migrated._rows(
|
||||
"SELECT source_id, languages_required, deadline_host, deadline FROM roles")}
|
||||
assert (rows["a"]["deadline_host"], rows["a"]["deadline"]) == ("2026-10-18", None)
|
||||
assert rows["a"]["languages_required"] is None and rows["b"]["deadline_host"] == ""
|
||||
[data] = migrated.agents()
|
||||
assert (data.cv_text, data.cv_languages, data.avoid_text, data.must_text) == \
|
||||
("", [], "", "")
|
||||
assert (data.cv_text, data.cv_languages, data.avoid_text) == ("", [], "")
|
||||
prep.run(migrated, remap=True, dupes=False)
|
||||
assert migrated.role(posting("a").uid).deadline == "2026-10-18"
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("PRAGMA user_version = 27")
|
||||
assert Store(path).migrations_applied == [28, 29, 30, 31, 32]
|
||||
assert Store(path).migrations_applied == [28, 29, 30, 31, 32, 33, 34, 35]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- deadline
|
||||
@@ -229,8 +225,8 @@ def test_the_language_flag() -> None:
|
||||
|
||||
|
||||
def test_the_cv_score_weighs_skills_and_career_above_experience() -> None:
|
||||
assert cv_score(100, 100, 100) == 100.0
|
||||
assert cv_score(80, 60, 40) == round((2400 + 1500 + 1200) / 85, 1)
|
||||
assert cv_score(10, 10, 10) == 100.0
|
||||
assert cv_score(8, 6, 4) == round(10 * (240 + 150 + 120) / 85, 1)
|
||||
|
||||
|
||||
def test_a_stored_language_list_survives_a_bad_entry(store: Store) -> None:
|
||||
@@ -243,20 +239,36 @@ def test_a_stored_language_list_survives_a_bad_entry(store: Store) -> None:
|
||||
assert [lang.language for lang in store.role(item.uid).languages_required] == ["da"]
|
||||
|
||||
|
||||
def test_v29_adds_the_musts_and_broken_columns_and_runs_twice(tmp_path: Path) -> None:
|
||||
path = tmp_path / "v28.db"
|
||||
Store(path)
|
||||
def test_v34_drops_the_must_haves_and_runs_twice(tmp_path: Path) -> None:
|
||||
"""A file from before migration 34 holds the agent's must text and the
|
||||
judge's verdicts on it; the columns go, the rest of the agent and the
|
||||
judgement stay."""
|
||||
path = tmp_path / "v33.db"
|
||||
store = Store(path)
|
||||
item = posting("1")
|
||||
upsert(store, item)
|
||||
agent = store.save_agent("Data", "data leadership", cv_text="CV.", like_text="Teams.")
|
||||
assert store.set_judgement(agent.id, item.uid, "key", 8, 7, 9, ["Good."], ["Bad."])
|
||||
with sqlite3.connect(path) as conn:
|
||||
for column in ("musts", "broken"):
|
||||
conn.execute(f"ALTER TABLE role_judgements DROP COLUMN {column}")
|
||||
conn.execute("PRAGMA user_version = 28")
|
||||
conn.execute("ALTER TABLE agents ADD COLUMN must_text TEXT NOT NULL DEFAULT ''")
|
||||
conn.execute("ALTER TABLE role_judgements ADD COLUMN musts TEXT NOT NULL DEFAULT '[]'")
|
||||
conn.execute("ALTER TABLE role_judgements ADD COLUMN broken INTEGER NOT NULL DEFAULT 0")
|
||||
conn.execute("UPDATE agents SET must_text = 'Remote.'")
|
||||
conn.execute("UPDATE role_judgements SET musts = ?, broken = 1",
|
||||
(json.dumps([{"id": "M1", "text": "Remote.", "verdict": "broken"}]),))
|
||||
conn.execute("PRAGMA user_version = 33")
|
||||
|
||||
assert Store(path).migrations_applied == [29, 30, 31, 32]
|
||||
with sqlite3.connect(path) as conn:
|
||||
columns = {r[1] for r in conn.execute("PRAGMA table_info(role_judgements)")}
|
||||
assert {"musts", "broken"} <= columns
|
||||
conn.execute("PRAGMA user_version = 28")
|
||||
assert Store(path).migrations_applied == [29, 30, 31, 32]
|
||||
for _ in range(2): # runs twice without harm
|
||||
migrated = Store(path)
|
||||
assert migrated.migrations_applied == [34, 35]
|
||||
with sqlite3.connect(path) as conn:
|
||||
assert "must_text" not in {r[1] for r in conn.execute("PRAGMA table_info(agents)")}
|
||||
assert not {"musts", "broken"} & {
|
||||
r[1] for r in conn.execute("PRAGMA table_info(role_judgements)")}
|
||||
conn.execute("PRAGMA user_version = 33")
|
||||
assert migrated.agent(agent.id).like_text == "Teams." # type: ignore[union-attr]
|
||||
found = migrated.judgement(agent.id, item.uid)
|
||||
assert found is not None and (found.career, found.cv_score) == (9, cv_score(8, 7, 9))
|
||||
|
||||
|
||||
def test_v30_adds_like_text_and_runs_twice(tmp_path: Path) -> None:
|
||||
@@ -268,15 +280,49 @@ def test_v30_adds_like_text_and_runs_twice(tmp_path: Path) -> None:
|
||||
|
||||
migrated = Store(path)
|
||||
|
||||
assert migrated.migrations_applied == [30, 31, 32]
|
||||
assert migrated.migrations_applied == [30, 31, 32, 33, 34, 35]
|
||||
[data] = migrated.agents()
|
||||
assert (data.like_text, data.avoid_text) == ("", "Reporting.")
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("PRAGMA user_version = 29")
|
||||
assert Store(path).migrations_applied == [30, 31, 32]
|
||||
assert Store(path).migrations_applied == [30, 31, 32, 33, 34, 35]
|
||||
|
||||
|
||||
def test_the_musts_text_is_one_must_have_per_line() -> None:
|
||||
assert must_lines("- A\n* B c\n\u2022 D\n1. E\n2) F\n \nG\n-H") == [
|
||||
"A", "B c", "D", "E", "F", "G", "-H"]
|
||||
assert must_lines("") == []
|
||||
def test_v33_clears_every_judgement_and_lowers_the_old_summarise_default(
|
||||
tmp_path: Path) -> None:
|
||||
"""A judgement on the 0 to 100 scale is never shown on the 0 to 10 one:
|
||||
migration 33 clears every row and gives the tries back, and the judge
|
||||
reads the pair again. `summarise_top` 50, migration 32's default, is 30;
|
||||
any other value, and a limits object without the key, stay."""
|
||||
path = tmp_path / "v32.db"
|
||||
store = Store(path)
|
||||
item = posting("1")
|
||||
upsert(store, item)
|
||||
old = store.save_agent("Old", "data leadership", cv_text="CV.")
|
||||
own = store.save_agent("Own", "data leadership")
|
||||
bare = store.save_agent("Bare", "data leadership")
|
||||
assert store.set_judgement(old.id, item.uid, "key", 80, 70, 90, ["Good."], ["Bad."])
|
||||
store.judge_missed([(own.id, item.uid)])
|
||||
with sqlite3.connect(path) as conn:
|
||||
conn.execute("UPDATE agents SET limits = json_set(limits, '$.summarise_top', 50) "
|
||||
"WHERE id = ?", (old.id,))
|
||||
conn.execute("UPDATE agents SET limits = json_set(limits, '$.summarise_top', 12) "
|
||||
"WHERE id = ?", (own.id,))
|
||||
conn.execute("UPDATE agents SET limits = '{}' WHERE id = ?", (bare.id,))
|
||||
conn.execute("PRAGMA user_version = 32")
|
||||
|
||||
for _ in range(2): # runs twice without harm
|
||||
migrated = Store(path)
|
||||
assert migrated.migrations_applied == [33, 34, 35]
|
||||
assert migrated.judgement(old.id, item.uid) is None
|
||||
with sqlite3.connect(path) as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT skills, experience, career, cv_score, strengths, gaps, "
|
||||
"input_key, tries, judged_at FROM role_judgements").fetchall()
|
||||
assert rows == [(None, None, None, None, "[]", "[]", "", 0, None)] * 2
|
||||
conn.execute("PRAGMA user_version = 32")
|
||||
assert {a.name: a.limits.summarise_top for a in migrated.agents()} == {
|
||||
"Old": 30, "Own": 12, "Bare": 30}
|
||||
with sqlite3.connect(path) as conn:
|
||||
assert conn.execute("SELECT limits FROM agents WHERE id = ?",
|
||||
(bare.id,)).fetchone() == ("{}",)
|
||||
@@ -268,7 +268,7 @@ def test_cv_match_orders_the_unjudged_by_closeness_then_score(store: Store) -> N
|
||||
assert [r.uid for r in store.roles(agent=owner.id, by_cv=True)] == [
|
||||
judged, close, far, none_b, none_a]
|
||||
assert [r.uid for r in store.roles_page(agent=owner.id)] == [
|
||||
far, none_b, none_a, close, judged], "Agent match ignores it"
|
||||
far, none_b, none_a, close, judged], "What I'm looking for ignores it"
|
||||
found = store.role(close, owner.id)
|
||||
assert found is not None and found.cv_fit == 0.9
|
||||
|
||||
@@ -282,7 +282,7 @@ def test_migration_31_adds_the_column_and_the_table_and_runs_twice(tmp_path: Pat
|
||||
conn.execute("DROP TABLE agent_cv_vectors")
|
||||
conn.execute("ALTER TABLE scores DROP COLUMN cv_fit")
|
||||
conn.execute("PRAGMA user_version = 30")
|
||||
assert Store(path).migrations_applied == [31, 32]
|
||||
assert Store(path).migrations_applied == [31, 32, 33, 34, 35]
|
||||
step = {v: s for v, _, s in MIGRATIONS}[31]
|
||||
with sqlite3.connect(path) as conn:
|
||||
assert callable(step)
|
||||
|
||||
@@ -4,6 +4,7 @@ no network. The live cases are the 2026-09-29 database's."""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
@@ -238,6 +239,31 @@ def test_folding_needs_the_same_title_a_link_and_texts_that_can_be_one_job() ->
|
||||
assert fold([pc("1", "Berlin", a), pc("2", "Munich", "")]).placed == {"2"}
|
||||
|
||||
|
||||
def test_the_per_place_fold_keeps_to_one_country() -> None:
|
||||
"""A federal sales role posted in Canberra and in Washington has alike
|
||||
text and is two openings; Berlin and Munich are one job in two offices.
|
||||
A row with no country joins either."""
|
||||
text = "Sell to federal agencies, run the public sector pipeline and partners."
|
||||
got = fold([replace(pc("au", "Canberra", text), country="AU"),
|
||||
replace(pc("us", "Washington", text), country="US"),
|
||||
pc("b", "Berlin", text), pc("m", "Munich", text),
|
||||
replace(pc("n", "", text), country="")])
|
||||
assert got.dup_of == {"au": None, "us": None, "b": None, "m": "b", "n": "au"}
|
||||
|
||||
|
||||
def test_a_remote_listing_is_not_folded_into_an_office_one() -> None:
|
||||
"""The kept role takes a copy's places, not its remote, so the gate would
|
||||
read "Remote, Germany" as an office role in Vienna and reject it for an
|
||||
agent that takes remote roles. Two remote listings still fold."""
|
||||
text = "We build the payment platform for our bank customers in Kotlin."
|
||||
office = replace(pc("v", "Vienna", text), country="AT")
|
||||
remote = replace(pc("r", "Remote, Berlin", text), remote=True)
|
||||
other = replace(pc("s", "Remote, Hamburg", text), remote=True)
|
||||
got = fold([office, remote, other])
|
||||
assert got.dup_of == {"v": None, "r": None, "s": "r"}
|
||||
assert got.placed == {"s"}
|
||||
|
||||
|
||||
def test_a_copy_of_a_folded_row_follows_it_to_the_new_kept_role() -> None:
|
||||
"""Berlin is kept over Munich; a second Munich row that was Munich's copy
|
||||
now points at Berlin, and is not a place of its own."""
|
||||
|
||||
+22
-19
@@ -103,10 +103,11 @@ def test_berlin_and_munich_of_one_host_fold_into_one_role_with_two_places(
|
||||
assert (row.uid, row.place, row.copies) == (kept, f"{got.places[0].text} +1", 0)
|
||||
|
||||
|
||||
def test_a_place_copy_in_europe_meets_a_europe_gate_for_its_kept_role(store: Store) -> None:
|
||||
"""OpenAI posts one role in San Francisco and in Dublin. The kept role
|
||||
holds both places, so a Europe gate and the Place filter find it through
|
||||
Dublin, though `roles.place` and `roles.country` are San Francisco's."""
|
||||
def test_a_posting_per_office_in_two_countries_is_two_roles(store: Store) -> None:
|
||||
"""OpenAI posts one role in San Francisco and in Dublin. The fold keeps
|
||||
to one country (ADR 1.4), so each is a role of its own: a Europe gate
|
||||
takes Dublin and rejects San Francisco, and the Place filter finds
|
||||
Dublin alone."""
|
||||
from app import prep
|
||||
from app.models import Gate
|
||||
sf = role("sf", company="OpenAI", place="San Francisco", country="US", board="ashby")
|
||||
@@ -114,10 +115,11 @@ def test_a_place_copy_in_europe_meets_a_europe_gate_for_its_kept_role(store: Sto
|
||||
for item in (sf, dublin):
|
||||
store.upsert_role(item)
|
||||
prep.run(store)
|
||||
kept = kept_of(store, sf, dublin)
|
||||
assert dup_of(store) == {sf.uid: None, dublin.uid: None}
|
||||
agent = store.save_agent("A", gates=[Gate(field="country", values=["FR", "IE", "EU"])])
|
||||
assert {c.uid: c.gates for c in store.score_candidates(agent.id, agent.gates)} == {kept: [1]}
|
||||
assert {r.uid for r in store.roles(places=["country:IE"], by_score=False)} == {kept}
|
||||
assert {c.uid: c.gates for c in store.score_candidates(agent.id, agent.gates)} == {
|
||||
sf.uid: [0], dublin.uid: [1]}
|
||||
assert {r.uid for r in store.roles(places=["country:IE"], by_score=False)} == {dublin.uid}
|
||||
|
||||
|
||||
def test_places_of_two_hosts_or_two_different_texts_stay_apart(store: Store) -> None:
|
||||
@@ -279,7 +281,7 @@ def test_a_v17_file_gains_dup_of_with_its_first_rebuild(tmp_path: Path) -> None:
|
||||
|
||||
migrated = Store(path)
|
||||
|
||||
assert migrated.migrations_applied == [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
|
||||
assert migrated.migrations_applied == [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]
|
||||
assert dup_of(migrated) == {first.uid: None, second.uid: first.uid}
|
||||
with sqlite3.connect(path) as conn:
|
||||
assert conn.execute("SELECT 1 FROM sqlite_master WHERE name = 'idx_roles_dup_of'"
|
||||
@@ -305,30 +307,31 @@ def test_switching_a_board_off_shows_its_roles_copies_at_once(store: Store) -> N
|
||||
|
||||
|
||||
def test_a_role_in_several_cities_leads_with_the_one_nearest_the_agent(store: Store) -> None:
|
||||
"""Mistral posts one role per city. For an agent kept to Europe the kept
|
||||
Montréal role leads with Paris, in the row and the reader, and opens the
|
||||
Paris copy; without an agent the host's order stands."""
|
||||
"""A Danish company posts one role per city. For an agent whose towns
|
||||
hold Copenhagen the kept Aarhus role leads with Copenhagen, in the row
|
||||
and the reader, and opens the Copenhagen copy; without an agent the
|
||||
host's order stands."""
|
||||
from app import prep
|
||||
from app.models import Gate
|
||||
items = [role(c, company="Mistral AI", place=p, board="ashby") for c, p in (
|
||||
("mtl", "Montréal"), ("nyc", "New York"), ("par", "Paris"))]
|
||||
items = [role(c, company="Danske", place=p, board="ashby") for c, p in (
|
||||
("aar", "Aarhus"), ("bil", "Billund"), ("cph", "Copenhagen"))]
|
||||
for item in items:
|
||||
store.upsert_role(item)
|
||||
prep.run(store)
|
||||
kept = kept_of(store, *items)
|
||||
agent = store.save_agent("A", gates=[
|
||||
Gate(field="country", values=["FR", "DK", "EU"], mode="require"),
|
||||
Gate(field="country", values=["DK"], mode="any")])
|
||||
Gate(field="place", values=["copenhagen"], mode="any")])
|
||||
|
||||
got = store.role(kept, agent.id)
|
||||
assert got is not None
|
||||
own = next(p.text for p in got.places if not p.url)
|
||||
assert own != "Paris"
|
||||
assert (got.lead_place, got.lead_url) == ("Paris", "https://jobs.example/par")
|
||||
assert got.places[0].text == "Paris"
|
||||
assert own != "Copenhagen"
|
||||
assert (got.lead_place, got.lead_url) == ("Copenhagen", "https://jobs.example/cph")
|
||||
assert got.places[0].text == "Copenhagen"
|
||||
(row,) = store.roles_page(agent=agent.id, by_score=False)
|
||||
assert (row.place, row.lead_place, row.lead_url) == ("Paris +2", "Paris", got.lead_url)
|
||||
assert row.place_texts[0] == "Paris" and len(row.place_texts) == 3
|
||||
assert (row.place, row.lead_place, row.lead_url) == ("Copenhagen +2", "Copenhagen", got.lead_url)
|
||||
assert row.place_texts[0] == "Copenhagen" and len(row.place_texts) == 3
|
||||
|
||||
plain = store.role(kept)
|
||||
assert plain is not None
|
||||
|
||||
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user