Rank by the cut: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text renamed from enrich, the feed kept to the fetch gate, and migration 32

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-10-04 15:11:43 +02:00
1 parent 060ccf66e6
commit 617356e60c
35 files changed
+1054 -751

No files matched your search

+51 -47
View File
@@ -1,32 +1,29 @@
"""The tasks roledesk offers, and the order a pull runs them in.
pull → prep → embed → score → enrich → prep → embed → score → extract → score
→ summarise → judge
pull → prep → embed → score → fetch_text → prep → embed → score → extract
→ score → judge → summarise
newsdesk's cascade convention: a cascade is a tuple of task names, and the
final job of one task starts the next (`step` is the callback it carries).
Nothing is alive between two tasks. `prep` (`app/jobs/prep.py`) maps what
the pull stored and folds the copies before anything reads the roles; after
`enrich` it maps the roles their own pages filled. A stage is addressed by its position in
the tuple, not its name, since `score` runs twice.
`fetch_text` it maps the roles their own pages filled. A stage is addressed
by its position in the tuple, not its name, since `score` runs three times.
Every stage is always in the chain, and a stage that is switched off finds
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this pull does not
satisfy, is an extract stage with no batches whose final job moves straight
on. The free stages go first: embed and score cost no tokens, and the score
they produce is what bounds the rest. Enrich reads the best roles without
text on their own pages (`app/jobs/enrich.py`), and prep, embed and score
follow for the roles it filled; when it filled none, `step` passes over all
three. Extract
reads only the best
roles of each enabled agent, as deep and as low as that agent's own `limits`
say; the second score then applies the gates to the columns it filled.
Summarise follows, on the best roles by the same agent's summarise limits,
and judge runs last, reading each agent's best roles against its CV by its
judge limits. The `enrich` cascade, which follows the rescore after an agent
is saved, ends with judge too, so a changed CV, prompt or like text is read
again at once rather than at the next pull; a pair whose `input_key` did not
change costs nothing.
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this
pull does not satisfy, is a stage with no batches whose final job moves
straight on. The free stages go first: embed and score cost no tokens, and
score writes each agent's cut (`app.jobs.score`), which bounds the rest.
Fetch text reads the role's own page for the roles in the cut that came
without text (`app/jobs/fetch_text.py`), and prep, embed and score follow
for the roles it filled, so their levels and the cut are recomputed; when
it filled none, `step` passes over all three. Extract reads the cut; the
score after it applies the fetch gate and the cut again with the columns
extract filled, so a role can drop out or enter (one that enters is
extracted on the next pull). Judge reads each agent's cut as recomputed,
against its CV. Summarise runs last, on the top of each agent's feed by
final score and the roles liked or saved: it is no part of any score.
A batch that failed, or a stage somebody killed, does not stop the chain: the
work comes round on the next pull, and a rescore of what did arrive is worth
@@ -36,10 +33,13 @@ having either way.
and a cascade is started by one: the `pull` job's only work is to ask for the
chain. Its default schedule, `every 6h`, is declared here beside it.
`enrich` is a job and a cascade the same way, `enrich → prep → embed → score`, queued
behind the `score` that follows a change to the ranking other than a pull's
(`enrich_after`): an agent saved, a company's affinity changed. A vote moves
one agent's taste a little and queues neither.
`fetch_text` is a job and a cascade the same way, the pull's chain from
fetch text on, queued behind the `score` that follows a change to the
ranking other than a pull's (`fetch_text_after`): an agent saved, a
company's affinity or overrides changed. The cut may have changed, so its
new roles get their text, extract and judge at once rather than at the next
pull; a judgement whose `input_key` did not change costs nothing. A vote
moves one agent's taste a little and queues neither.
"""
from __future__ import annotations
@@ -49,19 +49,22 @@ from dataclasses import replace
from functools import partial
from typing import Any
from app.jobs import embed, enrich, extract, judge, prep, pull, score, summarise # noqa: F401 -- registers the jobs
from app.jobs import ( # noqa: F401 -- registers the jobs
embed, extract, fetch_text, judge, prep, pull, score, summarise,
)
from desk.tools import agent
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
STEP = "app.jobs.cascade:step"
CASCADES: dict[str, tuple[str, ...]] = {
"pull": ("pull", "prep", "embed", "score", "enrich", "prep", "embed", "score", "extract",
"score", "summarise", "judge"),
"enrich": ("enrich", "prep", "embed", "score", "judge"),
"pull": ("pull", "prep", "embed", "score", "fetch_text", "prep", "embed", "score",
"extract", "score", "judge", "summarise"),
"fetch_text": ("fetch_text", "prep", "embed", "score", "extract", "score", "judge",
"summarise"),
}
#: What follows `enrich` for the roles it filled, passed over when it filled none.
#: What follows `fetch_text` for the roles it filled, passed over when it filled none.
REFILL = ("prep", "embed", "score")
@@ -131,7 +134,7 @@ STAGES: dict[str, Callable[[str], Task]] = {
label="Mapping the roles' host words"),
"summarise": _summarise,
"judge": _judge,
"enrich": lambda cascade: enrich.enrich_task(),
"fetch_text": lambda cascade: fetch_text.fetch_text_task(),
}
@@ -157,7 +160,7 @@ def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
name, done = payload.get("cascade", ""), payload.get("task", "")
at = int(payload.get("at", -1))
tasks = CASCADES.get(name, ())
if done == "enrich" and not enrich.filled(tally) \
if done == "fetch_text" and not fetch_text.filled(tally) \
and tasks[at + 1:at + 1 + len(REFILL)] == REFILL:
at += len(REFILL)
following = follows(name, at)
@@ -222,24 +225,25 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
return {"started": " > ".join(CASCADES["pull"]), "run": run}
@job("enrich", "Reading the best roles on their own pages", solo=True, resource="http")
def run_enrich(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Ask for the enrich cascade, `enrich → prep → embed → score`. Its own work is
nil, as the `pull` job's. Under `NULL` there is no queue to take it."""
if not context.task(chain("enrich")):
raise RuntimeError("no queue to start the enrich cascade on")
return {"started": " > ".join(CASCADES["enrich"])}
@job("fetch_text", fetch_text.LABEL, solo=True, resource="http")
def run_fetch_text(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Ask for the fetch text cascade, the pull's chain from fetch text on.
Its own work is nil, as the `pull` job's. Under `NULL` there is no queue
to take it."""
if not context.task(chain("fetch_text")):
raise RuntimeError("no queue to start the fetch text cascade on")
return {"started": " > ".join(CASCADES["fetch_text"])}
def enrich_after(queue: Any, score_job: Any) -> Any:
"""Queue the `enrich` job behind `score_job`, the rescore that follows a
change to the ranking, however that score ends. One per score job
(`enrich:after:<id>`), so ten changes answered by one score are one
enrich. None without a score job, or when the queue refuses."""
def fetch_text_after(queue: Any, score_job: Any) -> Any:
"""Queue the `fetch_text` job behind `score_job`, the rescore that
follows a change to the ranking, however that score ends. One per score
job (`fetch_text:after:<id>`), so ten changes answered by one score are
one cascade. None without a score job, or when the queue refuses."""
if score_job is None:
return None
return queue.enqueue("enrich", depends_on=[score_job.id], after="done",
dedup_key=f"enrich:after:{score_job.id}")
return queue.enqueue("fetch_text", depends_on=[score_job.id], after="done",
dedup_key=f"fetch_text:after:{score_job.id}")
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
@@ -255,5 +259,5 @@ for _name in ("extract", "embed", "score", "summarise", "judge", "prep"):
# setting: a cadence edited or switched off under Runs survives
# every restart (see pipeline/schedule.py).
declare("pull", "pull", "every 6h",
label="Pull every board, then map, embed, score, read the best roles' pages, "
"extract, rescore, summarise and judge against the CVs")
label="Pull every board, then map, embed, score, fetch the text of the roles in the "
"cut, extract, rescore, judge against the CVs and summarise")
+6 -7
View File
@@ -12,13 +12,12 @@ is waiting while it is open, has not been given up on, has never been
answered (`extracted_at IS NULL`), has no model level (`seniority_model`
empty), was never asked where a remote hire may live (`remote_where_model`
NULL) or for its languages and deadline (`languages_required` NULL), and is
in the best of an enabled agent's
ungated scores as deep and as low as that agent's own `limits` say
(`extract_top`, `extract_min`). That bound is what keeps the cost
proportional to the rankings rather than to the boards:
the pull scores every role for free first, then reads the few hundred that
rank, and scores again so the gates see what was filled. A batch built for
named uids finishes what it was dealt whatever their rank. The answer only
in the cut of an enabled agent (`scores.cut`, see `app.jobs.score`). That
bound is what keeps the cost proportional to the agents rather than to the
boards: the pull scores every role for free first, then reads the cut, and
scores again so the fetch gate and the cut see what was filled; a role that
enters the cut then is read on the next pull. A batch built for named uids
finishes what it was dealt whatever their rank. The answer only
fills columns that are still empty: a board's structured field wins over what
a model read out of the prose. The exception is the level, which the model
always gives (the board's word and the title's go in as hints) and which
+36 -30
View File
@@ -1,32 +1,34 @@
"""`enrich`: the best roles without text, read on their own pages.
"""`fetch_text`: the roles in the cut without text, read on their own pages.
Some hosts list a role without its text and state it on the role's own page
only (`text="detail"`: Workday, SuccessFactors, Oracle, Phenom and the rest
with a `detail`, and a search provider such as Jobbank). The pull reads
listings only; this stage reads those pages after scoring, so the calls go
where the ranking is. A saved search whose reader has `detail` is treated as
a board under its target `provider:name`: the same bar, budgets and state
keys, and its failing record is `search:failing:<target>`.
listings only; this stage reads those pages after scoring, for the roles the
model pass will read, so each has its text first. A saved search whose
reader has `detail` is treated as a board under its target `provider:name`:
the same budgets and state keys, and its failing record is
`search:failing:<target>`. No model is asked.
1. `enrich_waiting` hands over the open roles without text that an enabled
agent with a prompt ranks ungated among its best `enrich.top` and at
`enrich.min_score` or above, best first by the highest such score. A role
below that bar for every agent gets no call.
1. `Store.fetch_text_waiting` hands over the open roles without text in the
cut of an enabled agent (`scores.cut`, see `app.jobs.score`), best first
by the highest final score. A role in no cut gets no call.
2. They are taken in that order, those whose read failed last, on boards and
searches whose reader has `detail`, at most `enrich.per_board` from one
board and `enrich.per_run` in all, leaving out the roles already read,
unless their read failed or gave no text `enrich.retry_days` ago
(`plugins.ats.detail_due`).
searches whose reader has `detail`, at most `fetch_text.per_board` from
one board and `fetch_text.per_run` in all, leaving out the roles already
read, unless their read failed or gave no text `fetch_text.retry_days`
ago (`plugins.ats.detail_due`).
3. One batch per board or search, on the `http` pool, reads them
(`plugins.ats.read_board_details`, which keeps `ats:detailed:<target>`,
`ats:detailfailed:<target>` and `ats:detailempty:<target>`) and stores
what each page filled. `Store.upsert_role` drops the vector of a role
whose text improved, so the next `embed` encodes those roles only.
In the pull cascade `embed` and `score` follow; `cascade.step` passes over
both when no role was filled. The same stage follows a rescore after an agent
is saved or a company's affinity changes (`cascade.enrich_after`), never after
a vote. Whether it runs is `stages.stage_on(config, store, "enrich")`.
In the pull cascade `prep`, `embed` and `score` follow, so the filled roles
are embedded again and their levels and the cut recomputed; `cascade.step`
passes over the three when no role was filled. The same stage starts the
cascade that follows a rescore after an agent is saved or a company's
affinity or overrides change (`cascade.fetch_text_after`), never after a
vote. Whether it runs is `stages.stage_on(config, store, "fetch_text")`.
"""
from __future__ import annotations
@@ -47,8 +49,8 @@ from pipeline import NULL, Batch, RunContext, Tally, Task, job
log = logging.getLogger(__name__)
KIND = "enrich_batch"
SETTLE = "app.jobs.enrich:settle"
KIND = "fetch_text_batch"
SETTLE = "app.jobs.fetch_text:settle"
def _ats() -> Any:
@@ -93,13 +95,14 @@ def _readable(store: Store) -> dict[int, Subscription]:
def plan(config: Config, store: Store) -> list[tuple[str, list[str]]]:
"""`(target, uids)` per board or search, best first, within the budgets:
the roles `enrich_waiting` ranks, on fetchable boards and searches whose
reader has `detail` and that are not failing, and due for a read."""
settings = config.enrich
the roles `fetch_text_waiting` hands over, on fetchable boards and
searches whose reader has `detail` and that are not failing, and due for
a read."""
settings = config.fetch_text
ats = _ats()
subs = _readable(store)
by_board: dict[str, list[dict[str, Any]]] = defaultdict(list)
for found in store.enrich_waiting(settings.top, settings.min_score):
for found in store.fetch_text_waiting():
sub = subs.get(found["subscription_id"])
if sub is None or f"{sub.source}:failing:{sub.target}" in store.state:
continue
@@ -135,7 +138,7 @@ def plan(config: Config, store: Store) -> list[tuple[str, list[str]]]:
def batches(config: Config, store: Store) -> list[Batch]:
"""The finder: one batch per board or search, nothing while the stage is
off."""
if not stages.stage_on(config, store, "enrich"):
if not stages.stage_on(config, store, "fetch_text"):
return []
return [Batch({"target": target, "uids": uids},
label=f"Reading {len(uids)} roles on {target}")
@@ -153,12 +156,12 @@ def run_batch(config: Config, store: Store, *, target: str, uids: list[str],
if not subs or not uids:
return {"target": target, "filled": 0, "texts": 0}
sub = subs[0]
found = {f["uid"]: f for f in store.enrich_waiting(0, 0.0, uids)}
found = {f["uid"]: f for f in store.fetch_text_waiting(uids)}
rows = [_row(found[uid], sub) for uid in uids if uid in found]
context.check()
with httpx.Client() as client:
filled = ats.read_board_details(store.state, target, rows, client=client,
cap=len(rows), retry_days=config.enrich.retry_days)
cap=len(rows), retry_days=config.fetch_text.retry_days)
label = sub.company or sub.label
texts = 0
for row in filled:
@@ -166,7 +169,7 @@ def run_batch(config: Config, store: Store, *, target: str, uids: list[str],
item.source = sub.source
store.upsert_role(item, sub)
texts += bool(row.get("description"))
log.info("enrich: %s: %d read, %d filled, %d with text", target, len(rows), len(filled),
log.info("fetch_text: %s: %d read, %d filled, %d with text", target, len(rows), len(filled),
texts)
return {"target": target, "read": len(rows), "filled": len(filled), "texts": texts}
@@ -183,6 +186,9 @@ def settle(config: Config, store: Store, tally: Tally, *,
"texts": sum(int(result.get("texts") or 0) for result in tally.results)}
def enrich_task() -> Task:
return Task(kind=KIND, finder=batches, name="enrich", settle=SETTLE,
label="Reading the best roles on their own pages")
#: What the task and the cascade's job are called on the Runs page.
LABEL = "Fetching the text of the roles in the cut"
def fetch_text_task() -> Task:
return Task(kind=KIND, finder=batches, name="fetch_text", settle=SETTLE, label=LABEL)
+11 -11
View File
@@ -1,4 +1,4 @@
"""`judge`: each of an agent's best roles read against its CV, as three numbers.
"""`judge`: each role in an agent's cut read against its CV, as three numbers.
A `ModelTask` beside `summarise`, after the `/rank` of
github.com/MadsLorentzen/ai-job-search: the model reads a posting against a
@@ -10,14 +10,15 @@ CV match sorts by. The model also reports a verdict on each of the
agent's must-haves (one per line of its must text, numbered M1..Mn in the
prompt); it does not apply them. The rule is code's: a "broken" one makes
career count as at most `app.models.BROKEN_CAP`, when the role is served. Culture and location are left out: the first
is guessed from ad text, the second is what the gates do.
is guessed from ad text, the second is what the fetch gate does.
The work is pairs, `(agent, role)`: an enabled agent with a CV, and a role
among its best ungated scores as deep and as low as its own `limits` say
(`judge_top`, `judge_min`), or among the `judge_cv_top` ungated roles closest
to its CV (`scores.cv_fit`) that those leave out, with some text, that has
no judgement yet, or one read from another CV, prompts, like, avoid or must
text or role text than today's (`input_key`). A pair is named by the composite key `"<agent_id>:<uid>"`,
The work is pairs, `(agent, role)`: an enabled agent with a CV and a prompt,
and an open role in its cut (`scores.cut`, see `app.jobs.score`: the best by
fit, the best by CV closeness that those leave out, and the liked and saved)
with some text, that has no judgement yet, or one read from another CV,
prompts, like, avoid or must text or role text than today's (`input_key`).
The cut is the only selection: the pull judges it after extract and the
rescore that follows, so it reads the cut as recomputed. A pair is named by the composite key `"<agent_id>:<uid>"`,
which is what `desk.modeltask` hands round as a uid, so `desk/` is untouched.
A batch holds one agent's pairs, since the profile is in the system prompt;
the finder cuts per agent (`_finder`) and the job runs one call per agent it
@@ -186,8 +187,7 @@ def input_key(agent: Agent, text: str) -> str:
def _judged(agent: Agent) -> bool:
"""Whether `judge` reads for this agent at all: enabled, with a CV and a
prompt with text (without one its scores have no fit, so its best roles
are its newest)."""
prompt with text (without one nothing of its cut is in by fit)."""
return agent.enabled and bool(agent.cv_text.strip()) \
and any(p.text.strip() for p in agent.prompts)
@@ -195,7 +195,7 @@ def _judged(agent: Agent) -> bool:
def waiting(store: Store, keys: Sequence[str] | None = None,
limit: int | None = None) -> list[tuple[Agent, Any, str, str]]:
"""The pairs `judge` has yet to read, as `(agent, row, text, input_key)`,
each agent's best first (`Store.judge_rows`). Given `keys`, only those
each agent's cut in its order (`Store.judge_rows`). Given `keys`, only those
of them still waiting, whatever their rank."""
wanted: dict[int, list[str]] | None = None
if keys is not None:
+1 -1
View File
@@ -135,7 +135,7 @@ def _detail_target(role: Role) -> str:
def _detail_text(role: Role, target: str) -> str:
"""The text of a search role, read by its reader's `detail` on the ad's
own page as the pull's enrich stage reads it. Raises `Unreadable` when
own page as the pull's fetch text stage reads it. Raises `Unreadable` when
the read fails or the page states no text."""
from plugins import ats
+1 -1
View File
@@ -3,7 +3,7 @@
In the pull cascade it follows the pull, whose settle has closed what is
gone, and maps every role the pull changed, then folds the copies across
the whole database (`Store.rebuild_dupes`, which the pull's settle used to
run). After `enrich` it maps the roles their own pages filled, since only
run). After `fetch_text` it maps the roles their own pages filled, since only
those are due. `solo`, so two cascades never write the same rows at once.
"""
+132 -67
View File
@@ -1,21 +1,29 @@
"""`score`: every open role, scored for every enabled search agent.
"""`score`: every open role, scored for every enabled search agent, and each
agent's cut.
A role is scored per agent, `(uid, agent) -> score`, because a role is only
good for a given search. For each agent:
good for a given search. Each agent first applies its fetch gate:
require gates a role that fails one keeps its row, with score 0 and the
reason ("remote not remote")
any gates one group: a role that matches none of them, and has the
column filled for at least one, is gated ("none of
country DK / remote remote")
prefer gates `weight` when the role matches, 0 when it does not
reason ("remote not remote"); the region, which nothing
lifts
any gates one group, the place part: a role that matches none of
them, and has the column filled for at least one, is
gated ("none of country DK / remote remote"); skipped for
a company set to All countries
exclude gates a role that matches one of the values keeps its row, with
score 0 and the value ("title account executive")
score 0 and the value ("title account executive"); those
on `title` are skipped for a company set to All roles
A gate rejects on evidence, never on absence: a role whose column is empty
is not gated by it. `extract` is what fills an empty column, and it reads
only the roles that rank, so gating on an empty column would keep those
roles out of extract's reach for good.
only the roles in the cut, so gating on an empty column would keep those
roles out of extract's reach for good. A hidden company (`affinity = -1`)
is not scored at all; a favourite passes the gate like any other company.
The final score, the feed's order, is the weighted sum of four terms over
the sum of the weights, so it stays 0..1:
fit cos(role vector, prompt vector) as a level: the embedding
model's calibrated `lo` is 0 and `hi` 1, clipped outside
(see `scale`), so a role's fit does not depend on what
@@ -29,25 +37,29 @@ roles out of extract's reach for good.
over this agent's votes; with only likes it is the liked mean,
with only passes the negated passed mean; no votes, no term
Beside the score, for an agent with a CV, `cv_fit`: how close the role is to
the CV, the best cosine between the role and the CV's chunks
(`embed.cv_chunks`, embedded as the prompts are), scaled like fit. It is no
term of the score; the judge reads the closest roles the score leaves out
(`Store.judge_rows`) and CV match orders the roles not judged yet by it.
A role with no vector is scored without `fit` and `taste`, unless `fit` is
all there is, when it is gated as "no vector". An agent with no votes scores
every role without `taste`; the direction is newsdesk's
`centroid(liked) - centroid(disliked)`. An agent with no prompt that has
text has no `fit` either, so nothing of it enters the cut by fit and its
scores do not decide what summarise reads (`Store._summarise_top`); a run
names such agents in one notice.
Every term is 0..1. The score is the weighted sum over the sum of the weights
(the agent's four and its prefer gates'), so it stays 0..1. A favourite
company (`affinity = 2`) skips the `any` group but not the `require` or
`exclude` gates, and
is scored like any other company with its affinity term at 1: nothing puts
its roles above a better fit. A hidden one (`-1`) is not scored at all. A
role with no vector is scored without `fit` and `taste`, unless `fit` is all
there is, when it is gated as "no vector".
An agent with no votes scores every role without `taste`; the direction is
newsdesk's `centroid(liked) - centroid(disliked)`. An agent with no prompt
that has text has no `fit` either: its scores are not a match, so it is left
out of what extract and summarise read (`Store._top_scope`), and a run names
such agents in one notice.
Beside the score, `fit` is stored as it is, and for an agent with a CV,
`cv_fit`: how close the role is to the CV, the best cosine between the role
and the CV's chunks (`embed.cv_chunks`, embedded as the prompts are), scaled
like fit. Neither carries the affinity term.
The cut is the roles that get fetch text, extract and judge (`cut_of`). It
is computed from two levels, each with the company's affinity as a bonus of
`w.affinity * affinity / 3` over the sum of the weights, the same size as in
the final score (a liked company about 0.05 above neutral, a favourite about
0.1): the fit level `fit + bonus` and the CV level `cv_fit + bonus`. Over
the roles that pass the fetch gate, the best `cut_fit_top` by fit level at
or above `cut_fit_min` are `fit`; the best `cut_cv_top` by CV level at or
above `cut_cv_min` that are not in yet are `cv`; every role this agent
liked or anybody saved that is not in yet is `kept`. Newness and taste are
no part of it, and the two lists are not merged into one number.
Recomputed in full on every run, and each agent's rows are replaced in one
transaction: nothing is incremental, so nothing can drift.
@@ -67,7 +79,7 @@ import numpy as np
from pipeline import NULL, Notice, RunContext, job
from app.config import Calibration, Config
from app.jobs import embed
from app.models import Agent, Candidate, Gate, Weights
from app.models import Agent, Candidate, Gate, Limits, Weights
from app.storage import Store
log = logging.getLogger(__name__)
@@ -130,10 +142,10 @@ def scale(cosines: dict[str, float],
embedding model's calibrated `band` (`app.config.CALIBRATION`).
Raw cosines on multilingual-e5-small sit between about 0.75 and 0.87 for
real postings against a real intent, so unscaled a prefer gate weighted
0.2 moved a score more than the whole spread of fit. The band stretches
that spread to 0..1 with fixed constants, so a role's level is the same
in every run and a `limits` bar is an absolute threshold.
real postings against a real intent, so unscaled the whole spread of fit
was smaller than a recency step. The band stretches that spread to 0..1
with fixed constants, so a role's level is the same in every run and a
minimum of the cut is an absolute threshold.
Without a band, for a model nobody calibrated, it falls back to min-max
across the run: the closest role is 1 and the farthest 0, which ranks
@@ -165,19 +177,20 @@ def _reason(gate: Gate) -> str:
def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
"""Why this role is gated, as reasons: every require gate it fails, every
exclude gate it matches, and the `any` group when it matches none of them.
A gate with an empty column (state -1) neither passes nor fails, so the
group fails only when some member had evidence. A favourite company's
roles skip the `any` group but not the require and exclude gates, so a
star widens a search to a company's roles within e.g. Europe, not to the
whole world, and not to the titles the agent leaves out."""
"""Why the fetch gate rejects this role, as reasons: every require gate
it fails, every exclude gate it matches, and the `any` group when it
matches none of them. A gate with an empty column (state -1) neither
passes nor fails, so the group fails only when some member had evidence.
A company set to All countries skips the `any` group, the place part,
and one set to All roles the exclude gates on `title`, the titles to
leave out; the require gates, the region, apply to every company."""
states = list(zip(gates, candidate.gates, strict=True))
reasons = [_reason(g) for g, state in states if g.mode == "require" and state == 0]
reasons += [f"{g.field} {g.values[state - 1]}" for g, state in states
if g.mode == "exclude" and state > 0]
if g.mode == "exclude" and state > 0
and not (candidate.all_roles and g.field == "title")]
group = [(g, state) for g, state in states if g.mode == "any"]
if candidate.affinity != 2 and group and not any(state == 1 for _, state in group) \
if not candidate.all_countries and group and not any(state == 1 for _, state in group) \
and any(state == 0 for _, state in group):
reasons.append("none of " + ", ".join(_wanted(g) for g, _ in group))
return reasons
@@ -186,17 +199,15 @@ def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
def score_one(candidate: Candidate, gates: Sequence[Gate], weights: Weights,
fit: float | None, now: datetime, *, taste: float | None = None,
missing: str = "no vector") -> tuple[float, str]:
"""One role for one agent: `(score, gated_reason)`. `fit` and `taste` are
already scaled to 0..1 (see `scale`), or None when there is nothing to
measure: no role vector, no prompt vector, no votes. A None term leaves
the denominator; `missing` says why fit is out."""
"""One role for one agent: `(final score, gated_reason)`. `fit` and
`taste` are already scaled to 0..1 (see `scale`), or None when there is
nothing to measure: no role vector, no prompt vector, no votes. A None
term leaves the denominator; `missing` says why fit is out."""
if failed := failed_gates(candidate, gates):
return 0.0, ", ".join(failed)
terms = [(weights.recency, recency(candidate.posted_at, now)),
(weights.affinity, affinity_term(candidate.affinity))]
terms += [(g.weight, 1.0 if state == 1 else 0.0)
for g, state in zip(gates, candidate.gates, strict=True) if g.mode == "prefer"]
if fit is not None:
terms.append((weights.fit, fit))
if taste is not None:
@@ -208,6 +219,44 @@ def score_one(candidate: Candidate, gates: Sequence[Gate], weights: Weights,
return sum(w * t for w, t in terms) / total, ""
def bonus(affinity: int, weights: Weights) -> float:
"""What a company's affinity adds to both levels of the cut: the share
the affinity term gives it in the final score above a company at -1, so
a liked company sits about 0.05 above a neutral one and a favourite
about 0.1 on the default weights. 0 when every weight is 0."""
total = weights.fit + weights.recency + weights.affinity + weights.taste
return weights.affinity * affinity / 3 / total if total else 0.0
def cut_of(candidates: Sequence[Candidate], passed: set[str], fits: dict[str, float],
near: dict[str, float], kept: set[str], weights: Weights,
limits: Limits) -> dict[str, str]:
"""`uid -> why` for the roles in the cut, among `candidates` whose uid
is in `passed` (the fetch gate let them through): `fit` for the best
`limits.cut_fit_top` by fit level at or above `cut_fit_min`, `cv` for
the best `cut_cv_top` by CV level at or above `cut_cv_min` not in yet,
and `kept` for those in `kept` (liked or saved) not in yet. A level is
`fits` or `near` plus the company's `bonus`; a role without one is in
no list by it. Ties go to the newer role, then the lower uid."""
by_uid = {c.uid: c for c in candidates if c.uid in passed}
# Newest first, then uid, as the sort below keeps equal levels in order.
ordered = sorted(sorted(by_uid.values(), key=lambda c: c.uid),
key=lambda c: c.posted_at, reverse=True)
cut: dict[str, str] = {}
for why, levels, top, floor in (("fit", fits, limits.cut_fit_top, limits.cut_fit_min),
("cv", near, limits.cut_cv_top, limits.cut_cv_min)):
scored = [(levels[c.uid] + bonus(c.affinity, weights), c.uid) for c in ordered
if c.uid in levels and c.uid not in cut]
scored = [(level, uid) for level, uid in scored if level >= floor]
scored.sort(key=lambda pair: pair[0], reverse=True)
for _, uid in scored[:top]:
cut[uid] = why
for c in ordered:
if c.uid in kept and c.uid not in cut:
cut[c.uid] = "kept"
return cut
# ------------------------------------------------------------------ vectors
def intent_key(model: str, intent: str) -> str:
@@ -301,12 +350,13 @@ def _cv(store: Store, agent: Agent, model: str, encoder: _Encoder) -> np.ndarray
def no_intent(names: Sequence[str]) -> Notice:
"""The notice for enabled agents without a prompt that has text: their
scores have no fit, so `Store._top_scope` keeps them from feeding extract
and summarise. The code keeps its name from when an agent had one intent."""
scores have no fit, so nothing of theirs enters the cut by fit and
their final scores do not choose what summarise reads. The code keeps
its name from when an agent had one intent."""
return Notice(
code="score.no_intent", level="warning", kind="expected", key="score",
text=f"No prompt with text, so ranked without fit and not read by extract "
f"or summarise: {', '.join(names)}.",
text=f"No prompt with text, so ranked without fit and nothing in the cut by fit: "
f"{', '.join(names)}.",
fix="Give the agent a prompt under Settings, Search agents.",
where="app.jobs.score")
@@ -314,7 +364,8 @@ def no_intent(names: Sequence[str]) -> Notice:
def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
prompts: Sequence[tuple[str, np.ndarray]], now: datetime,
calibration: Calibration | None = None,
cv: np.ndarray | None = None) -> dict[str, int]:
cv: np.ndarray | None = None,
saved: set[str] | None = None) -> dict[str, int]:
"""Two passes: the cosines of the roles this agent scores, to each of its
`prompts` and to its taste direction, then every score with both scaled
by the model's `calibration`, or across the run without one. A role's fit
@@ -327,9 +378,12 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
share. The voted roles are scored like any other; a liked one sits near
the direction it helped make, which is what a like should do. With `cv`
(`_cv`), each of those roles also gets its `cv_fit`, its `closeness` to
the CV scaled like fit, beside the score and not in it."""
the CV scaled like fit. Then the cut (`cut_of`), with this agent's likes
and the `saved` roles (`Store.saved_uids`, read here when not given)
kept."""
candidates = store.score_candidates(agent.id, agent.gates)
direction = taste_direction(vectors, store.votes(agent.id))
votes = store.votes(agent.id)
direction = taste_direction(vectors, votes)
cosines: dict[str, dict[str, float]] = {name: {} for name, _ in prompts}
tastes: dict[str, float] = {}
near: dict[str, float] = {}
@@ -351,22 +405,29 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
fits[uid], best[uid] = value, name
tastes = scale(tastes, calibration.taste if calibration else None)
near = scale(near, calibration.cv_band if calibration else None)
rows: list[tuple[str, float, str, str, float | None]] = []
gated = 0
scored: list[tuple[str, float, str, float | None]] = []
passed: set[str] = set()
for candidate in candidates:
fit = fits.get(candidate.uid)
value, reason = score_one(
candidate, agent.gates, agent.weights, fit, now,
taste=tastes.get(candidate.uid),
missing="no vector" if prompts else "no prompt vector")
gated += bool(reason)
if not reason:
passed.add(candidate.uid)
scored.append((candidate.uid, value, reason, fit))
kept = {uid for uid, vote in votes.items() if vote == 1}
kept |= store.saved_uids() if saved is None else saved
cut = cut_of(candidates, passed, fits, near, kept, agent.weights, agent.limits)
rows = []
for uid, value, reason, fit in scored:
counted = fit is not None and agent.weights.fit and not reason
cv_fit = near.get(candidate.uid)
rows.append((candidate.uid, round(value, 6), reason,
best[candidate.uid] if counted else "",
None if cv_fit is None else round(cv_fit, 6)))
cv_fit = near.get(uid)
rows.append((uid, round(value, 6), reason, best[uid] if counted else "",
None if cv_fit is None else round(cv_fit, 6),
None if fit is None else round(fit, 6), cut.get(uid, "")))
store.replace_scores(agent.id, rows)
return {"scored": len(rows), "gated": gated}
return {"scored": len(rows), "gated": len(rows) - len(passed), "cut": len(cut)}
@job("score", "Scoring roles for every agent", solo=True, resource="cpu")
@@ -386,17 +447,21 @@ def run_score(config: Config, store: Store, *,
if blind := [agent.name for agent in agents
if not any(p.text.strip() for p in agent.prompts)]:
context.notice(no_intent(blind))
scored = gated = 0
scored = gated = cut = 0
saved = store.saved_uids()
for n, agent in enumerate(agents):
context.check()
context.progress(n, len(agents), f"scoring for {agent.name}, {n + 1} of {len(agents)}")
counts = score_agent(store, agent, vectors, _prompts(store, agent, model, encoder),
now, calibration, cv=_cv(store, agent, model, encoder))
now, calibration, cv=_cv(store, agent, model, encoder),
saved=saved)
scored += counts["scored"]
gated += counts["gated"]
log.info("%-20s %6d scored, %6d gated", agent.name, counts["scored"], counts["gated"])
cut += counts["cut"]
log.info("%-20s %6d scored, %6d gated, %5d in the cut", agent.name, counts["scored"],
counts["gated"], counts["cut"])
context.progress(len(agents), len(agents), "")
return {"agents": len(agents), "scored": scored, "gated": gated,
return {"agents": len(agents), "scored": scored, "gated": gated, "cut": cut,
"vectors": len(vectors)}
+13 -12
View File
@@ -1,5 +1,5 @@
"""Whether the three model stages of a pull, `extract`, `summarise` and
`judge`, and the `enrich` stage run.
`judge`, and the `fetch_text` stage run.
`extract.enabled`, `summarise.enabled` and `judge.enabled` in config.yaml are
read at startup and say nothing on the page, so the switch lives in the
@@ -10,16 +10,17 @@ deployment that never uses the switch behaves as it did. `stage_on` is the
one place that decides, and the tasks' finders ask it
(`extract.extract_task`, `summarise.summarise_task`, `judge.judge_task`).
`enrich` (reading the best roles on their own pages, `app/jobs/enrich.py`)
asks no model and is on by default (`enrich.enabled`); its switch is
`stage:enrich`, set the same way as the others: by the Runs page
(`PUT /api/stages/enrich`) and by `app stages enrich on|off`.
`fetch_text` (reading the role's own page for the roles in the cut,
`app/jobs/fetch_text.py`) asks no model and is on by default
(`fetch_text.enabled`); its switch is `stage:fetch_text`, set the same way
as the others: by the Runs page (`PUT /api/stages/fetch_text`) and by
`app stages fetch_text on|off`.
`describe` is what `GET /api/stages` and `app stages` show: for each stage
with a switch, the model stages and `enrich`, whether it is on, where that comes from, the model it uses, and who
would answer. The model is the one a call is made with: `agent.model` when
set, since it overrides the per-task model everywhere (`desk/modeltask.py`),
else the stage's own.
with a switch, the model stages and `fetch_text`, whether it is on, where
that comes from, the model it uses, and who would answer. The model is the
one a call is made with: `agent.model` when set, since it overrides the
per-task model everywhere (`desk/modeltask.py`), else the stage's own.
"""
from __future__ import annotations
@@ -34,8 +35,8 @@ from desk.tools import agent
NAMES = ("extract", "summarise", "judge")
#: Every stage with a switch: the model stages and `enrich`.
SWITCHES = (*NAMES, "enrich")
#: Every stage with a switch: the model stages and `fetch_text`.
SWITCHES = (*NAMES, "fetch_text")
#: What `describe_one` says of a stage that asks no model.
NO_ASK = "reads role pages, asks no model"
@@ -53,7 +54,7 @@ class Stage(BaseModel):
`backend` is `agent.backend`'s answer, "" for nobody, and `answer` says
it in words."""
name: Literal["extract", "summarise", "judge", "enrich"]
name: Literal["extract", "summarise", "judge", "fetch_text"]
on: bool
source: Literal["stored", "config"]
model: str
+7 -6
View File
@@ -6,12 +6,13 @@ what the model is told, the schema it answers in, what to do with the
answer), and running it is `desk.modeltask`'s.
A role is waiting while it has no summary, has some text, is open and no copy,
and is either in the top of an enabled agent's ungated scores as deep and as
low as that agent's own `limits` say (`summarise_top`, `summarise_min`), or
one the owner cares about whatever its rank: saved (it has an application, the
star) or voted up by an enabled agent. The limits keep the cost flat: a pull
summarises the roles somebody will look at, not the 30k that are open, and the
saved and liked roles are a handful the owner picked. Each is summarised
and is either in the top of an enabled agent's feed, the best `summarise_top`
final scores among the roles its fetch gate passes (its `limits`), or one the
owner cares about whatever its rank: saved (it has an application, the star)
or voted up by an enabled agent. Summarise is no part of any score and runs
last in the pull, once the final scores are set. The limit keeps the cost
flat: a pull summarises the roles somebody will look at, not the 30k that are
open, and the saved and liked roles are a handful the owner picked. Each is summarised
once, with the same give-up rule (`GIVE_UP_AFTER` replies that left it
without). Saving or liking a role queues a batch for that one uid at once,
while the stage is on (`web.api`), rather than waiting for the next pull. A