Rank by the cut: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text renamed from enrich, the feed kept to the fetch gate, and migration 32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
060ccf66e6
commit
617356e60c
35 files changed
+1054
-751
No files matched your search
+51
-47
@@ -1,32 +1,29 @@
|
||||
"""The tasks roledesk offers, and the order a pull runs them in.
|
||||
|
||||
pull → prep → embed → score → enrich → prep → embed → score → extract → score
|
||||
→ summarise → judge
|
||||
pull → prep → embed → score → fetch_text → prep → embed → score → extract
|
||||
→ score → judge → summarise
|
||||
|
||||
newsdesk's cascade convention: a cascade is a tuple of task names, and the
|
||||
final job of one task starts the next (`step` is the callback it carries).
|
||||
Nothing is alive between two tasks. `prep` (`app/jobs/prep.py`) maps what
|
||||
the pull stored and folds the copies before anything reads the roles; after
|
||||
`enrich` it maps the roles their own pages filled. A stage is addressed by its position in
|
||||
the tuple, not its name, since `score` runs twice.
|
||||
`fetch_text` it maps the roles their own pages filled. A stage is addressed
|
||||
by its position in the tuple, not its name, since `score` runs three times.
|
||||
|
||||
Every stage is always in the chain, and a stage that is switched off finds
|
||||
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this pull does not
|
||||
satisfy, is an extract stage with no batches whose final job moves straight
|
||||
on. The free stages go first: embed and score cost no tokens, and the score
|
||||
they produce is what bounds the rest. Enrich reads the best roles without
|
||||
text on their own pages (`app/jobs/enrich.py`), and prep, embed and score
|
||||
follow for the roles it filled; when it filled none, `step` passes over all
|
||||
three. Extract
|
||||
reads only the best
|
||||
roles of each enabled agent, as deep and as low as that agent's own `limits`
|
||||
say; the second score then applies the gates to the columns it filled.
|
||||
Summarise follows, on the best roles by the same agent's summarise limits,
|
||||
and judge runs last, reading each agent's best roles against its CV by its
|
||||
judge limits. The `enrich` cascade, which follows the rescore after an agent
|
||||
is saved, ends with judge too, so a changed CV, prompt or like text is read
|
||||
again at once rather than at the next pull; a pair whose `input_key` did not
|
||||
change costs nothing.
|
||||
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this
|
||||
pull does not satisfy, is a stage with no batches whose final job moves
|
||||
straight on. The free stages go first: embed and score cost no tokens, and
|
||||
score writes each agent's cut (`app.jobs.score`), which bounds the rest.
|
||||
Fetch text reads the role's own page for the roles in the cut that came
|
||||
without text (`app/jobs/fetch_text.py`), and prep, embed and score follow
|
||||
for the roles it filled, so their levels and the cut are recomputed; when
|
||||
it filled none, `step` passes over all three. Extract reads the cut; the
|
||||
score after it applies the fetch gate and the cut again with the columns
|
||||
extract filled, so a role can drop out or enter (one that enters is
|
||||
extracted on the next pull). Judge reads each agent's cut as recomputed,
|
||||
against its CV. Summarise runs last, on the top of each agent's feed by
|
||||
final score and the roles liked or saved: it is no part of any score.
|
||||
|
||||
A batch that failed, or a stage somebody killed, does not stop the chain: the
|
||||
work comes round on the next pull, and a rescore of what did arrive is worth
|
||||
@@ -36,10 +33,13 @@ having either way.
|
||||
and a cascade is started by one: the `pull` job's only work is to ask for the
|
||||
chain. Its default schedule, `every 6h`, is declared here beside it.
|
||||
|
||||
`enrich` is a job and a cascade the same way, `enrich → prep → embed → score`, queued
|
||||
behind the `score` that follows a change to the ranking other than a pull's
|
||||
(`enrich_after`): an agent saved, a company's affinity changed. A vote moves
|
||||
one agent's taste a little and queues neither.
|
||||
`fetch_text` is a job and a cascade the same way, the pull's chain from
|
||||
fetch text on, queued behind the `score` that follows a change to the
|
||||
ranking other than a pull's (`fetch_text_after`): an agent saved, a
|
||||
company's affinity or overrides changed. The cut may have changed, so its
|
||||
new roles get their text, extract and judge at once rather than at the next
|
||||
pull; a judgement whose `input_key` did not change costs nothing. A vote
|
||||
moves one agent's taste a little and queues neither.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -49,19 +49,22 @@ from dataclasses import replace
|
||||
from functools import partial
|
||||
from typing import Any
|
||||
|
||||
from app.jobs import embed, enrich, extract, judge, prep, pull, score, summarise # noqa: F401 -- registers the jobs
|
||||
from app.jobs import ( # noqa: F401 -- registers the jobs
|
||||
embed, extract, fetch_text, judge, prep, pull, score, summarise,
|
||||
)
|
||||
from desk.tools import agent
|
||||
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
|
||||
|
||||
STEP = "app.jobs.cascade:step"
|
||||
|
||||
CASCADES: dict[str, tuple[str, ...]] = {
|
||||
"pull": ("pull", "prep", "embed", "score", "enrich", "prep", "embed", "score", "extract",
|
||||
"score", "summarise", "judge"),
|
||||
"enrich": ("enrich", "prep", "embed", "score", "judge"),
|
||||
"pull": ("pull", "prep", "embed", "score", "fetch_text", "prep", "embed", "score",
|
||||
"extract", "score", "judge", "summarise"),
|
||||
"fetch_text": ("fetch_text", "prep", "embed", "score", "extract", "score", "judge",
|
||||
"summarise"),
|
||||
}
|
||||
|
||||
#: What follows `enrich` for the roles it filled, passed over when it filled none.
|
||||
#: What follows `fetch_text` for the roles it filled, passed over when it filled none.
|
||||
REFILL = ("prep", "embed", "score")
|
||||
|
||||
|
||||
@@ -131,7 +134,7 @@ STAGES: dict[str, Callable[[str], Task]] = {
|
||||
label="Mapping the roles' host words"),
|
||||
"summarise": _summarise,
|
||||
"judge": _judge,
|
||||
"enrich": lambda cascade: enrich.enrich_task(),
|
||||
"fetch_text": lambda cascade: fetch_text.fetch_text_task(),
|
||||
}
|
||||
|
||||
|
||||
@@ -157,7 +160,7 @@ def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
|
||||
name, done = payload.get("cascade", ""), payload.get("task", "")
|
||||
at = int(payload.get("at", -1))
|
||||
tasks = CASCADES.get(name, ())
|
||||
if done == "enrich" and not enrich.filled(tally) \
|
||||
if done == "fetch_text" and not fetch_text.filled(tally) \
|
||||
and tasks[at + 1:at + 1 + len(REFILL)] == REFILL:
|
||||
at += len(REFILL)
|
||||
following = follows(name, at)
|
||||
@@ -222,24 +225,25 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
|
||||
return {"started": " > ".join(CASCADES["pull"]), "run": run}
|
||||
|
||||
|
||||
@job("enrich", "Reading the best roles on their own pages", solo=True, resource="http")
|
||||
def run_enrich(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Ask for the enrich cascade, `enrich → prep → embed → score`. Its own work is
|
||||
nil, as the `pull` job's. Under `NULL` there is no queue to take it."""
|
||||
if not context.task(chain("enrich")):
|
||||
raise RuntimeError("no queue to start the enrich cascade on")
|
||||
return {"started": " > ".join(CASCADES["enrich"])}
|
||||
@job("fetch_text", fetch_text.LABEL, solo=True, resource="http")
|
||||
def run_fetch_text(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Ask for the fetch text cascade, the pull's chain from fetch text on.
|
||||
Its own work is nil, as the `pull` job's. Under `NULL` there is no queue
|
||||
to take it."""
|
||||
if not context.task(chain("fetch_text")):
|
||||
raise RuntimeError("no queue to start the fetch text cascade on")
|
||||
return {"started": " > ".join(CASCADES["fetch_text"])}
|
||||
|
||||
|
||||
def enrich_after(queue: Any, score_job: Any) -> Any:
|
||||
"""Queue the `enrich` job behind `score_job`, the rescore that follows a
|
||||
change to the ranking, however that score ends. One per score job
|
||||
(`enrich:after:<id>`), so ten changes answered by one score are one
|
||||
enrich. None without a score job, or when the queue refuses."""
|
||||
def fetch_text_after(queue: Any, score_job: Any) -> Any:
|
||||
"""Queue the `fetch_text` job behind `score_job`, the rescore that
|
||||
follows a change to the ranking, however that score ends. One per score
|
||||
job (`fetch_text:after:<id>`), so ten changes answered by one score are
|
||||
one cascade. None without a score job, or when the queue refuses."""
|
||||
if score_job is None:
|
||||
return None
|
||||
return queue.enqueue("enrich", depends_on=[score_job.id], after="done",
|
||||
dedup_key=f"enrich:after:{score_job.id}")
|
||||
return queue.enqueue("fetch_text", depends_on=[score_job.id], after="done",
|
||||
dedup_key=f"fetch_text:after:{score_job.id}")
|
||||
|
||||
|
||||
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
|
||||
@@ -255,5 +259,5 @@ for _name in ("extract", "embed", "score", "summarise", "judge", "prep"):
|
||||
# setting: a cadence edited or switched off under Runs survives
|
||||
# every restart (see pipeline/schedule.py).
|
||||
declare("pull", "pull", "every 6h",
|
||||
label="Pull every board, then map, embed, score, read the best roles' pages, "
|
||||
"extract, rescore, summarise and judge against the CVs")
|
||||
label="Pull every board, then map, embed, score, fetch the text of the roles in the "
|
||||
"cut, extract, rescore, judge against the CVs and summarise")
|
||||
+6
-7
@@ -12,13 +12,12 @@ is waiting while it is open, has not been given up on, has never been
|
||||
answered (`extracted_at IS NULL`), has no model level (`seniority_model`
|
||||
empty), was never asked where a remote hire may live (`remote_where_model`
|
||||
NULL) or for its languages and deadline (`languages_required` NULL), and is
|
||||
in the best of an enabled agent's
|
||||
ungated scores as deep and as low as that agent's own `limits` say
|
||||
(`extract_top`, `extract_min`). That bound is what keeps the cost
|
||||
proportional to the rankings rather than to the boards:
|
||||
the pull scores every role for free first, then reads the few hundred that
|
||||
rank, and scores again so the gates see what was filled. A batch built for
|
||||
named uids finishes what it was dealt whatever their rank. The answer only
|
||||
in the cut of an enabled agent (`scores.cut`, see `app.jobs.score`). That
|
||||
bound is what keeps the cost proportional to the agents rather than to the
|
||||
boards: the pull scores every role for free first, then reads the cut, and
|
||||
scores again so the fetch gate and the cut see what was filled; a role that
|
||||
enters the cut then is read on the next pull. A batch built for named uids
|
||||
finishes what it was dealt whatever their rank. The answer only
|
||||
fills columns that are still empty: a board's structured field wins over what
|
||||
a model read out of the prose. The exception is the level, which the model
|
||||
always gives (the board's word and the title's go in as hints) and which
|
||||
|
||||
@@ -1,32 +1,34 @@
|
||||
"""`enrich`: the best roles without text, read on their own pages.
|
||||
"""`fetch_text`: the roles in the cut without text, read on their own pages.
|
||||
|
||||
Some hosts list a role without its text and state it on the role's own page
|
||||
only (`text="detail"`: Workday, SuccessFactors, Oracle, Phenom and the rest
|
||||
with a `detail`, and a search provider such as Jobbank). The pull reads
|
||||
listings only; this stage reads those pages after scoring, so the calls go
|
||||
where the ranking is. A saved search whose reader has `detail` is treated as
|
||||
a board under its target `provider:name`: the same bar, budgets and state
|
||||
keys, and its failing record is `search:failing:<target>`.
|
||||
listings only; this stage reads those pages after scoring, for the roles the
|
||||
model pass will read, so each has its text first. A saved search whose
|
||||
reader has `detail` is treated as a board under its target `provider:name`:
|
||||
the same budgets and state keys, and its failing record is
|
||||
`search:failing:<target>`. No model is asked.
|
||||
|
||||
1. `enrich_waiting` hands over the open roles without text that an enabled
|
||||
agent with a prompt ranks ungated among its best `enrich.top` and at
|
||||
`enrich.min_score` or above, best first by the highest such score. A role
|
||||
below that bar for every agent gets no call.
|
||||
1. `Store.fetch_text_waiting` hands over the open roles without text in the
|
||||
cut of an enabled agent (`scores.cut`, see `app.jobs.score`), best first
|
||||
by the highest final score. A role in no cut gets no call.
|
||||
2. They are taken in that order, those whose read failed last, on boards and
|
||||
searches whose reader has `detail`, at most `enrich.per_board` from one
|
||||
board and `enrich.per_run` in all, leaving out the roles already read,
|
||||
unless their read failed or gave no text `enrich.retry_days` ago
|
||||
(`plugins.ats.detail_due`).
|
||||
searches whose reader has `detail`, at most `fetch_text.per_board` from
|
||||
one board and `fetch_text.per_run` in all, leaving out the roles already
|
||||
read, unless their read failed or gave no text `fetch_text.retry_days`
|
||||
ago (`plugins.ats.detail_due`).
|
||||
3. One batch per board or search, on the `http` pool, reads them
|
||||
(`plugins.ats.read_board_details`, which keeps `ats:detailed:<target>`,
|
||||
`ats:detailfailed:<target>` and `ats:detailempty:<target>`) and stores
|
||||
what each page filled. `Store.upsert_role` drops the vector of a role
|
||||
whose text improved, so the next `embed` encodes those roles only.
|
||||
|
||||
In the pull cascade `embed` and `score` follow; `cascade.step` passes over
|
||||
both when no role was filled. The same stage follows a rescore after an agent
|
||||
is saved or a company's affinity changes (`cascade.enrich_after`), never after
|
||||
a vote. Whether it runs is `stages.stage_on(config, store, "enrich")`.
|
||||
In the pull cascade `prep`, `embed` and `score` follow, so the filled roles
|
||||
are embedded again and their levels and the cut recomputed; `cascade.step`
|
||||
passes over the three when no role was filled. The same stage starts the
|
||||
cascade that follows a rescore after an agent is saved or a company's
|
||||
affinity or overrides change (`cascade.fetch_text_after`), never after a
|
||||
vote. Whether it runs is `stages.stage_on(config, store, "fetch_text")`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -47,8 +49,8 @@ from pipeline import NULL, Batch, RunContext, Tally, Task, job
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
KIND = "enrich_batch"
|
||||
SETTLE = "app.jobs.enrich:settle"
|
||||
KIND = "fetch_text_batch"
|
||||
SETTLE = "app.jobs.fetch_text:settle"
|
||||
|
||||
|
||||
def _ats() -> Any:
|
||||
@@ -93,13 +95,14 @@ def _readable(store: Store) -> dict[int, Subscription]:
|
||||
|
||||
def plan(config: Config, store: Store) -> list[tuple[str, list[str]]]:
|
||||
"""`(target, uids)` per board or search, best first, within the budgets:
|
||||
the roles `enrich_waiting` ranks, on fetchable boards and searches whose
|
||||
reader has `detail` and that are not failing, and due for a read."""
|
||||
settings = config.enrich
|
||||
the roles `fetch_text_waiting` hands over, on fetchable boards and
|
||||
searches whose reader has `detail` and that are not failing, and due for
|
||||
a read."""
|
||||
settings = config.fetch_text
|
||||
ats = _ats()
|
||||
subs = _readable(store)
|
||||
by_board: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for found in store.enrich_waiting(settings.top, settings.min_score):
|
||||
for found in store.fetch_text_waiting():
|
||||
sub = subs.get(found["subscription_id"])
|
||||
if sub is None or f"{sub.source}:failing:{sub.target}" in store.state:
|
||||
continue
|
||||
@@ -135,7 +138,7 @@ def plan(config: Config, store: Store) -> list[tuple[str, list[str]]]:
|
||||
def batches(config: Config, store: Store) -> list[Batch]:
|
||||
"""The finder: one batch per board or search, nothing while the stage is
|
||||
off."""
|
||||
if not stages.stage_on(config, store, "enrich"):
|
||||
if not stages.stage_on(config, store, "fetch_text"):
|
||||
return []
|
||||
return [Batch({"target": target, "uids": uids},
|
||||
label=f"Reading {len(uids)} roles on {target}")
|
||||
@@ -153,12 +156,12 @@ def run_batch(config: Config, store: Store, *, target: str, uids: list[str],
|
||||
if not subs or not uids:
|
||||
return {"target": target, "filled": 0, "texts": 0}
|
||||
sub = subs[0]
|
||||
found = {f["uid"]: f for f in store.enrich_waiting(0, 0.0, uids)}
|
||||
found = {f["uid"]: f for f in store.fetch_text_waiting(uids)}
|
||||
rows = [_row(found[uid], sub) for uid in uids if uid in found]
|
||||
context.check()
|
||||
with httpx.Client() as client:
|
||||
filled = ats.read_board_details(store.state, target, rows, client=client,
|
||||
cap=len(rows), retry_days=config.enrich.retry_days)
|
||||
cap=len(rows), retry_days=config.fetch_text.retry_days)
|
||||
label = sub.company or sub.label
|
||||
texts = 0
|
||||
for row in filled:
|
||||
@@ -166,7 +169,7 @@ def run_batch(config: Config, store: Store, *, target: str, uids: list[str],
|
||||
item.source = sub.source
|
||||
store.upsert_role(item, sub)
|
||||
texts += bool(row.get("description"))
|
||||
log.info("enrich: %s: %d read, %d filled, %d with text", target, len(rows), len(filled),
|
||||
log.info("fetch_text: %s: %d read, %d filled, %d with text", target, len(rows), len(filled),
|
||||
texts)
|
||||
return {"target": target, "read": len(rows), "filled": len(filled), "texts": texts}
|
||||
|
||||
@@ -183,6 +186,9 @@ def settle(config: Config, store: Store, tally: Tally, *,
|
||||
"texts": sum(int(result.get("texts") or 0) for result in tally.results)}
|
||||
|
||||
|
||||
def enrich_task() -> Task:
|
||||
return Task(kind=KIND, finder=batches, name="enrich", settle=SETTLE,
|
||||
label="Reading the best roles on their own pages")
|
||||
#: What the task and the cascade's job are called on the Runs page.
|
||||
LABEL = "Fetching the text of the roles in the cut"
|
||||
|
||||
|
||||
def fetch_text_task() -> Task:
|
||||
return Task(kind=KIND, finder=batches, name="fetch_text", settle=SETTLE, label=LABEL)
|
||||
+11
-11
@@ -1,4 +1,4 @@
|
||||
"""`judge`: each of an agent's best roles read against its CV, as three numbers.
|
||||
"""`judge`: each role in an agent's cut read against its CV, as three numbers.
|
||||
|
||||
A `ModelTask` beside `summarise`, after the `/rank` of
|
||||
github.com/MadsLorentzen/ai-job-search: the model reads a posting against a
|
||||
@@ -10,14 +10,15 @@ CV match sorts by. The model also reports a verdict on each of the
|
||||
agent's must-haves (one per line of its must text, numbered M1..Mn in the
|
||||
prompt); it does not apply them. The rule is code's: a "broken" one makes
|
||||
career count as at most `app.models.BROKEN_CAP`, when the role is served. Culture and location are left out: the first
|
||||
is guessed from ad text, the second is what the gates do.
|
||||
is guessed from ad text, the second is what the fetch gate does.
|
||||
|
||||
The work is pairs, `(agent, role)`: an enabled agent with a CV, and a role
|
||||
among its best ungated scores as deep and as low as its own `limits` say
|
||||
(`judge_top`, `judge_min`), or among the `judge_cv_top` ungated roles closest
|
||||
to its CV (`scores.cv_fit`) that those leave out, with some text, that has
|
||||
no judgement yet, or one read from another CV, prompts, like, avoid or must
|
||||
text or role text than today's (`input_key`). A pair is named by the composite key `"<agent_id>:<uid>"`,
|
||||
The work is pairs, `(agent, role)`: an enabled agent with a CV and a prompt,
|
||||
and an open role in its cut (`scores.cut`, see `app.jobs.score`: the best by
|
||||
fit, the best by CV closeness that those leave out, and the liked and saved)
|
||||
with some text, that has no judgement yet, or one read from another CV,
|
||||
prompts, like, avoid or must text or role text than today's (`input_key`).
|
||||
The cut is the only selection: the pull judges it after extract and the
|
||||
rescore that follows, so it reads the cut as recomputed. A pair is named by the composite key `"<agent_id>:<uid>"`,
|
||||
which is what `desk.modeltask` hands round as a uid, so `desk/` is untouched.
|
||||
A batch holds one agent's pairs, since the profile is in the system prompt;
|
||||
the finder cuts per agent (`_finder`) and the job runs one call per agent it
|
||||
@@ -186,8 +187,7 @@ def input_key(agent: Agent, text: str) -> str:
|
||||
|
||||
def _judged(agent: Agent) -> bool:
|
||||
"""Whether `judge` reads for this agent at all: enabled, with a CV and a
|
||||
prompt with text (without one its scores have no fit, so its best roles
|
||||
are its newest)."""
|
||||
prompt with text (without one nothing of its cut is in by fit)."""
|
||||
return agent.enabled and bool(agent.cv_text.strip()) \
|
||||
and any(p.text.strip() for p in agent.prompts)
|
||||
|
||||
@@ -195,7 +195,7 @@ def _judged(agent: Agent) -> bool:
|
||||
def waiting(store: Store, keys: Sequence[str] | None = None,
|
||||
limit: int | None = None) -> list[tuple[Agent, Any, str, str]]:
|
||||
"""The pairs `judge` has yet to read, as `(agent, row, text, input_key)`,
|
||||
each agent's best first (`Store.judge_rows`). Given `keys`, only those
|
||||
each agent's cut in its order (`Store.judge_rows`). Given `keys`, only those
|
||||
of them still waiting, whatever their rank."""
|
||||
wanted: dict[int, list[str]] | None = None
|
||||
if keys is not None:
|
||||
|
||||
+1
-1
@@ -135,7 +135,7 @@ def _detail_target(role: Role) -> str:
|
||||
|
||||
def _detail_text(role: Role, target: str) -> str:
|
||||
"""The text of a search role, read by its reader's `detail` on the ad's
|
||||
own page as the pull's enrich stage reads it. Raises `Unreadable` when
|
||||
own page as the pull's fetch text stage reads it. Raises `Unreadable` when
|
||||
the read fails or the page states no text."""
|
||||
from plugins import ats
|
||||
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
In the pull cascade it follows the pull, whose settle has closed what is
|
||||
gone, and maps every role the pull changed, then folds the copies across
|
||||
the whole database (`Store.rebuild_dupes`, which the pull's settle used to
|
||||
run). After `enrich` it maps the roles their own pages filled, since only
|
||||
run). After `fetch_text` it maps the roles their own pages filled, since only
|
||||
those are due. `solo`, so two cascades never write the same rows at once.
|
||||
"""
|
||||
|
||||
|
||||
+132
-67
@@ -1,21 +1,29 @@
|
||||
"""`score`: every open role, scored for every enabled search agent.
|
||||
"""`score`: every open role, scored for every enabled search agent, and each
|
||||
agent's cut.
|
||||
|
||||
A role is scored per agent, `(uid, agent) -> score`, because a role is only
|
||||
good for a given search. For each agent:
|
||||
good for a given search. Each agent first applies its fetch gate:
|
||||
|
||||
require gates a role that fails one keeps its row, with score 0 and the
|
||||
reason ("remote not remote")
|
||||
any gates one group: a role that matches none of them, and has the
|
||||
column filled for at least one, is gated ("none of
|
||||
country DK / remote remote")
|
||||
prefer gates `weight` when the role matches, 0 when it does not
|
||||
reason ("remote not remote"); the region, which nothing
|
||||
lifts
|
||||
any gates one group, the place part: a role that matches none of
|
||||
them, and has the column filled for at least one, is
|
||||
gated ("none of country DK / remote remote"); skipped for
|
||||
a company set to All countries
|
||||
exclude gates a role that matches one of the values keeps its row, with
|
||||
score 0 and the value ("title account executive")
|
||||
score 0 and the value ("title account executive"); those
|
||||
on `title` are skipped for a company set to All roles
|
||||
|
||||
A gate rejects on evidence, never on absence: a role whose column is empty
|
||||
is not gated by it. `extract` is what fills an empty column, and it reads
|
||||
only the roles that rank, so gating on an empty column would keep those
|
||||
roles out of extract's reach for good.
|
||||
only the roles in the cut, so gating on an empty column would keep those
|
||||
roles out of extract's reach for good. A hidden company (`affinity = -1`)
|
||||
is not scored at all; a favourite passes the gate like any other company.
|
||||
|
||||
The final score, the feed's order, is the weighted sum of four terms over
|
||||
the sum of the weights, so it stays 0..1:
|
||||
|
||||
fit cos(role vector, prompt vector) as a level: the embedding
|
||||
model's calibrated `lo` is 0 and `hi` 1, clipped outside
|
||||
(see `scale`), so a role's fit does not depend on what
|
||||
@@ -29,25 +37,29 @@ roles out of extract's reach for good.
|
||||
over this agent's votes; with only likes it is the liked mean,
|
||||
with only passes the negated passed mean; no votes, no term
|
||||
|
||||
Beside the score, for an agent with a CV, `cv_fit`: how close the role is to
|
||||
the CV, the best cosine between the role and the CV's chunks
|
||||
(`embed.cv_chunks`, embedded as the prompts are), scaled like fit. It is no
|
||||
term of the score; the judge reads the closest roles the score leaves out
|
||||
(`Store.judge_rows`) and CV match orders the roles not judged yet by it.
|
||||
A role with no vector is scored without `fit` and `taste`, unless `fit` is
|
||||
all there is, when it is gated as "no vector". An agent with no votes scores
|
||||
every role without `taste`; the direction is newsdesk's
|
||||
`centroid(liked) - centroid(disliked)`. An agent with no prompt that has
|
||||
text has no `fit` either, so nothing of it enters the cut by fit and its
|
||||
scores do not decide what summarise reads (`Store._summarise_top`); a run
|
||||
names such agents in one notice.
|
||||
|
||||
Every term is 0..1. The score is the weighted sum over the sum of the weights
|
||||
(the agent's four and its prefer gates'), so it stays 0..1. A favourite
|
||||
company (`affinity = 2`) skips the `any` group but not the `require` or
|
||||
`exclude` gates, and
|
||||
is scored like any other company with its affinity term at 1: nothing puts
|
||||
its roles above a better fit. A hidden one (`-1`) is not scored at all. A
|
||||
role with no vector is scored without `fit` and `taste`, unless `fit` is all
|
||||
there is, when it is gated as "no vector".
|
||||
An agent with no votes scores every role without `taste`; the direction is
|
||||
newsdesk's `centroid(liked) - centroid(disliked)`. An agent with no prompt
|
||||
that has text has no `fit` either: its scores are not a match, so it is left
|
||||
out of what extract and summarise read (`Store._top_scope`), and a run names
|
||||
such agents in one notice.
|
||||
Beside the score, `fit` is stored as it is, and for an agent with a CV,
|
||||
`cv_fit`: how close the role is to the CV, the best cosine between the role
|
||||
and the CV's chunks (`embed.cv_chunks`, embedded as the prompts are), scaled
|
||||
like fit. Neither carries the affinity term.
|
||||
|
||||
The cut is the roles that get fetch text, extract and judge (`cut_of`). It
|
||||
is computed from two levels, each with the company's affinity as a bonus of
|
||||
`w.affinity * affinity / 3` over the sum of the weights, the same size as in
|
||||
the final score (a liked company about 0.05 above neutral, a favourite about
|
||||
0.1): the fit level `fit + bonus` and the CV level `cv_fit + bonus`. Over
|
||||
the roles that pass the fetch gate, the best `cut_fit_top` by fit level at
|
||||
or above `cut_fit_min` are `fit`; the best `cut_cv_top` by CV level at or
|
||||
above `cut_cv_min` that are not in yet are `cv`; every role this agent
|
||||
liked or anybody saved that is not in yet is `kept`. Newness and taste are
|
||||
no part of it, and the two lists are not merged into one number.
|
||||
|
||||
Recomputed in full on every run, and each agent's rows are replaced in one
|
||||
transaction: nothing is incremental, so nothing can drift.
|
||||
@@ -67,7 +79,7 @@ import numpy as np
|
||||
from pipeline import NULL, Notice, RunContext, job
|
||||
from app.config import Calibration, Config
|
||||
from app.jobs import embed
|
||||
from app.models import Agent, Candidate, Gate, Weights
|
||||
from app.models import Agent, Candidate, Gate, Limits, Weights
|
||||
from app.storage import Store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -130,10 +142,10 @@ def scale(cosines: dict[str, float],
|
||||
embedding model's calibrated `band` (`app.config.CALIBRATION`).
|
||||
|
||||
Raw cosines on multilingual-e5-small sit between about 0.75 and 0.87 for
|
||||
real postings against a real intent, so unscaled a prefer gate weighted
|
||||
0.2 moved a score more than the whole spread of fit. The band stretches
|
||||
that spread to 0..1 with fixed constants, so a role's level is the same
|
||||
in every run and a `limits` bar is an absolute threshold.
|
||||
real postings against a real intent, so unscaled the whole spread of fit
|
||||
was smaller than a recency step. The band stretches that spread to 0..1
|
||||
with fixed constants, so a role's level is the same in every run and a
|
||||
minimum of the cut is an absolute threshold.
|
||||
|
||||
Without a band, for a model nobody calibrated, it falls back to min-max
|
||||
across the run: the closest role is 1 and the farthest 0, which ranks
|
||||
@@ -165,19 +177,20 @@ def _reason(gate: Gate) -> str:
|
||||
|
||||
|
||||
def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
|
||||
"""Why this role is gated, as reasons: every require gate it fails, every
|
||||
exclude gate it matches, and the `any` group when it matches none of them.
|
||||
A gate with an empty column (state -1) neither passes nor fails, so the
|
||||
group fails only when some member had evidence. A favourite company's
|
||||
roles skip the `any` group but not the require and exclude gates, so a
|
||||
star widens a search to a company's roles within e.g. Europe, not to the
|
||||
whole world, and not to the titles the agent leaves out."""
|
||||
"""Why the fetch gate rejects this role, as reasons: every require gate
|
||||
it fails, every exclude gate it matches, and the `any` group when it
|
||||
matches none of them. A gate with an empty column (state -1) neither
|
||||
passes nor fails, so the group fails only when some member had evidence.
|
||||
A company set to All countries skips the `any` group, the place part,
|
||||
and one set to All roles the exclude gates on `title`, the titles to
|
||||
leave out; the require gates, the region, apply to every company."""
|
||||
states = list(zip(gates, candidate.gates, strict=True))
|
||||
reasons = [_reason(g) for g, state in states if g.mode == "require" and state == 0]
|
||||
reasons += [f"{g.field} {g.values[state - 1]}" for g, state in states
|
||||
if g.mode == "exclude" and state > 0]
|
||||
if g.mode == "exclude" and state > 0
|
||||
and not (candidate.all_roles and g.field == "title")]
|
||||
group = [(g, state) for g, state in states if g.mode == "any"]
|
||||
if candidate.affinity != 2 and group and not any(state == 1 for _, state in group) \
|
||||
if not candidate.all_countries and group and not any(state == 1 for _, state in group) \
|
||||
and any(state == 0 for _, state in group):
|
||||
reasons.append("none of " + ", ".join(_wanted(g) for g, _ in group))
|
||||
return reasons
|
||||
@@ -186,17 +199,15 @@ def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
|
||||
def score_one(candidate: Candidate, gates: Sequence[Gate], weights: Weights,
|
||||
fit: float | None, now: datetime, *, taste: float | None = None,
|
||||
missing: str = "no vector") -> tuple[float, str]:
|
||||
"""One role for one agent: `(score, gated_reason)`. `fit` and `taste` are
|
||||
already scaled to 0..1 (see `scale`), or None when there is nothing to
|
||||
measure: no role vector, no prompt vector, no votes. A None term leaves
|
||||
the denominator; `missing` says why fit is out."""
|
||||
"""One role for one agent: `(final score, gated_reason)`. `fit` and
|
||||
`taste` are already scaled to 0..1 (see `scale`), or None when there is
|
||||
nothing to measure: no role vector, no prompt vector, no votes. A None
|
||||
term leaves the denominator; `missing` says why fit is out."""
|
||||
if failed := failed_gates(candidate, gates):
|
||||
return 0.0, ", ".join(failed)
|
||||
|
||||
terms = [(weights.recency, recency(candidate.posted_at, now)),
|
||||
(weights.affinity, affinity_term(candidate.affinity))]
|
||||
terms += [(g.weight, 1.0 if state == 1 else 0.0)
|
||||
for g, state in zip(gates, candidate.gates, strict=True) if g.mode == "prefer"]
|
||||
if fit is not None:
|
||||
terms.append((weights.fit, fit))
|
||||
if taste is not None:
|
||||
@@ -208,6 +219,44 @@ def score_one(candidate: Candidate, gates: Sequence[Gate], weights: Weights,
|
||||
return sum(w * t for w, t in terms) / total, ""
|
||||
|
||||
|
||||
def bonus(affinity: int, weights: Weights) -> float:
|
||||
"""What a company's affinity adds to both levels of the cut: the share
|
||||
the affinity term gives it in the final score above a company at -1, so
|
||||
a liked company sits about 0.05 above a neutral one and a favourite
|
||||
about 0.1 on the default weights. 0 when every weight is 0."""
|
||||
total = weights.fit + weights.recency + weights.affinity + weights.taste
|
||||
return weights.affinity * affinity / 3 / total if total else 0.0
|
||||
|
||||
|
||||
def cut_of(candidates: Sequence[Candidate], passed: set[str], fits: dict[str, float],
|
||||
near: dict[str, float], kept: set[str], weights: Weights,
|
||||
limits: Limits) -> dict[str, str]:
|
||||
"""`uid -> why` for the roles in the cut, among `candidates` whose uid
|
||||
is in `passed` (the fetch gate let them through): `fit` for the best
|
||||
`limits.cut_fit_top` by fit level at or above `cut_fit_min`, `cv` for
|
||||
the best `cut_cv_top` by CV level at or above `cut_cv_min` not in yet,
|
||||
and `kept` for those in `kept` (liked or saved) not in yet. A level is
|
||||
`fits` or `near` plus the company's `bonus`; a role without one is in
|
||||
no list by it. Ties go to the newer role, then the lower uid."""
|
||||
by_uid = {c.uid: c for c in candidates if c.uid in passed}
|
||||
# Newest first, then uid, as the sort below keeps equal levels in order.
|
||||
ordered = sorted(sorted(by_uid.values(), key=lambda c: c.uid),
|
||||
key=lambda c: c.posted_at, reverse=True)
|
||||
cut: dict[str, str] = {}
|
||||
for why, levels, top, floor in (("fit", fits, limits.cut_fit_top, limits.cut_fit_min),
|
||||
("cv", near, limits.cut_cv_top, limits.cut_cv_min)):
|
||||
scored = [(levels[c.uid] + bonus(c.affinity, weights), c.uid) for c in ordered
|
||||
if c.uid in levels and c.uid not in cut]
|
||||
scored = [(level, uid) for level, uid in scored if level >= floor]
|
||||
scored.sort(key=lambda pair: pair[0], reverse=True)
|
||||
for _, uid in scored[:top]:
|
||||
cut[uid] = why
|
||||
for c in ordered:
|
||||
if c.uid in kept and c.uid not in cut:
|
||||
cut[c.uid] = "kept"
|
||||
return cut
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ vectors
|
||||
|
||||
def intent_key(model: str, intent: str) -> str:
|
||||
@@ -301,12 +350,13 @@ def _cv(store: Store, agent: Agent, model: str, encoder: _Encoder) -> np.ndarray
|
||||
|
||||
def no_intent(names: Sequence[str]) -> Notice:
|
||||
"""The notice for enabled agents without a prompt that has text: their
|
||||
scores have no fit, so `Store._top_scope` keeps them from feeding extract
|
||||
and summarise. The code keeps its name from when an agent had one intent."""
|
||||
scores have no fit, so nothing of theirs enters the cut by fit and
|
||||
their final scores do not choose what summarise reads. The code keeps
|
||||
its name from when an agent had one intent."""
|
||||
return Notice(
|
||||
code="score.no_intent", level="warning", kind="expected", key="score",
|
||||
text=f"No prompt with text, so ranked without fit and not read by extract "
|
||||
f"or summarise: {', '.join(names)}.",
|
||||
text=f"No prompt with text, so ranked without fit and nothing in the cut by fit: "
|
||||
f"{', '.join(names)}.",
|
||||
fix="Give the agent a prompt under Settings, Search agents.",
|
||||
where="app.jobs.score")
|
||||
|
||||
@@ -314,7 +364,8 @@ def no_intent(names: Sequence[str]) -> Notice:
|
||||
def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
prompts: Sequence[tuple[str, np.ndarray]], now: datetime,
|
||||
calibration: Calibration | None = None,
|
||||
cv: np.ndarray | None = None) -> dict[str, int]:
|
||||
cv: np.ndarray | None = None,
|
||||
saved: set[str] | None = None) -> dict[str, int]:
|
||||
"""Two passes: the cosines of the roles this agent scores, to each of its
|
||||
`prompts` and to its taste direction, then every score with both scaled
|
||||
by the model's `calibration`, or across the run without one. A role's fit
|
||||
@@ -327,9 +378,12 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
share. The voted roles are scored like any other; a liked one sits near
|
||||
the direction it helped make, which is what a like should do. With `cv`
|
||||
(`_cv`), each of those roles also gets its `cv_fit`, its `closeness` to
|
||||
the CV scaled like fit, beside the score and not in it."""
|
||||
the CV scaled like fit. Then the cut (`cut_of`), with this agent's likes
|
||||
and the `saved` roles (`Store.saved_uids`, read here when not given)
|
||||
kept."""
|
||||
candidates = store.score_candidates(agent.id, agent.gates)
|
||||
direction = taste_direction(vectors, store.votes(agent.id))
|
||||
votes = store.votes(agent.id)
|
||||
direction = taste_direction(vectors, votes)
|
||||
cosines: dict[str, dict[str, float]] = {name: {} for name, _ in prompts}
|
||||
tastes: dict[str, float] = {}
|
||||
near: dict[str, float] = {}
|
||||
@@ -351,22 +405,29 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
fits[uid], best[uid] = value, name
|
||||
tastes = scale(tastes, calibration.taste if calibration else None)
|
||||
near = scale(near, calibration.cv_band if calibration else None)
|
||||
rows: list[tuple[str, float, str, str, float | None]] = []
|
||||
gated = 0
|
||||
scored: list[tuple[str, float, str, float | None]] = []
|
||||
passed: set[str] = set()
|
||||
for candidate in candidates:
|
||||
fit = fits.get(candidate.uid)
|
||||
value, reason = score_one(
|
||||
candidate, agent.gates, agent.weights, fit, now,
|
||||
taste=tastes.get(candidate.uid),
|
||||
missing="no vector" if prompts else "no prompt vector")
|
||||
gated += bool(reason)
|
||||
if not reason:
|
||||
passed.add(candidate.uid)
|
||||
scored.append((candidate.uid, value, reason, fit))
|
||||
kept = {uid for uid, vote in votes.items() if vote == 1}
|
||||
kept |= store.saved_uids() if saved is None else saved
|
||||
cut = cut_of(candidates, passed, fits, near, kept, agent.weights, agent.limits)
|
||||
rows = []
|
||||
for uid, value, reason, fit in scored:
|
||||
counted = fit is not None and agent.weights.fit and not reason
|
||||
cv_fit = near.get(candidate.uid)
|
||||
rows.append((candidate.uid, round(value, 6), reason,
|
||||
best[candidate.uid] if counted else "",
|
||||
None if cv_fit is None else round(cv_fit, 6)))
|
||||
cv_fit = near.get(uid)
|
||||
rows.append((uid, round(value, 6), reason, best[uid] if counted else "",
|
||||
None if cv_fit is None else round(cv_fit, 6),
|
||||
None if fit is None else round(fit, 6), cut.get(uid, "")))
|
||||
store.replace_scores(agent.id, rows)
|
||||
return {"scored": len(rows), "gated": gated}
|
||||
return {"scored": len(rows), "gated": len(rows) - len(passed), "cut": len(cut)}
|
||||
|
||||
|
||||
@job("score", "Scoring roles for every agent", solo=True, resource="cpu")
|
||||
@@ -386,17 +447,21 @@ def run_score(config: Config, store: Store, *,
|
||||
if blind := [agent.name for agent in agents
|
||||
if not any(p.text.strip() for p in agent.prompts)]:
|
||||
context.notice(no_intent(blind))
|
||||
scored = gated = 0
|
||||
scored = gated = cut = 0
|
||||
saved = store.saved_uids()
|
||||
for n, agent in enumerate(agents):
|
||||
context.check()
|
||||
context.progress(n, len(agents), f"scoring for {agent.name}, {n + 1} of {len(agents)}")
|
||||
counts = score_agent(store, agent, vectors, _prompts(store, agent, model, encoder),
|
||||
now, calibration, cv=_cv(store, agent, model, encoder))
|
||||
now, calibration, cv=_cv(store, agent, model, encoder),
|
||||
saved=saved)
|
||||
scored += counts["scored"]
|
||||
gated += counts["gated"]
|
||||
log.info("%-20s %6d scored, %6d gated", agent.name, counts["scored"], counts["gated"])
|
||||
cut += counts["cut"]
|
||||
log.info("%-20s %6d scored, %6d gated, %5d in the cut", agent.name, counts["scored"],
|
||||
counts["gated"], counts["cut"])
|
||||
context.progress(len(agents), len(agents), "")
|
||||
return {"agents": len(agents), "scored": scored, "gated": gated,
|
||||
return {"agents": len(agents), "scored": scored, "gated": gated, "cut": cut,
|
||||
"vectors": len(vectors)}
|
||||
|
||||
|
||||
|
||||
+13
-12
@@ -1,5 +1,5 @@
|
||||
"""Whether the three model stages of a pull, `extract`, `summarise` and
|
||||
`judge`, and the `enrich` stage run.
|
||||
`judge`, and the `fetch_text` stage run.
|
||||
|
||||
`extract.enabled`, `summarise.enabled` and `judge.enabled` in config.yaml are
|
||||
read at startup and say nothing on the page, so the switch lives in the
|
||||
@@ -10,16 +10,17 @@ deployment that never uses the switch behaves as it did. `stage_on` is the
|
||||
one place that decides, and the tasks' finders ask it
|
||||
(`extract.extract_task`, `summarise.summarise_task`, `judge.judge_task`).
|
||||
|
||||
`enrich` (reading the best roles on their own pages, `app/jobs/enrich.py`)
|
||||
asks no model and is on by default (`enrich.enabled`); its switch is
|
||||
`stage:enrich`, set the same way as the others: by the Runs page
|
||||
(`PUT /api/stages/enrich`) and by `app stages enrich on|off`.
|
||||
`fetch_text` (reading the role's own page for the roles in the cut,
|
||||
`app/jobs/fetch_text.py`) asks no model and is on by default
|
||||
(`fetch_text.enabled`); its switch is `stage:fetch_text`, set the same way
|
||||
as the others: by the Runs page (`PUT /api/stages/fetch_text`) and by
|
||||
`app stages fetch_text on|off`.
|
||||
|
||||
`describe` is what `GET /api/stages` and `app stages` show: for each stage
|
||||
with a switch, the model stages and `enrich`, whether it is on, where that comes from, the model it uses, and who
|
||||
would answer. The model is the one a call is made with: `agent.model` when
|
||||
set, since it overrides the per-task model everywhere (`desk/modeltask.py`),
|
||||
else the stage's own.
|
||||
with a switch, the model stages and `fetch_text`, whether it is on, where
|
||||
that comes from, the model it uses, and who would answer. The model is the
|
||||
one a call is made with: `agent.model` when set, since it overrides the
|
||||
per-task model everywhere (`desk/modeltask.py`), else the stage's own.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -34,8 +35,8 @@ from desk.tools import agent
|
||||
|
||||
NAMES = ("extract", "summarise", "judge")
|
||||
|
||||
#: Every stage with a switch: the model stages and `enrich`.
|
||||
SWITCHES = (*NAMES, "enrich")
|
||||
#: Every stage with a switch: the model stages and `fetch_text`.
|
||||
SWITCHES = (*NAMES, "fetch_text")
|
||||
|
||||
#: What `describe_one` says of a stage that asks no model.
|
||||
NO_ASK = "reads role pages, asks no model"
|
||||
@@ -53,7 +54,7 @@ class Stage(BaseModel):
|
||||
`backend` is `agent.backend`'s answer, "" for nobody, and `answer` says
|
||||
it in words."""
|
||||
|
||||
name: Literal["extract", "summarise", "judge", "enrich"]
|
||||
name: Literal["extract", "summarise", "judge", "fetch_text"]
|
||||
on: bool
|
||||
source: Literal["stored", "config"]
|
||||
model: str
|
||||
|
||||
@@ -6,12 +6,13 @@ what the model is told, the schema it answers in, what to do with the
|
||||
answer), and running it is `desk.modeltask`'s.
|
||||
|
||||
A role is waiting while it has no summary, has some text, is open and no copy,
|
||||
and is either in the top of an enabled agent's ungated scores as deep and as
|
||||
low as that agent's own `limits` say (`summarise_top`, `summarise_min`), or
|
||||
one the owner cares about whatever its rank: saved (it has an application, the
|
||||
star) or voted up by an enabled agent. The limits keep the cost flat: a pull
|
||||
summarises the roles somebody will look at, not the 30k that are open, and the
|
||||
saved and liked roles are a handful the owner picked. Each is summarised
|
||||
and is either in the top of an enabled agent's feed, the best `summarise_top`
|
||||
final scores among the roles its fetch gate passes (its `limits`), or one the
|
||||
owner cares about whatever its rank: saved (it has an application, the star)
|
||||
or voted up by an enabled agent. Summarise is no part of any score and runs
|
||||
last in the pull, once the final scores are set. The limit keeps the cost
|
||||
flat: a pull summarises the roles somebody will look at, not the 30k that are
|
||||
open, and the saved and liked roles are a handful the owner picked. Each is summarised
|
||||
once, with the same give-up rule (`GIVE_UP_AFTER` replies that left it
|
||||
without). Saving or liking a role queues a batch for that one uid at once,
|
||||
while the stage is on (`web.api`), rather than waiting for the next pull. A
|
||||
|
||||
Reference in new issue
Block a user