Rank by the cut: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text renamed from enrich, the feed kept to the fetch gate, and migration 32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
060ccf66e6
commit
617356e60c
35 files changed
+1054
-751
No files matched your search
+132
-67
@@ -1,21 +1,29 @@
|
||||
"""`score`: every open role, scored for every enabled search agent.
|
||||
"""`score`: every open role, scored for every enabled search agent, and each
|
||||
agent's cut.
|
||||
|
||||
A role is scored per agent, `(uid, agent) -> score`, because a role is only
|
||||
good for a given search. For each agent:
|
||||
good for a given search. Each agent first applies its fetch gate:
|
||||
|
||||
require gates a role that fails one keeps its row, with score 0 and the
|
||||
reason ("remote not remote")
|
||||
any gates one group: a role that matches none of them, and has the
|
||||
column filled for at least one, is gated ("none of
|
||||
country DK / remote remote")
|
||||
prefer gates `weight` when the role matches, 0 when it does not
|
||||
reason ("remote not remote"); the region, which nothing
|
||||
lifts
|
||||
any gates one group, the place part: a role that matches none of
|
||||
them, and has the column filled for at least one, is
|
||||
gated ("none of country DK / remote remote"); skipped for
|
||||
a company set to All countries
|
||||
exclude gates a role that matches one of the values keeps its row, with
|
||||
score 0 and the value ("title account executive")
|
||||
score 0 and the value ("title account executive"); those
|
||||
on `title` are skipped for a company set to All roles
|
||||
|
||||
A gate rejects on evidence, never on absence: a role whose column is empty
|
||||
is not gated by it. `extract` is what fills an empty column, and it reads
|
||||
only the roles that rank, so gating on an empty column would keep those
|
||||
roles out of extract's reach for good.
|
||||
only the roles in the cut, so gating on an empty column would keep those
|
||||
roles out of extract's reach for good. A hidden company (`affinity = -1`)
|
||||
is not scored at all; a favourite passes the gate like any other company.
|
||||
|
||||
The final score, the feed's order, is the weighted sum of four terms over
|
||||
the sum of the weights, so it stays 0..1:
|
||||
|
||||
fit cos(role vector, prompt vector) as a level: the embedding
|
||||
model's calibrated `lo` is 0 and `hi` 1, clipped outside
|
||||
(see `scale`), so a role's fit does not depend on what
|
||||
@@ -29,25 +37,29 @@ roles out of extract's reach for good.
|
||||
over this agent's votes; with only likes it is the liked mean,
|
||||
with only passes the negated passed mean; no votes, no term
|
||||
|
||||
Beside the score, for an agent with a CV, `cv_fit`: how close the role is to
|
||||
the CV, the best cosine between the role and the CV's chunks
|
||||
(`embed.cv_chunks`, embedded as the prompts are), scaled like fit. It is no
|
||||
term of the score; the judge reads the closest roles the score leaves out
|
||||
(`Store.judge_rows`) and CV match orders the roles not judged yet by it.
|
||||
A role with no vector is scored without `fit` and `taste`, unless `fit` is
|
||||
all there is, when it is gated as "no vector". An agent with no votes scores
|
||||
every role without `taste`; the direction is newsdesk's
|
||||
`centroid(liked) - centroid(disliked)`. An agent with no prompt that has
|
||||
text has no `fit` either, so nothing of it enters the cut by fit and its
|
||||
scores do not decide what summarise reads (`Store._summarise_top`); a run
|
||||
names such agents in one notice.
|
||||
|
||||
Every term is 0..1. The score is the weighted sum over the sum of the weights
|
||||
(the agent's four and its prefer gates'), so it stays 0..1. A favourite
|
||||
company (`affinity = 2`) skips the `any` group but not the `require` or
|
||||
`exclude` gates, and
|
||||
is scored like any other company with its affinity term at 1: nothing puts
|
||||
its roles above a better fit. A hidden one (`-1`) is not scored at all. A
|
||||
role with no vector is scored without `fit` and `taste`, unless `fit` is all
|
||||
there is, when it is gated as "no vector".
|
||||
An agent with no votes scores every role without `taste`; the direction is
|
||||
newsdesk's `centroid(liked) - centroid(disliked)`. An agent with no prompt
|
||||
that has text has no `fit` either: its scores are not a match, so it is left
|
||||
out of what extract and summarise read (`Store._top_scope`), and a run names
|
||||
such agents in one notice.
|
||||
Beside the score, `fit` is stored as it is, and for an agent with a CV,
|
||||
`cv_fit`: how close the role is to the CV, the best cosine between the role
|
||||
and the CV's chunks (`embed.cv_chunks`, embedded as the prompts are), scaled
|
||||
like fit. Neither carries the affinity term.
|
||||
|
||||
The cut is the roles that get fetch text, extract and judge (`cut_of`). It
|
||||
is computed from two levels, each with the company's affinity as a bonus of
|
||||
`w.affinity * affinity / 3` over the sum of the weights, the same size as in
|
||||
the final score (a liked company about 0.05 above neutral, a favourite about
|
||||
0.1): the fit level `fit + bonus` and the CV level `cv_fit + bonus`. Over
|
||||
the roles that pass the fetch gate, the best `cut_fit_top` by fit level at
|
||||
or above `cut_fit_min` are `fit`; the best `cut_cv_top` by CV level at or
|
||||
above `cut_cv_min` that are not in yet are `cv`; every role this agent
|
||||
liked or anybody saved that is not in yet is `kept`. Newness and taste are
|
||||
no part of it, and the two lists are not merged into one number.
|
||||
|
||||
Recomputed in full on every run, and each agent's rows are replaced in one
|
||||
transaction: nothing is incremental, so nothing can drift.
|
||||
@@ -67,7 +79,7 @@ import numpy as np
|
||||
from pipeline import NULL, Notice, RunContext, job
|
||||
from app.config import Calibration, Config
|
||||
from app.jobs import embed
|
||||
from app.models import Agent, Candidate, Gate, Weights
|
||||
from app.models import Agent, Candidate, Gate, Limits, Weights
|
||||
from app.storage import Store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -130,10 +142,10 @@ def scale(cosines: dict[str, float],
|
||||
embedding model's calibrated `band` (`app.config.CALIBRATION`).
|
||||
|
||||
Raw cosines on multilingual-e5-small sit between about 0.75 and 0.87 for
|
||||
real postings against a real intent, so unscaled a prefer gate weighted
|
||||
0.2 moved a score more than the whole spread of fit. The band stretches
|
||||
that spread to 0..1 with fixed constants, so a role's level is the same
|
||||
in every run and a `limits` bar is an absolute threshold.
|
||||
real postings against a real intent, so unscaled the whole spread of fit
|
||||
was smaller than a recency step. The band stretches that spread to 0..1
|
||||
with fixed constants, so a role's level is the same in every run and a
|
||||
minimum of the cut is an absolute threshold.
|
||||
|
||||
Without a band, for a model nobody calibrated, it falls back to min-max
|
||||
across the run: the closest role is 1 and the farthest 0, which ranks
|
||||
@@ -165,19 +177,20 @@ def _reason(gate: Gate) -> str:
|
||||
|
||||
|
||||
def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
|
||||
"""Why this role is gated, as reasons: every require gate it fails, every
|
||||
exclude gate it matches, and the `any` group when it matches none of them.
|
||||
A gate with an empty column (state -1) neither passes nor fails, so the
|
||||
group fails only when some member had evidence. A favourite company's
|
||||
roles skip the `any` group but not the require and exclude gates, so a
|
||||
star widens a search to a company's roles within e.g. Europe, not to the
|
||||
whole world, and not to the titles the agent leaves out."""
|
||||
"""Why the fetch gate rejects this role, as reasons: every require gate
|
||||
it fails, every exclude gate it matches, and the `any` group when it
|
||||
matches none of them. A gate with an empty column (state -1) neither
|
||||
passes nor fails, so the group fails only when some member had evidence.
|
||||
A company set to All countries skips the `any` group, the place part,
|
||||
and one set to All roles the exclude gates on `title`, the titles to
|
||||
leave out; the require gates, the region, apply to every company."""
|
||||
states = list(zip(gates, candidate.gates, strict=True))
|
||||
reasons = [_reason(g) for g, state in states if g.mode == "require" and state == 0]
|
||||
reasons += [f"{g.field} {g.values[state - 1]}" for g, state in states
|
||||
if g.mode == "exclude" and state > 0]
|
||||
if g.mode == "exclude" and state > 0
|
||||
and not (candidate.all_roles and g.field == "title")]
|
||||
group = [(g, state) for g, state in states if g.mode == "any"]
|
||||
if candidate.affinity != 2 and group and not any(state == 1 for _, state in group) \
|
||||
if not candidate.all_countries and group and not any(state == 1 for _, state in group) \
|
||||
and any(state == 0 for _, state in group):
|
||||
reasons.append("none of " + ", ".join(_wanted(g) for g, _ in group))
|
||||
return reasons
|
||||
@@ -186,17 +199,15 @@ def failed_gates(candidate: Candidate, gates: Sequence[Gate]) -> list[str]:
|
||||
def score_one(candidate: Candidate, gates: Sequence[Gate], weights: Weights,
|
||||
fit: float | None, now: datetime, *, taste: float | None = None,
|
||||
missing: str = "no vector") -> tuple[float, str]:
|
||||
"""One role for one agent: `(score, gated_reason)`. `fit` and `taste` are
|
||||
already scaled to 0..1 (see `scale`), or None when there is nothing to
|
||||
measure: no role vector, no prompt vector, no votes. A None term leaves
|
||||
the denominator; `missing` says why fit is out."""
|
||||
"""One role for one agent: `(final score, gated_reason)`. `fit` and
|
||||
`taste` are already scaled to 0..1 (see `scale`), or None when there is
|
||||
nothing to measure: no role vector, no prompt vector, no votes. A None
|
||||
term leaves the denominator; `missing` says why fit is out."""
|
||||
if failed := failed_gates(candidate, gates):
|
||||
return 0.0, ", ".join(failed)
|
||||
|
||||
terms = [(weights.recency, recency(candidate.posted_at, now)),
|
||||
(weights.affinity, affinity_term(candidate.affinity))]
|
||||
terms += [(g.weight, 1.0 if state == 1 else 0.0)
|
||||
for g, state in zip(gates, candidate.gates, strict=True) if g.mode == "prefer"]
|
||||
if fit is not None:
|
||||
terms.append((weights.fit, fit))
|
||||
if taste is not None:
|
||||
@@ -208,6 +219,44 @@ def score_one(candidate: Candidate, gates: Sequence[Gate], weights: Weights,
|
||||
return sum(w * t for w, t in terms) / total, ""
|
||||
|
||||
|
||||
def bonus(affinity: int, weights: Weights) -> float:
|
||||
"""What a company's affinity adds to both levels of the cut: the share
|
||||
the affinity term gives it in the final score above a company at -1, so
|
||||
a liked company sits about 0.05 above a neutral one and a favourite
|
||||
about 0.1 on the default weights. 0 when every weight is 0."""
|
||||
total = weights.fit + weights.recency + weights.affinity + weights.taste
|
||||
return weights.affinity * affinity / 3 / total if total else 0.0
|
||||
|
||||
|
||||
def cut_of(candidates: Sequence[Candidate], passed: set[str], fits: dict[str, float],
|
||||
near: dict[str, float], kept: set[str], weights: Weights,
|
||||
limits: Limits) -> dict[str, str]:
|
||||
"""`uid -> why` for the roles in the cut, among `candidates` whose uid
|
||||
is in `passed` (the fetch gate let them through): `fit` for the best
|
||||
`limits.cut_fit_top` by fit level at or above `cut_fit_min`, `cv` for
|
||||
the best `cut_cv_top` by CV level at or above `cut_cv_min` not in yet,
|
||||
and `kept` for those in `kept` (liked or saved) not in yet. A level is
|
||||
`fits` or `near` plus the company's `bonus`; a role without one is in
|
||||
no list by it. Ties go to the newer role, then the lower uid."""
|
||||
by_uid = {c.uid: c for c in candidates if c.uid in passed}
|
||||
# Newest first, then uid, as the sort below keeps equal levels in order.
|
||||
ordered = sorted(sorted(by_uid.values(), key=lambda c: c.uid),
|
||||
key=lambda c: c.posted_at, reverse=True)
|
||||
cut: dict[str, str] = {}
|
||||
for why, levels, top, floor in (("fit", fits, limits.cut_fit_top, limits.cut_fit_min),
|
||||
("cv", near, limits.cut_cv_top, limits.cut_cv_min)):
|
||||
scored = [(levels[c.uid] + bonus(c.affinity, weights), c.uid) for c in ordered
|
||||
if c.uid in levels and c.uid not in cut]
|
||||
scored = [(level, uid) for level, uid in scored if level >= floor]
|
||||
scored.sort(key=lambda pair: pair[0], reverse=True)
|
||||
for _, uid in scored[:top]:
|
||||
cut[uid] = why
|
||||
for c in ordered:
|
||||
if c.uid in kept and c.uid not in cut:
|
||||
cut[c.uid] = "kept"
|
||||
return cut
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ vectors
|
||||
|
||||
def intent_key(model: str, intent: str) -> str:
|
||||
@@ -301,12 +350,13 @@ def _cv(store: Store, agent: Agent, model: str, encoder: _Encoder) -> np.ndarray
|
||||
|
||||
def no_intent(names: Sequence[str]) -> Notice:
|
||||
"""The notice for enabled agents without a prompt that has text: their
|
||||
scores have no fit, so `Store._top_scope` keeps them from feeding extract
|
||||
and summarise. The code keeps its name from when an agent had one intent."""
|
||||
scores have no fit, so nothing of theirs enters the cut by fit and
|
||||
their final scores do not choose what summarise reads. The code keeps
|
||||
its name from when an agent had one intent."""
|
||||
return Notice(
|
||||
code="score.no_intent", level="warning", kind="expected", key="score",
|
||||
text=f"No prompt with text, so ranked without fit and not read by extract "
|
||||
f"or summarise: {', '.join(names)}.",
|
||||
text=f"No prompt with text, so ranked without fit and nothing in the cut by fit: "
|
||||
f"{', '.join(names)}.",
|
||||
fix="Give the agent a prompt under Settings, Search agents.",
|
||||
where="app.jobs.score")
|
||||
|
||||
@@ -314,7 +364,8 @@ def no_intent(names: Sequence[str]) -> Notice:
|
||||
def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
prompts: Sequence[tuple[str, np.ndarray]], now: datetime,
|
||||
calibration: Calibration | None = None,
|
||||
cv: np.ndarray | None = None) -> dict[str, int]:
|
||||
cv: np.ndarray | None = None,
|
||||
saved: set[str] | None = None) -> dict[str, int]:
|
||||
"""Two passes: the cosines of the roles this agent scores, to each of its
|
||||
`prompts` and to its taste direction, then every score with both scaled
|
||||
by the model's `calibration`, or across the run without one. A role's fit
|
||||
@@ -327,9 +378,12 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
share. The voted roles are scored like any other; a liked one sits near
|
||||
the direction it helped make, which is what a like should do. With `cv`
|
||||
(`_cv`), each of those roles also gets its `cv_fit`, its `closeness` to
|
||||
the CV scaled like fit, beside the score and not in it."""
|
||||
the CV scaled like fit. Then the cut (`cut_of`), with this agent's likes
|
||||
and the `saved` roles (`Store.saved_uids`, read here when not given)
|
||||
kept."""
|
||||
candidates = store.score_candidates(agent.id, agent.gates)
|
||||
direction = taste_direction(vectors, store.votes(agent.id))
|
||||
votes = store.votes(agent.id)
|
||||
direction = taste_direction(vectors, votes)
|
||||
cosines: dict[str, dict[str, float]] = {name: {} for name, _ in prompts}
|
||||
tastes: dict[str, float] = {}
|
||||
near: dict[str, float] = {}
|
||||
@@ -351,22 +405,29 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
fits[uid], best[uid] = value, name
|
||||
tastes = scale(tastes, calibration.taste if calibration else None)
|
||||
near = scale(near, calibration.cv_band if calibration else None)
|
||||
rows: list[tuple[str, float, str, str, float | None]] = []
|
||||
gated = 0
|
||||
scored: list[tuple[str, float, str, float | None]] = []
|
||||
passed: set[str] = set()
|
||||
for candidate in candidates:
|
||||
fit = fits.get(candidate.uid)
|
||||
value, reason = score_one(
|
||||
candidate, agent.gates, agent.weights, fit, now,
|
||||
taste=tastes.get(candidate.uid),
|
||||
missing="no vector" if prompts else "no prompt vector")
|
||||
gated += bool(reason)
|
||||
if not reason:
|
||||
passed.add(candidate.uid)
|
||||
scored.append((candidate.uid, value, reason, fit))
|
||||
kept = {uid for uid, vote in votes.items() if vote == 1}
|
||||
kept |= store.saved_uids() if saved is None else saved
|
||||
cut = cut_of(candidates, passed, fits, near, kept, agent.weights, agent.limits)
|
||||
rows = []
|
||||
for uid, value, reason, fit in scored:
|
||||
counted = fit is not None and agent.weights.fit and not reason
|
||||
cv_fit = near.get(candidate.uid)
|
||||
rows.append((candidate.uid, round(value, 6), reason,
|
||||
best[candidate.uid] if counted else "",
|
||||
None if cv_fit is None else round(cv_fit, 6)))
|
||||
cv_fit = near.get(uid)
|
||||
rows.append((uid, round(value, 6), reason, best[uid] if counted else "",
|
||||
None if cv_fit is None else round(cv_fit, 6),
|
||||
None if fit is None else round(fit, 6), cut.get(uid, "")))
|
||||
store.replace_scores(agent.id, rows)
|
||||
return {"scored": len(rows), "gated": gated}
|
||||
return {"scored": len(rows), "gated": len(rows) - len(passed), "cut": len(cut)}
|
||||
|
||||
|
||||
@job("score", "Scoring roles for every agent", solo=True, resource="cpu")
|
||||
@@ -386,17 +447,21 @@ def run_score(config: Config, store: Store, *,
|
||||
if blind := [agent.name for agent in agents
|
||||
if not any(p.text.strip() for p in agent.prompts)]:
|
||||
context.notice(no_intent(blind))
|
||||
scored = gated = 0
|
||||
scored = gated = cut = 0
|
||||
saved = store.saved_uids()
|
||||
for n, agent in enumerate(agents):
|
||||
context.check()
|
||||
context.progress(n, len(agents), f"scoring for {agent.name}, {n + 1} of {len(agents)}")
|
||||
counts = score_agent(store, agent, vectors, _prompts(store, agent, model, encoder),
|
||||
now, calibration, cv=_cv(store, agent, model, encoder))
|
||||
now, calibration, cv=_cv(store, agent, model, encoder),
|
||||
saved=saved)
|
||||
scored += counts["scored"]
|
||||
gated += counts["gated"]
|
||||
log.info("%-20s %6d scored, %6d gated", agent.name, counts["scored"], counts["gated"])
|
||||
cut += counts["cut"]
|
||||
log.info("%-20s %6d scored, %6d gated, %5d in the cut", agent.name, counts["scored"],
|
||||
counts["gated"], counts["cut"])
|
||||
context.progress(len(agents), len(agents), "")
|
||||
return {"agents": len(agents), "scored": scored, "gated": gated,
|
||||
return {"agents": len(agents), "scored": scored, "gated": gated, "cut": cut,
|
||||
"vectors": len(vectors)}
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user