Calibrate fit and taste per embedding model instead of min-max per run
scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants from CALIBRATION in app/config.py, keyed by model and replaceable under embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19 (taste from synthetic votes). An uncalibrated model falls back to min-max and logs a warning once per run. The extract and summarise bars default to 0.55 and 0.65, about 300 and 35 roles for the starter agent on the measured data; stored limits are kept. An agent without an intent is left out of the model-stage scope, and each score run names such agents in one score.no_intent notice. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
706b127bc0
commit
b021838959
12 files changed
+336
-76
No files matched your search
+58
-22
@@ -14,13 +14,15 @@ A gate rejects on evidence, never on absence: a role whose column is empty
|
||||
is not gated by it. `extract` is what fills an empty column, and it reads
|
||||
only the roles that rank, so gating on an empty column would keep those
|
||||
roles out of extract's reach for good.
|
||||
fit cos(role vector, intent vector), min-max scaled across
|
||||
the roles this agent scores in this run: the closest is 1,
|
||||
the farthest 0, a lone role 1
|
||||
fit cos(role vector, intent vector) as a level: the embedding
|
||||
model's calibrated `lo` is 0 and `hi` 1, clipped outside
|
||||
(see `scale`), so a role's fit does not depend on what
|
||||
else is open
|
||||
recency 0.5 ** (days since posted_at / 14)
|
||||
affinity (the agent's affinity for the company + 1) / 3
|
||||
taste cos(role vector, liked direction), min-max scaled like fit,
|
||||
where the direction is mean(liked vectors) - mean(passed vectors)
|
||||
taste cos(role vector, liked direction), scaled like fit with the
|
||||
model's taste calibration, where the direction is
|
||||
mean(liked vectors) - mean(passed vectors)
|
||||
over this agent's votes; with only likes it is the liked mean,
|
||||
with only passes the negated passed mean; no votes, no term
|
||||
|
||||
@@ -30,7 +32,10 @@ company (`affinity = 2`) skips the gates and is floored at 0.9; a hidden one
|
||||
(`-1`) is not scored at all. A role with no vector is scored without `fit`
|
||||
and `taste`, unless `fit` is all there is, when it is gated as "no vector".
|
||||
An agent with no votes scores every role without `taste`; the direction is
|
||||
newsdesk's `centroid(liked) - centroid(disliked)`.
|
||||
newsdesk's `centroid(liked) - centroid(disliked)`. An agent with no intent
|
||||
has no `fit` either: its scores are not a match, so it is left out of what
|
||||
extract and summarise read (`Store._top_scope`), and a run names such agents
|
||||
in one notice.
|
||||
|
||||
Recomputed in full on every run, and each agent's rows are replaced in one
|
||||
transaction: nothing is incremental, so nothing can drift.
|
||||
@@ -47,8 +52,8 @@ from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from pipeline import NULL, RunContext, job
|
||||
from app.config import Config
|
||||
from pipeline import NULL, Notice, RunContext, job
|
||||
from app.config import Calibration, Config
|
||||
from app.jobs import embed
|
||||
from app.models import Agent, Candidate, Gate, Weights
|
||||
from app.storage import Store
|
||||
@@ -110,19 +115,28 @@ def taste_direction(vectors: dict[str, np.ndarray],
|
||||
return direction
|
||||
|
||||
|
||||
def scale(cosines: dict[str, float]) -> dict[str, float]:
|
||||
"""Min-max to 0..1 across one agent's run, so `fit` ranks roles rather
|
||||
than stating a level.
|
||||
def scale(cosines: dict[str, float],
|
||||
band: tuple[float, float] | None) -> dict[str, float]:
|
||||
"""Cosines to 0..1 levels: `clip((cos - lo) / (hi - lo), 0, 1)` with the
|
||||
embedding model's calibrated `band` (`app.config.CALIBRATION`).
|
||||
|
||||
`(cos + 1) / 2` was the level, and on multilingual-e5-small it spans about
|
||||
0.87 to 0.95 for real postings against a real intent (0.55 to 0.79 on the
|
||||
all-MiniLM-L6-v2 this replaced), so a prefer gate weighted 0.2 moved a
|
||||
score more than the whole spread of fit did either way. Scaled, the
|
||||
closest role is 1 and the farthest 0. A single role, or roles that are all
|
||||
equally close, get 1: there is nothing to tell them apart by.
|
||||
Raw cosines on multilingual-e5-small sit between about 0.75 and 0.87 for
|
||||
real postings against a real intent, so unscaled a prefer gate weighted
|
||||
0.2 moved a score more than the whole spread of fit. The band stretches
|
||||
that spread to 0..1 with fixed constants, so a role's level is the same
|
||||
in every run and a `limits` bar is an absolute threshold.
|
||||
|
||||
Without a band, for a model nobody calibrated, it falls back to min-max
|
||||
across the run: the closest role is 1 and the farthest 0, which ranks
|
||||
rather than states a level. A single role, or roles that are all equally
|
||||
close, then get 1: there is nothing to tell them apart by.
|
||||
"""
|
||||
if not cosines:
|
||||
return {}
|
||||
if band is not None:
|
||||
lo, hi = band
|
||||
return {uid: min(max((value - lo) / (hi - lo), 0.0), 1.0)
|
||||
for uid, value in cosines.items()}
|
||||
low, high = min(cosines.values()), max(cosines.values())
|
||||
if high - low < 1e-9:
|
||||
return dict.fromkeys(cosines, 1.0)
|
||||
@@ -231,11 +245,24 @@ def _intent(store: Store, agent: Agent, model: str,
|
||||
|
||||
# ------------------------------------------------------------------- the job
|
||||
|
||||
def no_intent(names: Sequence[str]) -> Notice:
|
||||
"""The notice for enabled agents without an intent: their scores have no
|
||||
fit, so `Store._top_scope` keeps them from feeding extract and summarise."""
|
||||
return Notice(
|
||||
code="score.no_intent", level="warning", kind="expected", key="score",
|
||||
text=f"No intent, so ranked without fit and not read by extract or "
|
||||
f"summarise: {', '.join(names)}.",
|
||||
fix="Give the agent an intent under Settings, Search agents.",
|
||||
where="app.jobs.score")
|
||||
|
||||
|
||||
def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
intent: np.ndarray | None, now: datetime) -> dict[str, int]:
|
||||
intent: np.ndarray | None, now: datetime,
|
||||
calibration: Calibration | None = None) -> dict[str, int]:
|
||||
"""Two passes: the cosines of the roles this agent scores, to its intent
|
||||
and to its taste direction, then every score with both scaled across
|
||||
them. A gated role is left out of the scaling, so a role that fails a
|
||||
and to its taste direction, then every score with both scaled by the
|
||||
model's `calibration`, or across the run without one. A gated role is
|
||||
left out of the cosines, so under the fallback a role that fails a
|
||||
require gate cannot stretch the range the ranked roles share. The voted
|
||||
roles are scored like any other; a liked one sits near the direction it
|
||||
helped make, which is what a like should do."""
|
||||
@@ -251,7 +278,8 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
cosines[candidate.uid] = cosine(role_vec, intent)
|
||||
if direction is not None:
|
||||
tastes[candidate.uid] = cosine(role_vec, direction)
|
||||
fits, tastes = scale(cosines), scale(tastes)
|
||||
fits = scale(cosines, calibration.fit if calibration else None)
|
||||
tastes = scale(tastes, calibration.taste if calibration else None)
|
||||
rows: list[tuple[str, float, str]] = []
|
||||
gated = 0
|
||||
for candidate in candidates:
|
||||
@@ -274,11 +302,19 @@ def run_score(config: Config, store: Store, *,
|
||||
vectors = store.vectors(model) if agents else {}
|
||||
encoder = _Encoder(model, context)
|
||||
now = store.clock()
|
||||
calibration = config.embed.calibrated()
|
||||
if agents and calibration is None:
|
||||
log.warning("no calibration for %s: fit and taste are min-max scaled across "
|
||||
"each agent's roles in this run, so they rank rather than state a "
|
||||
"level; add it under embed.calibration", model)
|
||||
if blind := [agent.name for agent in agents if not agent.intent.strip()]:
|
||||
context.notice(no_intent(blind))
|
||||
scored = gated = 0
|
||||
for n, agent in enumerate(agents):
|
||||
context.check()
|
||||
context.progress(n, len(agents), f"scoring for {agent.name}, {n + 1} of {len(agents)}")
|
||||
counts = score_agent(store, agent, vectors, _intent(store, agent, model, encoder), now)
|
||||
counts = score_agent(store, agent, vectors, _intent(store, agent, model, encoder), now,
|
||||
calibration)
|
||||
scored += counts["scored"]
|
||||
gated += counts["gated"]
|
||||
log.info("%-20s %6d scored, %6d gated", agent.name, counts["scored"], counts["gated"])
|
||||
|
||||
Reference in new issue
Block a user