Calibrate fit and taste per embedding model instead of min-max per run

scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants
from CALIBRATION in app/config.py, keyed by model and replaceable under
embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19
(taste from synthetic votes). An uncalibrated model falls back to min-max
and logs a warning once per run.

The extract and summarise bars default to 0.55 and 0.65, about 300 and 35
roles for the starter agent on the measured data; stored limits are kept.
An agent without an intent is left out of the model-stage scope, and each
score run names such agents in one score.no_intent notice.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 14:59:34 +02:00
1 parent 706b127bc0
commit b021838959
12 files changed
+336 -76

No files matched your search

+58 -22
View File
@@ -14,13 +14,15 @@ A gate rejects on evidence, never on absence: a role whose column is empty
is not gated by it. `extract` is what fills an empty column, and it reads
only the roles that rank, so gating on an empty column would keep those
roles out of extract's reach for good.
fit cos(role vector, intent vector), min-max scaled across
the roles this agent scores in this run: the closest is 1,
the farthest 0, a lone role 1
fit cos(role vector, intent vector) as a level: the embedding
model's calibrated `lo` is 0 and `hi` 1, clipped outside
(see `scale`), so a role's fit does not depend on what
else is open
recency 0.5 ** (days since posted_at / 14)
affinity (the agent's affinity for the company + 1) / 3
taste cos(role vector, liked direction), min-max scaled like fit,
where the direction is mean(liked vectors) - mean(passed vectors)
taste cos(role vector, liked direction), scaled like fit with the
model's taste calibration, where the direction is
mean(liked vectors) - mean(passed vectors)
over this agent's votes; with only likes it is the liked mean,
with only passes the negated passed mean; no votes, no term
@@ -30,7 +32,10 @@ company (`affinity = 2`) skips the gates and is floored at 0.9; a hidden one
(`-1`) is not scored at all. A role with no vector is scored without `fit`
and `taste`, unless `fit` is all there is, when it is gated as "no vector".
An agent with no votes scores every role without `taste`; the direction is
newsdesk's `centroid(liked) - centroid(disliked)`.
newsdesk's `centroid(liked) - centroid(disliked)`. An agent with no intent
has no `fit` either: its scores are not a match, so it is left out of what
extract and summarise read (`Store._top_scope`), and a run names such agents
in one notice.
Recomputed in full on every run, and each agent's rows are replaced in one
transaction: nothing is incremental, so nothing can drift.
@@ -47,8 +52,8 @@ from typing import Any
import numpy as np
from pipeline import NULL, RunContext, job
from app.config import Config
from pipeline import NULL, Notice, RunContext, job
from app.config import Calibration, Config
from app.jobs import embed
from app.models import Agent, Candidate, Gate, Weights
from app.storage import Store
@@ -110,19 +115,28 @@ def taste_direction(vectors: dict[str, np.ndarray],
return direction
def scale(cosines: dict[str, float]) -> dict[str, float]:
"""Min-max to 0..1 across one agent's run, so `fit` ranks roles rather
than stating a level.
def scale(cosines: dict[str, float],
band: tuple[float, float] | None) -> dict[str, float]:
"""Cosines to 0..1 levels: `clip((cos - lo) / (hi - lo), 0, 1)` with the
embedding model's calibrated `band` (`app.config.CALIBRATION`).
`(cos + 1) / 2` was the level, and on multilingual-e5-small it spans about
0.87 to 0.95 for real postings against a real intent (0.55 to 0.79 on the
all-MiniLM-L6-v2 this replaced), so a prefer gate weighted 0.2 moved a
score more than the whole spread of fit did either way. Scaled, the
closest role is 1 and the farthest 0. A single role, or roles that are all
equally close, get 1: there is nothing to tell them apart by.
Raw cosines on multilingual-e5-small sit between about 0.75 and 0.87 for
real postings against a real intent, so unscaled a prefer gate weighted
0.2 moved a score more than the whole spread of fit. The band stretches
that spread to 0..1 with fixed constants, so a role's level is the same
in every run and a `limits` bar is an absolute threshold.
Without a band, for a model nobody calibrated, it falls back to min-max
across the run: the closest role is 1 and the farthest 0, which ranks
rather than states a level. A single role, or roles that are all equally
close, then get 1: there is nothing to tell them apart by.
"""
if not cosines:
return {}
if band is not None:
lo, hi = band
return {uid: min(max((value - lo) / (hi - lo), 0.0), 1.0)
for uid, value in cosines.items()}
low, high = min(cosines.values()), max(cosines.values())
if high - low < 1e-9:
return dict.fromkeys(cosines, 1.0)
@@ -231,11 +245,24 @@ def _intent(store: Store, agent: Agent, model: str,
# ------------------------------------------------------------------- the job
def no_intent(names: Sequence[str]) -> Notice:
"""The notice for enabled agents without an intent: their scores have no
fit, so `Store._top_scope` keeps them from feeding extract and summarise."""
return Notice(
code="score.no_intent", level="warning", kind="expected", key="score",
text=f"No intent, so ranked without fit and not read by extract or "
f"summarise: {', '.join(names)}.",
fix="Give the agent an intent under Settings, Search agents.",
where="app.jobs.score")
def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
intent: np.ndarray | None, now: datetime) -> dict[str, int]:
intent: np.ndarray | None, now: datetime,
calibration: Calibration | None = None) -> dict[str, int]:
"""Two passes: the cosines of the roles this agent scores, to its intent
and to its taste direction, then every score with both scaled across
them. A gated role is left out of the scaling, so a role that fails a
and to its taste direction, then every score with both scaled by the
model's `calibration`, or across the run without one. A gated role is
left out of the cosines, so under the fallback a role that fails a
require gate cannot stretch the range the ranked roles share. The voted
roles are scored like any other; a liked one sits near the direction it
helped make, which is what a like should do."""
@@ -251,7 +278,8 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
cosines[candidate.uid] = cosine(role_vec, intent)
if direction is not None:
tastes[candidate.uid] = cosine(role_vec, direction)
fits, tastes = scale(cosines), scale(tastes)
fits = scale(cosines, calibration.fit if calibration else None)
tastes = scale(tastes, calibration.taste if calibration else None)
rows: list[tuple[str, float, str]] = []
gated = 0
for candidate in candidates:
@@ -274,11 +302,19 @@ def run_score(config: Config, store: Store, *,
vectors = store.vectors(model) if agents else {}
encoder = _Encoder(model, context)
now = store.clock()
calibration = config.embed.calibrated()
if agents and calibration is None:
log.warning("no calibration for %s: fit and taste are min-max scaled across "
"each agent's roles in this run, so they rank rather than state a "
"level; add it under embed.calibration", model)
if blind := [agent.name for agent in agents if not agent.intent.strip()]:
context.notice(no_intent(blind))
scored = gated = 0
for n, agent in enumerate(agents):
context.check()
context.progress(n, len(agents), f"scoring for {agent.name}, {n + 1} of {len(agents)}")
counts = score_agent(store, agent, vectors, _intent(store, agent, model, encoder), now)
counts = score_agent(store, agent, vectors, _intent(store, agent, model, encoder), now,
calibration)
scored += counts["scored"]
gated += counts["gated"]
log.info("%-20s %6d scored, %6d gated", agent.name, counts["scored"], counts["gated"])