Score each role's closeness to the agent's CV, let the judge also read the 50 closest the score leaves out, and order CV match's unjudged roles by it

The CV is cut into chunks inside e5's window, embedded like a prompt and
cached per agent under a hash of the model, chunking and text. A role's
cv_fit (scores.cv_fit, migration 31) is its best chunk cosine, scaled with
its own calibration band: fit's clipped 88 of agent 2's ungated roles to 1.
It is no term of the Agent match score. judge_cv_top (default 50) sits
beside judge_top and judge_min on the agents page and card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-10-03 23:35:21 +02:00
1 parent b481cb5692
commit c23594507a
23 files changed
+603 -85

No files matched your search

+25 -9
View File
@@ -72,8 +72,9 @@ class JudgeConfig(ModelTaskConfig):
"""Each search agent's best roles read against its CV, as three numbers
and a total. See app/jobs/judge.py. Off by default like extract; on, a
pull judges the best of each enabled agent with a CV, as deep and as low
as that agent's own `limits` say (`judge_top`, `judge_min`), and only
the roles whose judgement is missing or was read from another CV or
as that agent's own `limits` say (`judge_top`, `judge_min`), and the
`judge_cv_top` ungated ones closest to its CV that those leave out, and
only the roles whose judgement is missing or was read from another CV or
text."""
enabled: bool = False
@@ -103,21 +104,30 @@ class EnrichConfig(Section):
class Calibration(Section):
"""Where one embedding model's cosines sit, as `(lo, hi)` for fit and for
taste: `score.scale` maps `lo` to 0 and `hi` to 1 and clips outside, so a
term is a level rather than a rank within the run."""
"""Where one embedding model's cosines sit, as `(lo, hi)` for fit, for
taste and for a role's closeness to a CV (`cv`, fit's when not given):
`score.scale` maps `lo` to 0 and `hi` to 1 and clips outside, so a term
is a level rather than a rank within the run."""
fit: tuple[float, float]
taste: tuple[float, float]
cv: tuple[float, float] | None = None
@field_validator("fit", "taste")
@field_validator("fit", "taste", "cv")
@classmethod
def _lo_below_hi(cls, band: tuple[float, float]) -> tuple[float, float]:
def _lo_below_hi(cls, band: tuple[float, float] | None) -> tuple[float, float] | None:
if band is None:
return band
lo, hi = band
if not -1 <= lo < hi <= 1:
raise ValueError(f"needs -1 <= lo < hi <= 1, got [{lo}, {hi}]")
return band
@property
def cv_band(self) -> tuple[float, float]:
"""The band `cv_fit` is scaled with: `cv`, else fit's."""
return self.cv or self.fit
#: The built-in calibration, by embedding model. Measured on a copy of the
#: live database, 16,841 open roles (see docs/decisions.md). fit: pooled over
@@ -125,9 +135,15 @@ class Calibration(Section):
#: every single intent's p99.9, so only about 5 of the starter agent's 4,156
#: ungated roles clip to 1. taste: from synthetic votes, 30 liked and 30
#: passed roles for each of three intents, since the copy had none (pooled p1
#: -0.140, p99.9 0.192); re-measure it once real votes exist.
#: -0.140, p99.9 0.192); re-measure it once real votes exist. cv: the best
#: cosine over the chunks of the one CV in the copy of 2026-10-03, over that
#: agent's 7,270 ungated roles: p1 0.770, median 0.832, p99.9 0.891, max
#: 0.902. A CV chunk is a longer text than a prompt and sits closer to every
#: posting, so fit's band clipped 88 of them to 1 and the 50 closest tied;
#: with this one, 1 clips. Re-measure it once more agents hold a CV.
CALIBRATION: dict[str, Calibration] = {
"intfloat/multilingual-e5-small": Calibration(fit=(0.75, 0.88), taste=(-0.14, 0.19)),
"intfloat/multilingual-e5-small": Calibration(fit=(0.75, 0.88), taste=(-0.14, 0.19),
cv=(0.77, 0.90)),
}