Score each role's closeness to the agent's CV, let the judge also read the 50 closest the score leaves out, and order CV match's unjudged roles by it
The CV is cut into chunks inside e5's window, embedded like a prompt and cached per agent under a hash of the model, chunking and text. A role's cv_fit (scores.cv_fit, migration 31) is its best chunk cosine, scaled with its own calibration band: fit's clipped 88 of agent 2's ungated roles to 1. It is no term of the Agent match score. judge_cv_top (default 50) sits beside judge_top and judge_min on the agents page and card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b481cb5692
commit
c23594507a
23 files changed
+603
-85
No files matched your search
+25
-9
@@ -72,8 +72,9 @@ class JudgeConfig(ModelTaskConfig):
|
||||
"""Each search agent's best roles read against its CV, as three numbers
|
||||
and a total. See app/jobs/judge.py. Off by default like extract; on, a
|
||||
pull judges the best of each enabled agent with a CV, as deep and as low
|
||||
as that agent's own `limits` say (`judge_top`, `judge_min`), and only
|
||||
the roles whose judgement is missing or was read from another CV or
|
||||
as that agent's own `limits` say (`judge_top`, `judge_min`), and the
|
||||
`judge_cv_top` ungated ones closest to its CV that those leave out, and
|
||||
only the roles whose judgement is missing or was read from another CV or
|
||||
text."""
|
||||
|
||||
enabled: bool = False
|
||||
@@ -103,21 +104,30 @@ class EnrichConfig(Section):
|
||||
|
||||
|
||||
class Calibration(Section):
|
||||
"""Where one embedding model's cosines sit, as `(lo, hi)` for fit and for
|
||||
taste: `score.scale` maps `lo` to 0 and `hi` to 1 and clips outside, so a
|
||||
term is a level rather than a rank within the run."""
|
||||
"""Where one embedding model's cosines sit, as `(lo, hi)` for fit, for
|
||||
taste and for a role's closeness to a CV (`cv`, fit's when not given):
|
||||
`score.scale` maps `lo` to 0 and `hi` to 1 and clips outside, so a term
|
||||
is a level rather than a rank within the run."""
|
||||
|
||||
fit: tuple[float, float]
|
||||
taste: tuple[float, float]
|
||||
cv: tuple[float, float] | None = None
|
||||
|
||||
@field_validator("fit", "taste")
|
||||
@field_validator("fit", "taste", "cv")
|
||||
@classmethod
|
||||
def _lo_below_hi(cls, band: tuple[float, float]) -> tuple[float, float]:
|
||||
def _lo_below_hi(cls, band: tuple[float, float] | None) -> tuple[float, float] | None:
|
||||
if band is None:
|
||||
return band
|
||||
lo, hi = band
|
||||
if not -1 <= lo < hi <= 1:
|
||||
raise ValueError(f"needs -1 <= lo < hi <= 1, got [{lo}, {hi}]")
|
||||
return band
|
||||
|
||||
@property
|
||||
def cv_band(self) -> tuple[float, float]:
|
||||
"""The band `cv_fit` is scaled with: `cv`, else fit's."""
|
||||
return self.cv or self.fit
|
||||
|
||||
|
||||
#: The built-in calibration, by embedding model. Measured on a copy of the
|
||||
#: live database, 16,841 open roles (see docs/decisions.md). fit: pooled over
|
||||
@@ -125,9 +135,15 @@ class Calibration(Section):
|
||||
#: every single intent's p99.9, so only about 5 of the starter agent's 4,156
|
||||
#: ungated roles clip to 1. taste: from synthetic votes, 30 liked and 30
|
||||
#: passed roles for each of three intents, since the copy had none (pooled p1
|
||||
#: -0.140, p99.9 0.192); re-measure it once real votes exist.
|
||||
#: -0.140, p99.9 0.192); re-measure it once real votes exist. cv: the best
|
||||
#: cosine over the chunks of the one CV in the copy of 2026-10-03, over that
|
||||
#: agent's 7,270 ungated roles: p1 0.770, median 0.832, p99.9 0.891, max
|
||||
#: 0.902. A CV chunk is a longer text than a prompt and sits closer to every
|
||||
#: posting, so fit's band clipped 88 of them to 1 and the 50 closest tied;
|
||||
#: with this one, 1 clips. Re-measure it once more agents hold a CV.
|
||||
CALIBRATION: dict[str, Calibration] = {
|
||||
"intfloat/multilingual-e5-small": Calibration(fit=(0.75, 0.88), taste=(-0.14, 0.19)),
|
||||
"intfloat/multilingual-e5-small": Calibration(fit=(0.75, 0.88), taste=(-0.14, 0.19),
|
||||
cv=(0.77, 0.90)),
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user