Calibrate fit and taste per embedding model instead of min-max per run

scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants
from CALIBRATION in app/config.py, keyed by model and replaceable under
embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19
(taste from synthetic votes). An uncalibrated model falls back to min-max
and logs a warning once per run.

The extract and summarise bars default to 0.55 and 0.65, about 300 and 35
roles for the starter agent on the measured data; stored limits are kept.
An agent without an intent is left out of the model-stage scope, and each
score run names such agents in one score.no_intent notice.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 14:59:34 +02:00
1 parent 706b127bc0
commit b021838959
12 files changed
+336 -76

No files matched your search

+11 -6
View File
@@ -499,15 +499,20 @@ class Limits(Model):
"""How far down an agent's ranking the two model stages read. Each pair
is a top N, the backstop, and a minimum score, the real stop: a role
is read when it is among the agent's best `extract_top` ungated scores
and not below `extract_min`, and summarised likewise. A fresh role with
the worst fit scores about 0.27 on the default weights, a good one 0.6
or more, so extract's floor is loose and summarise's, the long answer,
stricter."""
and not below `extract_min`, and summarised likewise. Fit is a
calibrated level (`score.scale`), so a bar means the same in every run.
A fresh role with the worst fit scores about 0.27 on the default
weights, a perfect one about 0.87. Measured on 16,841 open roles for the
"Copenhagen engineering" starter agent (4,156 ungated, no votes), 0.55
admits about 300 roles to extract and 0.65 about 35 to summarise, the
long answer; the top N catches an agent whose bar admits far more. An
agent keeps the limits stored with it; only a key its JSON lacks
follows these defaults."""
extract_top: int = Field(500, ge=0)
extract_min: float = Field(0.3, ge=0, le=1)
extract_min: float = Field(0.55, ge=0, le=1)
summarise_top: int = Field(30, ge=0)
summarise_min: float = Field(0.5, ge=0, le=1)
summarise_min: float = Field(0.65, ge=0, le=1)
class Agent(Model):