Calibrate fit and taste per embedding model instead of min-max per run
scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants from CALIBRATION in app/config.py, keyed by model and replaceable under embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19 (taste from synthetic votes). An uncalibrated model falls back to min-max and logs a warning once per run. The extract and summarise bars default to 0.55 and 0.65, about 300 and 35 roles for the starter agent on the measured data; stored limits are kept. An agent without an intent is left out of the model-stage scope, and each score run names such agents in one score.no_intent notice. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
706b127bc0
commit
b021838959
12 files changed
+336
-76
No files matched your search
+11
-6
@@ -499,15 +499,20 @@ class Limits(Model):
|
||||
"""How far down an agent's ranking the two model stages read. Each pair
|
||||
is a top N, the backstop, and a minimum score, the real stop: a role
|
||||
is read when it is among the agent's best `extract_top` ungated scores
|
||||
and not below `extract_min`, and summarised likewise. A fresh role with
|
||||
the worst fit scores about 0.27 on the default weights, a good one 0.6
|
||||
or more, so extract's floor is loose and summarise's, the long answer,
|
||||
stricter."""
|
||||
and not below `extract_min`, and summarised likewise. Fit is a
|
||||
calibrated level (`score.scale`), so a bar means the same in every run.
|
||||
A fresh role with the worst fit scores about 0.27 on the default
|
||||
weights, a perfect one about 0.87. Measured on 16,841 open roles for the
|
||||
"Copenhagen engineering" starter agent (4,156 ungated, no votes), 0.55
|
||||
admits about 300 roles to extract and 0.65 about 35 to summarise, the
|
||||
long answer; the top N catches an agent whose bar admits far more. An
|
||||
agent keeps the limits stored with it; only a key its JSON lacks
|
||||
follows these defaults."""
|
||||
|
||||
extract_top: int = Field(500, ge=0)
|
||||
extract_min: float = Field(0.3, ge=0, le=1)
|
||||
extract_min: float = Field(0.55, ge=0, le=1)
|
||||
summarise_top: int = Field(30, ge=0)
|
||||
summarise_min: float = Field(0.5, ge=0, le=1)
|
||||
summarise_min: float = Field(0.65, ge=0, le=1)
|
||||
|
||||
|
||||
class Agent(Model):
|
||||
|
||||
Reference in new issue
Block a user