Calibrate fit and taste per embedding model instead of min-max per run

scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants
from CALIBRATION in app/config.py, keyed by model and replaceable under
embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19
(taste from synthetic votes). An uncalibrated model falls back to min-max
and logs a warning once per run.

The extract and summarise bars default to 0.55 and 0.65, about 300 and 35
roles for the starter agent on the measured data; stored limits are kept.
An agent without an intent is left out of the model-stage scope, and each
score run names such agents in one score.no_intent notice.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 14:59:34 +02:00
1 parent 706b127bc0
commit b021838959
12 files changed
+336 -76

No files matched your search

+16 -6
View File
@@ -74,9 +74,16 @@ summarise:
# Danish-country roles (see docs/decisions.md), for about 3x the CPU time. An
# e5 model's "query: " / "passage: " prefixes are added automatically, keyed
# off this name.
# calibration maps a model's cosines to 0..1 levels, as [lo, hi] for fit and
# for taste: lo is 0, hi is 1, clipped outside. multilingual-e5-small has
# built-in constants (app/config.py, CALIBRATION); an entry here replaces
# them for that model. A model with none is min-max scaled per run instead,
# so its fit is a rank, not a level, and the log says so.
embed:
model: intfloat/multilingual-e5-small
per_run: 50000
calibration: {}
# intfloat/multilingual-e5-small: {fit: [0.75, 0.88], taste: [-0.14, 0.19]}
# One block per plugin, keyed by the plugin's name. The ats and search plugins
# read their boards and searches from the subscriptions table, so they need
@@ -99,11 +106,14 @@ sources: {}
# limits is how deep into this agent's ranking the two model stages read,
# each a top N (the backstop) and a minimum score (the real stop): extract
# reads the best extract_top roles at extract_min or more, summarise the best
# summarise_top at summarise_min or more. A fresh role with the worst fit
# scores about 0.27 on the default weights and a good one 0.6 or more. The
# defaults are {extract_top: 500, extract_min: 0.3, summarise_top: 30,
# summarise_min: 0.5}; a wide monitoring search raises them, a narrow hunt
# lowers them.
# summarise_top at summarise_min or more. Fit is a calibrated level, so a bar
# means the same in every run: a fresh role with the worst fit scores about
# 0.27 on the default weights, a perfect one about 0.87. The defaults are
# {extract_top: 500, extract_min: 0.55, summarise_top: 30, summarise_min:
# 0.65}; measured for "Copenhagen engineering" on 16,841 open roles, 0.55
# admits about 300 roles to extract and 0.65 about 35 to summarise. A wide
# monitoring search lowers the bars and raises the counts, a narrow hunt the
# reverse. An agent without an intent has no fit and feeds neither stage.
# The Search agents sub-tab lists and edits them; the order menu on Roles
# picks one by name. Two more, beside the starter in config.yaml:
agents: []
@@ -117,7 +127,7 @@ agents: []
# - {field: country, values: [AT, BE, BG, HR, CY, CZ, DK, EE, FI, FR, DE, GR, HU, IE, IT,
# LV, LT, LU, MT, NL, PL, PT, RO, SK, SI, ES, SE, IS, LI, NO,
# GB, CH, EU], mode: require}
# limits: {extract_top: 1000, extract_min: 0.3, summarise_top: 50, summarise_min: 0.5}
# limits: {extract_top: 1000, extract_min: 0.45, summarise_top: 50, summarise_min: 0.6}
# - name: Europe hybrid
# intent: >-
# Senior backend or platform engineer in a European tech hub, hybrid or