Calibrate fit and taste per embedding model instead of min-max per run
scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants from CALIBRATION in app/config.py, keyed by model and replaceable under embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19 (taste from synthetic votes). An uncalibrated model falls back to min-max and logs a warning once per run. The extract and summarise bars default to 0.55 and 0.65, about 300 and 35 roles for the starter agent on the measured data; stored limits are kept. An agent without an intent is left out of the model-stage scope, and each score run names such agents in one score.no_intent notice. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
706b127bc0
commit
b021838959
12 files changed
+336
-76
No files matched your search
+16
-6
@@ -74,9 +74,16 @@ summarise:
|
||||
# Danish-country roles (see docs/decisions.md), for about 3x the CPU time. An
|
||||
# e5 model's "query: " / "passage: " prefixes are added automatically, keyed
|
||||
# off this name.
|
||||
# calibration maps a model's cosines to 0..1 levels, as [lo, hi] for fit and
|
||||
# for taste: lo is 0, hi is 1, clipped outside. multilingual-e5-small has
|
||||
# built-in constants (app/config.py, CALIBRATION); an entry here replaces
|
||||
# them for that model. A model with none is min-max scaled per run instead,
|
||||
# so its fit is a rank, not a level, and the log says so.
|
||||
embed:
|
||||
model: intfloat/multilingual-e5-small
|
||||
per_run: 50000
|
||||
calibration: {}
|
||||
# intfloat/multilingual-e5-small: {fit: [0.75, 0.88], taste: [-0.14, 0.19]}
|
||||
|
||||
# One block per plugin, keyed by the plugin's name. The ats and search plugins
|
||||
# read their boards and searches from the subscriptions table, so they need
|
||||
@@ -99,11 +106,14 @@ sources: {}
|
||||
# limits is how deep into this agent's ranking the two model stages read,
|
||||
# each a top N (the backstop) and a minimum score (the real stop): extract
|
||||
# reads the best extract_top roles at extract_min or more, summarise the best
|
||||
# summarise_top at summarise_min or more. A fresh role with the worst fit
|
||||
# scores about 0.27 on the default weights and a good one 0.6 or more. The
|
||||
# defaults are {extract_top: 500, extract_min: 0.3, summarise_top: 30,
|
||||
# summarise_min: 0.5}; a wide monitoring search raises them, a narrow hunt
|
||||
# lowers them.
|
||||
# summarise_top at summarise_min or more. Fit is a calibrated level, so a bar
|
||||
# means the same in every run: a fresh role with the worst fit scores about
|
||||
# 0.27 on the default weights, a perfect one about 0.87. The defaults are
|
||||
# {extract_top: 500, extract_min: 0.55, summarise_top: 30, summarise_min:
|
||||
# 0.65}; measured for "Copenhagen engineering" on 16,841 open roles, 0.55
|
||||
# admits about 300 roles to extract and 0.65 about 35 to summarise. A wide
|
||||
# monitoring search lowers the bars and raises the counts, a narrow hunt the
|
||||
# reverse. An agent without an intent has no fit and feeds neither stage.
|
||||
# The Search agents sub-tab lists and edits them; the order menu on Roles
|
||||
# picks one by name. Two more, beside the starter in config.yaml:
|
||||
agents: []
|
||||
@@ -117,7 +127,7 @@ agents: []
|
||||
# - {field: country, values: [AT, BE, BG, HR, CY, CZ, DK, EE, FI, FR, DE, GR, HU, IE, IT,
|
||||
# LV, LT, LU, MT, NL, PL, PT, RO, SK, SI, ES, SE, IS, LI, NO,
|
||||
# GB, CH, EU], mode: require}
|
||||
# limits: {extract_top: 1000, extract_min: 0.3, summarise_top: 50, summarise_min: 0.5}
|
||||
# limits: {extract_top: 1000, extract_min: 0.45, summarise_top: 50, summarise_min: 0.6}
|
||||
# - name: Europe hybrid
|
||||
# intent: >-
|
||||
# Senior backend or platform engineer in a European tech hub, hybrid or
|
||||
|
||||
Reference in new issue
Block a user