Switch the embed model to multilingual-e5-small for the European corpus
Measured on the live roles against the "Copenhagen engineering" agent's intent: the English-only all-MiniLM-L6-v2 separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles (a Danish engineering role beats the rest's non-engineering median only 68% of the time). multilingual-e5-small reaches 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. paraphrase-multilingual-MiniLM-L12-v2 measured worse than the English-only baseline (AUC 0.60) and was dropped. e5 needs "query: "/"passage: " prefixes; app/jobs/embed.py adds them by matching the model name so a model change can't leave a stale prefix setting behind. Added tests for the prefixing and for the two invariants the model change relies on: embed drops the old model's vectors before re-embedding, and an agent's cached intent vector is keyed by model so a model change is a cache miss, never a stale-dimension read. Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
1 parent
0558ef19ed
commit
b560cdce2a
7 files changed
+214
-15
No files matched your search
+7
-6
@@ -111,11 +111,12 @@ def scale(cosines: dict[str, float]) -> dict[str, float]:
|
||||
"""Min-max to 0..1 across one agent's run, so `fit` ranks roles rather
|
||||
than stating a level.
|
||||
|
||||
`(cos + 1) / 2` was the level, and on MiniLM it spans about 0.55 to 0.79
|
||||
for real postings against a real intent, so a prefer gate weighted 0.2
|
||||
moved a score more than the whole spread of fit did. Scaled, the closest
|
||||
role is 1 and the farthest 0. A single role, or roles that are all equally
|
||||
close, get 1: there is nothing to tell them apart by.
|
||||
`(cos + 1) / 2` was the level, and on multilingual-e5-small it spans about
|
||||
0.87 to 0.95 for real postings against a real intent (0.55 to 0.79 on the
|
||||
all-MiniLM-L6-v2 this replaced), so a prefer gate weighted 0.2 moved a
|
||||
score more than the whole spread of fit did either way. Scaled, the
|
||||
closest role is 1 and the farthest 0. A single role, or roles that are all
|
||||
equally close, get 1: there is nothing to tell them apart by.
|
||||
"""
|
||||
if not cosines:
|
||||
return {}
|
||||
@@ -219,7 +220,7 @@ def _intent(store: Store, agent: Agent, model: str,
|
||||
cached = store.intent_vector(agent.id, key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
vec = encoder.encode(agent.intent)
|
||||
vec = encoder.encode(embed.intent_text(model, agent.intent))
|
||||
if vec is not None:
|
||||
store.set_intent_vector(agent.id, key, vec)
|
||||
return vec
|
||||
|
||||
Reference in new issue
Block a user