Switch the embed model to multilingual-e5-small for the European corpus
Measured on the live roles against the "Copenhagen engineering" agent's intent: the English-only all-MiniLM-L6-v2 separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles (a Danish engineering role beats the rest's non-engineering median only 68% of the time). multilingual-e5-small reaches 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. paraphrase-multilingual-MiniLM-L12-v2 measured worse than the English-only baseline (AUC 0.60) and was dropped. e5 needs "query: "/"passage: " prefixes; app/jobs/embed.py adds them by matching the model name so a model change can't leave a stale prefix setting behind. Added tests for the prefixing and for the two invariants the model change relies on: embed drops the old model's vectors before re-embedding, and an agent's cached intent vector is keyed by model so a model change is a cache miss, never a stale-dimension read. Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
1 parent
0558ef19ed
commit
b560cdce2a
7 files changed
+214
-15
No files matched your search
+14
-3
@@ -69,11 +69,22 @@ class SummariseConfig(ModelTaskConfig):
|
||||
class EmbedConfig(Section):
|
||||
"""Turning a role into a vector for `fit`. Needs sentence-transformers;
|
||||
without it `embed` says so and does nothing, and `score` leaves fit out.
|
||||
See app/jobs/embed.py."""
|
||||
See app/jobs/embed.py.
|
||||
|
||||
`intfloat/multilingual-e5-small`, not the English-only
|
||||
`sentence-transformers/all-MiniLM-L6-v2` roledesk started with: measured
|
||||
on the live roles against the "Copenhagen engineering" agent's intent,
|
||||
MiniLM separates engineering from non-engineering roles with an AUC of
|
||||
0.68 (0.74 restricted to Danish-country roles), multilingual-e5-small
|
||||
with an AUC of 0.70 (1.00 restricted to Danish-country roles), for about
|
||||
3x the CPU time. See docs/decisions.md.
|
||||
"""
|
||||
|
||||
#: Recorded on every vector. Changing it drops the old vectors, which are
|
||||
#: a different geometry and never comparable.
|
||||
model: str = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
#: a different geometry and never comparable. An e5 model needs its
|
||||
#: "query: " / "passage: " prefixes, which app/jobs/embed.py adds by
|
||||
#: matching this name, not a separate setting.
|
||||
model: str = "intfloat/multilingual-e5-small"
|
||||
#: Roles embedded per run at most, newest first.
|
||||
per_run: int = Field(50_000, ge=1)
|
||||
|
||||
|
||||
Reference in new issue
Block a user