Switch the embed model to multilingual-e5-small for the European corpus
Measured on the live roles against the "Copenhagen engineering" agent's intent: the English-only all-MiniLM-L6-v2 separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles (a Danish engineering role beats the rest's non-engineering median only 68% of the time). multilingual-e5-small reaches 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. paraphrase-multilingual-MiniLM-L12-v2 measured worse than the English-only baseline (AUC 0.60) and was dropped. e5 needs "query: "/"passage: " prefixes; app/jobs/embed.py adds them by matching the model name so a model change can't leave a stale prefix setting behind. Added tests for the prefixing and for the two invariants the model change relies on: embed drops the old model's vectors before re-embedding, and an agent's cached intent vector is keyed by model so a model change is a cache miss, never a stale-dimension read. Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
1 parent
0558ef19ed
commit
b560cdce2a
7 files changed
+214
-15
No files matched your search
+14
-3
@@ -69,11 +69,22 @@ class SummariseConfig(ModelTaskConfig):
|
||||
class EmbedConfig(Section):
|
||||
"""Turning a role into a vector for `fit`. Needs sentence-transformers;
|
||||
without it `embed` says so and does nothing, and `score` leaves fit out.
|
||||
See app/jobs/embed.py."""
|
||||
See app/jobs/embed.py.
|
||||
|
||||
`intfloat/multilingual-e5-small`, not the English-only
|
||||
`sentence-transformers/all-MiniLM-L6-v2` roledesk started with: measured
|
||||
on the live roles against the "Copenhagen engineering" agent's intent,
|
||||
MiniLM separates engineering from non-engineering roles with an AUC of
|
||||
0.68 (0.74 restricted to Danish-country roles), multilingual-e5-small
|
||||
with an AUC of 0.70 (1.00 restricted to Danish-country roles), for about
|
||||
3x the CPU time. See docs/decisions.md.
|
||||
"""
|
||||
|
||||
#: Recorded on every vector. Changing it drops the old vectors, which are
|
||||
#: a different geometry and never comparable.
|
||||
model: str = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
#: a different geometry and never comparable. An e5 model needs its
|
||||
#: "query: " / "passage: " prefixes, which app/jobs/embed.py adds by
|
||||
#: matching this name, not a separate setting.
|
||||
model: str = "intfloat/multilingual-e5-small"
|
||||
#: Roles embedded per run at most, newest first.
|
||||
per_run: int = Field(50_000, ge=1)
|
||||
|
||||
|
||||
+40
-1
@@ -7,11 +7,21 @@ the description, which is what says what a role is.
|
||||
sentence-transformers is optional, as in newsdesk. Without it this task puts
|
||||
one notice on screen and does nothing; `score` then leaves `fit` out and ranks
|
||||
on what it has.
|
||||
|
||||
The default, `intfloat/multilingual-e5-small`, was picked over the English-only
|
||||
MiniLM once the corpus turned European: measured on the live roles against the
|
||||
"Copenhagen engineering" agent's intent, MiniLM separated engineering from
|
||||
non-engineering roles with an AUC of 0.68 (0.74 restricted to Danish-country
|
||||
roles), multilingual-e5-small with an AUC of 0.70 (1.00 restricted to
|
||||
Danish-country roles), for about 3x the CPU time. `paraphrase-multilingual-
|
||||
MiniLM-L12-v2`, the other multilingual candidate measured, scored *worse*
|
||||
than plain MiniLM (AUC 0.60) and was not chosen.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from desk.tools import embedding
|
||||
@@ -29,11 +39,40 @@ BODY_CHARS = 2000
|
||||
#: Roles encoded per call, and per commit: a kill loses at most one chunk.
|
||||
CHUNK = 64
|
||||
|
||||
#: intfloat's e5 family is trained with a "query: " / "passage: " prefix baked
|
||||
#: into the pretraining objective; encoding without it measurably hurts
|
||||
#: retrieval. Matched against the model id itself, not a second config field,
|
||||
#: so switching `embed.model` to an e5 variant can't leave a stale prefix
|
||||
#: setting behind it -- the one place that says which model to use is also
|
||||
#: the one place that decides whether it wants a prefix. Every other model
|
||||
#: (MiniLM and friends) has no such convention, so the match is narrow: the
|
||||
#: model id's own name must contain "e5" set off by "-", "_" or "/".
|
||||
_E5_NAME = re.compile(r"(?:^|[-_/])e5(?:[-_]|$)", re.IGNORECASE)
|
||||
|
||||
|
||||
def wants_e5_prefix(model: str) -> bool:
|
||||
return bool(_E5_NAME.search(model))
|
||||
|
||||
|
||||
def document(role: Role) -> str:
|
||||
return f"{role.title}\n{role.description[:BODY_CHARS]}"
|
||||
|
||||
|
||||
def role_text(model: str, role: Role) -> str:
|
||||
"""What gets encoded for a role: `document(role)`, prefixed "passage: "
|
||||
for an e5 model since that is the side e5 calls the passage."""
|
||||
text = document(role)
|
||||
return f"passage: {text}" if wants_e5_prefix(model) else text
|
||||
|
||||
|
||||
def intent_text(model: str, intent: str) -> str:
|
||||
"""What gets encoded for an agent's intent: prefixed "query: " for an e5
|
||||
model, since the intent is what a role is matched against, e5's "query"
|
||||
side. `score._intent` calls this so the cached vector and a freshly
|
||||
encoded one are always the same text for a given model."""
|
||||
return f"query: {intent}" if wants_e5_prefix(model) else intent
|
||||
|
||||
|
||||
def load_embedder(model: str) -> embedding.Embedder:
|
||||
"""The encoder. A seam of its own so a test hands in a fake and nothing
|
||||
downloads a model; `score` loads through it too."""
|
||||
@@ -72,7 +111,7 @@ def run_embed(config: Config, store: Store, *,
|
||||
context.check()
|
||||
context.progress(at, len(roles), f"embedding roles, {at} of {len(roles)}")
|
||||
chunk = roles[at:at + CHUNK]
|
||||
vectors = embedder.encode([document(r) for r in chunk])
|
||||
vectors = embedder.encode([role_text(model, r) for r in chunk])
|
||||
store.put_vectors(model, [(r.uid, v) for r, v in zip(chunk, vectors, strict=True)])
|
||||
context.progress(len(roles), len(roles),
|
||||
f"embedding roles, {len(roles)} of {len(roles)}")
|
||||
|
||||
+7
-6
@@ -111,11 +111,12 @@ def scale(cosines: dict[str, float]) -> dict[str, float]:
|
||||
"""Min-max to 0..1 across one agent's run, so `fit` ranks roles rather
|
||||
than stating a level.
|
||||
|
||||
`(cos + 1) / 2` was the level, and on MiniLM it spans about 0.55 to 0.79
|
||||
for real postings against a real intent, so a prefer gate weighted 0.2
|
||||
moved a score more than the whole spread of fit did. Scaled, the closest
|
||||
role is 1 and the farthest 0. A single role, or roles that are all equally
|
||||
close, get 1: there is nothing to tell them apart by.
|
||||
`(cos + 1) / 2` was the level, and on multilingual-e5-small it spans about
|
||||
0.87 to 0.95 for real postings against a real intent (0.55 to 0.79 on the
|
||||
all-MiniLM-L6-v2 this replaced), so a prefer gate weighted 0.2 moved a
|
||||
score more than the whole spread of fit did either way. Scaled, the
|
||||
closest role is 1 and the farthest 0. A single role, or roles that are all
|
||||
equally close, get 1: there is nothing to tell them apart by.
|
||||
"""
|
||||
if not cosines:
|
||||
return {}
|
||||
@@ -219,7 +220,7 @@ def _intent(store: Store, agent: Agent, model: str,
|
||||
cached = store.intent_vector(agent.id, key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
vec = encoder.encode(agent.intent)
|
||||
vec = encoder.encode(embed.intent_text(model, agent.intent))
|
||||
if vec is not None:
|
||||
store.set_intent_vector(agent.id, key, vec)
|
||||
return vec
|
||||
|
||||
Reference in new issue
Block a user