Calibrate fit and taste per embedding model instead of min-max per run

scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants
from CALIBRATION in app/config.py, keyed by model and replaceable under
embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19
(taste from synthetic votes). An uncalibrated model falls back to min-max
and logs a warning once per run.

The extract and summarise bars default to 0.55 and 0.65, about 300 and 35
roles for the starter agent on the measured data; stored limits are kept.
An agent without an intent is left out of the model-stage scope, and each
score run names such agents in one score.no_intent notice.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 14:59:34 +02:00
1 parent 706b127bc0
commit b021838959
12 files changed
+336 -76

No files matched your search

+10 -4
View File
@@ -90,7 +90,13 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the gates and floors the score, `-1` mutes.
**`fit` is the cosine between a role and the agent's intent, min-max scaled across the roles that agent scores in the run.** `(cos + 1) / 2` spanned only about 0.55 to 0.79 in the real run, so a prefer gate at 0.2 outweighed all of fit. A single role gets 1, gated roles stay out of the range, and the starter agents' prefer weights are 0.05.
**`fit` is the cosine between a role and the agent's intent, mapped to a level by constants calibrated per embedding model: `clip((cos - lo) / (hi - lo), 0, 1)`.** `(cos + 1) / 2` spanned only about 0.87 to 0.95 on multilingual-e5-small, so a prefer gate at 0.2 outweighed all of fit. Min-max across the roles an agent scored in the run fixed the spread but made fit a rank: the best role of a poor set got 1, a role's fit and the 0.9 favourite floor's meaning drifted from run to run, and the `limits` bars that bound the paid stages compared an absolute number with a relative one. The constants are `CALIBRATION` in `app/config.py`, keyed by model name, and an entry under `embed.calibration` replaces a model's built-in one; keyed by model, a change of `embed.model` cannot carry another geometry's constants along. A model with none falls back to min-max per run, with gated roles left out of the range, and the log warns once per run. `taste` is mapped the same way with its own pair. The starter agents' prefer weights are 0.05.
**multilingual-e5-small's constants are fit 0.75 to 0.88 and taste -0.14 to 0.19, measured on a copy of the live database.** Over 16,841 open roles and five different intents the raw fit cosine pooled to p1 0.7528, p50 0.7989, p99 0.8528, p99.5 0.8595, p99.9 0.8711 and max 0.9031; per intent p1 ranged 0.7456 to 0.7700 and p99.9 0.8413 to 0.8773. `hi` is above every intent's p99.9, so about 5 of the "Copenhagen engineering" agent's 4,156 ungated roles clip to 1; at 0.86 (p99.5) 75 would, and lose their order where it matters most. The span 0.13 is close to the 0.145 min-max span it replaces, so a prefer weight keeps about its old size against fit. The copy had no votes, so taste comes from synthetic ones, 30 liked and 30 passed roles for each of three intents (pooled p1 -0.1404, p50 0.0429, p99.9 0.1924), and should be re-measured once real votes exist.
**The default bars are 0.55 for extract and 0.65 for summarise.** On the same data, the Copenhagen agent's ungated roles clearing each bar, min-max then calibrated, were: 0.3: 3,467 and 3,520; 0.4: 1,837 and 2,090; 0.5: 519 and 728; 0.55: 200 and 307; 0.6: 56 and 111; 0.65: 15 and 35. So 0.55 sends about 300 roles to extract and 0.65 about 35 to summarise, which its top 30 trims; the top-N backstops (500 and 30) stay for an agent whose bar admits more. The old defaults, 0.3 and 0.5, admit 3,520 and 728 on the calibrated scale, where only the top N would stop them. An agent keeps the limits stored with it, so one saved under the old defaults keeps 0.3 and 0.5 until it is edited: a migration cannot tell a stored default from a choice.
**An agent without an intent feeds neither model stage.** Its scores have no fit term, so they order roles by recency, affinity and prefer gates, and a fresh role of any kind clears a bar. `Store._top_scope` leaves such an agent out, and each score run raises one `score.no_intent` notice naming the enabled agents without an intent. The agent still ranks its own list.
**The embedding backend is effectively required.** The intent is a CV or a paragraph per agent, and TF-IDF cannot relate "Kubernetes" to "container orchestration". The model is `intfloat/multilingual-e5-small`, not the English-only `all-MiniLM-L6-v2` roledesk started with: the corpus turned European, and on a sample of the live roles (all open Danish-country roles plus a random sample of the rest, language labelled by a stop-word count) against the "Copenhagen engineering" agent's intent, MiniLM separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles, where a Danish-country engineering role beats the rest's non-engineering median only 68% of the time; multilingual-e5-small reaches an AUC of 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. A second multilingual candidate, `paraphrase-multilingual-MiniLM-L12-v2`, measured *worse* than plain MiniLM (AUC 0.60) and was dropped. e5 needs its own "query: " / "passage: " prefixes, which `app/jobs/embed.py` adds by matching the model name, not a separate setting.
@@ -112,7 +118,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**The pull cascade is `pull → embed → score → extract → score → summarise`.** Every stage is always in the chain and a stage that is off finds nothing, as in newsdesk. The free stages run first: embed and score cost no tokens, and their score is what bounds the two model stages. Extract reads only what ranks, then score runs again so the gates read what it filled. A stage is addressed by its position in the tuple, since `score` is in it twice. Extract reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly.
**Extract is bounded like summarise: the top roles of each agent, by that agent's own `limits`.** Extract has short answers but would otherwise touch every open role, and the seed boards alone produced 29,000; at Haiku's rates that is tens of dollars for a first pass over roles nobody will look at. Bounded to the ranking, a pull reads a few hundred. `extract_top` is the backstop (default 500) and `extract_min` the real stop (default 0.3, loose): a fixed count stays flat whatever the boards produce, where a percentage would not. The bounds are per agent rather than config, because one agent may be a wide monitoring search that should read a thousand and another a narrow hunt that wants fifty; one global number fits neither. A batch dealt named uids finishes them whatever their rank, so a running batch is not cut short by a rescore.
**Extract is bounded like summarise: the top roles of each agent, by that agent's own `limits`.** Extract has short answers but would otherwise touch every open role, and the seed boards alone produced 29,000; at Haiku's rates that is tens of dollars for a first pass over roles nobody will look at. Bounded to the ranking, a pull reads a few hundred. `extract_top` is the backstop (default 500) and `extract_min` the real stop (default 0.55, about 300 roles for the starter agent): a fixed count stays flat whatever the boards produce, where a percentage would not. The bounds are per agent rather than config, because one agent may be a wide monitoring search that should read a thousand and another a narrow hunt that wants fifty; one global number fits neither. A batch dealt named uids finishes them whatever their rank, so a running batch is not cut short by a rescore.
**`extract_batch` and `summarise_batch` are `isolate=True`, with a timeout.** `context.check()` only runs between the calls `modeltask.one` makes, not inside one, so a call that hangs on either backend was otherwise unkillable short of restarting the server; a process boundary is what lets a Kill, or the timeout itself, actually end it. One call is already bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for summarise): a batch still running after that is stuck, not slow. The cost: an isolated job's context has no working `stop`, `task` or `fan_out` (see `pipeline/child.py`'s `context_for`), so a refusal or an account-level failure inside a batch can no longer stop the rest of that task's queued siblings — they run on to the same failure instead of being spared it.
@@ -142,7 +148,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Affinity and votes are per agent.** A company hidden for one hunt may be a favourite for another, and a vote says what one agent should learn from, so both carry the agent id, and the Roles tab and the company list ask for one agent's. Without an agent id the API uses the default agent, the enabled one with the lowest id, so the CLI and older callers keep working.
**`taste` is the liked-minus-passed centroid, as in newsdesk.** An agent's votes become one vector, the mean of the liked roles' embeddings minus the mean of the passed ones', and a role's taste term is its cosine to that vector, weighted beside fit, recency and affinity. It is the one term an agent learns rather than is told, and it costs no model call.
**`taste` is the liked-minus-passed centroid, as in newsdesk.** An agent's votes become one vector, the mean of the liked roles' embeddings minus the mean of the passed ones', and a role's taste term is its cosine to that vector, calibrated like fit, weighted beside fit, recency and affinity. It is the one term an agent learns rather than is told, and it costs no model call.
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest.
@@ -160,7 +166,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` scrapes `original_url` only when the read page asks for it, because following Jobindex's redirect for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair.
**Summarising is bounded to the top roles of each agent, by that agent's own `limits`, and off by default.** The pull's `summarise` stage only considers roles with no summary yet that are in the top `summarise_top` (default 30) of an enabled agent's scores and at `summarise_min` (default 0.5, stricter than extract's since a summary is the long answer), so the cost stays flat whatever the backlog of open roles is. The bounds live on the agent, beside its gates and weights, for the same reason as extract's: a wide monitoring search and a narrow hunt want different depths, and the agent is where the difference is already expressed. `summarise.enabled: false` matches `extract`'s own default; the read page's Summarise button asks for one role's summary either way.
**Summarising is bounded to the top roles of each agent, by that agent's own `limits`, and off by default.** The pull's `summarise` stage only considers roles with no summary yet that are in the top `summarise_top` (default 30) of an enabled agent's scores and at `summarise_min` (default 0.65, about 35 roles for the starter agent, stricter than extract's since a summary is the long answer), so the cost stays flat whatever the backlog of open roles is. The bounds live on the agent, beside its gates and weights, for the same reason as extract's: a wide monitoring search and a narrow hunt want different depths, and the agent is where the difference is already expressed. `summarise.enabled: false` matches `extract`'s own default; the read page's Summarise button asks for one role's summary either way.
**The schedules element's narrow layout is fixed upstream, in `../boilerplate`.** `<desk-schedules>` is a shadow root the app cannot lay out from outside, and `pipeline/` is a downstream copy refreshed with `git archive`, so a fix landing only in roledesk would be reverted by the next refresh.