Score each role's closeness to the agent's CV, let the judge also read the 50 closest the score leaves out, and order CV match's unjudged roles by it
The CV is cut into chunks inside e5's window, embedded like a prompt and cached per agent under a hash of the model, chunking and text. A role's cv_fit (scores.cv_fit, migration 31) is its best chunk cosine, scaled with its own calibration band: fit's clipped 88 of agent 2's ungated roles to 1. It is no term of the Agent match score. judge_cv_top (default 50) sits beside judge_top and judge_min on the agents page and card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b481cb5692
commit
c23594507a
23 files changed
+603
-85
No files matched your search
+25
-9
@@ -72,8 +72,9 @@ class JudgeConfig(ModelTaskConfig):
|
||||
"""Each search agent's best roles read against its CV, as three numbers
|
||||
and a total. See app/jobs/judge.py. Off by default like extract; on, a
|
||||
pull judges the best of each enabled agent with a CV, as deep and as low
|
||||
as that agent's own `limits` say (`judge_top`, `judge_min`), and only
|
||||
the roles whose judgement is missing or was read from another CV or
|
||||
as that agent's own `limits` say (`judge_top`, `judge_min`), and the
|
||||
`judge_cv_top` ungated ones closest to its CV that those leave out, and
|
||||
only the roles whose judgement is missing or was read from another CV or
|
||||
text."""
|
||||
|
||||
enabled: bool = False
|
||||
@@ -103,21 +104,30 @@ class EnrichConfig(Section):
|
||||
|
||||
|
||||
class Calibration(Section):
|
||||
"""Where one embedding model's cosines sit, as `(lo, hi)` for fit and for
|
||||
taste: `score.scale` maps `lo` to 0 and `hi` to 1 and clips outside, so a
|
||||
term is a level rather than a rank within the run."""
|
||||
"""Where one embedding model's cosines sit, as `(lo, hi)` for fit, for
|
||||
taste and for a role's closeness to a CV (`cv`, fit's when not given):
|
||||
`score.scale` maps `lo` to 0 and `hi` to 1 and clips outside, so a term
|
||||
is a level rather than a rank within the run."""
|
||||
|
||||
fit: tuple[float, float]
|
||||
taste: tuple[float, float]
|
||||
cv: tuple[float, float] | None = None
|
||||
|
||||
@field_validator("fit", "taste")
|
||||
@field_validator("fit", "taste", "cv")
|
||||
@classmethod
|
||||
def _lo_below_hi(cls, band: tuple[float, float]) -> tuple[float, float]:
|
||||
def _lo_below_hi(cls, band: tuple[float, float] | None) -> tuple[float, float] | None:
|
||||
if band is None:
|
||||
return band
|
||||
lo, hi = band
|
||||
if not -1 <= lo < hi <= 1:
|
||||
raise ValueError(f"needs -1 <= lo < hi <= 1, got [{lo}, {hi}]")
|
||||
return band
|
||||
|
||||
@property
|
||||
def cv_band(self) -> tuple[float, float]:
|
||||
"""The band `cv_fit` is scaled with: `cv`, else fit's."""
|
||||
return self.cv or self.fit
|
||||
|
||||
|
||||
#: The built-in calibration, by embedding model. Measured on a copy of the
|
||||
#: live database, 16,841 open roles (see docs/decisions.md). fit: pooled over
|
||||
@@ -125,9 +135,15 @@ class Calibration(Section):
|
||||
#: every single intent's p99.9, so only about 5 of the starter agent's 4,156
|
||||
#: ungated roles clip to 1. taste: from synthetic votes, 30 liked and 30
|
||||
#: passed roles for each of three intents, since the copy had none (pooled p1
|
||||
#: -0.140, p99.9 0.192); re-measure it once real votes exist.
|
||||
#: -0.140, p99.9 0.192); re-measure it once real votes exist. cv: the best
|
||||
#: cosine over the chunks of the one CV in the copy of 2026-10-03, over that
|
||||
#: agent's 7,270 ungated roles: p1 0.770, median 0.832, p99.9 0.891, max
|
||||
#: 0.902. A CV chunk is a longer text than a prompt and sits closer to every
|
||||
#: posting, so fit's band clipped 88 of them to 1 and the 50 closest tied;
|
||||
#: with this one, 1 clips. Re-measure it once more agents hold a CV.
|
||||
CALIBRATION: dict[str, Calibration] = {
|
||||
"intfloat/multilingual-e5-small": Calibration(fit=(0.75, 0.88), taste=(-0.14, 0.19)),
|
||||
"intfloat/multilingual-e5-small": Calibration(fit=(0.75, 0.88), taste=(-0.14, 0.19),
|
||||
cv=(0.77, 0.90)),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -77,6 +77,62 @@ def intent_text(model: str, intent: str) -> str:
|
||||
return f"query: {intent}" if wants_e5_prefix(model) else intent
|
||||
|
||||
|
||||
#: The most of a CV one vector reads. multilingual-e5-small reads 512 tokens
|
||||
#: and cuts the rest; on the owner's English CV a paragraph runs about four
|
||||
#: characters a token (1,896 characters were 471 tokens with "query: "), so
|
||||
#: 1,200 characters stay inside the window with room for a denser language.
|
||||
CV_CHUNK_CHARS = 1200
|
||||
|
||||
#: The version of `cv_chunks`, part of the CV vectors' cache key
|
||||
#: (`score.cv_key`): a change to how a CV is cut embeds every CV again.
|
||||
CV_CHUNKING = 1
|
||||
|
||||
#: Where a long paragraph is cut, coarsest first, and what joins its pieces
|
||||
#: again: lines, then sentences.
|
||||
_CUTS = ((re.compile(r"\n+"), "\n"), (re.compile(r"(?<=[.!?;])\s+"), " "))
|
||||
|
||||
|
||||
def _pieces(text: str, limit: int) -> list[str]:
|
||||
"""`text` cut to pieces of at most `limit` characters: by line, then by
|
||||
sentence, then at the last space before the limit, else at the limit."""
|
||||
if len(text) <= limit:
|
||||
return [text]
|
||||
for pattern, glue in _CUTS:
|
||||
parts = [p.strip() for p in pattern.split(text) if p.strip()]
|
||||
if len(parts) > 1:
|
||||
return _merge([piece for part in parts for piece in _pieces(part, limit)],
|
||||
limit, glue)
|
||||
cut = text.rfind(" ", 0, limit + 1)
|
||||
cut = cut if cut > 0 else limit
|
||||
return [text[:cut].strip(), *_pieces(text[cut:].strip(), limit)]
|
||||
|
||||
|
||||
def _merge(pieces: list[str], limit: int, glue: str) -> list[str]:
|
||||
"""Neighbouring pieces joined with `glue` while the join fits `limit`."""
|
||||
merged: list[str] = []
|
||||
for piece in pieces:
|
||||
if merged and len(merged[-1]) + len(glue) + len(piece) <= limit:
|
||||
merged[-1] += glue + piece
|
||||
else:
|
||||
merged.append(piece)
|
||||
return merged
|
||||
|
||||
|
||||
def cv_chunks(cv_text: str, limit: int | None = None) -> list[str]:
|
||||
"""A CV as the texts its vectors are made from: its paragraphs (blank
|
||||
lines apart), a longer one cut by line, then by sentence, and neighbours
|
||||
joined again while they fit `limit` (`CV_CHUNK_CHARS`), so a heading
|
||||
travels with what follows it and no chunk runs past what the model
|
||||
reads."""
|
||||
limit = limit or CV_CHUNK_CHARS
|
||||
pieces = []
|
||||
for paragraph in re.split(r"\n\s*\n", cv_text):
|
||||
lines = "\n".join(" ".join(line.split()) for line in paragraph.splitlines()).strip()
|
||||
if lines:
|
||||
pieces += _pieces(lines, limit)
|
||||
return _merge(pieces, limit, "\n\n")
|
||||
|
||||
|
||||
def load_embedder(model: str) -> embedding.Embedder:
|
||||
"""The encoder. A seam of its own so a test hands in a fake and nothing
|
||||
downloads a model; `score` loads through it too."""
|
||||
|
||||
+4
-3
@@ -14,9 +14,10 @@ is guessed from ad text, the second is what the gates do.
|
||||
|
||||
The work is pairs, `(agent, role)`: an enabled agent with a CV, and a role
|
||||
among its best ungated scores as deep and as low as its own `limits` say
|
||||
(`judge_top`, `judge_min`) with some text, that has no judgement yet, or one
|
||||
read from another CV, prompts, like, avoid or must text or role text than today's
|
||||
(`input_key`). A pair is named by the composite key `"<agent_id>:<uid>"`,
|
||||
(`judge_top`, `judge_min`), or among the `judge_cv_top` ungated roles closest
|
||||
to its CV (`scores.cv_fit`) that those leave out, with some text, that has
|
||||
no judgement yet, or one read from another CV, prompts, like, avoid or must
|
||||
text or role text than today's (`input_key`). A pair is named by the composite key `"<agent_id>:<uid>"`,
|
||||
which is what `desk.modeltask` hands round as a uid, so `desk/` is untouched.
|
||||
A batch holds one agent's pairs, since the profile is in the system prompt;
|
||||
the finder cuts per agent (`_finder`) and the job runs one call per agent it
|
||||
|
||||
+61
-6
@@ -29,6 +29,12 @@ roles out of extract's reach for good.
|
||||
over this agent's votes; with only likes it is the liked mean,
|
||||
with only passes the negated passed mean; no votes, no term
|
||||
|
||||
Beside the score, for an agent with a CV, `cv_fit`: how close the role is to
|
||||
the CV, the best cosine between the role and the CV's chunks
|
||||
(`embed.cv_chunks`, embedded as the prompts are), scaled like fit. It is no
|
||||
term of the score; the judge reads the closest roles the score leaves out
|
||||
(`Store.judge_rows`) and CV match orders the roles not judged yet by it.
|
||||
|
||||
Every term is 0..1. The score is the weighted sum over the sum of the weights
|
||||
(the agent's four and its prefer gates'), so it stays 0..1. A favourite
|
||||
company (`affinity = 2`) skips the `any` group but not the `require` or
|
||||
@@ -208,6 +214,21 @@ def intent_key(model: str, intent: str) -> str:
|
||||
return hashlib.sha1(f"{model}\n{intent}".encode()).hexdigest()
|
||||
|
||||
|
||||
def cv_key(model: str, cv_text: str) -> str:
|
||||
"""What the CV's vectors are cached under: the model, how the CV is cut
|
||||
(`embed.CV_CHUNKING`) and the text."""
|
||||
return hashlib.sha1(f"{model}\n{embed.CV_CHUNKING}\n{cv_text}".encode()).hexdigest()
|
||||
|
||||
|
||||
def closeness(role: np.ndarray, chunks: np.ndarray) -> float:
|
||||
"""The best raw cosine between a role and the CV's chunks, -1..1, with
|
||||
`chunks` one unit-length row each (`_cv`); 0 for a zero role vector."""
|
||||
norm = float(np.linalg.norm(role))
|
||||
if norm == 0 or not len(chunks):
|
||||
return 0.0
|
||||
return float(np.max(chunks @ role)) / norm
|
||||
|
||||
|
||||
class _Encoder:
|
||||
"""The embedder, loaded on first use only: an agent whose prompt vectors
|
||||
are cached costs no model load at all."""
|
||||
@@ -219,6 +240,10 @@ class _Encoder:
|
||||
self.failed = False
|
||||
|
||||
def encode(self, text: str) -> np.ndarray | None:
|
||||
found = self.encode_many([text])
|
||||
return None if found is None else found[0]
|
||||
|
||||
def encode_many(self, texts: Sequence[str]) -> np.ndarray | None:
|
||||
if self.failed:
|
||||
return None
|
||||
if self._embedder is None:
|
||||
@@ -228,7 +253,7 @@ class _Encoder:
|
||||
self.failed = True
|
||||
self.context.notice(embed.unavailable(exc))
|
||||
return None
|
||||
return np.asarray(self._embedder.encode([text])[0])
|
||||
return np.asarray(self._embedder.encode(list(texts)))
|
||||
|
||||
|
||||
def _prompts(store: Store, agent: Agent, model: str,
|
||||
@@ -251,6 +276,27 @@ def _prompts(store: Store, agent: Agent, model: str,
|
||||
return found
|
||||
|
||||
|
||||
def _cv(store: Store, agent: Agent, model: str, encoder: _Encoder) -> np.ndarray | None:
|
||||
"""The agent's CV as one unit-length row per chunk (`embed.cv_chunks`),
|
||||
each encoded as the prompts are ("query: " for an e5 model), or None
|
||||
without a CV or an embedder. Cached under `cv_key`, so a run embeds the
|
||||
CV only after it changed."""
|
||||
if not agent.cv_text.strip():
|
||||
return None
|
||||
key = cv_key(model, agent.cv_text)
|
||||
found = store.cv_vectors(agent.id, key)
|
||||
if found is None:
|
||||
chunks = embed.cv_chunks(agent.cv_text)
|
||||
made = encoder.encode_many([embed.intent_text(model, c) for c in chunks])
|
||||
if made is None or not len(made):
|
||||
return None
|
||||
found = list(made)
|
||||
store.set_cv_vectors(agent.id, key, found)
|
||||
rows = np.vstack(found).astype(np.float64)
|
||||
norms = np.linalg.norm(rows, axis=1, keepdims=True)
|
||||
return rows / np.where(norms == 0, 1.0, norms)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- the job
|
||||
|
||||
def no_intent(names: Sequence[str]) -> Notice:
|
||||
@@ -267,7 +313,8 @@ def no_intent(names: Sequence[str]) -> Notice:
|
||||
|
||||
def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
prompts: Sequence[tuple[str, np.ndarray]], now: datetime,
|
||||
calibration: Calibration | None = None) -> dict[str, int]:
|
||||
calibration: Calibration | None = None,
|
||||
cv: np.ndarray | None = None) -> dict[str, int]:
|
||||
"""Two passes: the cosines of the roles this agent scores, to each of its
|
||||
`prompts` and to its taste direction, then every score with both scaled
|
||||
by the model's `calibration`, or across the run without one. A role's fit
|
||||
@@ -278,11 +325,14 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
A gated role is left out of the cosines, so under the fallback a role
|
||||
that fails a require gate cannot stretch the range the ranked roles
|
||||
share. The voted roles are scored like any other; a liked one sits near
|
||||
the direction it helped make, which is what a like should do."""
|
||||
the direction it helped make, which is what a like should do. With `cv`
|
||||
(`_cv`), each of those roles also gets its `cv_fit`, its `closeness` to
|
||||
the CV scaled like fit, beside the score and not in it."""
|
||||
candidates = store.score_candidates(agent.id, agent.gates)
|
||||
direction = taste_direction(vectors, store.votes(agent.id))
|
||||
cosines: dict[str, dict[str, float]] = {name: {} for name, _ in prompts}
|
||||
tastes: dict[str, float] = {}
|
||||
near: dict[str, float] = {}
|
||||
for candidate in candidates:
|
||||
role_vec = vectors.get(candidate.uid)
|
||||
if role_vec is None or failed_gates(candidate, agent.gates):
|
||||
@@ -291,6 +341,8 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
cosines[name][candidate.uid] = cosine(role_vec, prompt_vec)
|
||||
if direction is not None:
|
||||
tastes[candidate.uid] = cosine(role_vec, direction)
|
||||
if cv is not None:
|
||||
near[candidate.uid] = closeness(role_vec, cv)
|
||||
fits: dict[str, float] = {}
|
||||
best: dict[str, str] = {}
|
||||
for name, levels in cosines.items():
|
||||
@@ -298,7 +350,8 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
if uid not in fits or value > fits[uid]:
|
||||
fits[uid], best[uid] = value, name
|
||||
tastes = scale(tastes, calibration.taste if calibration else None)
|
||||
rows: list[tuple[str, float, str, str]] = []
|
||||
near = scale(near, calibration.cv_band if calibration else None)
|
||||
rows: list[tuple[str, float, str, str, float | None]] = []
|
||||
gated = 0
|
||||
for candidate in candidates:
|
||||
fit = fits.get(candidate.uid)
|
||||
@@ -308,8 +361,10 @@ def score_agent(store: Store, agent: Agent, vectors: dict[str, np.ndarray],
|
||||
missing="no vector" if prompts else "no prompt vector")
|
||||
gated += bool(reason)
|
||||
counted = fit is not None and agent.weights.fit and not reason
|
||||
cv_fit = near.get(candidate.uid)
|
||||
rows.append((candidate.uid, round(value, 6), reason,
|
||||
best[candidate.uid] if counted else ""))
|
||||
best[candidate.uid] if counted else "",
|
||||
None if cv_fit is None else round(cv_fit, 6)))
|
||||
store.replace_scores(agent.id, rows)
|
||||
return {"scored": len(rows), "gated": gated}
|
||||
|
||||
@@ -336,7 +391,7 @@ def run_score(config: Config, store: Store, *,
|
||||
context.check()
|
||||
context.progress(n, len(agents), f"scoring for {agent.name}, {n + 1} of {len(agents)}")
|
||||
counts = score_agent(store, agent, vectors, _prompts(store, agent, model, encoder),
|
||||
now, calibration)
|
||||
now, calibration, cv=_cv(store, agent, model, encoder))
|
||||
scored += counts["scored"]
|
||||
gated += counts["gated"]
|
||||
log.info("%-20s %6d scored, %6d gated", agent.name, counts["scored"], counts["gated"])
|
||||
|
||||
@@ -1244,6 +1244,9 @@ class ScoredRole(Role):
|
||||
#: judged; `cv_score` is its total.
|
||||
judgement: Judgement | None = None
|
||||
cv_score: float | None = None
|
||||
#: How close the role is to the agent's CV, 0..1 (`scores.cv_fit`), None
|
||||
#: without a CV, a vector or a score, and for a gated role.
|
||||
cv_fit: float | None = None
|
||||
#: The posting's languages against the agent's (`language_flag`).
|
||||
lang_flag: LangFlag = ""
|
||||
lang_note: str = ""
|
||||
@@ -1328,6 +1331,8 @@ class RoleRow(Model):
|
||||
lang_flag: LangFlag = ""
|
||||
lang_note: str = ""
|
||||
cv_score: float | None = None
|
||||
#: As `ScoredRole.cv_fit`: how close the role is to the agent's CV, 0..1.
|
||||
cv_fit: float | None = None
|
||||
|
||||
@computed_field # type: ignore[prop-decorator]
|
||||
@property
|
||||
@@ -1490,6 +1495,10 @@ class Limits(Model):
|
||||
#: so between summarise's 30 and extract's 500.
|
||||
judge_top: int = Field(150, ge=0)
|
||||
judge_min: float = Field(0.55, ge=0, le=1)
|
||||
#: And the agent's ungated roles closest to its CV (`scores.cv_fit`, see
|
||||
#: `app.jobs.score`) that the pair above leaves out, as many as this:
|
||||
#: a role the prompts rank low can still be one the CV fits.
|
||||
judge_cv_top: int = Field(50, ge=0)
|
||||
|
||||
|
||||
class Prompt(Model):
|
||||
|
||||
@@ -43,9 +43,11 @@
|
||||
// What the model reads after a pull, per agent: the best `extract_top`
|
||||
// roles scoring at least `extract_min` get their gate columns filled, the
|
||||
// best `summarise_top` at `summarise_min` or more a summary, and so does any
|
||||
// role that is saved or liked, whatever its score.
|
||||
// role that is saved or liked, whatever its score. The judge reads the best
|
||||
// `judge_top` at `judge_min` or more and the `judge_cv_top` ungated roles
|
||||
// closest to the CV that those leave out.
|
||||
const DEFAULT_LIMITS = { extract_top: 500, extract_min: 0.55, summarise_top: 30, summarise_min: 0.65,
|
||||
judge_top: 150, judge_min: 0.55 };
|
||||
judge_top: 150, judge_min: 0.55, judge_cv_top: 50 };
|
||||
// The levels of a language, the order of the server's ladder, and how big a
|
||||
// CV file the browser reads into the box.
|
||||
const LEVELS = ["basic", "conversational", "professional", "fluent", "native"];
|
||||
@@ -301,10 +303,12 @@
|
||||
function limitsLine(row) {
|
||||
const l = Object.assign({}, DEFAULT_LIMITS, row.limits);
|
||||
const p = el("p", "agent-limits");
|
||||
const judged = `${l.judge_top} at ${l.judge_min}`
|
||||
+ (l.judge_cv_top ? ` and the ${l.judge_cv_top} closest to the CV` : "");
|
||||
const clauses = [
|
||||
["extract", `reads ${l.extract_top} at ${l.extract_min}`, `${l.extract_top} at ${l.extract_min}`],
|
||||
["summarise", `summarises ${l.summarise_top} at ${l.summarise_min}`, `${l.summarise_top} at ${l.summarise_min}`],
|
||||
["judge", `judges ${l.judge_top} at ${l.judge_min}`, `${l.judge_top} at ${l.judge_min}`],
|
||||
["judge", `judges ${judged}`, judged],
|
||||
];
|
||||
clauses.forEach(([name, on, would], i) => {
|
||||
if (i) p.append(" · ");
|
||||
@@ -788,6 +792,7 @@
|
||||
$("#a-summarise-min", f).value = limits.summarise_min;
|
||||
$("#a-judge-top", f).value = limits.judge_top;
|
||||
$("#a-judge-min", f).value = limits.judge_min;
|
||||
$("#a-judge-cv-top", f).value = limits.judge_cv_top;
|
||||
$("#a-delete", f).hidden = !row;
|
||||
f.addEventListener("input", () => { f.dataset.dirty = "true"; view(); });
|
||||
f.addEventListener("change", view);
|
||||
@@ -899,6 +904,7 @@
|
||||
summarise_min: Number($("#a-summarise-min", f).value),
|
||||
judge_top: Number($("#a-judge-top", f).value),
|
||||
judge_min: Number($("#a-judge-min", f).value),
|
||||
judge_cv_top: Number($("#a-judge-cv-top", f).value),
|
||||
},
|
||||
fetch_terms: read.terms(),
|
||||
cv_text: $("#a-cv-text", f).value,
|
||||
|
||||
@@ -717,6 +717,9 @@ body[data-reader="open"] .row[data-open="true"] {
|
||||
padding-top: .2rem; text-align: right;
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
/* Under CV match, a role not judged yet shows its closeness to the CV,
|
||||
fainter, beside an empty bar: a measure, not the model's reading. */
|
||||
.score.near { color: var(--faint); font-style: italic; }
|
||||
|
||||
/* Newest hides the score, because in that order it is not what put the
|
||||
row where it is. Done here rather than in the row template so the markup is
|
||||
|
||||
+18
-7
@@ -1041,15 +1041,26 @@
|
||||
}
|
||||
|
||||
/* The gutter's number, of 100, and what it says: the agent's score, or
|
||||
under CV match the CV match, where a role `judge` has not read yet
|
||||
shows a dash and an empty bar. */
|
||||
under CV match the CV match. A role `judge` has not read yet has an
|
||||
empty bar and, faint, how close it is to the CV (`cv_fit`), which is
|
||||
what orders it there, else a dash. */
|
||||
function gutterOf(role) {
|
||||
if (sortShown() === "cv") {
|
||||
const judged = role.cv_score !== null && role.cv_score !== undefined;
|
||||
if (judged) {
|
||||
return {
|
||||
pct: Math.round(Math.min(100, Math.max(0, role.cv_score))),
|
||||
shown: String(Math.round(role.cv_score)),
|
||||
title: "How well it matches the CV, of 100",
|
||||
};
|
||||
}
|
||||
const near = role.cv_fit !== null && role.cv_fit !== undefined;
|
||||
return {
|
||||
pct: judged ? Math.round(Math.min(100, Math.max(0, role.cv_score))) : 0,
|
||||
shown: judged ? String(Math.round(role.cv_score)) : "–",
|
||||
title: judged ? "How well it matches the CV, of 100" : "Not judged against the CV yet",
|
||||
pct: 0,
|
||||
shown: near ? String(Math.round(Math.min(1, Math.max(0, role.cv_fit)) * 100)) : "–",
|
||||
title: near ? "Not judged against the CV yet; how close it is to the CV, of 100"
|
||||
: "Not judged against the CV yet",
|
||||
near,
|
||||
};
|
||||
}
|
||||
const scored = role.score !== null && role.score !== undefined;
|
||||
@@ -1058,11 +1069,11 @@
|
||||
}
|
||||
|
||||
function drawRow(el, role) {
|
||||
const { pct, shown, title } = gutterOf(role);
|
||||
const { pct, shown, title, near } = gutterOf(role);
|
||||
const text = excerpt(role);
|
||||
el.innerHTML = `
|
||||
<div class="gutter"><i style="height:${pct}%"></i></div>
|
||||
<div class="score" title="${esc(title)}">${esc(shown)}</div>
|
||||
<div class="score${near ? " near" : ""}" title="${esc(title)}">${esc(shown)}</div>
|
||||
<div class="body">
|
||||
<a class="headline" href="${esc(pathFor(role.uid))}">${esc(role.title || "(untitled)")}</a>
|
||||
<div class="meta">
|
||||
|
||||
@@ -387,6 +387,8 @@ CREATE TABLE IF NOT EXISTS role_votes (
|
||||
|
||||
-- A role is scored per agent. A gated role keeps its row, with the reason.
|
||||
-- prompt names the agent's prompt that gave the fit, '' when fit did not count.
|
||||
-- cv_fit is how close the role is to the agent's CV, 0..1, NULL without a CV
|
||||
-- or a vector and for a gated role; it is no term of score.
|
||||
CREATE TABLE IF NOT EXISTS scores (
|
||||
uid TEXT NOT NULL REFERENCES roles(uid) ON DELETE CASCADE,
|
||||
agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
|
||||
@@ -394,9 +396,21 @@ CREATE TABLE IF NOT EXISTS scores (
|
||||
gated_reason TEXT NOT NULL DEFAULT '',
|
||||
scored_at TEXT NOT NULL,
|
||||
prompt TEXT NOT NULL DEFAULT '',
|
||||
cv_fit REAL,
|
||||
PRIMARY KEY (uid, agent_id)
|
||||
);
|
||||
|
||||
-- The vectors of an agent's CV, one per chunk in order, for scores.cv_fit.
|
||||
-- vec_key is sha1 of the embedding model, the chunking and the CV text, the
|
||||
-- same on every row of an agent, so an unchanged CV is not embedded again.
|
||||
CREATE TABLE IF NOT EXISTS agent_cv_vectors (
|
||||
agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
|
||||
position INTEGER NOT NULL,
|
||||
vec_key TEXT NOT NULL,
|
||||
vec BLOB NOT NULL,
|
||||
PRIMARY KEY (agent_id, position)
|
||||
);
|
||||
|
||||
-- One agent's judge reading of a role against its CV. Its own table, since
|
||||
-- replace_scores rewrites scores every run and a judgement outlives that
|
||||
-- and the role closing. skills, experience and career are 0 to 100,
|
||||
@@ -713,6 +727,10 @@ UPDATE roles SET seniority = COALESCE(NULLIF(seniority_model, ''),
|
||||
# reads every role again (judge.FORMAT).
|
||||
(30, "agents.like_text: what the person likes, beside what they do not like",
|
||||
lambda conn: _add_like_text(conn)),
|
||||
# A Python step: it adds the column and the table only where they are
|
||||
# missing. The next score fills them.
|
||||
(31, "scores.cv_fit and agent_cv_vectors: how close each role is to the agent's CV",
|
||||
lambda conn: _add_cv_fit(conn)),
|
||||
)
|
||||
|
||||
#: What a file created from `SCHEMA` today is already at.
|
||||
@@ -1061,6 +1079,21 @@ def _add_like_text(conn: sqlite3.Connection) -> None:
|
||||
conn.execute("ALTER TABLE agents ADD COLUMN like_text TEXT NOT NULL DEFAULT ''")
|
||||
|
||||
|
||||
def _add_cv_fit(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 31's step: `scores.cv_fit`, NULL on every row until the
|
||||
next score, and `agent_cv_vectors`, empty until then. Runs twice without
|
||||
harm."""
|
||||
columns = {r[1] for r in conn.execute("PRAGMA table_info(scores)")}
|
||||
if "cv_fit" not in columns:
|
||||
conn.execute("ALTER TABLE scores ADD COLUMN cv_fit REAL")
|
||||
conn.execute("""CREATE TABLE IF NOT EXISTS agent_cv_vectors (
|
||||
agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
|
||||
position INTEGER NOT NULL,
|
||||
vec_key TEXT NOT NULL,
|
||||
vec BLOB NOT NULL,
|
||||
PRIMARY KEY (agent_id, position))""")
|
||||
|
||||
|
||||
def _add_all_roles(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 21's step. The column; the tag for every company that is a
|
||||
favourite of any agent, since favourite no longer widens what a pull
|
||||
|
||||
+67
-27
@@ -146,8 +146,10 @@ def _cv_sql(weights: Weights | None) -> str:
|
||||
|
||||
def _by_cv(expression: str) -> str:
|
||||
"""The order of CV match: the judged roles by `cv_score` (its SQL
|
||||
`expression`), then the rest by score, as `roles` orders them."""
|
||||
return f"{expression} IS NULL, {expression} DESC, "
|
||||
`expression`), then the rest by how close they are to the CV
|
||||
(`scores.cv_fit`), then by score, as `roles` orders them."""
|
||||
return (f"{expression} IS NULL, {expression} DESC, "
|
||||
"s.cv_fit IS NULL, s.cv_fit DESC, ")
|
||||
|
||||
#: How many days after today the feed's Closing soon filter reaches.
|
||||
CLOSING_DAYS = 4
|
||||
@@ -900,15 +902,15 @@ class Store:
|
||||
|
||||
def role(self, uid: str, agent_id: int | None = None) -> ScoredRole | None:
|
||||
"""One role with its tags, and with `vote`, `voted_at`, `score`,
|
||||
`gated_reason` and `prompt` as `agent_id` judged and scored it, and
|
||||
its `judgement`, `cv_score` (under the agent's weights as they are
|
||||
now) and language flag for that agent; None
|
||||
`gated_reason`, `prompt` and `cv_fit` as `agent_id` judged and scored
|
||||
it, and its `judgement`, `cv_score` (under the agent's weights as they
|
||||
are now) and language flag for that agent; None
|
||||
(unjudged, unscored) without an agent. The score is moved by the vote
|
||||
as `roles` serves it."""
|
||||
row = self._row(
|
||||
f"SELECT roles.*, v.vote, v.voted_at, {_VOTED_SCORE} AS score, "
|
||||
"COALESCE(s.gated_reason, '') AS gated_reason, COALESCE(s.prompt, '') AS prompt "
|
||||
f"FROM roles {_WITH_VOTES} "
|
||||
"COALESCE(s.gated_reason, '') AS gated_reason, COALESCE(s.prompt, '') AS prompt, "
|
||||
f"s.cv_fit FROM roles {_WITH_VOTES} "
|
||||
"LEFT JOIN scores s ON s.uid = roles.uid AND s.agent_id = ? WHERE roles.uid=?",
|
||||
(agent_id, agent_id, uid))
|
||||
if row is None:
|
||||
@@ -1143,7 +1145,8 @@ class Store:
|
||||
no score.
|
||||
|
||||
With `by_cv` as well, the roles the agent's `judge` read come first
|
||||
by `cv_score`, then the others by score (CV match). Every row
|
||||
by `cv_score`, then the others by how close they are to the CV
|
||||
(`cv_fit`), then by score (CV match). Every row
|
||||
carries the agent's `cv_score` (None when not judged), computed
|
||||
from the three numbers under the agent's weights at read time, and the
|
||||
language flag of its languages against the agent's
|
||||
@@ -1167,7 +1170,7 @@ class Store:
|
||||
order = "roles.posted_at DESC, roles.uid"
|
||||
scored = agent is not None and by_score
|
||||
if scored:
|
||||
select += f", {_VOTED_SCORE} AS score, s.gated_reason, s.prompt"
|
||||
select += f", {_VOTED_SCORE} AS score, s.gated_reason, s.prompt, s.cv_fit"
|
||||
order = (_by_cv(cv) if by_cv else "") + "score DESC, " + order
|
||||
elif open_only:
|
||||
# Newest first, open only: idx_roles_posted covers this exactly
|
||||
@@ -1221,7 +1224,8 @@ class Store:
|
||||
by uid, see `role`.
|
||||
|
||||
`lang_flag` and `lang_note` are the role's languages against the
|
||||
agent's (`language_flag`), and `cv_score` its `judge` total.
|
||||
agent's (`language_flag`), `cv_score` its `judge` total and `cv_fit`
|
||||
how close it is to the agent's CV.
|
||||
|
||||
`excerpt` is `app.models.excerpt` over a `SUBSTR` of `_EXCERPT_SOURCE`
|
||||
(`_EXCERPT_RAW` characters, trimmed further in Python), so the source
|
||||
@@ -1244,7 +1248,7 @@ class Store:
|
||||
order = "roles.posted_at DESC, roles.uid"
|
||||
scored = agent is not None and by_score
|
||||
if scored:
|
||||
select += f", {_VOTED_SCORE} AS score, s.gated_reason, s.prompt"
|
||||
select += f", {_VOTED_SCORE} AS score, s.gated_reason, s.prompt, s.cv_fit"
|
||||
order = (_by_cv(cv) if by_cv else "") + "score DESC, " + order
|
||||
elif open_only:
|
||||
# Same reasoning as `roles`: the planner otherwise sorts every
|
||||
@@ -1259,6 +1263,8 @@ class Store:
|
||||
row["excerpt"] = excerpt(row["excerpt"])
|
||||
if row["cv_score"] is not None:
|
||||
row["cv_score"] = round(row["cv_score"], 1)
|
||||
if row.get("cv_fit") is not None:
|
||||
row["cv_fit"] = round(row["cv_fit"], 3)
|
||||
row["lang_flag"], row["lang_note"] = language_flag(
|
||||
role_languages(row.pop("languages_required")), mine)
|
||||
found.append(RoleRow.from_row(row))
|
||||
@@ -1871,8 +1877,10 @@ class Store:
|
||||
def judge_rows(self, agent: Agent, uids: Sequence[str] | None = None) -> list[sqlite3.Row]:
|
||||
"""What `judge` may read for one agent: its best ungated scores of
|
||||
open roles that are no copy, as deep and as low as its `limits` say
|
||||
(`judge_top`, `judge_min`), best first, each with some text
|
||||
(`article_text` or `description`). Given `uids`, those of them with
|
||||
(`judge_top`, `judge_min`), best first, then the `judge_cv_top`
|
||||
ungated ones closest to its CV (`scores.cv_fit`) that the first set
|
||||
leaves out, closest first, each with some text (`article_text` or
|
||||
`description`). Given `uids`, those of them with
|
||||
text, whatever their rank or state. Each row carries `uid`,
|
||||
`title`, `company`, `place`, `article_text`, `description`, and the
|
||||
stored judgement's `input_key`, `judged_at` and `tries` (NULL, NULL
|
||||
@@ -1887,16 +1895,23 @@ class Store:
|
||||
return self._rows(
|
||||
f"{select} roles r {joined} WHERE r.uid IN (SELECT value FROM json_each(?)) "
|
||||
f"AND {text} ORDER BY r.uid", (agent.id, json.dumps(list(uids))))
|
||||
ranked = ("FROM scores s JOIN roles top ON top.uid = s.uid AND top.closed_at IS NULL "
|
||||
"AND top.dup_of IS NULL WHERE s.agent_id = ? AND s.gated_reason = ''")
|
||||
return self._rows(
|
||||
f"""{select} (SELECT s.uid, s.score FROM scores s
|
||||
JOIN roles top ON top.uid = s.uid AND top.closed_at IS NULL
|
||||
AND top.dup_of IS NULL
|
||||
WHERE s.agent_id = ? AND s.gated_reason = ''
|
||||
ORDER BY s.score DESC, top.posted_at DESC, s.uid LIMIT ?) best
|
||||
JOIN roles r ON r.uid = best.uid {joined}
|
||||
WHERE best.score >= ? AND {text}
|
||||
ORDER BY best.score DESC, r.posted_at DESC, r.uid""",
|
||||
(agent.id, agent.limits.judge_top, agent.id, agent.limits.judge_min))
|
||||
f"""WITH best AS (SELECT s.uid, s.score {ranked}
|
||||
ORDER BY s.score DESC, top.posted_at DESC, s.uid LIMIT ?),
|
||||
by_score AS (SELECT uid, 0 AS part, score AS rank FROM best
|
||||
WHERE score >= ?),
|
||||
by_cv AS (SELECT s.uid, 1 AS part, s.cv_fit AS rank {ranked}
|
||||
AND s.cv_fit IS NOT NULL
|
||||
AND s.uid NOT IN (SELECT uid FROM by_score)
|
||||
ORDER BY s.cv_fit DESC, top.posted_at DESC, s.uid LIMIT ?)
|
||||
{select} (SELECT * FROM by_score UNION ALL SELECT * FROM by_cv) w
|
||||
JOIN roles r ON r.uid = w.uid {joined}
|
||||
WHERE {text}
|
||||
ORDER BY w.part, w.rank DESC, r.posted_at DESC, r.uid""",
|
||||
(agent.id, agent.limits.judge_top, agent.limits.judge_min, agent.id,
|
||||
agent.limits.judge_cv_top, agent.id))
|
||||
|
||||
def set_judgement(self, agent_id: int, uid: str, input_key: str, skills: int,
|
||||
experience: int, career: int, strengths: Sequence[str],
|
||||
@@ -2189,6 +2204,27 @@ class Store:
|
||||
self._write("UPDATE agent_prompts SET vec_key=?, vec=? WHERE agent_id=? AND name=?",
|
||||
(key, pack(vec), agent_id, name))
|
||||
|
||||
def cv_vectors(self, agent_id: int, key: str) -> list[np.ndarray] | None:
|
||||
"""The cached vectors of the agent's CV, one per chunk in order, if
|
||||
they were made under `key`; None otherwise."""
|
||||
rows = self._rows("SELECT vec_key, vec FROM agent_cv_vectors WHERE agent_id=? "
|
||||
"ORDER BY position", (agent_id,))
|
||||
if not rows or any(r["vec_key"] != key for r in rows):
|
||||
return None
|
||||
return [unpack(r["vec"]) for r in rows]
|
||||
|
||||
def set_cv_vectors(self, agent_id: int, key: str, vectors: Sequence[np.ndarray]) -> None:
|
||||
"""Replace the agent's CV vectors with these, made under `key`. An
|
||||
agent deleted meanwhile gets nothing written, as in `replace_scores`."""
|
||||
with self.transaction() as conn:
|
||||
if conn.execute("SELECT 1 FROM agents WHERE id=?", (agent_id,)).fetchone() is None:
|
||||
return
|
||||
conn.execute("DELETE FROM agent_cv_vectors WHERE agent_id=?", (agent_id,))
|
||||
conn.executemany(
|
||||
"INSERT INTO agent_cv_vectors (agent_id, position, vec_key, vec) "
|
||||
"VALUES (?,?,?,?)",
|
||||
[(agent_id, n, key, pack(vec)) for n, vec in enumerate(vectors)])
|
||||
|
||||
# --------------------------------------------------------------- scores
|
||||
|
||||
def score_candidates(self, agent_id: int, gates: Sequence[Gate]) -> list[Candidate]:
|
||||
@@ -2226,11 +2262,13 @@ class Store:
|
||||
for r in rows]
|
||||
|
||||
def replace_scores(self, agent_id: int,
|
||||
rows: Sequence[tuple[str, float, str] | tuple[str, float, str, str]]
|
||||
rows: Sequence[tuple[str, float, str] | tuple[str, float, str, str]
|
||||
| tuple[str, float, str, str, float | None]]
|
||||
) -> int:
|
||||
"""`(uid, score, gated_reason)` for every role this agent scored,
|
||||
with the name of the prompt that gave the fit as a fourth item ('' when
|
||||
left out), replacing whatever it had: a role closed or muted since the last run
|
||||
left out) and the role's closeness to the agent's CV as a fifth (None
|
||||
when left out), replacing whatever it had: a role closed or muted since the last run
|
||||
drops out rather than keeping a stale score. One transaction, so a
|
||||
reader sees the old list or the new one. An agent deleted while its
|
||||
run was scoring gets nothing written and 0 back: its rows are gone
|
||||
@@ -2241,9 +2279,11 @@ class Store:
|
||||
return 0
|
||||
conn.execute("DELETE FROM scores WHERE agent_id=?", (agent_id,))
|
||||
conn.executemany(
|
||||
"""INSERT INTO scores (uid, agent_id, score, gated_reason, prompt, scored_at)
|
||||
VALUES (?,?,?,?,?,?)""",
|
||||
[(row[0], agent_id, row[1], row[2], row[3] if len(row) > 3 else "", stamp)
|
||||
"""INSERT INTO scores (uid, agent_id, score, gated_reason, prompt, cv_fit,
|
||||
scored_at)
|
||||
VALUES (?,?,?,?,?,?,?)""",
|
||||
[(row[0], agent_id, row[1], row[2], row[3] if len(row) > 3 else "",
|
||||
row[4] if len(row) > 4 else None, stamp)
|
||||
for row in rows])
|
||||
return len(rows)
|
||||
|
||||
|
||||
@@ -121,8 +121,9 @@
|
||||
<label>down to a score of <input type="number" id="a-summarise-min" min="0" max="1" step="0.05" value="0.65"></label>
|
||||
<label>Judge against the CV the best <input type="number" id="a-judge-top" min="0" step="10" value="150"></label>
|
||||
<label>down to a score of <input type="number" id="a-judge-min" min="0" max="1" step="0.05" value="0.55"></label>
|
||||
<label>and the closest to the CV <input type="number" id="a-judge-cv-top" min="0" step="10" value="50"></label>
|
||||
</fieldset>
|
||||
<span class="hint">Every pull scores every open role for free; these bound what the model then reads, per agent. The count is the backstop, the score the real stop. Fit is a level, the same in every run: a new role with the worst fit scores about 0.27, a perfect one about 0.87. Roles you saved or liked are summarised too, whatever their score. An agent with no prompt feeds neither extract nor summarise, and one with no CV feeds not the judge.</span>
|
||||
<span class="hint">Every pull scores every open role for free; these bound what the model then reads, per agent. The count is the backstop, the score the real stop. Fit is a level, the same in every run: a new role with the worst fit scores about 0.27, a perfect one about 0.87. Roles you saved or liked are summarised too, whatever their score. The judge also reads that many of the roles closest to the CV which the score leaves out, so a role the prompts rank low but the CV fits still gets read; closeness is a free local measure and no part of the score. An agent with no prompt feeds neither extract nor summarise, and one with no CV feeds not the judge.</span>
|
||||
<details class="advanced">
|
||||
<summary>Advanced</summary>
|
||||
<div class="advanced-body">
|
||||
|
||||
Reference in new issue
Block a user