Let the model report a verdict per must-have and cap career at 39 in code when one is broken

The judge numbers an agent's must-haves (one per line) and answers met, broken or unclear for each; career is stored as the model gave it and counts as at most BROKEN_CAP when a role is served, in cv_score, Store.judgement and the SQL of Best CV match. Migration 29 adds role_judgements.musts and broken; the judge's input_key carries a format version so earlier judgements are made again. The reader lists the must-haves with their verdicts and the cap.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Sonnet 5.5 committed 2026-10-03 22:40:54 +02:00
1 parent 64fecf57ad
commit 001fc9002b
19 files changed
+336 -61

No files matched your search

+65 -11
View File
@@ -6,7 +6,10 @@ written profile and scores skills, experience and career 0 to 100, with one
to three strengths and gaps. The total is `app.models.cv_score`, the three
numbers under the agent's `cv_*` weights, worked out when a role is served and
not stored, so the weights are no part of `input_key`; it is what the feed's
Best CV match sorts by. Culture and location are left out: the first
Best CV match sorts by. The model also reports a verdict on each of the
agent's must-haves (one per line of its must text, numbered M1..Mn in the
prompt); it does not apply them. The rule is code's: a "broken" one makes
career count as at most `app.models.BROKEN_CAP`, when the role is served. Culture and location are left out: the first
is guessed from ad text, the second is what the gates do.
The work is pairs, `(agent, role)`: an enabled agent with a CV, and a role
@@ -44,13 +47,20 @@ from pipeline import NULL, RunContext, Task, job
from pipeline.task import Batch
from app.config import Config
from app.jobs import stages
from app.models import LANGUAGE_LEVELS, Agent, CvLanguage, cv_languages
from app.models import (
LANGUAGE_LEVELS, MUST_VERDICTS, Agent, CvLanguage, cv_languages, must_lines,
)
from app.storage import Store
#: How much of a posting goes in one item: summarise's 8000, since the
#: judgement needs the whole role, not its opening and closing lines.
EXCERPT = 8000
#: The format of what the judge asks and stores, part of `input_key`: a
#: change to the prompt's contract makes every judgement be made again.
#: 2: the verdict on each numbered must-have (`musts`), the cap in code.
FORMAT = 2
#: How many strengths and gaps are kept, and how long each may be.
MOST, LONGEST = 3, 300
@@ -73,9 +83,17 @@ Score three dimensions, each 0 to 100:
aligned with their direction, with clear growth; 60-79 a good role, partly
aligned; 40-59 a decent job that does not build toward their goals; 0-39 a
dead end or a step backwards. Judge whether the tasks would energise them,
not only whether they could do them. A posting that breaks one of their
stated deal-breakers scores career 0-39, and the deal-breaker is named as a
gap.
not only whether they could do them.
Their must-haves and deal-breakers are numbered M1, M2 and so on. Give a
verdict on every one of them, in `musts`, for each item:
- "met": the posting text says it is met.
- "broken": only when the posting text clearly contradicts it.
- "unclear": the text does not say. Never infer from silence: a posting that
is silent on a must-have is "unclear", not "broken".
Judge a must-have by its own words, not by how well the role fits otherwise.
Answer musts as [] when there are none. The verdicts do not change the three
scores, which follow their rubrics; name a broken must-have as a gap.
Then 1 to 3 strengths and 1 to 3 gaps, each one short sentence grounded in
the posting text. State the gaps honestly: a prestigious role that fits
@@ -100,8 +118,21 @@ SCHEMA: dict[str, Any] = {
"career": {"type": "integer"},
"strengths": {"type": "array", "items": {"type": "string"}},
"gaps": {"type": "array", "items": {"type": "string"}},
"musts": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {"type": "string"},
"verdict": {"type": "string", "enum": list(MUST_VERDICTS)},
},
"required": ["id", "verdict"],
"additionalProperties": False,
},
},
},
"required": ["key", "skills", "experience", "career", "strengths", "gaps"],
"required": ["key", "skills", "experience", "career", "strengths", "gaps",
"musts"],
"additionalProperties": False,
},
},
@@ -144,7 +175,7 @@ def input_key(agent: Agent, text: str) -> str:
"""A hash of everything a judgement reads: the CV, the prompts, the
avoid and must texts, and the role's text as sent. A judgement stored
under another key is read again."""
blob = json.dumps([agent.cv_text, [[p.name, p.text] for p in agent.prompts],
blob = json.dumps([FORMAT, agent.cv_text, [[p.name, p.text] for p in agent.prompts],
agent.avoid_text, agent.must_text, text], ensure_ascii=False)
return hashlib.sha1(blob.encode()).hexdigest()[:16]
@@ -199,9 +230,9 @@ def system(agent: Agent) -> str:
if agent.avoid_text.strip():
parts.append("\n<avoid title=\"what drains them and environments to avoid\">\n"
f"{agent.avoid_text.strip()}\n</avoid>")
if agent.must_text.strip():
parts.append("\n<must title=\"must-haves and deal-breakers\">\n"
f"{agent.must_text.strip()}\n</must>")
if musts := must_lines(agent.must_text):
lines = "\n".join(f"M{n}: {line}" for n, line in enumerate(musts, start=1))
parts.append(f"\n<must title=\"must-haves and deal-breakers\">\n{lines}\n</must>")
return "\n".join(parts)
@@ -243,6 +274,25 @@ def _lines(value: Any) -> list[str]:
return [line for line in found if line][:MOST]
def _musts(agent: Agent | None, value: Any) -> list[dict[str, str]]:
"""The verdicts on the agent's must-haves as they are stored: one
`{id, text, verdict}` per line of its must text, in order. A must-have
the model left out, or gave a verdict outside "met", "broken", "unclear",
is "unclear": nothing is broken by silence. An id that is not one of the
agent's is ignored, the first answer for an id counts."""
said: dict[str, str] = {}
for one in value if isinstance(value, list) else []:
if isinstance(one, dict):
said.setdefault(str(one.get("id") or "").strip().upper(),
str(one.get("verdict") or "").strip().lower())
found = []
for n, line in enumerate(must_lines(agent.must_text if agent else ""), start=1):
verdict = said.get(f"M{n}", "")
found.append({"id": f"M{n}", "text": line,
"verdict": verdict if verdict in MUST_VERDICTS else "unclear"})
return found
def apply_batch(store: Store, answer: dict[str, Any],
batch: dict[str, Any] | None = None,
config: Config | None = None) -> JudgeResult:
@@ -251,6 +301,7 @@ def apply_batch(store: Store, answer: dict[str, Any],
all three numbers; the pair stays waiting."""
keys: dict[str, str] = dict((batch or {}).get("keys") or {})
stats = JudgeResult()
agents: dict[int, Agent | None] = {}
for one in answer.get("judgements") or []:
key = str(one.get("key") or "")
pair = pair_of(key)
@@ -259,8 +310,11 @@ def apply_batch(store: Store, answer: dict[str, Any],
stats.rejected.append(key or "?")
continue
skills, experience, career = (int(m) for m in marks if m is not None)
if pair[0] not in agents:
agents[pair[0]] = store.agent(pair[0])
if not store.set_judgement(pair[0], pair[1], keys.pop(key), skills, experience, career,
_lines(one.get("strengths")), _lines(one.get("gaps"))):
_lines(one.get("strengths")), _lines(one.get("gaps")),
_musts(agents[pair[0]], one.get("musts"))):
stats.rejected.append(key)
continue
stats.judged += 1
+53 -7
View File
@@ -1007,30 +1007,76 @@ def language_flag(required: list[RoleLanguage],
return "", ""
#: What career counts as at most when a posting breaks one of the agent's
#: must-haves: the top of the rubric's 0-39 band for a dead end. The model
#: reports whether each must-have is met; this cap is applied in code, when a
#: role is served (`cv_score`, `Store.judgement`, `store._cv_sql`).
BROKEN_CAP = 39
#: A must-have's verdict, as the judge answers it.
MUST_VERDICTS = ("met", "broken", "unclear")
_BULLET = re.compile(r"^\s*(?:[-*\u2022]+|\(?\d+[.)])\s+")
def must_lines(text: str) -> list[str]:
"""The agent's must-haves text read as one must-have per line: a leading
bullet (`-`, `*`, `•`) or number (`1.`, `2)`) is dropped, so are empty
lines."""
found = []
for line in text.splitlines():
line = " ".join(_BULLET.sub("", line, count=1).split())
if line:
found.append(line)
return found
def capped_career(career: int, broken: int = 0) -> int:
"""`career` as it counts: at most `BROKEN_CAP` when any must-have is
broken."""
return min(career, BROKEN_CAP) if broken else career
def cv_score(skills: int, experience: int, career: int,
weights: "Weights | None" = None) -> float | None:
weights: "Weights | None" = None, broken: int = 0) -> float | None:
"""The one number of a CV match, 0 to 100: the three numbers of the
judge averaged by the agent's `cv_skills`, `cv_experience` and
`cv_career` weights (30, 25 and 30 by default). None when all three
weights are 0."""
`cv_career` weights (30, 25 and 30 by default), career as
`capped_career` has it. None when all three weights are 0."""
w = weights or Weights()
total = w.cv_skills + w.cv_experience + w.cv_career
if not total:
return None
career = capped_career(career, broken)
return round((w.cv_skills * skills + w.cv_experience * experience
+ w.cv_career * career) / total, 1)
class Must(Model):
"""One must-have of an agent as the judge read it against a role: its
`id` (`M1`, by line), the line's `text` and the `verdict`."""
id: str
text: str = ""
verdict: Literal["met", "broken", "unclear"] = "unclear"
class Judgement(Model):
"""What `judge` read of one role for one agent (`role_judgements`): the
three dimensions 0 to 100, `cv_score` (`cv_score`) and 1 to 3 strengths
and gaps grounded in the posting. `cv_score` is not read from the table:
`Store.judgement` sets it from the agent's weights; None when all three
are 0."""
three dimensions 0 to 100 as the model gave them, `cv_score`
(`cv_score`) and 1 to 3 strengths and gaps grounded in the posting.
`musts` is the verdict on each must-have and `broken` how many are
broken; any makes `career_capped` the lower of `career` and
`BROKEN_CAP`. `cv_score` and `career_capped` are not read from the
table: `Store.judgement` sets them from the agent's weights; `cv_score`
is None when all three are 0."""
skills: int
experience: int
career: int
career_capped: int | None = None
broken: int = 0
musts: Annotated[list[Must], BeforeValidator(lambda v: _json(v, "[]"))] = []
cv_score: float | None = None
strengths: JsonStrings = []
gaps: JsonStrings = []
+3
View File
@@ -21,6 +21,9 @@
* The CV fieldset holds what the judge stage reads a role against: the CV as
* text (typed, or read from a .md/.txt file by the browser), the languages it
* states as rows of a code and a level, and the avoid and must-have texts.
* The must-have text is one must-have per line, which the judge answers one
* verdict each; a broken one caps career at 39 in code (the hint in
* settings/agents.html says so).
* Every save sends all four, and the card's switch re-posts the row as the
* server gave it, so neither drops them.
*
+8
View File
@@ -928,6 +928,14 @@ body[data-reader="open"] .company-strip { display: none; }
.reader-cv .cv-total { margin: 0 0 .45rem; font-variant-numeric: tabular-nums; }
.reader-cv .cv-total b { font-size: 1.25rem; }
.reader-cv .cv-total span { display: inline-block; margin-left: .4rem; color: var(--dim); font-size: .88rem; }
/* The must-haves, one line each with its verdict's mark. */
.cv-musts { list-style: none; margin: 0 0 .55rem; padding: 0; font-size: .92rem; line-height: 1.4; }
.cv-musts li { display: flex; gap: .45rem; }
.cv-musts li + li { margin-top: .15rem; }
.cv-musts .mark { flex: none; width: 1rem; text-align: center; font-weight: 700; }
.cv-musts .must-met .mark { color: var(--good); }
.cv-musts .must-broken .mark { color: var(--error); }
.cv-musts .must-unclear .mark { color: var(--faint); }
.cv-lists { display: grid; grid-template-columns: 1fr 1fr; gap: .5rem 1.2rem; }
.cv-lists ul { margin: 0; padding-left: 1.1rem; font-size: .92rem; line-height: 1.45; }
.cv-lists li + li { margin-top: .2rem; }
+16 -2
View File
@@ -1412,8 +1412,12 @@
try { return languageNames?.of(code) || code; } catch (err) { return code; }
}
const MUST_MARKS = { met: ["✓", "met"], broken: ["✗", "broken"], unclear: ["?", "unclear"] };
/* The agent's `judge` reading against its CV: the total and its three
numbers, then the strengths and gaps it named. A box as the summary is,
numbers, career as the server counted it (capped at 39 when a must-have
is broken, with what the model said), the must-haves with a verdict
each, then the strengths and gaps it named. A box as the summary is,
since it is the same kind of thing: a model's reading, not the posting. */
function cvMatch(r) {
const j = r.judgement;
@@ -1421,10 +1425,20 @@
const list = (items, none) => (items?.length
? `<ul>${items.map((x) => `<li>${esc(x)}</li>`).join("")}</ul>`
: `<p class="muted small">${none}</p>`);
const counted = j.career_capped ?? j.career;
const career = counted < j.career
? `career ${counted} (capped: breaks a must-have; model said ${j.career})` : `career ${j.career}`;
const musts = Array.isArray(j.musts) && j.musts.length
? `<ul class="cv-musts">${j.musts.map((m) => {
const [mark, word] = MUST_MARKS[m.verdict] || MUST_MARKS.unclear;
return `<li class="must-${word}"><span class="mark" role="img"
aria-label="${word}" title="${word}">${mark}</span><span>${esc(m.text)}</span></li>`;
}).join("")}</ul>` : "";
return `<section class="reader-summary reader-cv">
<h2 class="reader-label">CV match</h2>
<p class="cv-total"><b>${Math.round(j.cv_score)}</b> of 100
<span>skills ${j.skills} · experience ${j.experience} · career ${j.career}</span></p>
<span>skills ${j.skills} · experience ${j.experience} · ${career}</span></p>
${musts}
<div class="cv-lists">
<div><h3 class="reader-label">Strengths</h3>${list(j.strengths, "None named.")}</div>
<div><h3 class="reader-label">Gaps</h3>${list(j.gaps, "None named.")}</div>
+24 -2
View File
@@ -400,8 +400,11 @@ CREATE TABLE IF NOT EXISTS scores (
-- and the role closing. skills, experience and career are 0 to 100,
-- cv_score their weighted total, strengths and gaps JSON lists. input_key
-- is a hash of the CV, prompts, avoid text and the role's text it read: a
-- different one is read again. tries counts replies that left the role
-- without one since; at 3 it is given up until the agent is saved again. A
-- different one is read again. musts is a JSON list of {id, text, verdict},
-- the model's verdict on each of the agent's must-haves (one per line of its
-- must text), broken how many are "broken": career counts as at most 39 then
-- (app.models.BROKEN_CAP), applied when the role is served. tries counts
-- replies that left the role without one since; at 3 it is given up until the agent is saved again. A
-- row with judged_at NULL holds only tries.
CREATE TABLE IF NOT EXISTS role_judgements (
uid TEXT NOT NULL REFERENCES roles(uid) ON DELETE CASCADE,
@@ -412,6 +415,8 @@ CREATE TABLE IF NOT EXISTS role_judgements (
cv_score REAL,
strengths TEXT NOT NULL DEFAULT '[]',
gaps TEXT NOT NULL DEFAULT '[]',
musts TEXT NOT NULL DEFAULT '[]',
broken INTEGER NOT NULL DEFAULT 0,
input_key TEXT NOT NULL DEFAULT '',
tries INTEGER NOT NULL DEFAULT 0,
judged_at TEXT,
@@ -698,6 +703,10 @@ UPDATE roles SET seniority = COALESCE(NULLIF(seniority_model, ''),
(28, "agents.cv_text, cv_languages, avoid_text and must_text, roles.languages_required and the "
"deadline columns, and role_judgements: the CV match, the language flag and the "
"deadline", lambda conn: _add_cv_match(conn)),
# A Python step: it adds the columns only where they are missing. Rows
# keep their numbers; the judge reads them again (judge.FORMAT).
(29, "role_judgements.musts and broken: the verdict on each must-have, and how many are "
"broken", lambda conn: _add_musts(conn)),
)
#: What a file created from `SCHEMA` today is already at.
@@ -1025,6 +1034,19 @@ def _add_cv_match(conn: sqlite3.Connection) -> None:
PRIMARY KEY (uid, agent_id))""")
def _add_musts(conn: sqlite3.Connection) -> None:
"""Migration 29's step: `role_judgements.musts`, a JSON list, and
`broken`, the count of must-haves it holds as broken, empty and 0 on
every row. `judge.input_key` changed with it, so each judgement is read
again; until then it serves with no must-haves and no cap. Runs twice
without harm."""
columns = {r[1] for r in conn.execute("PRAGMA table_info(role_judgements)")}
for column, ddl in (("musts", "TEXT NOT NULL DEFAULT '[]'"),
("broken", "INTEGER NOT NULL DEFAULT 0")):
if column not in columns:
conn.execute(f"ALTER TABLE role_judgements ADD COLUMN {column} {ddl}")
def _add_all_roles(conn: sqlite3.Connection) -> None:
"""Migration 21's step. The column; the tag for every company that is a
favourite of any agent, since favourite no longer widens what a pull
+29 -16
View File
@@ -11,7 +11,7 @@ from __future__ import annotations
import json
import sqlite3
from collections import Counter
from collections.abc import Callable, Iterable, Iterator, MutableMapping, Sequence
from collections.abc import Callable, Iterable, Iterator, Mapping, MutableMapping, Sequence
from contextlib import closing, contextmanager
from datetime import UTC, datetime, timedelta
from pathlib import Path
@@ -26,7 +26,7 @@ from app.models import (
AFFINITY, HAND_REASONS, LEVELS, STATUSES, Agent, Application,
ApplicationEvent, Candidate, Company, CvLanguage, Gate, Judgement, Limits, Prompt, Role,
RoleCopy, RolePlace, RoleRow, ScoredRole, cv_languages as normal_languages, cv_score,
language_flag, role_languages,
BROKEN_CAP, capped_career, language_flag, role_languages,
Subscription, Weights, check_prompts, company_fold, company_key, country_of, excerpt, place_key,
fetch_terms as normal_terms, host_word, iso_day, posted_at, posted_host, remote_of, title_key,
)
@@ -120,7 +120,7 @@ _VOTED_SCORE = ("MIN(1.0, s.score * CASE v.vote WHEN -1 THEN 0.5 "
_WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?"
#: The agent's `judge` readings beside every role, the same way: `j.skills`,
#: `j.experience` and `j.career` are NULL for a role not judged and with no
#: `j.experience`, `j.career` and `j.broken` are NULL for a role not judged and with no
#: agent.
_WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agent_id = ? "
"AND j.judged_at IS NOT NULL")
@@ -129,15 +129,19 @@ _WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agen
def _cv_sql(weights: Weights | None) -> str:
"""`cv_score` as SQL for one agent's weights, unrounded (`app.models.cv_score`
rounds it): NULL for a role not judged, and for all three weights 0. The
rounds it): NULL for a role not judged, and for all three weights 0. Career
counts as at most `BROKEN_CAP` when the judgement holds a broken
must-have (`j.broken`), as `app.models.capped_career` has it. The
numbers are the model's validated, finite floats, so they are written in
rather than bound, the select's placeholders being then no concern."""
w = weights or Weights()
total = w.cv_skills + w.cv_experience + w.cv_career
if not total:
return "NULL"
career = (f"(CASE WHEN j.broken > 0 AND j.career > {BROKEN_CAP} THEN {BROKEN_CAP} "
"ELSE j.career END)")
return (f"(({w.cv_skills!r} * j.skills + {w.cv_experience!r} * j.experience "
f"+ {w.cv_career!r} * j.career) / {total!r})")
f"+ {w.cv_career!r} * {career}) / {total!r})")
def _by_cv(expression: str) -> str:
@@ -1888,27 +1892,34 @@ class Store:
def set_judgement(self, agent_id: int, uid: str, input_key: str, skills: int,
experience: int, career: int, strengths: Sequence[str],
gaps: Sequence[str]) -> bool:
gaps: Sequence[str], musts: Sequence[Mapping[str, str]] = ()) -> bool:
"""Store `judge`'s reading of one role for one agent, replacing the
one before, with `cv_score` (`app.models.cv_score`, at the default weights: the
table's column is not read, see `judgement`) and its tries
back at 0. False for an unknown role or agent."""
one before: the three numbers as the model gave them, the verdict on
each of the agent's must-haves (`musts`, `{id, text, verdict}`) and how
many are broken, `cv_score` (`app.models.cv_score`, at the default
weights and with the cap: the table's column is not read, see
`judgement`) and its tries back at 0. False for an unknown role or
agent."""
broken = sum(1 for m in musts if m.get("verdict") == "broken")
try:
with self.transaction() as conn:
conn.execute(
"""INSERT INTO role_judgements (uid, agent_id, skills, experience, career,
cv_score, strengths, gaps, input_key, tries,
judged_at)
VALUES (?,?,?,?,?,?,?,?,?,0,?)
cv_score, strengths, gaps, musts, broken,
input_key, tries, judged_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,0,?)
ON CONFLICT(uid, agent_id) DO UPDATE SET
skills=excluded.skills, experience=excluded.experience,
career=excluded.career, cv_score=excluded.cv_score,
strengths=excluded.strengths, gaps=excluded.gaps,
musts=excluded.musts, broken=excluded.broken,
input_key=excluded.input_key, tries=0, judged_at=excluded.judged_at""",
(uid, agent_id, skills, experience, career,
cv_score(skills, experience, career),
cv_score(skills, experience, career, broken=broken),
json.dumps(list(strengths), ensure_ascii=False),
json.dumps(list(gaps), ensure_ascii=False), input_key, self.now()))
json.dumps(list(gaps), ensure_ascii=False),
json.dumps([dict(m) for m in musts], ensure_ascii=False), broken,
input_key, self.now()))
except sqlite3.IntegrityError:
return False
return True
@@ -1935,7 +1946,8 @@ class Store:
def judgement(self, agent_id: int | None, uid: str) -> Judgement | None:
"""The agent's judgement of one role, None when it has none, its
`cv_score` the three numbers under the agent's weights as they are
now, not as stored."""
now, not as stored, and `career_capped` the model's career as the
broken must-haves leave it (`BROKEN_CAP`)."""
if agent_id is None:
return None
row = self._row("SELECT * FROM role_judgements WHERE agent_id = ? AND uid = ? "
@@ -1944,8 +1956,9 @@ class Store:
return None
found = Judgement.from_row(row)
agent = self.agent(agent_id)
found.career_capped = capped_career(found.career, found.broken)
found.cv_score = cv_score(found.skills, found.experience, found.career,
agent.weights if agent else None)
agent.weights if agent else None, found.broken)
return found
def set_article_text(self, uid: str, text: str) -> Role | None:
+2 -2
View File
@@ -96,8 +96,8 @@
<label>Avoid / drains me
<textarea id="a-avoid-text" rows="4" placeholder="Constant on-call, sales targets, large consultancies, open-plan offices"></textarea></label>
<label>Must-haves and deal-breakers
<textarea id="a-must-text" rows="4" placeholder="Permanent contract, a say in the roadmap, at most one day a week in the office"></textarea></label>
<span class="hint">Soft, read by the judge only: a role that breaks one of these scores low on career and the gap is named. Rules that must hold go under Where or Other rules instead.</span>
<textarea id="a-must-text" rows="4" placeholder="Permanent contract&#10;A say in the roadmap&#10;At most one day a week in the office"></textarea></label>
<span class="hint">One must-have per line. The judge reports for each whether a posting meets it, clearly breaks it or does not say; a broken one caps the role's career score at 39 in code, whatever the model scored. Rules that must hold go under Where or Other rules instead.</span>
</fieldset>
<fieldset class="weights">
<legend>How much each counts</legend>