Let the model report a verdict per must-have and cap career at 39 in code when one is broken

The judge numbers an agent's must-haves (one per line) and answers met, broken or unclear for each; career is stored as the model gave it and counts as at most BROKEN_CAP when a role is served, in cv_score, Store.judgement and the SQL of Best CV match. Migration 29 adds role_judgements.musts and broken; the judge's input_key carries a format version so earlier judgements are made again. The reader lists the must-haves with their verdicts and the cap.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Sonnet 5.5 committed 2026-10-03 22:40:54 +02:00
1 parent 64fecf57ad
commit 001fc9002b
19 files changed
+336 -61

No files matched your search

+65 -11
View File
@@ -6,7 +6,10 @@ written profile and scores skills, experience and career 0 to 100, with one
to three strengths and gaps. The total is `app.models.cv_score`, the three
numbers under the agent's `cv_*` weights, worked out when a role is served and
not stored, so the weights are no part of `input_key`; it is what the feed's
Best CV match sorts by. Culture and location are left out: the first
Best CV match sorts by. The model also reports a verdict on each of the
agent's must-haves (one per line of its must text, numbered M1..Mn in the
prompt); it does not apply them. The rule is code's: a "broken" one makes
career count as at most `app.models.BROKEN_CAP`, when the role is served. Culture and location are left out: the first
is guessed from ad text, the second is what the gates do.
The work is pairs, `(agent, role)`: an enabled agent with a CV, and a role
@@ -44,13 +47,20 @@ from pipeline import NULL, RunContext, Task, job
from pipeline.task import Batch
from app.config import Config
from app.jobs import stages
from app.models import LANGUAGE_LEVELS, Agent, CvLanguage, cv_languages
from app.models import (
LANGUAGE_LEVELS, MUST_VERDICTS, Agent, CvLanguage, cv_languages, must_lines,
)
from app.storage import Store
#: How much of a posting goes in one item: summarise's 8000, since the
#: judgement needs the whole role, not its opening and closing lines.
EXCERPT = 8000
#: The format of what the judge asks and stores, part of `input_key`: a
#: change to the prompt's contract makes every judgement be made again.
#: 2: the verdict on each numbered must-have (`musts`), the cap in code.
FORMAT = 2
#: How many strengths and gaps are kept, and how long each may be.
MOST, LONGEST = 3, 300
@@ -73,9 +83,17 @@ Score three dimensions, each 0 to 100:
aligned with their direction, with clear growth; 60-79 a good role, partly
aligned; 40-59 a decent job that does not build toward their goals; 0-39 a
dead end or a step backwards. Judge whether the tasks would energise them,
not only whether they could do them. A posting that breaks one of their
stated deal-breakers scores career 0-39, and the deal-breaker is named as a
gap.
not only whether they could do them.
Their must-haves and deal-breakers are numbered M1, M2 and so on. Give a
verdict on every one of them, in `musts`, for each item:
- "met": the posting text says it is met.
- "broken": only when the posting text clearly contradicts it.
- "unclear": the text does not say. Never infer from silence: a posting that
is silent on a must-have is "unclear", not "broken".
Judge a must-have by its own words, not by how well the role fits otherwise.
Answer musts as [] when there are none. The verdicts do not change the three
scores, which follow their rubrics; name a broken must-have as a gap.
Then 1 to 3 strengths and 1 to 3 gaps, each one short sentence grounded in
the posting text. State the gaps honestly: a prestigious role that fits
@@ -100,8 +118,21 @@ SCHEMA: dict[str, Any] = {
"career": {"type": "integer"},
"strengths": {"type": "array", "items": {"type": "string"}},
"gaps": {"type": "array", "items": {"type": "string"}},
"musts": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {"type": "string"},
"verdict": {"type": "string", "enum": list(MUST_VERDICTS)},
},
"required": ["id", "verdict"],
"additionalProperties": False,
},
},
},
"required": ["key", "skills", "experience", "career", "strengths", "gaps"],
"required": ["key", "skills", "experience", "career", "strengths", "gaps",
"musts"],
"additionalProperties": False,
},
},
@@ -144,7 +175,7 @@ def input_key(agent: Agent, text: str) -> str:
"""A hash of everything a judgement reads: the CV, the prompts, the
avoid and must texts, and the role's text as sent. A judgement stored
under another key is read again."""
blob = json.dumps([agent.cv_text, [[p.name, p.text] for p in agent.prompts],
blob = json.dumps([FORMAT, agent.cv_text, [[p.name, p.text] for p in agent.prompts],
agent.avoid_text, agent.must_text, text], ensure_ascii=False)
return hashlib.sha1(blob.encode()).hexdigest()[:16]
@@ -199,9 +230,9 @@ def system(agent: Agent) -> str:
if agent.avoid_text.strip():
parts.append("\n<avoid title=\"what drains them and environments to avoid\">\n"
f"{agent.avoid_text.strip()}\n</avoid>")
if agent.must_text.strip():
parts.append("\n<must title=\"must-haves and deal-breakers\">\n"
f"{agent.must_text.strip()}\n</must>")
if musts := must_lines(agent.must_text):
lines = "\n".join(f"M{n}: {line}" for n, line in enumerate(musts, start=1))
parts.append(f"\n<must title=\"must-haves and deal-breakers\">\n{lines}\n</must>")
return "\n".join(parts)
@@ -243,6 +274,25 @@ def _lines(value: Any) -> list[str]:
return [line for line in found if line][:MOST]
def _musts(agent: Agent | None, value: Any) -> list[dict[str, str]]:
"""The verdicts on the agent's must-haves as they are stored: one
`{id, text, verdict}` per line of its must text, in order. A must-have
the model left out, or gave a verdict outside "met", "broken", "unclear",
is "unclear": nothing is broken by silence. An id that is not one of the
agent's is ignored, the first answer for an id counts."""
said: dict[str, str] = {}
for one in value if isinstance(value, list) else []:
if isinstance(one, dict):
said.setdefault(str(one.get("id") or "").strip().upper(),
str(one.get("verdict") or "").strip().lower())
found = []
for n, line in enumerate(must_lines(agent.must_text if agent else ""), start=1):
verdict = said.get(f"M{n}", "")
found.append({"id": f"M{n}", "text": line,
"verdict": verdict if verdict in MUST_VERDICTS else "unclear"})
return found
def apply_batch(store: Store, answer: dict[str, Any],
batch: dict[str, Any] | None = None,
config: Config | None = None) -> JudgeResult:
@@ -251,6 +301,7 @@ def apply_batch(store: Store, answer: dict[str, Any],
all three numbers; the pair stays waiting."""
keys: dict[str, str] = dict((batch or {}).get("keys") or {})
stats = JudgeResult()
agents: dict[int, Agent | None] = {}
for one in answer.get("judgements") or []:
key = str(one.get("key") or "")
pair = pair_of(key)
@@ -259,8 +310,11 @@ def apply_batch(store: Store, answer: dict[str, Any],
stats.rejected.append(key or "?")
continue
skills, experience, career = (int(m) for m in marks if m is not None)
if pair[0] not in agents:
agents[pair[0]] = store.agent(pair[0])
if not store.set_judgement(pair[0], pair[1], keys.pop(key), skills, experience, career,
_lines(one.get("strengths")), _lines(one.get("gaps"))):
_lines(one.get("strengths")), _lines(one.get("gaps")),
_musts(agents[pair[0]], one.get("musts"))):
stats.rejected.append(key)
continue
stats.judged += 1