Rank by the cut: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text renamed from enrich, the feed kept to the fetch gate, and migration 32

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-10-04 15:11:43 +02:00
1 parent 060ccf66e6
commit 617356e60c
35 files changed
+1054 -751

No files matched your search

+51 -47
View File
@@ -1,32 +1,29 @@
"""The tasks roledesk offers, and the order a pull runs them in.
pull → prep → embed → score → enrich → prep → embed → score → extract → score
→ summarise → judge
pull → prep → embed → score → fetch_text → prep → embed → score → extract
→ score → judge → summarise
newsdesk's cascade convention: a cascade is a tuple of task names, and the
final job of one task starts the next (`step` is the callback it carries).
Nothing is alive between two tasks. `prep` (`app/jobs/prep.py`) maps what
the pull stored and folds the copies before anything reads the roles; after
`enrich` it maps the roles their own pages filled. A stage is addressed by its position in
the tuple, not its name, since `score` runs twice.
`fetch_text` it maps the roles their own pages filled. A stage is addressed
by its position in the tuple, not its name, since `score` runs three times.
Every stage is always in the chain, and a stage that is switched off finds
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this pull does not
satisfy, is an extract stage with no batches whose final job moves straight
on. The free stages go first: embed and score cost no tokens, and the score
they produce is what bounds the rest. Enrich reads the best roles without
text on their own pages (`app/jobs/enrich.py`), and prep, embed and score
follow for the roles it filled; when it filled none, `step` passes over all
three. Extract
reads only the best
roles of each enabled agent, as deep and as low as that agent's own `limits`
say; the second score then applies the gates to the columns it filled.
Summarise follows, on the best roles by the same agent's summarise limits,
and judge runs last, reading each agent's best roles against its CV by its
judge limits. The `enrich` cascade, which follows the rescore after an agent
is saved, ends with judge too, so a changed CV, prompt or like text is read
again at once rather than at the next pull; a pair whose `input_key` did not
change costs nothing.
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this
pull does not satisfy, is a stage with no batches whose final job moves
straight on. The free stages go first: embed and score cost no tokens, and
score writes each agent's cut (`app.jobs.score`), which bounds the rest.
Fetch text reads the role's own page for the roles in the cut that came
without text (`app/jobs/fetch_text.py`), and prep, embed and score follow
for the roles it filled, so their levels and the cut are recomputed; when
it filled none, `step` passes over all three. Extract reads the cut; the
score after it applies the fetch gate and the cut again with the columns
extract filled, so a role can drop out or enter (one that enters is
extracted on the next pull). Judge reads each agent's cut as recomputed,
against its CV. Summarise runs last, on the top of each agent's feed by
final score and the roles liked or saved: it is no part of any score.
A batch that failed, or a stage somebody killed, does not stop the chain: the
work comes round on the next pull, and a rescore of what did arrive is worth
@@ -36,10 +33,13 @@ having either way.
and a cascade is started by one: the `pull` job's only work is to ask for the
chain. Its default schedule, `every 6h`, is declared here beside it.
`enrich` is a job and a cascade the same way, `enrich → prep → embed → score`, queued
behind the `score` that follows a change to the ranking other than a pull's
(`enrich_after`): an agent saved, a company's affinity changed. A vote moves
one agent's taste a little and queues neither.
`fetch_text` is a job and a cascade the same way, the pull's chain from
fetch text on, queued behind the `score` that follows a change to the
ranking other than a pull's (`fetch_text_after`): an agent saved, a
company's affinity or overrides changed. The cut may have changed, so its
new roles get their text, extract and judge at once rather than at the next
pull; a judgement whose `input_key` did not change costs nothing. A vote
moves one agent's taste a little and queues neither.
"""
from __future__ import annotations
@@ -49,19 +49,22 @@ from dataclasses import replace
from functools import partial
from typing import Any
from app.jobs import embed, enrich, extract, judge, prep, pull, score, summarise # noqa: F401 -- registers the jobs
from app.jobs import ( # noqa: F401 -- registers the jobs
embed, extract, fetch_text, judge, prep, pull, score, summarise,
)
from desk.tools import agent
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
STEP = "app.jobs.cascade:step"
CASCADES: dict[str, tuple[str, ...]] = {
"pull": ("pull", "prep", "embed", "score", "enrich", "prep", "embed", "score", "extract",
"score", "summarise", "judge"),
"enrich": ("enrich", "prep", "embed", "score", "judge"),
"pull": ("pull", "prep", "embed", "score", "fetch_text", "prep", "embed", "score",
"extract", "score", "judge", "summarise"),
"fetch_text": ("fetch_text", "prep", "embed", "score", "extract", "score", "judge",
"summarise"),
}
#: What follows `enrich` for the roles it filled, passed over when it filled none.
#: What follows `fetch_text` for the roles it filled, passed over when it filled none.
REFILL = ("prep", "embed", "score")
@@ -131,7 +134,7 @@ STAGES: dict[str, Callable[[str], Task]] = {
label="Mapping the roles' host words"),
"summarise": _summarise,
"judge": _judge,
"enrich": lambda cascade: enrich.enrich_task(),
"fetch_text": lambda cascade: fetch_text.fetch_text_task(),
}
@@ -157,7 +160,7 @@ def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
name, done = payload.get("cascade", ""), payload.get("task", "")
at = int(payload.get("at", -1))
tasks = CASCADES.get(name, ())
if done == "enrich" and not enrich.filled(tally) \
if done == "fetch_text" and not fetch_text.filled(tally) \
and tasks[at + 1:at + 1 + len(REFILL)] == REFILL:
at += len(REFILL)
following = follows(name, at)
@@ -222,24 +225,25 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
return {"started": " > ".join(CASCADES["pull"]), "run": run}
@job("enrich", "Reading the best roles on their own pages", solo=True, resource="http")
def run_enrich(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Ask for the enrich cascade, `enrich → prep → embed → score`. Its own work is
nil, as the `pull` job's. Under `NULL` there is no queue to take it."""
if not context.task(chain("enrich")):
raise RuntimeError("no queue to start the enrich cascade on")
return {"started": " > ".join(CASCADES["enrich"])}
@job("fetch_text", fetch_text.LABEL, solo=True, resource="http")
def run_fetch_text(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Ask for the fetch text cascade, the pull's chain from fetch text on.
Its own work is nil, as the `pull` job's. Under `NULL` there is no queue
to take it."""
if not context.task(chain("fetch_text")):
raise RuntimeError("no queue to start the fetch text cascade on")
return {"started": " > ".join(CASCADES["fetch_text"])}
def enrich_after(queue: Any, score_job: Any) -> Any:
"""Queue the `enrich` job behind `score_job`, the rescore that follows a
change to the ranking, however that score ends. One per score job
(`enrich:after:<id>`), so ten changes answered by one score are one
enrich. None without a score job, or when the queue refuses."""
def fetch_text_after(queue: Any, score_job: Any) -> Any:
"""Queue the `fetch_text` job behind `score_job`, the rescore that
follows a change to the ranking, however that score ends. One per score
job (`fetch_text:after:<id>`), so ten changes answered by one score are
one cascade. None without a score job, or when the queue refuses."""
if score_job is None:
return None
return queue.enqueue("enrich", depends_on=[score_job.id], after="done",
dedup_key=f"enrich:after:{score_job.id}")
return queue.enqueue("fetch_text", depends_on=[score_job.id], after="done",
dedup_key=f"fetch_text:after:{score_job.id}")
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
@@ -255,5 +259,5 @@ for _name in ("extract", "embed", "score", "summarise", "judge", "prep"):
# setting: a cadence edited or switched off under Runs survives
# every restart (see pipeline/schedule.py).
declare("pull", "pull", "every 6h",
label="Pull every board, then map, embed, score, read the best roles' pages, "
"extract, rescore, summarise and judge against the CVs")
label="Pull every board, then map, embed, score, fetch the text of the roles in the "
"cut, extract, rescore, judge against the CVs and summarise")