Rank by the cut: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text renamed from enrich, the feed kept to the fetch gate, and migration 32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
060ccf66e6
commit
617356e60c
35 files changed
+1054
-751
No files matched your search
+51
-47
@@ -1,32 +1,29 @@
|
||||
"""The tasks roledesk offers, and the order a pull runs them in.
|
||||
|
||||
pull → prep → embed → score → enrich → prep → embed → score → extract → score
|
||||
→ summarise → judge
|
||||
pull → prep → embed → score → fetch_text → prep → embed → score → extract
|
||||
→ score → judge → summarise
|
||||
|
||||
newsdesk's cascade convention: a cascade is a tuple of task names, and the
|
||||
final job of one task starts the next (`step` is the callback it carries).
|
||||
Nothing is alive between two tasks. `prep` (`app/jobs/prep.py`) maps what
|
||||
the pull stored and folds the copies before anything reads the roles; after
|
||||
`enrich` it maps the roles their own pages filled. A stage is addressed by its position in
|
||||
the tuple, not its name, since `score` runs twice.
|
||||
`fetch_text` it maps the roles their own pages filled. A stage is addressed
|
||||
by its position in the tuple, not its name, since `score` runs three times.
|
||||
|
||||
Every stage is always in the chain, and a stage that is switched off finds
|
||||
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this pull does not
|
||||
satisfy, is an extract stage with no batches whose final job moves straight
|
||||
on. The free stages go first: embed and score cost no tokens, and the score
|
||||
they produce is what bounds the rest. Enrich reads the best roles without
|
||||
text on their own pages (`app/jobs/enrich.py`), and prep, embed and score
|
||||
follow for the roles it filled; when it filled none, `step` passes over all
|
||||
three. Extract
|
||||
reads only the best
|
||||
roles of each enabled agent, as deep and as low as that agent's own `limits`
|
||||
say; the second score then applies the gates to the columns it filled.
|
||||
Summarise follows, on the best roles by the same agent's summarise limits,
|
||||
and judge runs last, reading each agent's best roles against its CV by its
|
||||
judge limits. The `enrich` cascade, which follows the rescore after an agent
|
||||
is saved, ends with judge too, so a changed CV, prompt or like text is read
|
||||
again at once rather than at the next pull; a pair whose `input_key` did not
|
||||
change costs nothing.
|
||||
nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this
|
||||
pull does not satisfy, is a stage with no batches whose final job moves
|
||||
straight on. The free stages go first: embed and score cost no tokens, and
|
||||
score writes each agent's cut (`app.jobs.score`), which bounds the rest.
|
||||
Fetch text reads the role's own page for the roles in the cut that came
|
||||
without text (`app/jobs/fetch_text.py`), and prep, embed and score follow
|
||||
for the roles it filled, so their levels and the cut are recomputed; when
|
||||
it filled none, `step` passes over all three. Extract reads the cut; the
|
||||
score after it applies the fetch gate and the cut again with the columns
|
||||
extract filled, so a role can drop out or enter (one that enters is
|
||||
extracted on the next pull). Judge reads each agent's cut as recomputed,
|
||||
against its CV. Summarise runs last, on the top of each agent's feed by
|
||||
final score and the roles liked or saved: it is no part of any score.
|
||||
|
||||
A batch that failed, or a stage somebody killed, does not stop the chain: the
|
||||
work comes round on the next pull, and a rescore of what did arrive is worth
|
||||
@@ -36,10 +33,13 @@ having either way.
|
||||
and a cascade is started by one: the `pull` job's only work is to ask for the
|
||||
chain. Its default schedule, `every 6h`, is declared here beside it.
|
||||
|
||||
`enrich` is a job and a cascade the same way, `enrich → prep → embed → score`, queued
|
||||
behind the `score` that follows a change to the ranking other than a pull's
|
||||
(`enrich_after`): an agent saved, a company's affinity changed. A vote moves
|
||||
one agent's taste a little and queues neither.
|
||||
`fetch_text` is a job and a cascade the same way, the pull's chain from
|
||||
fetch text on, queued behind the `score` that follows a change to the
|
||||
ranking other than a pull's (`fetch_text_after`): an agent saved, a
|
||||
company's affinity or overrides changed. The cut may have changed, so its
|
||||
new roles get their text, extract and judge at once rather than at the next
|
||||
pull; a judgement whose `input_key` did not change costs nothing. A vote
|
||||
moves one agent's taste a little and queues neither.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -49,19 +49,22 @@ from dataclasses import replace
|
||||
from functools import partial
|
||||
from typing import Any
|
||||
|
||||
from app.jobs import embed, enrich, extract, judge, prep, pull, score, summarise # noqa: F401 -- registers the jobs
|
||||
from app.jobs import ( # noqa: F401 -- registers the jobs
|
||||
embed, extract, fetch_text, judge, prep, pull, score, summarise,
|
||||
)
|
||||
from desk.tools import agent
|
||||
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
|
||||
|
||||
STEP = "app.jobs.cascade:step"
|
||||
|
||||
CASCADES: dict[str, tuple[str, ...]] = {
|
||||
"pull": ("pull", "prep", "embed", "score", "enrich", "prep", "embed", "score", "extract",
|
||||
"score", "summarise", "judge"),
|
||||
"enrich": ("enrich", "prep", "embed", "score", "judge"),
|
||||
"pull": ("pull", "prep", "embed", "score", "fetch_text", "prep", "embed", "score",
|
||||
"extract", "score", "judge", "summarise"),
|
||||
"fetch_text": ("fetch_text", "prep", "embed", "score", "extract", "score", "judge",
|
||||
"summarise"),
|
||||
}
|
||||
|
||||
#: What follows `enrich` for the roles it filled, passed over when it filled none.
|
||||
#: What follows `fetch_text` for the roles it filled, passed over when it filled none.
|
||||
REFILL = ("prep", "embed", "score")
|
||||
|
||||
|
||||
@@ -131,7 +134,7 @@ STAGES: dict[str, Callable[[str], Task]] = {
|
||||
label="Mapping the roles' host words"),
|
||||
"summarise": _summarise,
|
||||
"judge": _judge,
|
||||
"enrich": lambda cascade: enrich.enrich_task(),
|
||||
"fetch_text": lambda cascade: fetch_text.fetch_text_task(),
|
||||
}
|
||||
|
||||
|
||||
@@ -157,7 +160,7 @@ def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
|
||||
name, done = payload.get("cascade", ""), payload.get("task", "")
|
||||
at = int(payload.get("at", -1))
|
||||
tasks = CASCADES.get(name, ())
|
||||
if done == "enrich" and not enrich.filled(tally) \
|
||||
if done == "fetch_text" and not fetch_text.filled(tally) \
|
||||
and tasks[at + 1:at + 1 + len(REFILL)] == REFILL:
|
||||
at += len(REFILL)
|
||||
following = follows(name, at)
|
||||
@@ -222,24 +225,25 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
|
||||
return {"started": " > ".join(CASCADES["pull"]), "run": run}
|
||||
|
||||
|
||||
@job("enrich", "Reading the best roles on their own pages", solo=True, resource="http")
|
||||
def run_enrich(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Ask for the enrich cascade, `enrich → prep → embed → score`. Its own work is
|
||||
nil, as the `pull` job's. Under `NULL` there is no queue to take it."""
|
||||
if not context.task(chain("enrich")):
|
||||
raise RuntimeError("no queue to start the enrich cascade on")
|
||||
return {"started": " > ".join(CASCADES["enrich"])}
|
||||
@job("fetch_text", fetch_text.LABEL, solo=True, resource="http")
|
||||
def run_fetch_text(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Ask for the fetch text cascade, the pull's chain from fetch text on.
|
||||
Its own work is nil, as the `pull` job's. Under `NULL` there is no queue
|
||||
to take it."""
|
||||
if not context.task(chain("fetch_text")):
|
||||
raise RuntimeError("no queue to start the fetch text cascade on")
|
||||
return {"started": " > ".join(CASCADES["fetch_text"])}
|
||||
|
||||
|
||||
def enrich_after(queue: Any, score_job: Any) -> Any:
|
||||
"""Queue the `enrich` job behind `score_job`, the rescore that follows a
|
||||
change to the ranking, however that score ends. One per score job
|
||||
(`enrich:after:<id>`), so ten changes answered by one score are one
|
||||
enrich. None without a score job, or when the queue refuses."""
|
||||
def fetch_text_after(queue: Any, score_job: Any) -> Any:
|
||||
"""Queue the `fetch_text` job behind `score_job`, the rescore that
|
||||
follows a change to the ranking, however that score ends. One per score
|
||||
job (`fetch_text:after:<id>`), so ten changes answered by one score are
|
||||
one cascade. None without a score job, or when the queue refuses."""
|
||||
if score_job is None:
|
||||
return None
|
||||
return queue.enqueue("enrich", depends_on=[score_job.id], after="done",
|
||||
dedup_key=f"enrich:after:{score_job.id}")
|
||||
return queue.enqueue("fetch_text", depends_on=[score_job.id], after="done",
|
||||
dedup_key=f"fetch_text:after:{score_job.id}")
|
||||
|
||||
|
||||
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
|
||||
@@ -255,5 +259,5 @@ for _name in ("extract", "embed", "score", "summarise", "judge", "prep"):
|
||||
# setting: a cadence edited or switched off under Runs survives
|
||||
# every restart (see pipeline/schedule.py).
|
||||
declare("pull", "pull", "every 6h",
|
||||
label="Pull every board, then map, embed, score, read the best roles' pages, "
|
||||
"extract, rescore, summarise and judge against the CVs")
|
||||
label="Pull every board, then map, embed, score, fetch the text of the roles in the "
|
||||
"cut, extract, rescore, judge against the CVs and summarise")
|
||||
Reference in new issue
Block a user