Files
jobboard/desk/modeltask.py
T
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

506 lines
24 KiB
Python

"""A task for the model: rows that are waiting, one call per chunk, answers
written back. The machine tagging and summarising both run in.
`agent.py` answers *how do I reach a model* — the API, or the local `claude`.
This module answers the layer above: *what do I send it, and what do I do with
what comes back*. Everything between those two lives here, once.
ModelTask the parts: waiting, build_batch, system, schema, apply,
missed
one one batch: one model call, one apply. What a job is.
of that as a `Task`: the batches, and how to find them
run every batch here and now, for a caller with no queue
Tagging and summarising used to be near-twins — the same loop, the same
failure handling, the same job body and the same resume, spelled out twice
and drifting by a line each merge. What actually differs between them is
small and is all *content*: which rows are waiting, what goes in the batch,
what the model is told, what shape it must answer in, what to do with the
answer. That is a `ModelTask`; this module is everything that is not one.
The third one — clustering, or whatever comes next — is a prompt, a schema
and two functions over the store, and no loop.
"""
from __future__ import annotations
import logging
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import Any
from desk.config import HasAgent, ModelTaskConfig
from pipeline.context import NULL, RunContext
from pipeline.task import Batch, Tally, Task, resolve
from desk.models import ModelTaskResult, Notice
from desk.notices import classify, stops_the_run
from desk.tools import agent
log = logging.getLogger(__name__)
#: `notices.py`'s code for "the model's safeguards would not take this text".
#: Named here because `one` has to recognise it by more than its wording: it
#: is the only failure that says something about a particular row, and so the
#: only one worth asking about again with fewer rows.
REFUSED = "agent.refused"
@dataclass(frozen=True, slots=True)
class ModelTask[R: ModelTaskResult]:
"""What one task for the model consists of. See tagging.TASK for one."""
#: The job kind.
kind: str
label: str
#: For progress lines and the idle result: "tagging" / "tags" /
#: "nothing untagged".
verb: str
noun: str
idle: str
#: The key under which `build_batch` lists its rows. Both of newsdesk's
#: say "items", and they used to say "articles" and "segments": a batch is
#: a mixed archive and naming it after one shape was a claim about the
#: rows in it that stopped being true.
items: str
#: Which config section governs it.
settings: Callable[[Any], ModelTaskConfig]
#: How many rows are waiting: `(config, store)`.
pending: Callable[[Any, Any], int]
#: Which rows are waiting, by uid, most recent first: `(config, store, limit)`.
#: What the task splits over — one batch per model call — so it asks for
#: identifiers rather than the payload, which each batch builds for itself
#: when it runs.
waiting: Callable[[Any, Any, int], list[str]]
#: The payload the model works from, its prompt included:
#: `(config, store, limit, uids)`. Given uids it builds that batch, and
#: asks the store again rather than trusting the list — a row may have been
#: done in the seconds since the task shared the work out.
build_batch: Callable[..., dict[str, Any]]
# All three take the config because *which rows are waiting* is no longer a
# question the store can answer on its own: summarising is governed by what
# each installed plugin declares about its own items, and the plugins are
# found through the config. A task that does not care ignores the argument.
#: The system prompt, given the store rather than fixed, because tagging's
#: carries the vocabulary and that grows between one run and the next.
system: Callable[[Any], str]
schema: dict[str, Any]
#: Write an answer back: `(store, answer, batch, config)`. Tolerant of a
#: sloppy response, and the only place the answer's shape is trusted.
apply: Callable[[Any, dict[str, Any], dict[str, Any] | None, Any], R]
#: `(store, uids, why="")`: the model was handed these rows and said
#: nothing usable about them. Count that against each one still waiting,
#: and answer how many of them that has now given up on. `why` is
#: `refused` when the safeguards turned the row down, so the Given up list
#: can say so.
#:
#: Every task needs one and there is no default, because the default is the
#: bug. A row the model cannot satisfy — a segment whose only honest
#: abstract is shorter than the guard, a uid it drops from every reply — is
#: otherwise found by the next `waiting`, sent again, and paid for again,
#: on every pull, for as long as the row exists. Where the count is kept is
#: the store's business; that there is a bound at all is this module's.
missed: Callable[..., int]
#: The result type, so totals start as the right kind of zero.
result: type[R]
#: `module:function` run once when every batch of the run is over, given
#: `(config, store, tally)` and the run context. "" for a task with
#: nothing to settle, which summarising is: an abstract is written for one
#: segment by the batch that read it, and no other batch has an opinion.
#:
#: For the decision a batch is in no position to make. Tagging's new tags
#: are the case: the cap is *per run*, and a batch that can see neither
#: the other thirty-nine nor what they proposed can only enforce it
#: forty times over. So a batch reports what it wants, and this is where
#: the run's own answer is worked out — once, over the whole tally.
#:
#: It returns the fields of `result` that settling added, and both paths
#: take it from there: queued, the final job reports it; direct, `_fold`
#: absorbs it into the total. Same function either way.
settle: str = ""
#: The field every answer in the reply names its row by. Both tasks say
#: `uid`, which is not a coincidence — an answer that cannot say which row
#: it is about is not applicable — and counting it in the streamed reply is
#: what moves the progress bar per item rather than per batch. A task whose
#: answers are not one-per-row sets it to "" and gets the old behaviour.
answer_key: str = "uid"
class _Answered:
"""Counts finished answers in a reply that is still arriving.
The model writes its answers one after another, so the `"uid"` that opens
each one is a reliable "another row is done" marker tens of seconds before
the request that carries it completes. Counting a substring is a blunt
instrument and deliberately so: the alternative is incremental JSON
parsing, which is a great deal of machinery to be *approximately as* right
about a number that only drives a bar. It cannot overcount — the key
appears once per answer, and the payload it is counted in is the model's
own output, not the batch we sent — and if it undercounts, the bar catches
up when the chunk lands and the real total is known.
`tail` is why this is a class rather than a closure: a key can be split
across two deltas, so the last few characters of each are carried into the
next. Carrying `len(needle) - 1` is exactly enough to catch one that spans
the join, and too few for anything already counted to be seen twice.
"""
def __init__(self, key: str, report: Callable[[int], None]) -> None:
self._needle = f'"{key}"'
self._report = report
self._tail = ""
self.count = 0
def __call__(self, piece: str) -> None:
window = self._tail + piece
found = window.count(self._needle)
self._tail = window[1 - len(self._needle):] if len(self._needle) > 1 else ""
if found:
self.count += found
self._report(self.count)
def one[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
uids: Sequence[str], *, context: RunContext = NULL) -> R:
"""One batch: one model call, one apply. What a job is.
The unit the queue admits. It used to be a whole backlog inside a single
job, which opened four threads of its own — so the queue counted one thing
running where four requests were in the air, and four model workers was a
number nobody was keeping. Now the job and the request are
the same thing and the count is true.
The prompt is built here rather than when the batch was handed out: for
tagging that carries the vocabulary as it stands when this batch starts,
which is what somebody adding a tag by hand mid-run should reach. What it
does *not* carry is what a sibling batch proposed a second ago — nothing
of this run's proposals exists yet, because they are settled once, when
every batch is over. See `settle`.
Raises `agent.AgentError` when no model can be reached, or when the one
that could gave nothing. The job fails, saying which — and the notice says
what to do about it, because both of those are things you fix rather than
things to work around.
A row the model answers is done with. One it leaves out, or answers in a
way `apply` will not take, is charged a turn against `task.missed` — so it
comes round again a bounded number of times rather than on every pull for
as long as it exists.
A batch the model's safeguards refuse outright is split and asked again in
halves rather than failed, because a refusal is about the text of one row
and the other rows in it are owed their answers. Down to a single row it
is charged like any other unanswered one and the run carries on. See
`_narrow`.
It counts its own rows and nothing else. As a job that is all there is; run
through `Task.run` instead, the caller shifts the progress into the whole
backlog, which is the only place that knows how big it is.
"""
context.check()
cfg = task.settings(config)
batch = task.build_batch(config, store, len(uids) or cfg.batch_size,
list(uids))
rows = batch[task.items]
if not rows:
# Done by someone else between the task sharing the work out and
# now, which is ordinary:
# every task rediscovers its backlog and the batches overlap at the
# edges. Nothing to ask, nothing to write.
return task.result(reason=task.idle)
name = agent.backend(config.agent)
if not name:
# The notice goes out from here rather than being left to the failure,
# because nothing was asked and so the `except` below never runs. Left
# to itself this reached the screen only as "Tagging a batch failed",
# filed as a bug, about a machine that is working correctly and is one
# install or one config line from working here too. The configured
# backend is what is passed, not the one that ran: there wasn't one,
# and `off` is the difference between a missing install and a switch.
unreachable = agent.AgentError("no model is reachable from here")
_report(context, classify(unreachable, backend=config.agent.backend))
raise unreachable
total = len(rows)
def report(done: int) -> None:
done = min(done, total)
context.progress(done, total, f"{task.verb} {task.items}, {done} of {total}")
context.progress(0, total, f"asking {name} about {total} {task.items}")
counted = (_Answered(task.answer_key, report) if task.answer_key else None)
try:
context.check()
reply = agent.ask_json(
config.agent,
system=task.system(store),
question={task.items: rows},
schema=task.schema,
model=config.agent.model or cfg.model,
thinking=cfg.thinking,
using=name,
on_partial=counted,
)
except agent.AgentError as exc:
notice = classify(exc, backend=name)
# A refusal is the one failure that is about the rows rather than
# about the call, and so the one that answers "which of them?" by
# being asked again with fewer. Everything else falls through to the
# raise below unchanged.
if notice.code == REFUSED and len(rows) > 1:
return _narrow(task, config, store, _uids(task, rows),
context=context)
log.exception("%s batch of %d failed on %s", task.verb, len(rows), name)
# The notice carries the sentence and the fix — "Claude Code is signed
# out" and the command that mends it, rather than the token that
# expired — to where you will see it. Then the batch fails: its rows
# keep their watermark and come round on the next run — uncharged,
# because nothing was asked of a model.
#
# Its siblings are unaffected only while the failure was this batch's.
# When it is the account's, `_report` stops the ones still queued: see
# `notices.stops_the_run`.
if notice.code == REFUSED:
# Down to one row, so the notice can name it — and has to. "The
# safeguards wouldn't take this text" is not a sentence anyone can
# act on without knowing which of the day's articles it is about,
# and the row is about to leave the backlog quietly. `key` is the
# title too, so two refusals in one run are two lines rather than
# one line with a count of two.
about = rows[0].get("title") or _uids(task, rows)[0]
notice = notice.model_copy(update={
"text": f"{notice.text} — {about}", "key": str(about)})
_report(context, notice)
if notice.code == REFUSED:
# Down to one row, and that row is the one. Charged and not
# raised: the rule about never charging a failure is a rule about
# failures that prove nothing, and this one has just proved
# exactly which row it is about. Left uncharged it is found by
# the next `waiting`, sent again, refused again, and poisons
# whichever batch it lands in for as long as the row exists —
# which is how two articles held eighteen others' abstracts.
totals = task.result(reason=notice.text)
totals.failed = 1
totals.given_up = task.missed(store, _uids(task, rows), why="refused")
totals.pending = task.pending(config, store)
return totals
raise
# Apply even if killed during the call: the spend has happened. The worker
# records cancellation at return while preserving the paid-for answer.
totals = task.apply(store, reply.data, batch, config)
# Whatever is still waiting now is what the model was asked about and did
# not settle, so it is charged one turn — before `pending` is read, so a
# row this call gave up on has already left the number the run reports.
# Charged on a reply and never on a failure: a batch that never reached a
# model cost nothing and proved nothing about its rows.
totals.given_up = task.missed(store, uids)
totals.batches = 1
totals.backend = reply.backend
totals.pending = task.pending(config, store)
report(total)
return totals
def _uids(task: ModelTask[Any], rows: Sequence[dict[str, Any]]) -> list[str]:
"""Which rows a built batch is about, from the batch itself.
`one` is given uids, but what it sent is `rows` — `build_batch` asks the
store again and drops whatever was finished in between. On the path this
serves, that difference is the whole point: a row nobody sent is a row
nobody may charge.
"""
return [str(row[task.answer_key]) for row in rows if row.get(task.answer_key)]
def _narrow[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
uids: Sequence[str], *,
context: RunContext) -> R:
"""Ask about these rows again in two halves, and add up what comes back.
A refusal names no row. It is one answer about the whole request, so the
only way to learn which row it is about is to send fewer — and the reason
to bother is that the rest of the batch is fine and is owed its abstracts.
The split costs a handful of extra calls, once, against one call that
summarises nobody and is made again on every pull.
Halves rather than one row at a time, because a clean half lands whole:
the cost is set by how many rows are flagged rather than by how many are
in the batch, and two flagged rows in one batch is ordinary — they arrive
from the same day's news.
Each half is a whole `one`, so a half that meets an unrelated failure
raises it exactly as it would have unsplit, and the batch fails the way it
always did.
The bad case, if the classifier ever took against the *prompt* rather than
any row: every single-row call is refused, every row is charged, and three
pulls empty the backlog into given-up. Bounded and visible — the status
line counts them and `reopen_given_up` asks again — but it is why the
notice names the text rather than the row.
"""
middle = len(uids) // 2
totals = task.result()
for half in (uids[:middle], uids[middle:]):
if not half:
continue
landed = one(task, config, store, half, context=context)
totals.absorb(landed)
# `absorb` leaves the fields that are about the run rather than the
# chunk, and which backend answered is one of them — the same reading
# `_fold` takes of the same shape.
totals.backend = landed.backend or totals.backend
totals.pending = task.pending(config, store)
return totals
def _report(context: RunContext, notice: Notice) -> None:
"""Put one failure on screen, and stop the run if it will meet the same one.
The two halves belong together because they are one judgement made once. A
batch that cannot reach a model, or is told the spend limit is reached, has
learned something about the *account* and not about its own rows — and the
twenty-two batches queued behind it would each pay a call to be told it
again, and each put the same line on screen. One batch fails, one notice
goes out, and the rest are cancelled rather than spent.
Which failures those are is the catalogue's to say and not this module's: a
timeout or a reply that was not JSON is this batch's bad luck, and the next
batch deserves its turn. See `notices.stops_the_run`.
"""
context.notice(notice)
if stops_the_run(notice):
context.stop(notice.text)
# ----------------------------------------------------------------- the task
def batches(task: ModelTask[Any], limit: int | None = None) -> Any:
"""The finder: what is waiting, cut into one payload per model call.
How many to a call is the task's own business — twenty-five articles is
what one call and one schema hold, and no generic queue could know it. How
many run at once is the queue's, because that is how many workers it has.
It asks for identifiers rather than the rows, and each batch asks the store
again when it runs: a row may have been done in the seconds since the work
was shared out.
`enabled: false` is answered here, and answered as nothing to do rather
than as a refusal. Here, because this is the one place every route to the
work passes through: the stage of a cascade, the button, and `run("tag")`
are three callers of this same finder, and a switch only one of them read
would be a switch that turned the task off in one place. As nothing to do,
because a task that finds nothing still queues its final job — so a
cascade whose tagging is off carries on to whatever follows rather than
ending on it.
"""
def find(config: HasAgent, store: Any) -> list[Batch]:
cfg = task.settings(config)
if not cfg.enabled:
return []
uids = task.waiting(config, store,
limit if limit is not None else 100_000)
size = max(cfg.batch_size, 1)
cut = [uids[at:at + size] for at in range(0, len(uids), size)]
return [Batch({"uids": chunk}, label=_label(task, len(chunk), n, len(cut)))
for n, chunk in enumerate(cut, start=1)]
return find
def _label(task: ModelTask[Any], rows: int, n: int, of: int) -> str:
"""What one batch is called in the queue: "Tagging 25 items, 2 of 9".
The kind's own label is "Tagging a batch" on all nine of them, which is
true and useless — the queue lists nine identical rows and nothing says
which is which or how much is left. The slice is the only thing that tells
them apart, and this is the one place it is known: the finder cut them, and
the batch that runs half an hour later knows only its own uids.
Its shape is the progress line's, capitalised because it is a name rather
than a sentence about what is happening. The tail is dropped for a task of
one batch, where "1 of 1" says nothing.
"""
said = f"{task.verb} {rows} {task.items}".capitalize()
return said if of < 2 else f"{said}, {n} of {of}"
def _fold(task: ModelTask[Any]) -> Any:
"""Add the batches up, for the direct path only. Under the queue each batch
is its own job with its own row, and there is nothing to add.
The settle runs here too, off the same tally shape the final job builds,
because "what may this run add" is a question about the run and not about
the queue. Otherwise `run("tag")` in a notebook would be the one caller
with no cap at all.
"""
def fold(config: HasAgent, store: Any, results: list[Any]) -> dict[str, Any]:
if not results:
return task.result(reason=task.idle).model_dump()
totals = task.result()
for result in results:
if isinstance(result, Exception):
# One batch that could not be answered is not a failed run: as
# eight jobs it would be one failing and seven landing. Its rows
# keep their watermark and come round next time.
totals.failed += 1
totals.reason = str(result)
continue
landed = task.result.model_validate(result)
totals.absorb(landed)
# `absorb` adds up counts and leaves the fields that are about the
# run rather than the batch. Which backend answered is one of
# those, and the run wants the last one to have said anything.
totals.backend = landed.backend or totals.backend
if task.settle:
answered = tuple(r for r in results if isinstance(r, dict))
settled = resolve(task.settle)(
config, store, Tally(done=len(answered), results=answered),
context=NULL) or {}
totals.absorb(task.result.model_validate(settled))
# After the settle, so a tag it added — and the rows it went back and
# tagged — are counted in what the run reports it did.
totals.pending = task.pending(config, store)
return totals.model_dump()
return fold
def of(task: ModelTask[Any], *, limit: int | None = None, name: str = "",
callback: str = "", carries: dict[str, Any] | None = None) -> Task:
"""The `Task` for one model task: its batches, and how to find them.
A `ModelTask` is the *content* — which rows are waiting, what the model is
told, what to do with the answer. This is that content as a step of the
pipeline, which is the same shape as every other step and is queued and
killed the same way.
"""
return Task(
kind=f"{task.kind}_batch", name=name or task.kind,
finder=batches(task, limit), callback=callback, settle=task.settle,
carries=dict(carries or {}),
size=lambda payload: len(payload.get("uids") or ()),
fold=_fold(task),
)
def run[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
limit: int | None = None, *, context: RunContext = NULL) -> dict[str, Any]:
"""Every batch, here and now, with no queue. The direct path and nothing else.
What a Claude Code session, a notebook and a test get: one call that blocks
and hands back the total. Under the queue this same task is `of(...)`
started against the queue, its batches are independent jobs, and there is
no total until the final job reads how they went — which is the whole point
of splitting it.
Same `one` either way, and the same batches from the same finder, so there
is no second implementation to drift.
"""
return of(task, limit=limit).run(config, store, context=context)