The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
506 lines
24 KiB
Python
506 lines
24 KiB
Python
"""A task for the model: rows that are waiting, one call per chunk, answers
|
|
written back. The machine tagging and summarising both run in.
|
|
|
|
`agent.py` answers *how do I reach a model* — the API, or the local `claude`.
|
|
This module answers the layer above: *what do I send it, and what do I do with
|
|
what comes back*. Everything between those two lives here, once.
|
|
|
|
ModelTask the parts: waiting, build_batch, system, schema, apply,
|
|
missed
|
|
one one batch: one model call, one apply. What a job is.
|
|
of that as a `Task`: the batches, and how to find them
|
|
run every batch here and now, for a caller with no queue
|
|
|
|
Tagging and summarising used to be near-twins — the same loop, the same
|
|
failure handling, the same job body and the same resume, spelled out twice
|
|
and drifting by a line each merge. What actually differs between them is
|
|
small and is all *content*: which rows are waiting, what goes in the batch,
|
|
what the model is told, what shape it must answer in, what to do with the
|
|
answer. That is a `ModelTask`; this module is everything that is not one.
|
|
|
|
The third one — clustering, or whatever comes next — is a prompt, a schema
|
|
and two functions over the store, and no loop.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from collections.abc import Callable, Sequence
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
from desk.config import HasAgent, ModelTaskConfig
|
|
from pipeline.context import NULL, RunContext
|
|
from pipeline.task import Batch, Tally, Task, resolve
|
|
from desk.models import ModelTaskResult, Notice
|
|
from desk.notices import classify, stops_the_run
|
|
from desk.tools import agent
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
#: `notices.py`'s code for "the model's safeguards would not take this text".
|
|
#: Named here because `one` has to recognise it by more than its wording: it
|
|
#: is the only failure that says something about a particular row, and so the
|
|
#: only one worth asking about again with fewer rows.
|
|
REFUSED = "agent.refused"
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class ModelTask[R: ModelTaskResult]:
|
|
"""What one task for the model consists of. See tagging.TASK for one."""
|
|
|
|
#: The job kind.
|
|
kind: str
|
|
label: str
|
|
#: For progress lines and the idle result: "tagging" / "tags" /
|
|
#: "nothing untagged".
|
|
verb: str
|
|
noun: str
|
|
idle: str
|
|
#: The key under which `build_batch` lists its rows. Both of newsdesk's
|
|
#: say "items", and they used to say "articles" and "segments": a batch is
|
|
#: a mixed archive and naming it after one shape was a claim about the
|
|
#: rows in it that stopped being true.
|
|
items: str
|
|
#: Which config section governs it.
|
|
settings: Callable[[Any], ModelTaskConfig]
|
|
#: How many rows are waiting: `(config, store)`.
|
|
pending: Callable[[Any, Any], int]
|
|
#: Which rows are waiting, by uid, most recent first: `(config, store, limit)`.
|
|
#: What the task splits over — one batch per model call — so it asks for
|
|
#: identifiers rather than the payload, which each batch builds for itself
|
|
#: when it runs.
|
|
waiting: Callable[[Any, Any, int], list[str]]
|
|
#: The payload the model works from, its prompt included:
|
|
#: `(config, store, limit, uids)`. Given uids it builds that batch, and
|
|
#: asks the store again rather than trusting the list — a row may have been
|
|
#: done in the seconds since the task shared the work out.
|
|
build_batch: Callable[..., dict[str, Any]]
|
|
|
|
# All three take the config because *which rows are waiting* is no longer a
|
|
# question the store can answer on its own: summarising is governed by what
|
|
# each installed plugin declares about its own items, and the plugins are
|
|
# found through the config. A task that does not care ignores the argument.
|
|
#: The system prompt, given the store rather than fixed, because tagging's
|
|
#: carries the vocabulary and that grows between one run and the next.
|
|
system: Callable[[Any], str]
|
|
schema: dict[str, Any]
|
|
#: Write an answer back: `(store, answer, batch, config)`. Tolerant of a
|
|
#: sloppy response, and the only place the answer's shape is trusted.
|
|
apply: Callable[[Any, dict[str, Any], dict[str, Any] | None, Any], R]
|
|
#: `(store, uids, why="")`: the model was handed these rows and said
|
|
#: nothing usable about them. Count that against each one still waiting,
|
|
#: and answer how many of them that has now given up on. `why` is
|
|
#: `refused` when the safeguards turned the row down, so the Given up list
|
|
#: can say so.
|
|
#:
|
|
#: Every task needs one and there is no default, because the default is the
|
|
#: bug. A row the model cannot satisfy — a segment whose only honest
|
|
#: abstract is shorter than the guard, a uid it drops from every reply — is
|
|
#: otherwise found by the next `waiting`, sent again, and paid for again,
|
|
#: on every pull, for as long as the row exists. Where the count is kept is
|
|
#: the store's business; that there is a bound at all is this module's.
|
|
missed: Callable[..., int]
|
|
#: The result type, so totals start as the right kind of zero.
|
|
result: type[R]
|
|
#: `module:function` run once when every batch of the run is over, given
|
|
#: `(config, store, tally)` and the run context. "" for a task with
|
|
#: nothing to settle, which summarising is: an abstract is written for one
|
|
#: segment by the batch that read it, and no other batch has an opinion.
|
|
#:
|
|
#: For the decision a batch is in no position to make. Tagging's new tags
|
|
#: are the case: the cap is *per run*, and a batch that can see neither
|
|
#: the other thirty-nine nor what they proposed can only enforce it
|
|
#: forty times over. So a batch reports what it wants, and this is where
|
|
#: the run's own answer is worked out — once, over the whole tally.
|
|
#:
|
|
#: It returns the fields of `result` that settling added, and both paths
|
|
#: take it from there: queued, the final job reports it; direct, `_fold`
|
|
#: absorbs it into the total. Same function either way.
|
|
settle: str = ""
|
|
#: The field every answer in the reply names its row by. Both tasks say
|
|
#: `uid`, which is not a coincidence — an answer that cannot say which row
|
|
#: it is about is not applicable — and counting it in the streamed reply is
|
|
#: what moves the progress bar per item rather than per batch. A task whose
|
|
#: answers are not one-per-row sets it to "" and gets the old behaviour.
|
|
answer_key: str = "uid"
|
|
|
|
|
|
class _Answered:
|
|
"""Counts finished answers in a reply that is still arriving.
|
|
|
|
The model writes its answers one after another, so the `"uid"` that opens
|
|
each one is a reliable "another row is done" marker tens of seconds before
|
|
the request that carries it completes. Counting a substring is a blunt
|
|
instrument and deliberately so: the alternative is incremental JSON
|
|
parsing, which is a great deal of machinery to be *approximately as* right
|
|
about a number that only drives a bar. It cannot overcount — the key
|
|
appears once per answer, and the payload it is counted in is the model's
|
|
own output, not the batch we sent — and if it undercounts, the bar catches
|
|
up when the chunk lands and the real total is known.
|
|
|
|
`tail` is why this is a class rather than a closure: a key can be split
|
|
across two deltas, so the last few characters of each are carried into the
|
|
next. Carrying `len(needle) - 1` is exactly enough to catch one that spans
|
|
the join, and too few for anything already counted to be seen twice.
|
|
"""
|
|
|
|
def __init__(self, key: str, report: Callable[[int], None]) -> None:
|
|
self._needle = f'"{key}"'
|
|
self._report = report
|
|
self._tail = ""
|
|
self.count = 0
|
|
|
|
def __call__(self, piece: str) -> None:
|
|
window = self._tail + piece
|
|
found = window.count(self._needle)
|
|
self._tail = window[1 - len(self._needle):] if len(self._needle) > 1 else ""
|
|
if found:
|
|
self.count += found
|
|
self._report(self.count)
|
|
|
|
|
|
def one[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
|
|
uids: Sequence[str], *, context: RunContext = NULL) -> R:
|
|
"""One batch: one model call, one apply. What a job is.
|
|
|
|
The unit the queue admits. It used to be a whole backlog inside a single
|
|
job, which opened four threads of its own — so the queue counted one thing
|
|
running where four requests were in the air, and four model workers was a
|
|
number nobody was keeping. Now the job and the request are
|
|
the same thing and the count is true.
|
|
|
|
The prompt is built here rather than when the batch was handed out: for
|
|
tagging that carries the vocabulary as it stands when this batch starts,
|
|
which is what somebody adding a tag by hand mid-run should reach. What it
|
|
does *not* carry is what a sibling batch proposed a second ago — nothing
|
|
of this run's proposals exists yet, because they are settled once, when
|
|
every batch is over. See `settle`.
|
|
|
|
Raises `agent.AgentError` when no model can be reached, or when the one
|
|
that could gave nothing. The job fails, saying which — and the notice says
|
|
what to do about it, because both of those are things you fix rather than
|
|
things to work around.
|
|
|
|
A row the model answers is done with. One it leaves out, or answers in a
|
|
way `apply` will not take, is charged a turn against `task.missed` — so it
|
|
comes round again a bounded number of times rather than on every pull for
|
|
as long as it exists.
|
|
|
|
A batch the model's safeguards refuse outright is split and asked again in
|
|
halves rather than failed, because a refusal is about the text of one row
|
|
and the other rows in it are owed their answers. Down to a single row it
|
|
is charged like any other unanswered one and the run carries on. See
|
|
`_narrow`.
|
|
|
|
It counts its own rows and nothing else. As a job that is all there is; run
|
|
through `Task.run` instead, the caller shifts the progress into the whole
|
|
backlog, which is the only place that knows how big it is.
|
|
"""
|
|
context.check()
|
|
cfg = task.settings(config)
|
|
batch = task.build_batch(config, store, len(uids) or cfg.batch_size,
|
|
list(uids))
|
|
rows = batch[task.items]
|
|
if not rows:
|
|
# Done by someone else between the task sharing the work out and
|
|
# now, which is ordinary:
|
|
# every task rediscovers its backlog and the batches overlap at the
|
|
# edges. Nothing to ask, nothing to write.
|
|
return task.result(reason=task.idle)
|
|
|
|
name = agent.backend(config.agent)
|
|
if not name:
|
|
# The notice goes out from here rather than being left to the failure,
|
|
# because nothing was asked and so the `except` below never runs. Left
|
|
# to itself this reached the screen only as "Tagging a batch failed",
|
|
# filed as a bug, about a machine that is working correctly and is one
|
|
# install or one config line from working here too. The configured
|
|
# backend is what is passed, not the one that ran: there wasn't one,
|
|
# and `off` is the difference between a missing install and a switch.
|
|
unreachable = agent.AgentError("no model is reachable from here")
|
|
_report(context, classify(unreachable, backend=config.agent.backend))
|
|
raise unreachable
|
|
|
|
total = len(rows)
|
|
|
|
def report(done: int) -> None:
|
|
done = min(done, total)
|
|
context.progress(done, total, f"{task.verb} {task.items}, {done} of {total}")
|
|
|
|
context.progress(0, total, f"asking {name} about {total} {task.items}")
|
|
counted = (_Answered(task.answer_key, report) if task.answer_key else None)
|
|
|
|
try:
|
|
context.check()
|
|
reply = agent.ask_json(
|
|
config.agent,
|
|
system=task.system(store),
|
|
question={task.items: rows},
|
|
schema=task.schema,
|
|
model=config.agent.model or cfg.model,
|
|
thinking=cfg.thinking,
|
|
using=name,
|
|
on_partial=counted,
|
|
)
|
|
except agent.AgentError as exc:
|
|
notice = classify(exc, backend=name)
|
|
# A refusal is the one failure that is about the rows rather than
|
|
# about the call, and so the one that answers "which of them?" by
|
|
# being asked again with fewer. Everything else falls through to the
|
|
# raise below unchanged.
|
|
if notice.code == REFUSED and len(rows) > 1:
|
|
return _narrow(task, config, store, _uids(task, rows),
|
|
context=context)
|
|
log.exception("%s batch of %d failed on %s", task.verb, len(rows), name)
|
|
# The notice carries the sentence and the fix — "Claude Code is signed
|
|
# out" and the command that mends it, rather than the token that
|
|
# expired — to where you will see it. Then the batch fails: its rows
|
|
# keep their watermark and come round on the next run — uncharged,
|
|
# because nothing was asked of a model.
|
|
#
|
|
# Its siblings are unaffected only while the failure was this batch's.
|
|
# When it is the account's, `_report` stops the ones still queued: see
|
|
# `notices.stops_the_run`.
|
|
if notice.code == REFUSED:
|
|
# Down to one row, so the notice can name it — and has to. "The
|
|
# safeguards wouldn't take this text" is not a sentence anyone can
|
|
# act on without knowing which of the day's articles it is about,
|
|
# and the row is about to leave the backlog quietly. `key` is the
|
|
# title too, so two refusals in one run are two lines rather than
|
|
# one line with a count of two.
|
|
about = rows[0].get("title") or _uids(task, rows)[0]
|
|
notice = notice.model_copy(update={
|
|
"text": f"{notice.text} — {about}", "key": str(about)})
|
|
_report(context, notice)
|
|
if notice.code == REFUSED:
|
|
# Down to one row, and that row is the one. Charged and not
|
|
# raised: the rule about never charging a failure is a rule about
|
|
# failures that prove nothing, and this one has just proved
|
|
# exactly which row it is about. Left uncharged it is found by
|
|
# the next `waiting`, sent again, refused again, and poisons
|
|
# whichever batch it lands in for as long as the row exists —
|
|
# which is how two articles held eighteen others' abstracts.
|
|
totals = task.result(reason=notice.text)
|
|
totals.failed = 1
|
|
totals.given_up = task.missed(store, _uids(task, rows), why="refused")
|
|
totals.pending = task.pending(config, store)
|
|
return totals
|
|
raise
|
|
|
|
# Apply even if killed during the call: the spend has happened. The worker
|
|
# records cancellation at return while preserving the paid-for answer.
|
|
totals = task.apply(store, reply.data, batch, config)
|
|
# Whatever is still waiting now is what the model was asked about and did
|
|
# not settle, so it is charged one turn — before `pending` is read, so a
|
|
# row this call gave up on has already left the number the run reports.
|
|
# Charged on a reply and never on a failure: a batch that never reached a
|
|
# model cost nothing and proved nothing about its rows.
|
|
totals.given_up = task.missed(store, uids)
|
|
totals.batches = 1
|
|
totals.backend = reply.backend
|
|
totals.pending = task.pending(config, store)
|
|
report(total)
|
|
return totals
|
|
|
|
|
|
def _uids(task: ModelTask[Any], rows: Sequence[dict[str, Any]]) -> list[str]:
|
|
"""Which rows a built batch is about, from the batch itself.
|
|
|
|
`one` is given uids, but what it sent is `rows` — `build_batch` asks the
|
|
store again and drops whatever was finished in between. On the path this
|
|
serves, that difference is the whole point: a row nobody sent is a row
|
|
nobody may charge.
|
|
"""
|
|
return [str(row[task.answer_key]) for row in rows if row.get(task.answer_key)]
|
|
|
|
|
|
def _narrow[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
|
|
uids: Sequence[str], *,
|
|
context: RunContext) -> R:
|
|
"""Ask about these rows again in two halves, and add up what comes back.
|
|
|
|
A refusal names no row. It is one answer about the whole request, so the
|
|
only way to learn which row it is about is to send fewer — and the reason
|
|
to bother is that the rest of the batch is fine and is owed its abstracts.
|
|
The split costs a handful of extra calls, once, against one call that
|
|
summarises nobody and is made again on every pull.
|
|
|
|
Halves rather than one row at a time, because a clean half lands whole:
|
|
the cost is set by how many rows are flagged rather than by how many are
|
|
in the batch, and two flagged rows in one batch is ordinary — they arrive
|
|
from the same day's news.
|
|
|
|
Each half is a whole `one`, so a half that meets an unrelated failure
|
|
raises it exactly as it would have unsplit, and the batch fails the way it
|
|
always did.
|
|
|
|
The bad case, if the classifier ever took against the *prompt* rather than
|
|
any row: every single-row call is refused, every row is charged, and three
|
|
pulls empty the backlog into given-up. Bounded and visible — the status
|
|
line counts them and `reopen_given_up` asks again — but it is why the
|
|
notice names the text rather than the row.
|
|
"""
|
|
middle = len(uids) // 2
|
|
totals = task.result()
|
|
for half in (uids[:middle], uids[middle:]):
|
|
if not half:
|
|
continue
|
|
landed = one(task, config, store, half, context=context)
|
|
totals.absorb(landed)
|
|
# `absorb` leaves the fields that are about the run rather than the
|
|
# chunk, and which backend answered is one of them — the same reading
|
|
# `_fold` takes of the same shape.
|
|
totals.backend = landed.backend or totals.backend
|
|
totals.pending = task.pending(config, store)
|
|
return totals
|
|
|
|
|
|
def _report(context: RunContext, notice: Notice) -> None:
|
|
"""Put one failure on screen, and stop the run if it will meet the same one.
|
|
|
|
The two halves belong together because they are one judgement made once. A
|
|
batch that cannot reach a model, or is told the spend limit is reached, has
|
|
learned something about the *account* and not about its own rows — and the
|
|
twenty-two batches queued behind it would each pay a call to be told it
|
|
again, and each put the same line on screen. One batch fails, one notice
|
|
goes out, and the rest are cancelled rather than spent.
|
|
|
|
Which failures those are is the catalogue's to say and not this module's: a
|
|
timeout or a reply that was not JSON is this batch's bad luck, and the next
|
|
batch deserves its turn. See `notices.stops_the_run`.
|
|
"""
|
|
context.notice(notice)
|
|
if stops_the_run(notice):
|
|
context.stop(notice.text)
|
|
|
|
|
|
# ----------------------------------------------------------------- the task
|
|
|
|
def batches(task: ModelTask[Any], limit: int | None = None) -> Any:
|
|
"""The finder: what is waiting, cut into one payload per model call.
|
|
|
|
How many to a call is the task's own business — twenty-five articles is
|
|
what one call and one schema hold, and no generic queue could know it. How
|
|
many run at once is the queue's, because that is how many workers it has.
|
|
|
|
It asks for identifiers rather than the rows, and each batch asks the store
|
|
again when it runs: a row may have been done in the seconds since the work
|
|
was shared out.
|
|
|
|
`enabled: false` is answered here, and answered as nothing to do rather
|
|
than as a refusal. Here, because this is the one place every route to the
|
|
work passes through: the stage of a cascade, the button, and `run("tag")`
|
|
are three callers of this same finder, and a switch only one of them read
|
|
would be a switch that turned the task off in one place. As nothing to do,
|
|
because a task that finds nothing still queues its final job — so a
|
|
cascade whose tagging is off carries on to whatever follows rather than
|
|
ending on it.
|
|
"""
|
|
|
|
def find(config: HasAgent, store: Any) -> list[Batch]:
|
|
cfg = task.settings(config)
|
|
if not cfg.enabled:
|
|
return []
|
|
uids = task.waiting(config, store,
|
|
limit if limit is not None else 100_000)
|
|
size = max(cfg.batch_size, 1)
|
|
cut = [uids[at:at + size] for at in range(0, len(uids), size)]
|
|
return [Batch({"uids": chunk}, label=_label(task, len(chunk), n, len(cut)))
|
|
for n, chunk in enumerate(cut, start=1)]
|
|
|
|
return find
|
|
|
|
|
|
def _label(task: ModelTask[Any], rows: int, n: int, of: int) -> str:
|
|
"""What one batch is called in the queue: "Tagging 25 items, 2 of 9".
|
|
|
|
The kind's own label is "Tagging a batch" on all nine of them, which is
|
|
true and useless — the queue lists nine identical rows and nothing says
|
|
which is which or how much is left. The slice is the only thing that tells
|
|
them apart, and this is the one place it is known: the finder cut them, and
|
|
the batch that runs half an hour later knows only its own uids.
|
|
|
|
Its shape is the progress line's, capitalised because it is a name rather
|
|
than a sentence about what is happening. The tail is dropped for a task of
|
|
one batch, where "1 of 1" says nothing.
|
|
"""
|
|
said = f"{task.verb} {rows} {task.items}".capitalize()
|
|
return said if of < 2 else f"{said}, {n} of {of}"
|
|
|
|
|
|
def _fold(task: ModelTask[Any]) -> Any:
|
|
"""Add the batches up, for the direct path only. Under the queue each batch
|
|
is its own job with its own row, and there is nothing to add.
|
|
|
|
The settle runs here too, off the same tally shape the final job builds,
|
|
because "what may this run add" is a question about the run and not about
|
|
the queue. Otherwise `run("tag")` in a notebook would be the one caller
|
|
with no cap at all.
|
|
"""
|
|
|
|
def fold(config: HasAgent, store: Any, results: list[Any]) -> dict[str, Any]:
|
|
if not results:
|
|
return task.result(reason=task.idle).model_dump()
|
|
totals = task.result()
|
|
for result in results:
|
|
if isinstance(result, Exception):
|
|
# One batch that could not be answered is not a failed run: as
|
|
# eight jobs it would be one failing and seven landing. Its rows
|
|
# keep their watermark and come round next time.
|
|
totals.failed += 1
|
|
totals.reason = str(result)
|
|
continue
|
|
landed = task.result.model_validate(result)
|
|
totals.absorb(landed)
|
|
# `absorb` adds up counts and leaves the fields that are about the
|
|
# run rather than the batch. Which backend answered is one of
|
|
# those, and the run wants the last one to have said anything.
|
|
totals.backend = landed.backend or totals.backend
|
|
if task.settle:
|
|
answered = tuple(r for r in results if isinstance(r, dict))
|
|
settled = resolve(task.settle)(
|
|
config, store, Tally(done=len(answered), results=answered),
|
|
context=NULL) or {}
|
|
totals.absorb(task.result.model_validate(settled))
|
|
# After the settle, so a tag it added — and the rows it went back and
|
|
# tagged — are counted in what the run reports it did.
|
|
totals.pending = task.pending(config, store)
|
|
return totals.model_dump()
|
|
|
|
return fold
|
|
|
|
|
|
def of(task: ModelTask[Any], *, limit: int | None = None, name: str = "",
|
|
callback: str = "", carries: dict[str, Any] | None = None) -> Task:
|
|
"""The `Task` for one model task: its batches, and how to find them.
|
|
|
|
A `ModelTask` is the *content* — which rows are waiting, what the model is
|
|
told, what to do with the answer. This is that content as a step of the
|
|
pipeline, which is the same shape as every other step and is queued and
|
|
killed the same way.
|
|
"""
|
|
return Task(
|
|
kind=f"{task.kind}_batch", name=name or task.kind,
|
|
finder=batches(task, limit), callback=callback, settle=task.settle,
|
|
carries=dict(carries or {}),
|
|
size=lambda payload: len(payload.get("uids") or ()),
|
|
fold=_fold(task),
|
|
)
|
|
|
|
|
|
def run[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
|
|
limit: int | None = None, *, context: RunContext = NULL) -> dict[str, Any]:
|
|
"""Every batch, here and now, with no queue. The direct path and nothing else.
|
|
|
|
What a Claude Code session, a notebook and a test get: one call that blocks
|
|
and hands back the total. Under the queue this same task is `of(...)`
|
|
started against the queue, its batches are independent jobs, and there is
|
|
no total until the final job reads how they went — which is the whole point
|
|
of splitting it.
|
|
|
|
Same `one` either way, and the same batches from the same finder, so there
|
|
is no second implementation to drift.
|
|
"""
|
|
return of(task, limit=limit).run(config, store, context=context)
|