Files
jobboard/pipeline/task.py
T
WolfandClaude Opus 5.5 9d666d0937 Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and
scheduler), auth (one password, sessions, passkeys), the deploy kit and its
docs, plus the coupling and packaging tests. Downstream copies, unedited.

Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:16 +02:00

496 lines
23 KiB
Python

"""A task: plain code that makes jobs, and is never a job itself.
task = Task(kind="body_batch", name="body", finder=needs_a_body)
task.start(config, store, queue) # queued: batches, then one final job
task.run(config, store) # here and now, no queue at all
A task is a business step — fetch every feed, give every article a body, tag
what has never been tagged. Given the config and the store it finds its own
work, cuts it into batch payloads, mints a run id for itself, queues one job
per payload under that id, and queues one final job behind them with
`after="done"` on all of them. Then it is over. Nothing of the task is alive
afterwards: only its jobs, and the callback the final job carries.
**It runs in whoever calls it.** The HTTP request behind a button, a final
job's handler, a Claude Code session, a test. Finding the work is a query, and
a query is cheap enough to do there — making the preparation asynchronous
would buy a few milliseconds and cost a whole job kind, a payload, a failure
mode and a row in the history that says nothing a person wants to read.
**A cascade is the same shape one level up**, and the pipeline does not tell
them apart: both are runs, both group jobs by an id, both are killed the same
way. The difference is a convention, and it is worth stating because nothing
enforces it:
a task queues everything it wants at once, and only its final job
calls back.
a cascade is where the breakpoints are: it queues the next task only
once the last one is over.
**The callback is a name, not a function.** Nothing in a table can hold a
function, and the final job's payload is a table row — so the name travels
and the worker that runs the job resolves it. `module:function`, imported
where it is used. A registry would work too, and was the other candidate; the
dotted name wins because it carries its own import, where a registry has to be
filled before the worker looks in it, which is exactly the import-order trap
`JOBS` already has to be careful about. It also reads as documentation in the
job's payload: `newsdesk.jobs.cascade:step` says where to go and look.
"""
from __future__ import annotations
import importlib
import logging
from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, field, replace
from typing import Any, Protocol
from pipeline.context import NULL, Cancelled, RunContext
from pipeline.registry import JOBS, job
from pipeline.table import on_its_own
log = logging.getLogger(__name__)
#: The job kind every task ends with. One kind for every task there will ever
#: be, because what it does is the same every time: read how the batches went
#: and call the name in its payload.
FINAL = "advance"
#: A payload for one batch: the keyword arguments its job will be called with.
Payload = dict[str, Any]
@dataclass(frozen=True, slots=True)
class Bound:
"""How many times one batch may be attempted, and how long it waits.
The defaults are one attempt and no wait, which is every batch of every
task that existed before this: a job with no policy fails at its first
exception exactly as it always did.
It exists because a bound is sometimes a property of the *batch* rather
than of the job kind — a plugin declares how long to wait before asking
its publisher again, so two batches of `body_batch` cut from two outlets
carry two different numbers. The values travel onto the row (see
`Jobs.enqueue`), because the attempt that has to decide may be running in
another process, after a restart, under a plugin file that has since
changed.
"""
#: Counting the first attempt, so 1 is "run it once".
max_attempts: int = 1
#: Seconds before the second attempt.
delay: float = 0.0
#: Multiplied into the delay once per attempt already made.
backoff: float = 1.0
@dataclass(frozen=True, slots=True)
class Batch:
"""One batch of a task, when its bound is its own.
A finder may return these instead of plain payloads. Two shapes rather
than one because the second is what nearly every task wants and reads as
what it is — `[{"uids": [...]}, ...]` is a list of batches, and wrapping
each of them to say "and the default bound" would be ceremony on every
task to serve the one that needs it.
"""
payload: Payload
bound: Bound = Bound()
#: What to call this one batch on screen, or "" to be called whatever its
#: job kind is called. The kind's label is the same string on every batch
#: of the task — nine rows reading "Tagging a batch" — because a label is
#: declared once, at registration, by something that cannot see how the
#: work was cut. Only the finder knows that, so this is where a batch that
#: can say which batch it is says so. See `desk/modeltask.py`.
label: str = ""
#: Given the app's config and store, what there is to do — one payload per
#: batch, or one `Batch` where the bound is the batch's own. An empty list is
#: a legitimate answer and the common one: a task with nothing waiting costs
#: one query, queues no batches, and its final job runs immediately and moves
#: the cascade on.
Finder = Callable[[Any, Any], list[Any]]
def _batch(found: Payload | Batch) -> Batch:
"""One thing a finder returned, as a batch. A bare payload gets the
default bound, which is the behaviour it had before bounds existed."""
return found if isinstance(found, Batch) else Batch(found)
#: name -> how to build that task, standing on its own. Filled by the app the
#: way `JOBS` is, and for the same reason: a task is as much a thing to start
#: as a job is, so whatever offers the one — `/api/jobs`, the Jobs view, a
#: script — has to be able to find the other by name.
TASKS: dict[str, Callable[[], Task]] = {}
class Sink(Protocol):
"""Where a task puts the jobs it makes.
Two of them: `JobQueue`, when a task is started from a request or a
handler, and nothing else. It is a protocol rather than the class so that
`Task` does not import the queue — the dependency runs the other way, and a
test can hand it a list.
"""
def new_run(self, kind: str, name: str = "", label: str = "") -> int: ...
def cancelled(self, run_id: int) -> bool: ...
def enqueue(self, kind: str, payload: Payload | None = None, *,
label: str = "", depends_on: Sequence[int] = (), after: str = "ok",
cascade_id: int = 0, task_id: int = 0,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> Any: ...
@dataclass(frozen=True, slots=True)
class Task:
"""One business step: what a batch of it is, how to find them, who hears.
Knows nothing about what the work *is* — that is the batch job's, and the
finder's. This is the shape they are held in.
"""
#: The job kind one batch runs as. A task of one batch names an ordinary
#: job kind (`pull`); a task of many names the batch kind (`body_batch`).
kind: str
#: How the work is found. Called once, in the caller, before anything is
#: queued.
finder: Finder
#: What this task is — `body`, `tag`. Its identity: what a cascade's list
#: matches against, and what the `runs` row is minted under. The job kind
#: when nobody says.
name: str = ""
#: How to say that on screen: "Fetching what the feeds left out". Written
#: onto the `runs` row, because a task is not a job and has no handler to
#: carry a label. The name when nobody says.
label: str = ""
#: `module:function` for the final job to call, or "" for a task that
#: reports to nobody — one started on its own, from a button or a script.
callback: str = ""
#: `module:function` the final job runs first, or "" for a task whose
#: batches finish their own work. A name for the same reason the callback
#: is one: it is a row in a table.
#:
#: The seam exists for a decision no batch can make on its own. A batch is
#: one of forty and cannot see the other thirty-nine, so anything that has
#: to hold across the whole run — a cap on what the run may add, a merge of
#: what several batches each asked for — has exactly one place it can be
#: made correctly, and that is the job that runs when they are all over.
#: It belongs to the task rather than to the callback because it *is* the
#: task's work: a task started on its own has no callback and still has to
#: settle. See `newsdesk/jobs/tagging.py` for the one that does.
settle: str = ""
#: What the callback is handed alongside the tally. Fixed when the task is
#: made, and stored in the final job's payload, so it survives a restart
#: and a worker in another process.
carries: dict[str, Any] = field(default_factory=dict)
#: How many items a payload is worth, for the direct path's progress bar.
#: Only the direct path asks: as a job a batch is on its own and reports
#: its own rows, which is what the boxes on screen are drawn from.
size: Callable[[Payload], int] | None = None
#: How the direct path's batch results become one result. The queue never
#: calls it — there each batch is its own job with its own row.
fold: Callable[[Any, Any, list[Any]], dict[str, Any]] | None = None
@property
def named(self) -> str:
return self.name or self.kind
@property
def called(self) -> str:
return self.label or self.named
# ------------------------------------------------------------- queued
def start(self, config: Any, store: Any, sink: Sink, *,
cascade_id: int = 0) -> int:
"""Find the work, queue it, and be over. Returns the task's run id.
The id is minted before a single job exists, because a task is a thing
before its jobs are — it is what decides how many there will be — and
naming it after whichever job it happened to queue first would be
describing it by one of its own consequences.
With no cascade given it begins one: something asked for this on its
own, so it is a cascade of one task. Named apart from any chain of the
same name — `pull on its own`, not `pull` — because pressing Tag is
not starting the chain that ends in tagging, and whoever asks whether
that chain is going has to be told no. See `on_its_own`.
Returns 0 when the cascade it belongs to has been killed. Nothing is
told; it tries, and the table says no. `Jobs.enqueue` is where that is
actually enforced, across processes and after a restart; this only
saves minting a run for jobs that would all be refused.
"""
task_id, _ = queue_batches(
sink, self.kind, lambda: self.finder(config, store),
name=self.named, label=self.called, then=self.callback,
settle=self.settle, carries=self.carries, cascade_id=cascade_id)
return task_id
# ------------------------------------------------------------- direct
def run(self, config: Any, store: Any, *,
context: RunContext = NULL) -> dict[str, Any]:
"""Do the batches here and now, in order, with no queue at all.
What a Claude Code session, a notebook and a test get. The same batch
handler the worker would call, with the same payloads from the same
finder — there is no second implementation to drift.
A batch that raises is kept and the rest carry on, which is what the
queue does with eight independent jobs of which one fails. Unless
*nothing* got through: then the first exception is raised, because a
run that returned "0 done" with the explanation only in the log would
be a failure reported as a success. A `Cancelled` travels either way —
it is an instruction from the person at the keyboard, not a batch that
did not land.
A batch's `Bound` is ignored here, and that is not an omission: a
bound is a number the *queue* enforces by putting a row back with a
time on it, and there is no queue and no row on this path. Asked for
directly, a batch is attempted once and its failure is returned to the
caller, who is standing right there.
"""
spec = JOBS.get(self.kind)
if spec is None:
raise KeyError(
f"{self.named!r} makes jobs of kind {self.kind!r}, which "
f"nothing registers (have: {', '.join(sorted(JOBS)) or 'none'})")
payloads = [_batch(found).payload for found in self.finder(config, store)]
sizes = [self._size(p) for p in payloads]
total = sum(sizes)
results: list[Any] = []
failure: Exception | None = None
at = 0
for payload, size in zip(payloads, sizes, strict=True):
context.check()
extra = {"context": self._placed(context, at, total, size)} \
if spec.wants_context else {}
try:
results.append(dict(spec.fn(config, store, **extra, **payload) or {}))
except Cancelled:
raise
except Exception as exc:
log.exception("%s: a batch of %s did not land", self.named, self.kind)
results.append(exc)
failure = failure or exc
finally:
# Position through the backlog, not what was answered: a batch
# the model skipped half of still moved the run that far, and a
# bar that goes backwards is worse than one that is approximate.
at += size
if failure is not None and not any(isinstance(r, dict) for r in results):
raise failure
if self.fold is not None:
return self.fold(config, store, results)
return {"batches": sum(1 for r in results if isinstance(r, dict)),
"failed": sum(1 for r in results if isinstance(r, Exception)),
"results": [r for r in results if isinstance(r, dict)]}
def _size(self, payload: Payload) -> int:
return int(self.size(payload)) if self.size is not None else 0
def _placed(self, context: RunContext, at: int, total: int,
size: int) -> RunContext:
"""The batch's context, with its progress shifted into the whole run.
A batch run as a job is on its own and counts its own rows. Run here it
is one of several in a row, and a bar that restarts per batch would be
a bar that goes backwards five times.
"""
if not total:
return context
return replace(context, on_progress=lambda done, _t, message:
context.progress(at + min(done, size), total, message))
# ------------------------------------------------------------- the final job
@dataclass(frozen=True, slots=True)
class Tally:
"""How a task's batches ended, as the callback is handed it.
Read off the jobs at the moment the final job runs, never counted up as
they landed — so there is nothing to increment and nothing to lose to a
restart half way through.
"""
done: int = 0
failed: int = 0
cancelled: int = 0
#: Whether the task itself was killed, as against however many of its
#: batches happen to have ended cancelled. The counts cannot say: one
#: batch killed by hand and every batch killed by the task going down look
#: identical there and mean opposite things — drop this batch, or skip
#: this whole step. Only the run's row knows, which is why it is asked.
killed: bool = False
#: What each batch returned, in the order they were queued.
results: tuple[dict[str, Any], ...] = ()
@property
def jobs(self) -> int:
return self.done + self.failed + self.cancelled
@classmethod
def of(cls, outcomes: Sequence[Any], *, killed: bool = False) -> Tally:
states = [getattr(j, "state", "") for j in outcomes]
return cls(
done=states.count("done"),
failed=states.count("failed"),
cancelled=states.count("cancelled"),
killed=killed,
results=tuple(dict(getattr(j, "result", {}) or {}) for j in outcomes),
)
def as_dict(self) -> dict[str, Any]:
return {"jobs": self.jobs, "done": self.done, "failed": self.failed,
"cancelled": self.cancelled, "killed": self.killed}
def queue_batches(sink: Sink, kind: str,
found: Callable[[], Iterable[Payload | Batch]], *,
name: str = "", label: str = "", then: str = "",
settle: str = "", carries: dict[str, Any] | None = None,
cascade_id: int = 0,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> tuple[int, tuple[int, ...]]:
"""Mint a task, queue one job per batch, and queue the job that reports.
The one implementation behind both `Task.start` and `fan_out`. Returns the
task's run id and the children's job ids; `(0, ())` when the cascade it
would join has been killed.
`found` is called only after the runs exist, which is the order a task has
always had: a task is a thing before its jobs are, since it is what decides
how many there will be.
A plain payload takes the kind's own retry policy, or the uniform one
passed here. A `Batch` carries its own `Bound`, which wins — that is what a
batch cut from one plugin profile needs when the profile declares how long
to leave its publisher alone.
With no cascade given it begins one: something asked for this on its own,
so it is a cascade of one task, named apart from any chain of the same name
(`pull on its own`, not `pull`). See `on_its_own`.
"""
if cascade_id and sink.cancelled(cascade_id):
return 0, ()
name = name or kind
label = label or name
cascade_id = cascade_id or sink.new_run("cascade", on_its_own(name), label)
task_id = sink.new_run("task", name, label)
ids: list[int] = []
for item in found():
if isinstance(item, Batch):
queued = sink.enqueue(kind, dict(item.payload), label=item.label,
cascade_id=cascade_id, task_id=task_id,
max_attempts=item.bound.max_attempts,
retry_delay=item.bound.delay,
retry_backoff=item.bound.backoff)
else:
queued = sink.enqueue(kind, dict(item),
cascade_id=cascade_id, task_id=task_id,
max_attempts=max_attempts,
retry_delay=retry_delay,
retry_backoff=retry_backoff)
if queued is not None:
ids.append(queued.id)
# Behind all of them, and part of the same task — killing the task spares
# it, because reporting is not work and "skip this task" is not "make it
# never end". That holds where the kill landed mid-loop too: `Jobs.enqueue`
# carves `after='done'` out of a task's kill, so this row is written and
# the task ends by saying it was killed. None here is the *cascade* going
# down, which takes the report with it by design: there is nothing left
# to report to. See `Jobs.cancel_run`.
sink.enqueue(FINAL, {"callback": then, "settle": settle,
"payload": dict(carries or {})},
depends_on=ids, after="done",
cascade_id=cascade_id, task_id=task_id)
return task_id, tuple(ids)
def fan_out(sink: Sink, kind: str, payloads: Sequence[Payload | Batch], *,
then: str = "", settle: str = "", label: str = "",
carries: dict[str, Any] | None = None,
cascade_id: int = 0,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> tuple[int, ...]:
"""Queue these payloads as jobs, and one job behind them that reports.
What `RunContext.fan_out` reaches. `Task` is the same call with a finder in
front, which is why a handler whose whole body is one of these *is* a task.
The children go into a run of their own inside the caller's cascade, so the
callback can fan out again and the result is a chain rather than a nest.
Returns the children's ids. The reporting job is not among them — it is
behind them by construction, and a caller that waited on it would be
holding a worker open to do so.
"""
_, ids = queue_batches(
sink, kind, lambda: payloads, label=label, then=then, settle=settle,
carries=carries, cascade_id=cascade_id, max_attempts=max_attempts,
retry_delay=retry_delay, retry_backoff=retry_backoff)
return ids
@job(FINAL, "Moving on")
def run_final(config: Any, store: Any, *, context: RunContext = NULL,
callback: str = "", settle: str = "",
payload: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""The one job every task ends with: tally the batches, call the name.
It runs for the milliseconds it takes to read how the batches went and
hand that to whoever asked — nothing is alive between two tasks, and
nothing was waiting while the last one ran.
`after="done"`, so it runs however the batches ended. That is the whole
reason the mode exists: a job whose purpose is to report that one of eight
failed cannot be held back by that failure.
A task with no callback stops here, which is what being asked for on its
own means.
The settle runs first and the callback second, because the settle is the
last of this task's own work and the callback is what comes after this
task: a cascade told to move on before the tags of the stage it is leaving
exist would be a stage reporting itself finished early.
It asks whether its own task was killed, as well as counting how the
batches ended, because those are two different facts that arrive looking
the same. A task killed while its batches ran leaves them all `cancelled`,
and so does a person killing every batch by hand; the first is "skip this
step", the second is "drop this work". Only the run's row can tell them
apart, and the callback needs to be told which it is — the whole reason
this job is spared the kill is so that somebody hears.
"""
tally = Tally.of(context.waited_for, killed=context.killed())
out = tally.as_dict()
if settle:
out |= resolve(settle)(config, store, tally, context=context) or {}
if not callback:
return out
return {**out, **(resolve(callback)(config, store, tally,
dict(payload or {}), context=context) or {})}
def resolve(name: str) -> Callable[..., dict[str, Any] | None]:
"""`module:function` to the function. See this module's docstring for why
a callback, or a settle, is a name at all."""
module, _, attr = name.partition(":")
if not module or not attr:
raise ValueError(f"a callback is 'module:function', not {name!r}")
return getattr(importlib.import_module(module), attr)