roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
496 lines
23 KiB
Python
496 lines
23 KiB
Python
"""A task: plain code that makes jobs, and is never a job itself.
|
|
|
|
task = Task(kind="body_batch", name="body", finder=needs_a_body)
|
|
task.start(config, store, queue) # queued: batches, then one final job
|
|
task.run(config, store) # here and now, no queue at all
|
|
|
|
A task is a business step — fetch every feed, give every article a body, tag
|
|
what has never been tagged. Given the config and the store it finds its own
|
|
work, cuts it into batch payloads, mints a run id for itself, queues one job
|
|
per payload under that id, and queues one final job behind them with
|
|
`after="done"` on all of them. Then it is over. Nothing of the task is alive
|
|
afterwards: only its jobs, and the callback the final job carries.
|
|
|
|
**It runs in whoever calls it.** The HTTP request behind a button, a final
|
|
job's handler, a Claude Code session, a test. Finding the work is a query, and
|
|
a query is cheap enough to do there — making the preparation asynchronous
|
|
would buy a few milliseconds and cost a whole job kind, a payload, a failure
|
|
mode and a row in the history that says nothing a person wants to read.
|
|
|
|
**A cascade is the same shape one level up**, and the pipeline does not tell
|
|
them apart: both are runs, both group jobs by an id, both are killed the same
|
|
way. The difference is a convention, and it is worth stating because nothing
|
|
enforces it:
|
|
|
|
a task queues everything it wants at once, and only its final job
|
|
calls back.
|
|
a cascade is where the breakpoints are: it queues the next task only
|
|
once the last one is over.
|
|
|
|
**The callback is a name, not a function.** Nothing in a table can hold a
|
|
function, and the final job's payload is a table row — so the name travels
|
|
and the worker that runs the job resolves it. `module:function`, imported
|
|
where it is used. A registry would work too, and was the other candidate; the
|
|
dotted name wins because it carries its own import, where a registry has to be
|
|
filled before the worker looks in it, which is exactly the import-order trap
|
|
`JOBS` already has to be careful about. It also reads as documentation in the
|
|
job's payload: `newsdesk.jobs.cascade:step` says where to go and look.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib
|
|
import logging
|
|
from collections.abc import Callable, Iterable, Sequence
|
|
from dataclasses import dataclass, field, replace
|
|
from typing import Any, Protocol
|
|
|
|
from pipeline.context import NULL, Cancelled, RunContext
|
|
from pipeline.registry import JOBS, job
|
|
from pipeline.table import on_its_own
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
#: The job kind every task ends with. One kind for every task there will ever
|
|
#: be, because what it does is the same every time: read how the batches went
|
|
#: and call the name in its payload.
|
|
FINAL = "advance"
|
|
|
|
#: A payload for one batch: the keyword arguments its job will be called with.
|
|
Payload = dict[str, Any]
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class Bound:
|
|
"""How many times one batch may be attempted, and how long it waits.
|
|
|
|
The defaults are one attempt and no wait, which is every batch of every
|
|
task that existed before this: a job with no policy fails at its first
|
|
exception exactly as it always did.
|
|
|
|
It exists because a bound is sometimes a property of the *batch* rather
|
|
than of the job kind — a plugin declares how long to wait before asking
|
|
its publisher again, so two batches of `body_batch` cut from two outlets
|
|
carry two different numbers. The values travel onto the row (see
|
|
`Jobs.enqueue`), because the attempt that has to decide may be running in
|
|
another process, after a restart, under a plugin file that has since
|
|
changed.
|
|
"""
|
|
|
|
#: Counting the first attempt, so 1 is "run it once".
|
|
max_attempts: int = 1
|
|
#: Seconds before the second attempt.
|
|
delay: float = 0.0
|
|
#: Multiplied into the delay once per attempt already made.
|
|
backoff: float = 1.0
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class Batch:
|
|
"""One batch of a task, when its bound is its own.
|
|
|
|
A finder may return these instead of plain payloads. Two shapes rather
|
|
than one because the second is what nearly every task wants and reads as
|
|
what it is — `[{"uids": [...]}, ...]` is a list of batches, and wrapping
|
|
each of them to say "and the default bound" would be ceremony on every
|
|
task to serve the one that needs it.
|
|
"""
|
|
|
|
payload: Payload
|
|
bound: Bound = Bound()
|
|
#: What to call this one batch on screen, or "" to be called whatever its
|
|
#: job kind is called. The kind's label is the same string on every batch
|
|
#: of the task — nine rows reading "Tagging a batch" — because a label is
|
|
#: declared once, at registration, by something that cannot see how the
|
|
#: work was cut. Only the finder knows that, so this is where a batch that
|
|
#: can say which batch it is says so. See `desk/modeltask.py`.
|
|
label: str = ""
|
|
|
|
|
|
#: Given the app's config and store, what there is to do — one payload per
|
|
#: batch, or one `Batch` where the bound is the batch's own. An empty list is
|
|
#: a legitimate answer and the common one: a task with nothing waiting costs
|
|
#: one query, queues no batches, and its final job runs immediately and moves
|
|
#: the cascade on.
|
|
Finder = Callable[[Any, Any], list[Any]]
|
|
|
|
|
|
def _batch(found: Payload | Batch) -> Batch:
|
|
"""One thing a finder returned, as a batch. A bare payload gets the
|
|
default bound, which is the behaviour it had before bounds existed."""
|
|
return found if isinstance(found, Batch) else Batch(found)
|
|
|
|
|
|
#: name -> how to build that task, standing on its own. Filled by the app the
|
|
#: way `JOBS` is, and for the same reason: a task is as much a thing to start
|
|
#: as a job is, so whatever offers the one — `/api/jobs`, the Jobs view, a
|
|
#: script — has to be able to find the other by name.
|
|
TASKS: dict[str, Callable[[], Task]] = {}
|
|
|
|
|
|
class Sink(Protocol):
|
|
"""Where a task puts the jobs it makes.
|
|
|
|
Two of them: `JobQueue`, when a task is started from a request or a
|
|
handler, and nothing else. It is a protocol rather than the class so that
|
|
`Task` does not import the queue — the dependency runs the other way, and a
|
|
test can hand it a list.
|
|
"""
|
|
|
|
def new_run(self, kind: str, name: str = "", label: str = "") -> int: ...
|
|
|
|
def cancelled(self, run_id: int) -> bool: ...
|
|
|
|
def enqueue(self, kind: str, payload: Payload | None = None, *,
|
|
label: str = "", depends_on: Sequence[int] = (), after: str = "ok",
|
|
cascade_id: int = 0, task_id: int = 0,
|
|
max_attempts: int | None = None,
|
|
retry_delay: float | None = None,
|
|
retry_backoff: float | None = None) -> Any: ...
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class Task:
|
|
"""One business step: what a batch of it is, how to find them, who hears.
|
|
|
|
Knows nothing about what the work *is* — that is the batch job's, and the
|
|
finder's. This is the shape they are held in.
|
|
"""
|
|
|
|
#: The job kind one batch runs as. A task of one batch names an ordinary
|
|
#: job kind (`pull`); a task of many names the batch kind (`body_batch`).
|
|
kind: str
|
|
#: How the work is found. Called once, in the caller, before anything is
|
|
#: queued.
|
|
finder: Finder
|
|
#: What this task is — `body`, `tag`. Its identity: what a cascade's list
|
|
#: matches against, and what the `runs` row is minted under. The job kind
|
|
#: when nobody says.
|
|
name: str = ""
|
|
#: How to say that on screen: "Fetching what the feeds left out". Written
|
|
#: onto the `runs` row, because a task is not a job and has no handler to
|
|
#: carry a label. The name when nobody says.
|
|
label: str = ""
|
|
#: `module:function` for the final job to call, or "" for a task that
|
|
#: reports to nobody — one started on its own, from a button or a script.
|
|
callback: str = ""
|
|
#: `module:function` the final job runs first, or "" for a task whose
|
|
#: batches finish their own work. A name for the same reason the callback
|
|
#: is one: it is a row in a table.
|
|
#:
|
|
#: The seam exists for a decision no batch can make on its own. A batch is
|
|
#: one of forty and cannot see the other thirty-nine, so anything that has
|
|
#: to hold across the whole run — a cap on what the run may add, a merge of
|
|
#: what several batches each asked for — has exactly one place it can be
|
|
#: made correctly, and that is the job that runs when they are all over.
|
|
#: It belongs to the task rather than to the callback because it *is* the
|
|
#: task's work: a task started on its own has no callback and still has to
|
|
#: settle. See `newsdesk/jobs/tagging.py` for the one that does.
|
|
settle: str = ""
|
|
#: What the callback is handed alongside the tally. Fixed when the task is
|
|
#: made, and stored in the final job's payload, so it survives a restart
|
|
#: and a worker in another process.
|
|
carries: dict[str, Any] = field(default_factory=dict)
|
|
#: How many items a payload is worth, for the direct path's progress bar.
|
|
#: Only the direct path asks: as a job a batch is on its own and reports
|
|
#: its own rows, which is what the boxes on screen are drawn from.
|
|
size: Callable[[Payload], int] | None = None
|
|
#: How the direct path's batch results become one result. The queue never
|
|
#: calls it — there each batch is its own job with its own row.
|
|
fold: Callable[[Any, Any, list[Any]], dict[str, Any]] | None = None
|
|
|
|
@property
|
|
def named(self) -> str:
|
|
return self.name or self.kind
|
|
|
|
@property
|
|
def called(self) -> str:
|
|
return self.label or self.named
|
|
|
|
# ------------------------------------------------------------- queued
|
|
|
|
def start(self, config: Any, store: Any, sink: Sink, *,
|
|
cascade_id: int = 0) -> int:
|
|
"""Find the work, queue it, and be over. Returns the task's run id.
|
|
|
|
The id is minted before a single job exists, because a task is a thing
|
|
before its jobs are — it is what decides how many there will be — and
|
|
naming it after whichever job it happened to queue first would be
|
|
describing it by one of its own consequences.
|
|
|
|
With no cascade given it begins one: something asked for this on its
|
|
own, so it is a cascade of one task. Named apart from any chain of the
|
|
same name — `pull on its own`, not `pull` — because pressing Tag is
|
|
not starting the chain that ends in tagging, and whoever asks whether
|
|
that chain is going has to be told no. See `on_its_own`.
|
|
|
|
Returns 0 when the cascade it belongs to has been killed. Nothing is
|
|
told; it tries, and the table says no. `Jobs.enqueue` is where that is
|
|
actually enforced, across processes and after a restart; this only
|
|
saves minting a run for jobs that would all be refused.
|
|
"""
|
|
task_id, _ = queue_batches(
|
|
sink, self.kind, lambda: self.finder(config, store),
|
|
name=self.named, label=self.called, then=self.callback,
|
|
settle=self.settle, carries=self.carries, cascade_id=cascade_id)
|
|
return task_id
|
|
|
|
# ------------------------------------------------------------- direct
|
|
|
|
def run(self, config: Any, store: Any, *,
|
|
context: RunContext = NULL) -> dict[str, Any]:
|
|
"""Do the batches here and now, in order, with no queue at all.
|
|
|
|
What a Claude Code session, a notebook and a test get. The same batch
|
|
handler the worker would call, with the same payloads from the same
|
|
finder — there is no second implementation to drift.
|
|
|
|
A batch that raises is kept and the rest carry on, which is what the
|
|
queue does with eight independent jobs of which one fails. Unless
|
|
*nothing* got through: then the first exception is raised, because a
|
|
run that returned "0 done" with the explanation only in the log would
|
|
be a failure reported as a success. A `Cancelled` travels either way —
|
|
it is an instruction from the person at the keyboard, not a batch that
|
|
did not land.
|
|
|
|
A batch's `Bound` is ignored here, and that is not an omission: a
|
|
bound is a number the *queue* enforces by putting a row back with a
|
|
time on it, and there is no queue and no row on this path. Asked for
|
|
directly, a batch is attempted once and its failure is returned to the
|
|
caller, who is standing right there.
|
|
"""
|
|
spec = JOBS.get(self.kind)
|
|
if spec is None:
|
|
raise KeyError(
|
|
f"{self.named!r} makes jobs of kind {self.kind!r}, which "
|
|
f"nothing registers (have: {', '.join(sorted(JOBS)) or 'none'})")
|
|
payloads = [_batch(found).payload for found in self.finder(config, store)]
|
|
sizes = [self._size(p) for p in payloads]
|
|
total = sum(sizes)
|
|
|
|
results: list[Any] = []
|
|
failure: Exception | None = None
|
|
at = 0
|
|
for payload, size in zip(payloads, sizes, strict=True):
|
|
context.check()
|
|
extra = {"context": self._placed(context, at, total, size)} \
|
|
if spec.wants_context else {}
|
|
try:
|
|
results.append(dict(spec.fn(config, store, **extra, **payload) or {}))
|
|
except Cancelled:
|
|
raise
|
|
except Exception as exc:
|
|
log.exception("%s: a batch of %s did not land", self.named, self.kind)
|
|
results.append(exc)
|
|
failure = failure or exc
|
|
finally:
|
|
# Position through the backlog, not what was answered: a batch
|
|
# the model skipped half of still moved the run that far, and a
|
|
# bar that goes backwards is worse than one that is approximate.
|
|
at += size
|
|
if failure is not None and not any(isinstance(r, dict) for r in results):
|
|
raise failure
|
|
if self.fold is not None:
|
|
return self.fold(config, store, results)
|
|
return {"batches": sum(1 for r in results if isinstance(r, dict)),
|
|
"failed": sum(1 for r in results if isinstance(r, Exception)),
|
|
"results": [r for r in results if isinstance(r, dict)]}
|
|
|
|
def _size(self, payload: Payload) -> int:
|
|
return int(self.size(payload)) if self.size is not None else 0
|
|
|
|
def _placed(self, context: RunContext, at: int, total: int,
|
|
size: int) -> RunContext:
|
|
"""The batch's context, with its progress shifted into the whole run.
|
|
|
|
A batch run as a job is on its own and counts its own rows. Run here it
|
|
is one of several in a row, and a bar that restarts per batch would be
|
|
a bar that goes backwards five times.
|
|
"""
|
|
if not total:
|
|
return context
|
|
return replace(context, on_progress=lambda done, _t, message:
|
|
context.progress(at + min(done, size), total, message))
|
|
|
|
|
|
# ------------------------------------------------------------- the final job
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class Tally:
|
|
"""How a task's batches ended, as the callback is handed it.
|
|
|
|
Read off the jobs at the moment the final job runs, never counted up as
|
|
they landed — so there is nothing to increment and nothing to lose to a
|
|
restart half way through.
|
|
"""
|
|
|
|
done: int = 0
|
|
failed: int = 0
|
|
cancelled: int = 0
|
|
#: Whether the task itself was killed, as against however many of its
|
|
#: batches happen to have ended cancelled. The counts cannot say: one
|
|
#: batch killed by hand and every batch killed by the task going down look
|
|
#: identical there and mean opposite things — drop this batch, or skip
|
|
#: this whole step. Only the run's row knows, which is why it is asked.
|
|
killed: bool = False
|
|
#: What each batch returned, in the order they were queued.
|
|
results: tuple[dict[str, Any], ...] = ()
|
|
|
|
@property
|
|
def jobs(self) -> int:
|
|
return self.done + self.failed + self.cancelled
|
|
|
|
@classmethod
|
|
def of(cls, outcomes: Sequence[Any], *, killed: bool = False) -> Tally:
|
|
states = [getattr(j, "state", "") for j in outcomes]
|
|
return cls(
|
|
done=states.count("done"),
|
|
failed=states.count("failed"),
|
|
cancelled=states.count("cancelled"),
|
|
killed=killed,
|
|
results=tuple(dict(getattr(j, "result", {}) or {}) for j in outcomes),
|
|
)
|
|
|
|
def as_dict(self) -> dict[str, Any]:
|
|
return {"jobs": self.jobs, "done": self.done, "failed": self.failed,
|
|
"cancelled": self.cancelled, "killed": self.killed}
|
|
|
|
|
|
def queue_batches(sink: Sink, kind: str,
|
|
found: Callable[[], Iterable[Payload | Batch]], *,
|
|
name: str = "", label: str = "", then: str = "",
|
|
settle: str = "", carries: dict[str, Any] | None = None,
|
|
cascade_id: int = 0,
|
|
max_attempts: int | None = None,
|
|
retry_delay: float | None = None,
|
|
retry_backoff: float | None = None) -> tuple[int, tuple[int, ...]]:
|
|
"""Mint a task, queue one job per batch, and queue the job that reports.
|
|
|
|
The one implementation behind both `Task.start` and `fan_out`. Returns the
|
|
task's run id and the children's job ids; `(0, ())` when the cascade it
|
|
would join has been killed.
|
|
|
|
`found` is called only after the runs exist, which is the order a task has
|
|
always had: a task is a thing before its jobs are, since it is what decides
|
|
how many there will be.
|
|
|
|
A plain payload takes the kind's own retry policy, or the uniform one
|
|
passed here. A `Batch` carries its own `Bound`, which wins — that is what a
|
|
batch cut from one plugin profile needs when the profile declares how long
|
|
to leave its publisher alone.
|
|
|
|
With no cascade given it begins one: something asked for this on its own,
|
|
so it is a cascade of one task, named apart from any chain of the same name
|
|
(`pull on its own`, not `pull`). See `on_its_own`.
|
|
"""
|
|
if cascade_id and sink.cancelled(cascade_id):
|
|
return 0, ()
|
|
name = name or kind
|
|
label = label or name
|
|
cascade_id = cascade_id or sink.new_run("cascade", on_its_own(name), label)
|
|
task_id = sink.new_run("task", name, label)
|
|
|
|
ids: list[int] = []
|
|
for item in found():
|
|
if isinstance(item, Batch):
|
|
queued = sink.enqueue(kind, dict(item.payload), label=item.label,
|
|
cascade_id=cascade_id, task_id=task_id,
|
|
max_attempts=item.bound.max_attempts,
|
|
retry_delay=item.bound.delay,
|
|
retry_backoff=item.bound.backoff)
|
|
else:
|
|
queued = sink.enqueue(kind, dict(item),
|
|
cascade_id=cascade_id, task_id=task_id,
|
|
max_attempts=max_attempts,
|
|
retry_delay=retry_delay,
|
|
retry_backoff=retry_backoff)
|
|
if queued is not None:
|
|
ids.append(queued.id)
|
|
# Behind all of them, and part of the same task — killing the task spares
|
|
# it, because reporting is not work and "skip this task" is not "make it
|
|
# never end". That holds where the kill landed mid-loop too: `Jobs.enqueue`
|
|
# carves `after='done'` out of a task's kill, so this row is written and
|
|
# the task ends by saying it was killed. None here is the *cascade* going
|
|
# down, which takes the report with it by design: there is nothing left
|
|
# to report to. See `Jobs.cancel_run`.
|
|
sink.enqueue(FINAL, {"callback": then, "settle": settle,
|
|
"payload": dict(carries or {})},
|
|
depends_on=ids, after="done",
|
|
cascade_id=cascade_id, task_id=task_id)
|
|
return task_id, tuple(ids)
|
|
|
|
|
|
def fan_out(sink: Sink, kind: str, payloads: Sequence[Payload | Batch], *,
|
|
then: str = "", settle: str = "", label: str = "",
|
|
carries: dict[str, Any] | None = None,
|
|
cascade_id: int = 0,
|
|
max_attempts: int | None = None,
|
|
retry_delay: float | None = None,
|
|
retry_backoff: float | None = None) -> tuple[int, ...]:
|
|
"""Queue these payloads as jobs, and one job behind them that reports.
|
|
|
|
What `RunContext.fan_out` reaches. `Task` is the same call with a finder in
|
|
front, which is why a handler whose whole body is one of these *is* a task.
|
|
The children go into a run of their own inside the caller's cascade, so the
|
|
callback can fan out again and the result is a chain rather than a nest.
|
|
|
|
Returns the children's ids. The reporting job is not among them — it is
|
|
behind them by construction, and a caller that waited on it would be
|
|
holding a worker open to do so.
|
|
"""
|
|
_, ids = queue_batches(
|
|
sink, kind, lambda: payloads, label=label, then=then, settle=settle,
|
|
carries=carries, cascade_id=cascade_id, max_attempts=max_attempts,
|
|
retry_delay=retry_delay, retry_backoff=retry_backoff)
|
|
return ids
|
|
|
|
|
|
@job(FINAL, "Moving on")
|
|
def run_final(config: Any, store: Any, *, context: RunContext = NULL,
|
|
callback: str = "", settle: str = "",
|
|
payload: dict[str, Any] | None = None,
|
|
) -> dict[str, Any]:
|
|
"""The one job every task ends with: tally the batches, call the name.
|
|
|
|
It runs for the milliseconds it takes to read how the batches went and
|
|
hand that to whoever asked — nothing is alive between two tasks, and
|
|
nothing was waiting while the last one ran.
|
|
|
|
`after="done"`, so it runs however the batches ended. That is the whole
|
|
reason the mode exists: a job whose purpose is to report that one of eight
|
|
failed cannot be held back by that failure.
|
|
|
|
A task with no callback stops here, which is what being asked for on its
|
|
own means.
|
|
|
|
The settle runs first and the callback second, because the settle is the
|
|
last of this task's own work and the callback is what comes after this
|
|
task: a cascade told to move on before the tags of the stage it is leaving
|
|
exist would be a stage reporting itself finished early.
|
|
|
|
It asks whether its own task was killed, as well as counting how the
|
|
batches ended, because those are two different facts that arrive looking
|
|
the same. A task killed while its batches ran leaves them all `cancelled`,
|
|
and so does a person killing every batch by hand; the first is "skip this
|
|
step", the second is "drop this work". Only the run's row can tell them
|
|
apart, and the callback needs to be told which it is — the whole reason
|
|
this job is spared the kill is so that somebody hears.
|
|
"""
|
|
tally = Tally.of(context.waited_for, killed=context.killed())
|
|
out = tally.as_dict()
|
|
if settle:
|
|
out |= resolve(settle)(config, store, tally, context=context) or {}
|
|
if not callback:
|
|
return out
|
|
return {**out, **(resolve(callback)(config, store, tally,
|
|
dict(payload or {}), context=context) or {})}
|
|
|
|
|
|
def resolve(name: str) -> Callable[..., dict[str, Any] | None]:
|
|
"""`module:function` to the function. See this module's docstring for why
|
|
a callback, or a settle, is a name at all."""
|
|
module, _, attr = name.partition(":")
|
|
if not module or not attr:
|
|
raise ValueError(f"a callback is 'module:function', not {name!r}")
|
|
return getattr(importlib.import_module(module), attr)
|