Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f436104f3
commit
9d666d0937
76 files changed
+15235
No files matched your search
@@ -0,0 +1,495 @@
|
||||
"""A task: plain code that makes jobs, and is never a job itself.
|
||||
|
||||
task = Task(kind="body_batch", name="body", finder=needs_a_body)
|
||||
task.start(config, store, queue) # queued: batches, then one final job
|
||||
task.run(config, store) # here and now, no queue at all
|
||||
|
||||
A task is a business step — fetch every feed, give every article a body, tag
|
||||
what has never been tagged. Given the config and the store it finds its own
|
||||
work, cuts it into batch payloads, mints a run id for itself, queues one job
|
||||
per payload under that id, and queues one final job behind them with
|
||||
`after="done"` on all of them. Then it is over. Nothing of the task is alive
|
||||
afterwards: only its jobs, and the callback the final job carries.
|
||||
|
||||
**It runs in whoever calls it.** The HTTP request behind a button, a final
|
||||
job's handler, a Claude Code session, a test. Finding the work is a query, and
|
||||
a query is cheap enough to do there — making the preparation asynchronous
|
||||
would buy a few milliseconds and cost a whole job kind, a payload, a failure
|
||||
mode and a row in the history that says nothing a person wants to read.
|
||||
|
||||
**A cascade is the same shape one level up**, and the pipeline does not tell
|
||||
them apart: both are runs, both group jobs by an id, both are killed the same
|
||||
way. The difference is a convention, and it is worth stating because nothing
|
||||
enforces it:
|
||||
|
||||
a task queues everything it wants at once, and only its final job
|
||||
calls back.
|
||||
a cascade is where the breakpoints are: it queues the next task only
|
||||
once the last one is over.
|
||||
|
||||
**The callback is a name, not a function.** Nothing in a table can hold a
|
||||
function, and the final job's payload is a table row — so the name travels
|
||||
and the worker that runs the job resolves it. `module:function`, imported
|
||||
where it is used. A registry would work too, and was the other candidate; the
|
||||
dotted name wins because it carries its own import, where a registry has to be
|
||||
filled before the worker looks in it, which is exactly the import-order trap
|
||||
`JOBS` already has to be careful about. It also reads as documentation in the
|
||||
job's payload: `newsdesk.jobs.cascade:step` says where to go and look.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
from collections.abc import Callable, Iterable, Sequence
|
||||
from dataclasses import dataclass, field, replace
|
||||
from typing import Any, Protocol
|
||||
|
||||
from pipeline.context import NULL, Cancelled, RunContext
|
||||
from pipeline.registry import JOBS, job
|
||||
from pipeline.table import on_its_own
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: The job kind every task ends with. One kind for every task there will ever
|
||||
#: be, because what it does is the same every time: read how the batches went
|
||||
#: and call the name in its payload.
|
||||
FINAL = "advance"
|
||||
|
||||
#: A payload for one batch: the keyword arguments its job will be called with.
|
||||
Payload = dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Bound:
|
||||
"""How many times one batch may be attempted, and how long it waits.
|
||||
|
||||
The defaults are one attempt and no wait, which is every batch of every
|
||||
task that existed before this: a job with no policy fails at its first
|
||||
exception exactly as it always did.
|
||||
|
||||
It exists because a bound is sometimes a property of the *batch* rather
|
||||
than of the job kind — a plugin declares how long to wait before asking
|
||||
its publisher again, so two batches of `body_batch` cut from two outlets
|
||||
carry two different numbers. The values travel onto the row (see
|
||||
`Jobs.enqueue`), because the attempt that has to decide may be running in
|
||||
another process, after a restart, under a plugin file that has since
|
||||
changed.
|
||||
"""
|
||||
|
||||
#: Counting the first attempt, so 1 is "run it once".
|
||||
max_attempts: int = 1
|
||||
#: Seconds before the second attempt.
|
||||
delay: float = 0.0
|
||||
#: Multiplied into the delay once per attempt already made.
|
||||
backoff: float = 1.0
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Batch:
|
||||
"""One batch of a task, when its bound is its own.
|
||||
|
||||
A finder may return these instead of plain payloads. Two shapes rather
|
||||
than one because the second is what nearly every task wants and reads as
|
||||
what it is — `[{"uids": [...]}, ...]` is a list of batches, and wrapping
|
||||
each of them to say "and the default bound" would be ceremony on every
|
||||
task to serve the one that needs it.
|
||||
"""
|
||||
|
||||
payload: Payload
|
||||
bound: Bound = Bound()
|
||||
#: What to call this one batch on screen, or "" to be called whatever its
|
||||
#: job kind is called. The kind's label is the same string on every batch
|
||||
#: of the task — nine rows reading "Tagging a batch" — because a label is
|
||||
#: declared once, at registration, by something that cannot see how the
|
||||
#: work was cut. Only the finder knows that, so this is where a batch that
|
||||
#: can say which batch it is says so. See `desk/modeltask.py`.
|
||||
label: str = ""
|
||||
|
||||
|
||||
#: Given the app's config and store, what there is to do — one payload per
|
||||
#: batch, or one `Batch` where the bound is the batch's own. An empty list is
|
||||
#: a legitimate answer and the common one: a task with nothing waiting costs
|
||||
#: one query, queues no batches, and its final job runs immediately and moves
|
||||
#: the cascade on.
|
||||
Finder = Callable[[Any, Any], list[Any]]
|
||||
|
||||
|
||||
def _batch(found: Payload | Batch) -> Batch:
|
||||
"""One thing a finder returned, as a batch. A bare payload gets the
|
||||
default bound, which is the behaviour it had before bounds existed."""
|
||||
return found if isinstance(found, Batch) else Batch(found)
|
||||
|
||||
|
||||
#: name -> how to build that task, standing on its own. Filled by the app the
|
||||
#: way `JOBS` is, and for the same reason: a task is as much a thing to start
|
||||
#: as a job is, so whatever offers the one — `/api/jobs`, the Jobs view, a
|
||||
#: script — has to be able to find the other by name.
|
||||
TASKS: dict[str, Callable[[], Task]] = {}
|
||||
|
||||
|
||||
class Sink(Protocol):
|
||||
"""Where a task puts the jobs it makes.
|
||||
|
||||
Two of them: `JobQueue`, when a task is started from a request or a
|
||||
handler, and nothing else. It is a protocol rather than the class so that
|
||||
`Task` does not import the queue — the dependency runs the other way, and a
|
||||
test can hand it a list.
|
||||
"""
|
||||
|
||||
def new_run(self, kind: str, name: str = "", label: str = "") -> int: ...
|
||||
|
||||
def cancelled(self, run_id: int) -> bool: ...
|
||||
|
||||
def enqueue(self, kind: str, payload: Payload | None = None, *,
|
||||
label: str = "", depends_on: Sequence[int] = (), after: str = "ok",
|
||||
cascade_id: int = 0, task_id: int = 0,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> Any: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Task:
|
||||
"""One business step: what a batch of it is, how to find them, who hears.
|
||||
|
||||
Knows nothing about what the work *is* — that is the batch job's, and the
|
||||
finder's. This is the shape they are held in.
|
||||
"""
|
||||
|
||||
#: The job kind one batch runs as. A task of one batch names an ordinary
|
||||
#: job kind (`pull`); a task of many names the batch kind (`body_batch`).
|
||||
kind: str
|
||||
#: How the work is found. Called once, in the caller, before anything is
|
||||
#: queued.
|
||||
finder: Finder
|
||||
#: What this task is — `body`, `tag`. Its identity: what a cascade's list
|
||||
#: matches against, and what the `runs` row is minted under. The job kind
|
||||
#: when nobody says.
|
||||
name: str = ""
|
||||
#: How to say that on screen: "Fetching what the feeds left out". Written
|
||||
#: onto the `runs` row, because a task is not a job and has no handler to
|
||||
#: carry a label. The name when nobody says.
|
||||
label: str = ""
|
||||
#: `module:function` for the final job to call, or "" for a task that
|
||||
#: reports to nobody — one started on its own, from a button or a script.
|
||||
callback: str = ""
|
||||
#: `module:function` the final job runs first, or "" for a task whose
|
||||
#: batches finish their own work. A name for the same reason the callback
|
||||
#: is one: it is a row in a table.
|
||||
#:
|
||||
#: The seam exists for a decision no batch can make on its own. A batch is
|
||||
#: one of forty and cannot see the other thirty-nine, so anything that has
|
||||
#: to hold across the whole run — a cap on what the run may add, a merge of
|
||||
#: what several batches each asked for — has exactly one place it can be
|
||||
#: made correctly, and that is the job that runs when they are all over.
|
||||
#: It belongs to the task rather than to the callback because it *is* the
|
||||
#: task's work: a task started on its own has no callback and still has to
|
||||
#: settle. See `newsdesk/jobs/tagging.py` for the one that does.
|
||||
settle: str = ""
|
||||
#: What the callback is handed alongside the tally. Fixed when the task is
|
||||
#: made, and stored in the final job's payload, so it survives a restart
|
||||
#: and a worker in another process.
|
||||
carries: dict[str, Any] = field(default_factory=dict)
|
||||
#: How many items a payload is worth, for the direct path's progress bar.
|
||||
#: Only the direct path asks: as a job a batch is on its own and reports
|
||||
#: its own rows, which is what the boxes on screen are drawn from.
|
||||
size: Callable[[Payload], int] | None = None
|
||||
#: How the direct path's batch results become one result. The queue never
|
||||
#: calls it — there each batch is its own job with its own row.
|
||||
fold: Callable[[Any, Any, list[Any]], dict[str, Any]] | None = None
|
||||
|
||||
@property
|
||||
def named(self) -> str:
|
||||
return self.name or self.kind
|
||||
|
||||
@property
|
||||
def called(self) -> str:
|
||||
return self.label or self.named
|
||||
|
||||
# ------------------------------------------------------------- queued
|
||||
|
||||
def start(self, config: Any, store: Any, sink: Sink, *,
|
||||
cascade_id: int = 0) -> int:
|
||||
"""Find the work, queue it, and be over. Returns the task's run id.
|
||||
|
||||
The id is minted before a single job exists, because a task is a thing
|
||||
before its jobs are — it is what decides how many there will be — and
|
||||
naming it after whichever job it happened to queue first would be
|
||||
describing it by one of its own consequences.
|
||||
|
||||
With no cascade given it begins one: something asked for this on its
|
||||
own, so it is a cascade of one task. Named apart from any chain of the
|
||||
same name — `pull on its own`, not `pull` — because pressing Tag is
|
||||
not starting the chain that ends in tagging, and whoever asks whether
|
||||
that chain is going has to be told no. See `on_its_own`.
|
||||
|
||||
Returns 0 when the cascade it belongs to has been killed. Nothing is
|
||||
told; it tries, and the table says no. `Jobs.enqueue` is where that is
|
||||
actually enforced, across processes and after a restart; this only
|
||||
saves minting a run for jobs that would all be refused.
|
||||
"""
|
||||
task_id, _ = queue_batches(
|
||||
sink, self.kind, lambda: self.finder(config, store),
|
||||
name=self.named, label=self.called, then=self.callback,
|
||||
settle=self.settle, carries=self.carries, cascade_id=cascade_id)
|
||||
return task_id
|
||||
|
||||
# ------------------------------------------------------------- direct
|
||||
|
||||
def run(self, config: Any, store: Any, *,
|
||||
context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Do the batches here and now, in order, with no queue at all.
|
||||
|
||||
What a Claude Code session, a notebook and a test get. The same batch
|
||||
handler the worker would call, with the same payloads from the same
|
||||
finder — there is no second implementation to drift.
|
||||
|
||||
A batch that raises is kept and the rest carry on, which is what the
|
||||
queue does with eight independent jobs of which one fails. Unless
|
||||
*nothing* got through: then the first exception is raised, because a
|
||||
run that returned "0 done" with the explanation only in the log would
|
||||
be a failure reported as a success. A `Cancelled` travels either way —
|
||||
it is an instruction from the person at the keyboard, not a batch that
|
||||
did not land.
|
||||
|
||||
A batch's `Bound` is ignored here, and that is not an omission: a
|
||||
bound is a number the *queue* enforces by putting a row back with a
|
||||
time on it, and there is no queue and no row on this path. Asked for
|
||||
directly, a batch is attempted once and its failure is returned to the
|
||||
caller, who is standing right there.
|
||||
"""
|
||||
spec = JOBS.get(self.kind)
|
||||
if spec is None:
|
||||
raise KeyError(
|
||||
f"{self.named!r} makes jobs of kind {self.kind!r}, which "
|
||||
f"nothing registers (have: {', '.join(sorted(JOBS)) or 'none'})")
|
||||
payloads = [_batch(found).payload for found in self.finder(config, store)]
|
||||
sizes = [self._size(p) for p in payloads]
|
||||
total = sum(sizes)
|
||||
|
||||
results: list[Any] = []
|
||||
failure: Exception | None = None
|
||||
at = 0
|
||||
for payload, size in zip(payloads, sizes, strict=True):
|
||||
context.check()
|
||||
extra = {"context": self._placed(context, at, total, size)} \
|
||||
if spec.wants_context else {}
|
||||
try:
|
||||
results.append(dict(spec.fn(config, store, **extra, **payload) or {}))
|
||||
except Cancelled:
|
||||
raise
|
||||
except Exception as exc:
|
||||
log.exception("%s: a batch of %s did not land", self.named, self.kind)
|
||||
results.append(exc)
|
||||
failure = failure or exc
|
||||
finally:
|
||||
# Position through the backlog, not what was answered: a batch
|
||||
# the model skipped half of still moved the run that far, and a
|
||||
# bar that goes backwards is worse than one that is approximate.
|
||||
at += size
|
||||
if failure is not None and not any(isinstance(r, dict) for r in results):
|
||||
raise failure
|
||||
if self.fold is not None:
|
||||
return self.fold(config, store, results)
|
||||
return {"batches": sum(1 for r in results if isinstance(r, dict)),
|
||||
"failed": sum(1 for r in results if isinstance(r, Exception)),
|
||||
"results": [r for r in results if isinstance(r, dict)]}
|
||||
|
||||
def _size(self, payload: Payload) -> int:
|
||||
return int(self.size(payload)) if self.size is not None else 0
|
||||
|
||||
def _placed(self, context: RunContext, at: int, total: int,
|
||||
size: int) -> RunContext:
|
||||
"""The batch's context, with its progress shifted into the whole run.
|
||||
|
||||
A batch run as a job is on its own and counts its own rows. Run here it
|
||||
is one of several in a row, and a bar that restarts per batch would be
|
||||
a bar that goes backwards five times.
|
||||
"""
|
||||
if not total:
|
||||
return context
|
||||
return replace(context, on_progress=lambda done, _t, message:
|
||||
context.progress(at + min(done, size), total, message))
|
||||
|
||||
|
||||
# ------------------------------------------------------------- the final job
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Tally:
|
||||
"""How a task's batches ended, as the callback is handed it.
|
||||
|
||||
Read off the jobs at the moment the final job runs, never counted up as
|
||||
they landed — so there is nothing to increment and nothing to lose to a
|
||||
restart half way through.
|
||||
"""
|
||||
|
||||
done: int = 0
|
||||
failed: int = 0
|
||||
cancelled: int = 0
|
||||
#: Whether the task itself was killed, as against however many of its
|
||||
#: batches happen to have ended cancelled. The counts cannot say: one
|
||||
#: batch killed by hand and every batch killed by the task going down look
|
||||
#: identical there and mean opposite things — drop this batch, or skip
|
||||
#: this whole step. Only the run's row knows, which is why it is asked.
|
||||
killed: bool = False
|
||||
#: What each batch returned, in the order they were queued.
|
||||
results: tuple[dict[str, Any], ...] = ()
|
||||
|
||||
@property
|
||||
def jobs(self) -> int:
|
||||
return self.done + self.failed + self.cancelled
|
||||
|
||||
@classmethod
|
||||
def of(cls, outcomes: Sequence[Any], *, killed: bool = False) -> Tally:
|
||||
states = [getattr(j, "state", "") for j in outcomes]
|
||||
return cls(
|
||||
done=states.count("done"),
|
||||
failed=states.count("failed"),
|
||||
cancelled=states.count("cancelled"),
|
||||
killed=killed,
|
||||
results=tuple(dict(getattr(j, "result", {}) or {}) for j in outcomes),
|
||||
)
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {"jobs": self.jobs, "done": self.done, "failed": self.failed,
|
||||
"cancelled": self.cancelled, "killed": self.killed}
|
||||
|
||||
|
||||
def queue_batches(sink: Sink, kind: str,
|
||||
found: Callable[[], Iterable[Payload | Batch]], *,
|
||||
name: str = "", label: str = "", then: str = "",
|
||||
settle: str = "", carries: dict[str, Any] | None = None,
|
||||
cascade_id: int = 0,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> tuple[int, tuple[int, ...]]:
|
||||
"""Mint a task, queue one job per batch, and queue the job that reports.
|
||||
|
||||
The one implementation behind both `Task.start` and `fan_out`. Returns the
|
||||
task's run id and the children's job ids; `(0, ())` when the cascade it
|
||||
would join has been killed.
|
||||
|
||||
`found` is called only after the runs exist, which is the order a task has
|
||||
always had: a task is a thing before its jobs are, since it is what decides
|
||||
how many there will be.
|
||||
|
||||
A plain payload takes the kind's own retry policy, or the uniform one
|
||||
passed here. A `Batch` carries its own `Bound`, which wins — that is what a
|
||||
batch cut from one plugin profile needs when the profile declares how long
|
||||
to leave its publisher alone.
|
||||
|
||||
With no cascade given it begins one: something asked for this on its own,
|
||||
so it is a cascade of one task, named apart from any chain of the same name
|
||||
(`pull on its own`, not `pull`). See `on_its_own`.
|
||||
"""
|
||||
if cascade_id and sink.cancelled(cascade_id):
|
||||
return 0, ()
|
||||
name = name or kind
|
||||
label = label or name
|
||||
cascade_id = cascade_id or sink.new_run("cascade", on_its_own(name), label)
|
||||
task_id = sink.new_run("task", name, label)
|
||||
|
||||
ids: list[int] = []
|
||||
for item in found():
|
||||
if isinstance(item, Batch):
|
||||
queued = sink.enqueue(kind, dict(item.payload), label=item.label,
|
||||
cascade_id=cascade_id, task_id=task_id,
|
||||
max_attempts=item.bound.max_attempts,
|
||||
retry_delay=item.bound.delay,
|
||||
retry_backoff=item.bound.backoff)
|
||||
else:
|
||||
queued = sink.enqueue(kind, dict(item),
|
||||
cascade_id=cascade_id, task_id=task_id,
|
||||
max_attempts=max_attempts,
|
||||
retry_delay=retry_delay,
|
||||
retry_backoff=retry_backoff)
|
||||
if queued is not None:
|
||||
ids.append(queued.id)
|
||||
# Behind all of them, and part of the same task — killing the task spares
|
||||
# it, because reporting is not work and "skip this task" is not "make it
|
||||
# never end". That holds where the kill landed mid-loop too: `Jobs.enqueue`
|
||||
# carves `after='done'` out of a task's kill, so this row is written and
|
||||
# the task ends by saying it was killed. None here is the *cascade* going
|
||||
# down, which takes the report with it by design: there is nothing left
|
||||
# to report to. See `Jobs.cancel_run`.
|
||||
sink.enqueue(FINAL, {"callback": then, "settle": settle,
|
||||
"payload": dict(carries or {})},
|
||||
depends_on=ids, after="done",
|
||||
cascade_id=cascade_id, task_id=task_id)
|
||||
return task_id, tuple(ids)
|
||||
|
||||
|
||||
def fan_out(sink: Sink, kind: str, payloads: Sequence[Payload | Batch], *,
|
||||
then: str = "", settle: str = "", label: str = "",
|
||||
carries: dict[str, Any] | None = None,
|
||||
cascade_id: int = 0,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> tuple[int, ...]:
|
||||
"""Queue these payloads as jobs, and one job behind them that reports.
|
||||
|
||||
What `RunContext.fan_out` reaches. `Task` is the same call with a finder in
|
||||
front, which is why a handler whose whole body is one of these *is* a task.
|
||||
The children go into a run of their own inside the caller's cascade, so the
|
||||
callback can fan out again and the result is a chain rather than a nest.
|
||||
|
||||
Returns the children's ids. The reporting job is not among them — it is
|
||||
behind them by construction, and a caller that waited on it would be
|
||||
holding a worker open to do so.
|
||||
"""
|
||||
_, ids = queue_batches(
|
||||
sink, kind, lambda: payloads, label=label, then=then, settle=settle,
|
||||
carries=carries, cascade_id=cascade_id, max_attempts=max_attempts,
|
||||
retry_delay=retry_delay, retry_backoff=retry_backoff)
|
||||
return ids
|
||||
|
||||
|
||||
@job(FINAL, "Moving on")
|
||||
def run_final(config: Any, store: Any, *, context: RunContext = NULL,
|
||||
callback: str = "", settle: str = "",
|
||||
payload: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""The one job every task ends with: tally the batches, call the name.
|
||||
|
||||
It runs for the milliseconds it takes to read how the batches went and
|
||||
hand that to whoever asked — nothing is alive between two tasks, and
|
||||
nothing was waiting while the last one ran.
|
||||
|
||||
`after="done"`, so it runs however the batches ended. That is the whole
|
||||
reason the mode exists: a job whose purpose is to report that one of eight
|
||||
failed cannot be held back by that failure.
|
||||
|
||||
A task with no callback stops here, which is what being asked for on its
|
||||
own means.
|
||||
|
||||
The settle runs first and the callback second, because the settle is the
|
||||
last of this task's own work and the callback is what comes after this
|
||||
task: a cascade told to move on before the tags of the stage it is leaving
|
||||
exist would be a stage reporting itself finished early.
|
||||
|
||||
It asks whether its own task was killed, as well as counting how the
|
||||
batches ended, because those are two different facts that arrive looking
|
||||
the same. A task killed while its batches ran leaves them all `cancelled`,
|
||||
and so does a person killing every batch by hand; the first is "skip this
|
||||
step", the second is "drop this work". Only the run's row can tell them
|
||||
apart, and the callback needs to be told which it is — the whole reason
|
||||
this job is spared the kill is so that somebody hears.
|
||||
"""
|
||||
tally = Tally.of(context.waited_for, killed=context.killed())
|
||||
out = tally.as_dict()
|
||||
if settle:
|
||||
out |= resolve(settle)(config, store, tally, context=context) or {}
|
||||
if not callback:
|
||||
return out
|
||||
return {**out, **(resolve(callback)(config, store, tally,
|
||||
dict(payload or {}), context=context) or {})}
|
||||
|
||||
|
||||
def resolve(name: str) -> Callable[..., dict[str, Any] | None]:
|
||||
"""`module:function` to the function. See this module's docstring for why
|
||||
a callback, or a settle, is a name at all."""
|
||||
module, _, attr = name.partition(":")
|
||||
if not module or not attr:
|
||||
raise ValueError(f"a callback is 'module:function', not {name!r}")
|
||||
return getattr(importlib.import_module(module), attr)
|
||||
Reference in new issue
Block a user