"""A task: plain code that makes jobs, and is never a job itself. task = Task(kind="body_batch", name="body", finder=needs_a_body) task.start(config, store, queue) # queued: batches, then one final job task.run(config, store) # here and now, no queue at all A task is a business step — fetch every feed, give every article a body, tag what has never been tagged. Given the config and the store it finds its own work, cuts it into batch payloads, mints a run id for itself, queues one job per payload under that id, and queues one final job behind them with `after="done"` on all of them. Then it is over. Nothing of the task is alive afterwards: only its jobs, and the callback the final job carries. **It runs in whoever calls it.** The HTTP request behind a button, a final job's handler, a Claude Code session, a test. Finding the work is a query, and a query is cheap enough to do there — making the preparation asynchronous would buy a few milliseconds and cost a whole job kind, a payload, a failure mode and a row in the history that says nothing a person wants to read. **A cascade is the same shape one level up**, and the pipeline does not tell them apart: both are runs, both group jobs by an id, both are killed the same way. The difference is a convention, and it is worth stating because nothing enforces it: a task queues everything it wants at once, and only its final job calls back. a cascade is where the breakpoints are: it queues the next task only once the last one is over. **The callback is a name, not a function.** Nothing in a table can hold a function, and the final job's payload is a table row — so the name travels and the worker that runs the job resolves it. `module:function`, imported where it is used. A registry would work too, and was the other candidate; the dotted name wins because it carries its own import, where a registry has to be filled before the worker looks in it, which is exactly the import-order trap `JOBS` already has to be careful about. It also reads as documentation in the job's payload: `newsdesk.jobs.cascade:step` says where to go and look. """ from __future__ import annotations import importlib import logging from collections.abc import Callable, Iterable, Sequence from dataclasses import dataclass, field, replace from typing import Any, Protocol from pipeline.context import NULL, Cancelled, RunContext from pipeline.registry import JOBS, job from pipeline.table import on_its_own log = logging.getLogger(__name__) #: The job kind every task ends with. One kind for every task there will ever #: be, because what it does is the same every time: read how the batches went #: and call the name in its payload. FINAL = "advance" #: A payload for one batch: the keyword arguments its job will be called with. Payload = dict[str, Any] @dataclass(frozen=True, slots=True) class Bound: """How many times one batch may be attempted, and how long it waits. The defaults are one attempt and no wait, which is every batch of every task that existed before this: a job with no policy fails at its first exception exactly as it always did. It exists because a bound is sometimes a property of the *batch* rather than of the job kind — a plugin declares how long to wait before asking its publisher again, so two batches of `body_batch` cut from two outlets carry two different numbers. The values travel onto the row (see `Jobs.enqueue`), because the attempt that has to decide may be running in another process, after a restart, under a plugin file that has since changed. """ #: Counting the first attempt, so 1 is "run it once". max_attempts: int = 1 #: Seconds before the second attempt. delay: float = 0.0 #: Multiplied into the delay once per attempt already made. backoff: float = 1.0 @dataclass(frozen=True, slots=True) class Batch: """One batch of a task, when its bound is its own. A finder may return these instead of plain payloads. Two shapes rather than one because the second is what nearly every task wants and reads as what it is — `[{"uids": [...]}, ...]` is a list of batches, and wrapping each of them to say "and the default bound" would be ceremony on every task to serve the one that needs it. """ payload: Payload bound: Bound = Bound() #: What to call this one batch on screen, or "" to be called whatever its #: job kind is called. The kind's label is the same string on every batch #: of the task — nine rows reading "Tagging a batch" — because a label is #: declared once, at registration, by something that cannot see how the #: work was cut. Only the finder knows that, so this is where a batch that #: can say which batch it is says so. See `desk/modeltask.py`. label: str = "" #: Given the app's config and store, what there is to do — one payload per #: batch, or one `Batch` where the bound is the batch's own. An empty list is #: a legitimate answer and the common one: a task with nothing waiting costs #: one query, queues no batches, and its final job runs immediately and moves #: the cascade on. Finder = Callable[[Any, Any], list[Any]] def _batch(found: Payload | Batch) -> Batch: """One thing a finder returned, as a batch. A bare payload gets the default bound, which is the behaviour it had before bounds existed.""" return found if isinstance(found, Batch) else Batch(found) #: name -> how to build that task, standing on its own. Filled by the app the #: way `JOBS` is, and for the same reason: a task is as much a thing to start #: as a job is, so whatever offers the one — `/api/jobs`, the Jobs view, a #: script — has to be able to find the other by name. TASKS: dict[str, Callable[[], Task]] = {} class Sink(Protocol): """Where a task puts the jobs it makes. Two of them: `JobQueue`, when a task is started from a request or a handler, and nothing else. It is a protocol rather than the class so that `Task` does not import the queue — the dependency runs the other way, and a test can hand it a list. """ def new_run(self, kind: str, name: str = "", label: str = "") -> int: ... def cancelled(self, run_id: int) -> bool: ... def enqueue(self, kind: str, payload: Payload | None = None, *, label: str = "", depends_on: Sequence[int] = (), after: str = "ok", cascade_id: int = 0, task_id: int = 0, max_attempts: int | None = None, retry_delay: float | None = None, retry_backoff: float | None = None) -> Any: ... @dataclass(frozen=True, slots=True) class Task: """One business step: what a batch of it is, how to find them, who hears. Knows nothing about what the work *is* — that is the batch job's, and the finder's. This is the shape they are held in. """ #: The job kind one batch runs as. A task of one batch names an ordinary #: job kind (`pull`); a task of many names the batch kind (`body_batch`). kind: str #: How the work is found. Called once, in the caller, before anything is #: queued. finder: Finder #: What this task is — `body`, `tag`. Its identity: what a cascade's list #: matches against, and what the `runs` row is minted under. The job kind #: when nobody says. name: str = "" #: How to say that on screen: "Fetching what the feeds left out". Written #: onto the `runs` row, because a task is not a job and has no handler to #: carry a label. The name when nobody says. label: str = "" #: `module:function` for the final job to call, or "" for a task that #: reports to nobody — one started on its own, from a button or a script. callback: str = "" #: `module:function` the final job runs first, or "" for a task whose #: batches finish their own work. A name for the same reason the callback #: is one: it is a row in a table. #: #: The seam exists for a decision no batch can make on its own. A batch is #: one of forty and cannot see the other thirty-nine, so anything that has #: to hold across the whole run — a cap on what the run may add, a merge of #: what several batches each asked for — has exactly one place it can be #: made correctly, and that is the job that runs when they are all over. #: It belongs to the task rather than to the callback because it *is* the #: task's work: a task started on its own has no callback and still has to #: settle. See `newsdesk/jobs/tagging.py` for the one that does. settle: str = "" #: What the callback is handed alongside the tally. Fixed when the task is #: made, and stored in the final job's payload, so it survives a restart #: and a worker in another process. carries: dict[str, Any] = field(default_factory=dict) #: How many items a payload is worth, for the direct path's progress bar. #: Only the direct path asks: as a job a batch is on its own and reports #: its own rows, which is what the boxes on screen are drawn from. size: Callable[[Payload], int] | None = None #: How the direct path's batch results become one result. The queue never #: calls it — there each batch is its own job with its own row. fold: Callable[[Any, Any, list[Any]], dict[str, Any]] | None = None @property def named(self) -> str: return self.name or self.kind @property def called(self) -> str: return self.label or self.named # ------------------------------------------------------------- queued def start(self, config: Any, store: Any, sink: Sink, *, cascade_id: int = 0) -> int: """Find the work, queue it, and be over. Returns the task's run id. The id is minted before a single job exists, because a task is a thing before its jobs are — it is what decides how many there will be — and naming it after whichever job it happened to queue first would be describing it by one of its own consequences. With no cascade given it begins one: something asked for this on its own, so it is a cascade of one task. Named apart from any chain of the same name — `pull on its own`, not `pull` — because pressing Tag is not starting the chain that ends in tagging, and whoever asks whether that chain is going has to be told no. See `on_its_own`. Returns 0 when the cascade it belongs to has been killed. Nothing is told; it tries, and the table says no. `Jobs.enqueue` is where that is actually enforced, across processes and after a restart; this only saves minting a run for jobs that would all be refused. """ task_id, _ = queue_batches( sink, self.kind, lambda: self.finder(config, store), name=self.named, label=self.called, then=self.callback, settle=self.settle, carries=self.carries, cascade_id=cascade_id) return task_id # ------------------------------------------------------------- direct def run(self, config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]: """Do the batches here and now, in order, with no queue at all. What a Claude Code session, a notebook and a test get. The same batch handler the worker would call, with the same payloads from the same finder — there is no second implementation to drift. A batch that raises is kept and the rest carry on, which is what the queue does with eight independent jobs of which one fails. Unless *nothing* got through: then the first exception is raised, because a run that returned "0 done" with the explanation only in the log would be a failure reported as a success. A `Cancelled` travels either way — it is an instruction from the person at the keyboard, not a batch that did not land. A batch's `Bound` is ignored here, and that is not an omission: a bound is a number the *queue* enforces by putting a row back with a time on it, and there is no queue and no row on this path. Asked for directly, a batch is attempted once and its failure is returned to the caller, who is standing right there. """ spec = JOBS.get(self.kind) if spec is None: raise KeyError( f"{self.named!r} makes jobs of kind {self.kind!r}, which " f"nothing registers (have: {', '.join(sorted(JOBS)) or 'none'})") payloads = [_batch(found).payload for found in self.finder(config, store)] sizes = [self._size(p) for p in payloads] total = sum(sizes) results: list[Any] = [] failure: Exception | None = None at = 0 for payload, size in zip(payloads, sizes, strict=True): context.check() extra = {"context": self._placed(context, at, total, size)} \ if spec.wants_context else {} try: results.append(dict(spec.fn(config, store, **extra, **payload) or {})) except Cancelled: raise except Exception as exc: log.exception("%s: a batch of %s did not land", self.named, self.kind) results.append(exc) failure = failure or exc finally: # Position through the backlog, not what was answered: a batch # the model skipped half of still moved the run that far, and a # bar that goes backwards is worse than one that is approximate. at += size if failure is not None and not any(isinstance(r, dict) for r in results): raise failure if self.fold is not None: return self.fold(config, store, results) return {"batches": sum(1 for r in results if isinstance(r, dict)), "failed": sum(1 for r in results if isinstance(r, Exception)), "results": [r for r in results if isinstance(r, dict)]} def _size(self, payload: Payload) -> int: return int(self.size(payload)) if self.size is not None else 0 def _placed(self, context: RunContext, at: int, total: int, size: int) -> RunContext: """The batch's context, with its progress shifted into the whole run. A batch run as a job is on its own and counts its own rows. Run here it is one of several in a row, and a bar that restarts per batch would be a bar that goes backwards five times. """ if not total: return context return replace(context, on_progress=lambda done, _t, message: context.progress(at + min(done, size), total, message)) # ------------------------------------------------------------- the final job @dataclass(frozen=True, slots=True) class Tally: """How a task's batches ended, as the callback is handed it. Read off the jobs at the moment the final job runs, never counted up as they landed — so there is nothing to increment and nothing to lose to a restart half way through. """ done: int = 0 failed: int = 0 cancelled: int = 0 #: Whether the task itself was killed, as against however many of its #: batches happen to have ended cancelled. The counts cannot say: one #: batch killed by hand and every batch killed by the task going down look #: identical there and mean opposite things — drop this batch, or skip #: this whole step. Only the run's row knows, which is why it is asked. killed: bool = False #: What each batch returned, in the order they were queued. results: tuple[dict[str, Any], ...] = () @property def jobs(self) -> int: return self.done + self.failed + self.cancelled @classmethod def of(cls, outcomes: Sequence[Any], *, killed: bool = False) -> Tally: states = [getattr(j, "state", "") for j in outcomes] return cls( done=states.count("done"), failed=states.count("failed"), cancelled=states.count("cancelled"), killed=killed, results=tuple(dict(getattr(j, "result", {}) or {}) for j in outcomes), ) def as_dict(self) -> dict[str, Any]: return {"jobs": self.jobs, "done": self.done, "failed": self.failed, "cancelled": self.cancelled, "killed": self.killed} def queue_batches(sink: Sink, kind: str, found: Callable[[], Iterable[Payload | Batch]], *, name: str = "", label: str = "", then: str = "", settle: str = "", carries: dict[str, Any] | None = None, cascade_id: int = 0, max_attempts: int | None = None, retry_delay: float | None = None, retry_backoff: float | None = None) -> tuple[int, tuple[int, ...]]: """Mint a task, queue one job per batch, and queue the job that reports. The one implementation behind both `Task.start` and `fan_out`. Returns the task's run id and the children's job ids; `(0, ())` when the cascade it would join has been killed. `found` is called only after the runs exist, which is the order a task has always had: a task is a thing before its jobs are, since it is what decides how many there will be. A plain payload takes the kind's own retry policy, or the uniform one passed here. A `Batch` carries its own `Bound`, which wins — that is what a batch cut from one plugin profile needs when the profile declares how long to leave its publisher alone. With no cascade given it begins one: something asked for this on its own, so it is a cascade of one task, named apart from any chain of the same name (`pull on its own`, not `pull`). See `on_its_own`. """ if cascade_id and sink.cancelled(cascade_id): return 0, () name = name or kind label = label or name cascade_id = cascade_id or sink.new_run("cascade", on_its_own(name), label) task_id = sink.new_run("task", name, label) ids: list[int] = [] for item in found(): if isinstance(item, Batch): queued = sink.enqueue(kind, dict(item.payload), label=item.label, cascade_id=cascade_id, task_id=task_id, max_attempts=item.bound.max_attempts, retry_delay=item.bound.delay, retry_backoff=item.bound.backoff) else: queued = sink.enqueue(kind, dict(item), cascade_id=cascade_id, task_id=task_id, max_attempts=max_attempts, retry_delay=retry_delay, retry_backoff=retry_backoff) if queued is not None: ids.append(queued.id) # Behind all of them, and part of the same task — killing the task spares # it, because reporting is not work and "skip this task" is not "make it # never end". That holds where the kill landed mid-loop too: `Jobs.enqueue` # carves `after='done'` out of a task's kill, so this row is written and # the task ends by saying it was killed. None here is the *cascade* going # down, which takes the report with it by design: there is nothing left # to report to. See `Jobs.cancel_run`. sink.enqueue(FINAL, {"callback": then, "settle": settle, "payload": dict(carries or {})}, depends_on=ids, after="done", cascade_id=cascade_id, task_id=task_id) return task_id, tuple(ids) def fan_out(sink: Sink, kind: str, payloads: Sequence[Payload | Batch], *, then: str = "", settle: str = "", label: str = "", carries: dict[str, Any] | None = None, cascade_id: int = 0, max_attempts: int | None = None, retry_delay: float | None = None, retry_backoff: float | None = None) -> tuple[int, ...]: """Queue these payloads as jobs, and one job behind them that reports. What `RunContext.fan_out` reaches. `Task` is the same call with a finder in front, which is why a handler whose whole body is one of these *is* a task. The children go into a run of their own inside the caller's cascade, so the callback can fan out again and the result is a chain rather than a nest. Returns the children's ids. The reporting job is not among them — it is behind them by construction, and a caller that waited on it would be holding a worker open to do so. """ _, ids = queue_batches( sink, kind, lambda: payloads, label=label, then=then, settle=settle, carries=carries, cascade_id=cascade_id, max_attempts=max_attempts, retry_delay=retry_delay, retry_backoff=retry_backoff) return ids @job(FINAL, "Moving on") def run_final(config: Any, store: Any, *, context: RunContext = NULL, callback: str = "", settle: str = "", payload: dict[str, Any] | None = None, ) -> dict[str, Any]: """The one job every task ends with: tally the batches, call the name. It runs for the milliseconds it takes to read how the batches went and hand that to whoever asked — nothing is alive between two tasks, and nothing was waiting while the last one ran. `after="done"`, so it runs however the batches ended. That is the whole reason the mode exists: a job whose purpose is to report that one of eight failed cannot be held back by that failure. A task with no callback stops here, which is what being asked for on its own means. The settle runs first and the callback second, because the settle is the last of this task's own work and the callback is what comes after this task: a cascade told to move on before the tags of the stage it is leaving exist would be a stage reporting itself finished early. It asks whether its own task was killed, as well as counting how the batches ended, because those are two different facts that arrive looking the same. A task killed while its batches ran leaves them all `cancelled`, and so does a person killing every batch by hand; the first is "skip this step", the second is "drop this work". Only the run's row can tell them apart, and the callback needs to be told which it is — the whole reason this job is spared the kill is so that somebody hears. """ tally = Tally.of(context.waited_for, killed=context.killed()) out = tally.as_dict() if settle: out |= resolve(settle)(config, store, tally, context=context) or {} if not callback: return out return {**out, **(resolve(callback)(config, store, tally, dict(payload or {}), context=context) or {})} def resolve(name: str) -> Callable[..., dict[str, Any] | None]: """`module:function` to the function. See this module's docstring for why a callback, or a settle, is a name at all.""" module, _, attr = name.partition(":") if not module or not attr: raise ValueError(f"a callback is 'module:function', not {name!r}") return getattr(importlib.import_module(module), attr)