"""What a running job is handed, and the two things it can raise. def handler(config, store, *, context=NULL, limit=None) -> dict This is the job-side half of the contract; `registry.py` is the other — how a function becomes queueable. A pipeline stage imports this to report progress from inside its own loops, and nothing else: no queue, no worker. Nothing here imports anything else from desk at module level, so it is safe to import from anywhere. Everything in here is optional to the job that receives it. Ignore `context` and the job still runs — it just cannot be stopped part-way and shows no progress. Use it and you get both, for two lines. """ from __future__ import annotations import logging import os import signal import subprocess import tempfile import time from collections.abc import Callable, Sequence from dataclasses import dataclass from typing import Any log = logging.getLogger(__name__) #: How often a spawned process is looked in on: whether it has finished, #: whether Kill has been pressed, whether its deadline has passed. Frequent #: enough that a Kill lands in well under a second, rare enough that waiting #: an hour for ffmpeg costs a few thousand cheap syscalls. POLL = 0.25 class Cancelled(Exception): """Raised by `RunContext.check` when the job has been asked to stop. Caught by the worker and recorded as `cancelled` rather than `failed`. A handler should let it travel — catching it is how you build the thing the Kill button can't kill. """ class Retry(Exception): """Raised by a handler that wants this same work attempted again later. An ordinary failure is already retried, within whatever bound the job was queued with — so this is not how a job asks to be retried at all. It is how it says *when*: `after` is the wait the other end named, a `Retry-After` header or a rate limit's own answer, and the queue uses it in place of the backoff it would otherwise have guessed. Guessing is the thing worth avoiding here — a minute is either a minute of doing nothing when the service was ready, or another refusal charged against the bound. It does not buy an attempt. A job queued with one attempt raises this and fails, exactly as any other exception would: the bound is ours to set and the wait is the world's to name, and a handler that could extend its own bound would be a policy nobody declared. """ def __init__(self, message: str = "", *, after: float | None = None) -> None: super().__init__(message) #: Seconds to wait, or None to leave it to the job's own backoff. self.after = after def _no_progress(done: int, total: int, message: str) -> None: return None def _no_task(task: Any) -> bool: return False def _no_stop(why: str) -> int: """Where "stop the rest of this run" goes when there is no run to stop. The direct path is one batch after another in the caller's own thread, and the caller is standing there: it sees each failure as it happens and can stop by not asking again. Nothing to cancel, so nothing is cancelled. """ return 0 def _no_fan_out(*_args: Any, **_kwargs: Any) -> tuple[int, ...]: """Where a fan-out goes when there is no queue to put it in. Empty rather than an error, and the same answer `task` gives for the same reason: a handler called directly — from a terminal, a notebook, a test — is being run by somebody who is standing there, and the useful behaviour is to do its own work and report that nothing was queued behind it. A handler that must know either way checks the return. """ return () def _log_only(notice: Any) -> None: """Where a notice goes when nothing is watching for one. A direct caller — `newsdesk job pull` in a terminal, a notebook, a test — has no socket, so the log is the destination. It is not a *lesser* one: it goes through the same `notices.render`, so the terminal gets the sentence, the command to run and the is-this-a-bug verdict, exactly as the browser does. """ from pipeline.notices import render log.warning("%s\n%s", notice.code, render(notice)) def _never() -> bool: return False def kill_tree(proc: subprocess.Popen[Any]) -> None: """Kill a spawned process *and* whatever it started. The children are the point. `proc.kill()` reaches the process we launched and nothing it launched itself, so killing a shell leaves the ffmpeg it ran holding the core — which is the exact failure a deadline exists to prevent. Both branches need the process to have been started detached, which `spawn` does. Best-effort by construction: between deciding to kill and killing, the process may have ended on its own and its id may have been reused. So every call is guarded, and a failure to kill something that is already gone is not news. """ if proc.poll() is not None: return try: if os.name == "nt": # No process groups to signal, so ask Windows to walk the tree. # `/T` is children, `/F` is without asking. subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)], capture_output=True, check=False, timeout=10) else: # Guarded by `os.name` above rather than by the type checker, # which is reading this file on Windows and knows these three do # not exist there. os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # type: ignore[attr-defined,unused-ignore] except Exception: log.debug("could not kill %s", proc.pid, exc_info=True) try: proc.wait(timeout=5) except Exception: log.warning("process %s did not go away after being killed", proc.pid) def spawn(argv: Sequence[str], *, check: Callable[[], None] | None = None, timeout: float | None = None, poll: float = POLL, **popen: Any) -> subprocess.CompletedProcess[bytes]: """Run a command, watching for Kill and for a deadline while it does. The building block under `RunContext.spawn` and under `isolate=True`, and the answer to the one thing a thread cannot do. A handler that blocks in `subprocess.run` cannot be stopped, because stopping it would mean tearing a thread out of a syscall; a handler that blocks *here* can, because what is expensive is the child and a child can be killed. The thread is only sleeping, which costs nothing. Output goes to temporary files rather than pipes. `PIPE` plus a poll loop is a deadlock: a child that fills the pipe buffer blocks on the write while we block on the poll, and neither side is wrong. A file cannot fill up in the way that matters, so the loop is free to look away. Started detached — its own process group, or Windows' equivalent — purely so `kill_tree` has something to aim at. """ detach: dict[str, Any] = ({"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP} if os.name == "nt" else {"start_new_session": True}) with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err: popen.setdefault("stdout", out) popen.setdefault("stderr", err) proc = subprocess.Popen(list(argv), **detach, **popen) deadline = time.monotonic() + timeout if timeout else None try: while proc.poll() is None: if check is not None: check() # raises Cancelled on Kill if deadline is not None and time.monotonic() > deadline: kill_tree(proc) raise TimeoutError( f"{argv[0]} was still running after {timeout:g}s") time.sleep(poll) except BaseException: kill_tree(proc) raise out.seek(0) err.seek(0) return subprocess.CompletedProcess( list(argv), proc.returncode, out.read() if popen["stdout"] is out else b"", err.read() if popen["stderr"] is err else b"") @dataclass(slots=True) class RunContext: """The handle a running job has on the run it is part of. Deliberately a handful of plain callables rather than a reference to the queue: a handler cannot reach back into the worker, reschedule itself or read other jobs, and a test can hand it a lambda without standing anything up. """ on_progress: Callable[[int, int, str], None] = _no_progress is_cancelled: Callable[[], bool] = _never #: Whether the *task* this job belongs to was killed — a different #: question from `is_cancelled`, which is about this job alone. See #: `killed`. is_killed: Callable[[], bool] = _never on_notice: Callable[[Any], None] = _log_only on_task: Callable[[Any], bool] = _no_task on_stop: Callable[[str], int] = _no_stop #: Where `fan_out` puts the jobs it asks for. Supplied by the queue, which #: is the only thing that can write rows; `_no_fan_out` under `NULL`. on_fan_out: Callable[..., tuple[int, ...]] = _no_fan_out #: How the jobs this one waited for ended, oldest first. Empty for a job #: that waited for nothing. Typed loosely for the same reason `notice` is: #: nothing in this module may import the rest of desk, and the shape it #: carries is `pipeline.models.Job`. waited_for: tuple[Any, ...] = () def upstream(self) -> dict[str, Any]: """What the jobs this one waited for returned, merged into one dict. The read half of a chain. `A -> B -> C` needs A's output to reach B, and this is how it arrives: B does not have its payload filled in by A, it reads A's result off A's row. Which is the whole difference — **a row is complete when it is written.** A payload with a hole in it meant for an earlier job to fill would be claimable the instant that job landed, so a worker could take it between the write and the commit; it could not carry a `dedup_key`, which has to be known at queue time; and it could not be re-run from its own row, which is what a retry does and what `max_attempts` is. Merged, later parents winning, because the overwhelmingly common case is one parent and `upstream()["text"]` is what that should read like. A job with several parents that care about the same key wants `waited_for` and to decide for itself. Empty under `NULL`, which is what makes the fallback work: def handler(config, store, *, context=NULL, html=""): html = html or context.upstream().get("html", "") From the queue nobody passes `html` and it comes from the parent; from a terminal you pass it yourself. Without that line a chained handler is the one kind in the pipeline you cannot simply call. """ merged: dict[str, Any] = {} for parent in self.waited_for: result = getattr(parent, "result", None) if isinstance(result, dict): merged |= result return merged def fan_out(self, kind: str, payloads: Sequence[dict[str, Any]], *, then: str = "", settle: str = "", label: str = "", carries: dict[str, Any] | None = None, max_attempts: int | None = None, retry_delay: float | None = None, retry_backoff: float | None = None) -> tuple[int, ...]: """Queue one job per payload, and one job behind them that reports. Returns the ids of the children. The reporting job is not among them: it is the pipeline's own kind, it is nothing an app writes, and it is the reason this method exists rather than a handler assembling a `depends_on` list by hand every time. def handler(config, store, *, context=NULL): chunks = chunk(needs_a_body(store)) context.fan_out("body_batch", [{"uids": c} for c in chunks], then="newsdesk.jobs.cascade:step") `then` is `module:function`, resolved where it is run, and is handed `(config, store, tally, carries)` once every child is over however it ended — see `Tally` in `task.py`. A name rather than a function because what carries it is a table row, and no row can hold a function. One transaction, so a fan-out is never half-queued. The children go into a run of their own within this job's cascade, which is what makes the callback able to fan out again: that is a cascade, and there is no cascade object anywhere — step N's callback queues step N+1. Nothing is queued under `NULL`, and `()` says so. """ return self.on_fan_out( kind, list(payloads), then=then, settle=settle, label=label, carries=dict(carries or {}), max_attempts=max_attempts, retry_delay=retry_delay, retry_backoff=retry_backoff) def spawn(self, argv: Sequence[str], *, timeout: float | None = None, **popen: Any) -> subprocess.CompletedProcess[bytes]: """Run an external command, killable, with this job's Kill wired in. What to reach for when the expensive thing is a binary — ffmpeg, stockfish, whisper.cpp. Because the work is in the child and not in this thread, Kill is no longer cooperative in the way `check` is: the child is killed outright, and its children with it. Blocking a worker thread on a process costs nothing, since the thread is only sleeping. `timeout` is a real deadline for the same reason, and the only kind of deadline the pipeline offers that can actually end the work. A job kind that wants one over Python code rather than over a command wants `isolate=True` — see `registry.py`. Raises `Cancelled` on Kill and `TimeoutError` on the deadline, having killed the process tree either way. Non-zero exit is *not* an error here: the returncode is handed back, because plenty of commands say something useful with it and only the handler knows which. """ return spawn(argv, check=self.check, timeout=timeout, **popen) def progress(self, done: int = 0, total: int = 0, message: str = "") -> None: """Say how far along this is. Units are the handler's own business — sources, files, minutes of audio — the UI renders `done/total`. The message has one shape, and every job in this codebase uses it: a lowercase participle, what it is being done to, and — when the job is walking a list — `, n of m`. "reading Golem, 3 of 12". "fetching the body from theverge.com, 40 of 212". "training on your votes" for a job that is one step and has no list to be three of. Not an em dash, not a middot, and never a bare identifier: this used to say `wordpress`, which named the module that reads the feed rather than the outlet, and read as a fragment of a log line that had escaped onto the status strip. `n` is the item being worked on now and so runs one ahead of `done`, which counts the ones that are finished. They are answering different questions — the fraction in the message belongs to the name beside it, and the bar belongs to the job — and a line saying "reading Golem, 0 of 12" while Golem is being read would be wrong about the only thing it was there to say. A phase that has nothing left to name ends on the collective — "reading every source, 12 of 12" — so the same participle carries the line from first item to last, and a full fraction is what says it is over. Past tense is not used: it dates a line that is about to be replaced by the next phase's. A job's *label* is the same grammar capitalised, because a label names the work rather than narrating it: "Tagging 25 items, 2 of 9". """ self.on_progress(done, total, message) def check(self) -> None: """Stop now if Kill has been pressed. Cancellation is cooperative because the work is synchronous: there is no safe way to tear a thread out of the middle of an HTTP fetch or a model load, so the job says where stopping is fine. Call it between units of work, not inside tight inner loops. """ if self.is_cancelled(): raise Cancelled() def killed(self) -> bool: """Whether the task this job belongs to was killed. Not the question `check` asks. That one is about this job, and killing one job is dropping one batch. This one is about the run, and killing a run is skipping the whole step — which the job that reports on the step has to be able to say, since it is spared the kill precisely so that somebody hears about it. The counts cannot tell the two apart: a batch someone killed by hand and a batch killed by the task going down both arrive as `cancelled`. Asked when asked, not read at the start, so a kill landing while the reporting job runs is still in what it reports. """ return bool(self.is_killed()) def step(self, done: int = 0, total: int = 0, message: str = "") -> None: """`progress` then `check` — what the top of a loop body always wants.""" self.progress(done, total, message) self.check() def task(self, task: Any) -> bool: """Queue a `Task` once this job has finished well. Recorded rather than queued: the worker starts the task after this job lands in `done`, so a task never outlives the failure of the job that described it, and a handler still cannot reach the queue. It names what it wants to happen and that is all. The task runs in the worker, synchronously, the moment the job is over — the same plain code a request would run. It is handed the queue as its sink and this job's cascade, so its batches land in the run this one belongs to. Returns whether anything took it. `False` under `NULL`, where there is no queue to take it: a callback invoked from a notebook or a Claude Code session has to do the work itself, and this is how it finds out that nobody else will. """ return bool(self.on_task(task)) def stop(self, why: str = "") -> int: """Cancel this task's remaining queued batches. Returns how many. For the failure that is not this batch's: a spend limit, a signed-out CLI, no model on the machine at all. The backend has answered, the answer is the same one every sibling is about to be given, and the only thing left to decide is whether to spend the other twenty-two calls finding that out. Twenty-three identical notices saying "looks like a bug" is what not deciding looked like. Only what is still *queued* stops. A batch already in flight is a request in the air and a bill either way, so it runs to its own end — and this one, the one that asked, keeps its own failure. The task's final job is spared for the reason `Jobs.cancel_run` spares it: reporting is not work, and a run that stopped early still has to say so. `why` is written onto each cancelled row, so a batch that never ran says on screen why it didn't. Not a kill. Nobody pressed anything, so the task's row is not flagged and `killed` stays false: the cascade goes on to the next stage, as it does after any batch that did not land. Whether the *next* task should stop too is that cascade's question, not this one's. """ return int(self.on_stop(why)) def notice(self, notice: Any) -> None: """Tell the user something went wrong. See `pipeline/notices.py`. The counterpart to `log.exception`, not a replacement for it: log the traceback for afterwards, raise a notice for now. Use it for what a person has to act on — a login to renew, a feed that has stopped resolving — and not for what the program can simply get on with. Under a queue it is written to the notice log and pushed to whoever is watching; called directly it is logged and nothing else. Safe to call repeatedly: the same `code` and `key` again is one row with a count, and only the first interrupts anyone. So give `key` whatever the failure is about — the source, the article — or two different failures of one kind are told as one. """ self.on_notice(notice) #: For calling a handler outside the queue. Reports nowhere, never cancels. NULL = RunContext()