Files
WolfandClaude Opus 5.5 515d2250fc Deploy from a WSL checkout, from boilerplate 1679e88
When the checkout lives under \\wsl.localhost, step 1 runs its tests in the
distro through deploy/test-commit.sh and db.ps1 push makes its snapshot
there, because the .venv is a Linux one. pipeline/context.py narrows by
sys.platform so mypy passes on Linux. See boilerplate docs/wsl.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 19:43:50 +02:00

452 lines
21 KiB
Python

"""What a running job is handed, and the two things it can raise.
def handler(config, store, *, context=NULL, limit=None) -> dict
This is the job-side half of the contract; `registry.py` is the other — how
a function becomes queueable. A pipeline stage imports this to report
progress from inside its own loops, and nothing else: no queue, no worker.
Nothing here imports anything else from desk at module level, so it is safe to
import from anywhere.
Everything in here is optional to the job that receives it. Ignore `context`
and the job still runs — it just cannot be stopped part-way and shows no
progress. Use it and you get both, for two lines.
"""
from __future__ import annotations
import logging
import os
import signal
import subprocess
import sys
import tempfile
import time
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import Any
log = logging.getLogger(__name__)
#: How often a spawned process is looked in on: whether it has finished,
#: whether Kill has been pressed, whether its deadline has passed. Frequent
#: enough that a Kill lands in well under a second, rare enough that waiting
#: an hour for ffmpeg costs a few thousand cheap syscalls.
POLL = 0.25
class Cancelled(Exception):
"""Raised by `RunContext.check` when the job has been asked to stop.
Caught by the worker and recorded as `cancelled` rather than `failed`. A
handler should let it travel — catching it is how you build the thing the
Kill button can't kill.
"""
class Retry(Exception):
"""Raised by a handler that wants this same work attempted again later.
An ordinary failure is already retried, within whatever bound the job was
queued with — so this is not how a job asks to be retried at all. It is how
it says *when*: `after` is the wait the other end named, a `Retry-After`
header or a rate limit's own answer, and the queue uses it in place of the
backoff it would otherwise have guessed. Guessing is the thing worth
avoiding here — a minute is either a minute of doing nothing when the
service was ready, or another refusal charged against the bound.
It does not buy an attempt. A job queued with one attempt raises this and
fails, exactly as any other exception would: the bound is ours to set and
the wait is the world's to name, and a handler that could extend its own
bound would be a policy nobody declared.
"""
def __init__(self, message: str = "", *, after: float | None = None) -> None:
super().__init__(message)
#: Seconds to wait, or None to leave it to the job's own backoff.
self.after = after
def _no_progress(done: int, total: int, message: str) -> None:
return None
def _no_task(task: Any) -> bool:
return False
def _no_stop(why: str) -> int:
"""Where "stop the rest of this run" goes when there is no run to stop.
The direct path is one batch after another in the caller's own thread, and
the caller is standing there: it sees each failure as it happens and can
stop by not asking again. Nothing to cancel, so nothing is cancelled.
"""
return 0
def _no_fan_out(*_args: Any, **_kwargs: Any) -> tuple[int, ...]:
"""Where a fan-out goes when there is no queue to put it in.
Empty rather than an error, and the same answer `task` gives for the same
reason: a handler called directly — from a terminal, a notebook, a test —
is being run by somebody who is standing there, and the useful behaviour
is to do its own work and report that nothing was queued behind it. A
handler that must know either way checks the return.
"""
return ()
def _log_only(notice: Any) -> None:
"""Where a notice goes when nothing is watching for one.
A direct caller — `newsdesk job pull` in a terminal, a notebook, a test — has
no socket, so the log is the destination. It is not a *lesser* one: it goes
through the same `notices.render`, so the terminal gets the sentence, the
command to run and the is-this-a-bug verdict, exactly as the browser does.
"""
from pipeline.notices import render
log.warning("%s\n%s", notice.code, render(notice))
def _never() -> bool:
return False
def kill_tree(proc: subprocess.Popen[Any]) -> None:
"""Kill a spawned process *and* whatever it started.
The children are the point. `proc.kill()` reaches the process we launched
and nothing it launched itself, so killing a shell leaves the ffmpeg it
ran holding the core — which is the exact failure a deadline exists to
prevent. Both branches need the process to have been started detached,
which `spawn` does.
Best-effort by construction: between deciding to kill and killing, the
process may have ended on its own and its id may have been reused. So
every call is guarded, and a failure to kill something that is already
gone is not news.
"""
if proc.poll() is not None:
return
try:
if os.name == "nt":
# No process groups to signal, so ask Windows to walk the tree.
# `/T` is children, `/F` is without asking.
subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)],
capture_output=True, check=False, timeout=10)
else:
# Guarded by `os.name` above rather than by the type checker,
# which is reading this file on Windows and knows these three do
# not exist there.
os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # type: ignore[attr-defined,unused-ignore]
except Exception:
log.debug("could not kill %s", proc.pid, exc_info=True)
try:
proc.wait(timeout=5)
except Exception:
log.warning("process %s did not go away after being killed", proc.pid)
def spawn(argv: Sequence[str], *, check: Callable[[], None] | None = None,
timeout: float | None = None, poll: float = POLL,
**popen: Any) -> subprocess.CompletedProcess[bytes]:
"""Run a command, watching for Kill and for a deadline while it does.
The building block under `RunContext.spawn` and under `isolate=True`, and
the answer to the one thing a thread cannot do. A handler that blocks in
`subprocess.run` cannot be stopped, because stopping it would mean tearing
a thread out of a syscall; a handler that blocks *here* can, because what
is expensive is the child and a child can be killed. The thread is only
sleeping, which costs nothing.
Output goes to temporary files rather than pipes. `PIPE` plus a poll loop
is a deadlock: a child that fills the pipe buffer blocks on the write
while we block on the poll, and neither side is wrong. A file cannot fill
up in the way that matters, so the loop is free to look away.
Started detached — its own process group, or Windows' equivalent — purely
so `kill_tree` has something to aim at.
"""
# An `if`, because mypy narrows `sys.platform` in statements only.
detach: dict[str, Any]
if sys.platform == "win32":
detach = {"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
else:
detach = {"start_new_session": True}
with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err:
popen.setdefault("stdout", out)
popen.setdefault("stderr", err)
proc = subprocess.Popen(list(argv), **detach, **popen)
deadline = time.monotonic() + timeout if timeout else None
try:
while proc.poll() is None:
if check is not None:
check() # raises Cancelled on Kill
if deadline is not None and time.monotonic() > deadline:
kill_tree(proc)
raise TimeoutError(
f"{argv[0]} was still running after {timeout:g}s")
time.sleep(poll)
except BaseException:
kill_tree(proc)
raise
out.seek(0)
err.seek(0)
return subprocess.CompletedProcess(
list(argv), proc.returncode,
out.read() if popen["stdout"] is out else b"",
err.read() if popen["stderr"] is err else b"")
@dataclass(slots=True)
class RunContext:
"""The handle a running job has on the run it is part of.
Deliberately a handful of plain callables rather than a reference to the
queue: a handler cannot reach back into the worker, reschedule itself or
read other jobs, and a test can hand it a lambda without standing anything
up.
"""
on_progress: Callable[[int, int, str], None] = _no_progress
is_cancelled: Callable[[], bool] = _never
#: Whether the *task* this job belongs to was killed — a different
#: question from `is_cancelled`, which is about this job alone. See
#: `killed`.
is_killed: Callable[[], bool] = _never
on_notice: Callable[[Any], None] = _log_only
on_task: Callable[[Any], bool] = _no_task
on_stop: Callable[[str], int] = _no_stop
#: Where `fan_out` puts the jobs it asks for. Supplied by the queue, which
#: is the only thing that can write rows; `_no_fan_out` under `NULL`.
on_fan_out: Callable[..., tuple[int, ...]] = _no_fan_out
#: How the jobs this one waited for ended, oldest first. Empty for a job
#: that waited for nothing. Typed loosely for the same reason `notice` is:
#: nothing in this module may import the rest of desk, and the shape it
#: carries is `pipeline.models.Job`.
waited_for: tuple[Any, ...] = ()
def upstream(self) -> dict[str, Any]:
"""What the jobs this one waited for returned, merged into one dict.
The read half of a chain. `A -> B -> C` needs A's output to reach B,
and this is how it arrives: B does not have its payload filled in by
A, it reads A's result off A's row. Which is the whole difference —
**a row is complete when it is written.** A payload with a hole in it
meant for an earlier job to fill would be claimable the instant that
job landed, so a worker could take it between the write and the
commit; it could not carry a `dedup_key`, which has to be known at
queue time; and it could not be re-run from its own row, which is
what a retry does and what `max_attempts` is.
Merged, later parents winning, because the overwhelmingly common case
is one parent and `upstream()["text"]` is what that should read like.
A job with several parents that care about the same key wants
`waited_for` and to decide for itself.
Empty under `NULL`, which is what makes the fallback work:
def handler(config, store, *, context=NULL, html=""):
html = html or context.upstream().get("html", "")
From the queue nobody passes `html` and it comes from the parent; from
a terminal you pass it yourself. Without that line a chained handler
is the one kind in the pipeline you cannot simply call.
"""
merged: dict[str, Any] = {}
for parent in self.waited_for:
result = getattr(parent, "result", None)
if isinstance(result, dict):
merged |= result
return merged
def fan_out(self, kind: str, payloads: Sequence[dict[str, Any]], *,
then: str = "", settle: str = "", label: str = "",
carries: dict[str, Any] | None = None,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> tuple[int, ...]:
"""Queue one job per payload, and one job behind them that reports.
Returns the ids of the children. The reporting job is not among them:
it is the pipeline's own kind, it is nothing an app writes, and it is
the reason this method exists rather than a handler assembling a
`depends_on` list by hand every time.
def handler(config, store, *, context=NULL):
chunks = chunk(needs_a_body(store))
context.fan_out("body_batch", [{"uids": c} for c in chunks],
then="newsdesk.jobs.cascade:step")
`then` is `module:function`, resolved where it is run, and is handed
`(config, store, tally, carries)` once every child is over however it
ended — see `Tally` in `task.py`. A name rather than a function
because what carries it is a table row, and no row can hold a
function.
One transaction, so a fan-out is never half-queued. The children go
into a run of their own within this job's cascade, which is what makes
the callback able to fan out again: that is a cascade, and there is no
cascade object anywhere — step N's callback queues step N+1.
Nothing is queued under `NULL`, and `()` says so.
"""
return self.on_fan_out(
kind, list(payloads), then=then, settle=settle, label=label,
carries=dict(carries or {}), max_attempts=max_attempts,
retry_delay=retry_delay, retry_backoff=retry_backoff)
def spawn(self, argv: Sequence[str], *, timeout: float | None = None,
**popen: Any) -> subprocess.CompletedProcess[bytes]:
"""Run an external command, killable, with this job's Kill wired in.
What to reach for when the expensive thing is a binary — ffmpeg,
stockfish, whisper.cpp. Because the work is in the child and not in
this thread, Kill is no longer cooperative in the way `check` is: the
child is killed outright, and its children with it. Blocking a worker
thread on a process costs nothing, since the thread is only sleeping.
`timeout` is a real deadline for the same reason, and the only kind of
deadline the pipeline offers that can actually end the work. A job
kind that wants one over Python code rather than over a command wants
`isolate=True` — see `registry.py`.
Raises `Cancelled` on Kill and `TimeoutError` on the deadline, having
killed the process tree either way. Non-zero exit is *not* an error
here: the returncode is handed back, because plenty of commands say
something useful with it and only the handler knows which.
"""
return spawn(argv, check=self.check, timeout=timeout, **popen)
def progress(self, done: int = 0, total: int = 0, message: str = "") -> None:
"""Say how far along this is. Units are the handler's own business —
sources, files, minutes of audio — the UI renders `done/total`.
The message has one shape, and every job in this codebase uses it:
a lowercase participle, what it is being done to, and — when the
job is walking a list — `, n of m`.
"reading Golem, 3 of 12". "fetching the body from theverge.com, 40 of
212". "training on your votes" for a job that is one step and has no
list to be three of. Not an em dash, not a middot, and never a bare
identifier: this used to say `wordpress`, which named the module that
reads the feed rather than the outlet, and read as a fragment of a log
line that had escaped onto the status strip.
`n` is the item being worked on now and so runs one ahead of `done`,
which counts the ones that are finished. They are answering different
questions — the fraction in the message belongs to the name beside it,
and the bar belongs to the job — and a line saying "reading Golem, 0 of
12" while Golem is being read would be wrong about the only thing it
was there to say. A phase that has nothing left to name ends on the
collective — "reading every source, 12 of 12" — so the same participle
carries the line from first item to last, and a full fraction is what
says it is over. Past tense is not used: it dates a line that is about
to be replaced by the next phase's.
A job's *label* is the same grammar capitalised, because a label names
the work rather than narrating it: "Tagging 25 items, 2 of 9".
"""
self.on_progress(done, total, message)
def check(self) -> None:
"""Stop now if Kill has been pressed.
Cancellation is cooperative because the work is synchronous: there is
no safe way to tear a thread out of the middle of an HTTP fetch or a
model load, so the job says where stopping is fine. Call it between
units of work, not inside tight inner loops.
"""
if self.is_cancelled():
raise Cancelled()
def killed(self) -> bool:
"""Whether the task this job belongs to was killed.
Not the question `check` asks. That one is about this job, and killing
one job is dropping one batch. This one is about the run, and killing
a run is skipping the whole step — which the job that reports on the
step has to be able to say, since it is spared the kill precisely so
that somebody hears about it. The counts cannot tell the two apart: a
batch someone killed by hand and a batch killed by the task going
down both arrive as `cancelled`.
Asked when asked, not read at the start, so a kill landing while the
reporting job runs is still in what it reports.
"""
return bool(self.is_killed())
def step(self, done: int = 0, total: int = 0, message: str = "") -> None:
"""`progress` then `check` — what the top of a loop body always wants."""
self.progress(done, total, message)
self.check()
def task(self, task: Any) -> bool:
"""Queue a `Task` once this job has finished well.
Recorded rather than queued: the worker starts the task after this job
lands in `done`, so a task never outlives the failure of the job that
described it, and a handler still cannot reach the queue. It names what
it wants to happen and that is all.
The task runs in the worker, synchronously, the moment the job is over
— the same plain code a request would run. It is handed the queue as
its sink and this job's cascade, so its batches land in the run this
one belongs to.
Returns whether anything took it. `False` under `NULL`, where there is
no queue to take it: a callback invoked from a notebook or a Claude
Code session has to do the work itself, and this is how it finds out
that nobody else will.
"""
return bool(self.on_task(task))
def stop(self, why: str = "") -> int:
"""Cancel this task's remaining queued batches. Returns how many.
For the failure that is not this batch's: a spend limit, a signed-out
CLI, no model on the machine at all. The backend has answered, the
answer is the same one every sibling is about to be given, and the
only thing left to decide is whether to spend the other twenty-two
calls finding that out. Twenty-three identical notices saying "looks
like a bug" is what not deciding looked like.
Only what is still *queued* stops. A batch already in flight is a
request in the air and a bill either way, so it runs to its own end —
and this one, the one that asked, keeps its own failure. The task's
final job is spared for the reason `Jobs.cancel_run` spares it:
reporting is not work, and a run that stopped early still has to say
so. `why` is written onto each cancelled row, so a batch that never
ran says on screen why it didn't.
Not a kill. Nobody pressed anything, so the task's row is not flagged
and `killed` stays false: the cascade goes on to the next stage, as it
does after any batch that did not land. Whether the *next* task should
stop too is that cascade's question, not this one's.
"""
return int(self.on_stop(why))
def notice(self, notice: Any) -> None:
"""Tell the user something went wrong. See `pipeline/notices.py`.
The counterpart to `log.exception`, not a replacement for it: log the
traceback for afterwards, raise a notice for now. Use it for what a
person has to act on — a login to renew, a feed that has stopped
resolving — and not for what the program can simply get on with.
Under a queue it is written to the notice log and pushed to whoever is
watching; called directly it is logged and nothing else. Safe to call
repeatedly: the same `code` and `key` again is one row with a count,
and only the first interrupts anyone. So give `key` whatever the
failure is about — the source, the article — or two different
failures of one kind are told as one.
"""
self.on_notice(notice)
#: For calling a handler outside the queue. Reports nowhere, never cancels.
NULL = RunContext()