roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
447 lines
21 KiB
Python
447 lines
21 KiB
Python
"""What a running job is handed, and the two things it can raise.
|
|
|
|
def handler(config, store, *, context=NULL, limit=None) -> dict
|
|
|
|
This is the job-side half of the contract; `registry.py` is the other — how
|
|
a function becomes queueable. A pipeline stage imports this to report
|
|
progress from inside its own loops, and nothing else: no queue, no worker.
|
|
Nothing here imports anything else from desk at module level, so it is safe to
|
|
import from anywhere.
|
|
|
|
Everything in here is optional to the job that receives it. Ignore `context`
|
|
and the job still runs — it just cannot be stopped part-way and shows no
|
|
progress. Use it and you get both, for two lines.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import os
|
|
import signal
|
|
import subprocess
|
|
import tempfile
|
|
import time
|
|
from collections.abc import Callable, Sequence
|
|
from dataclasses import dataclass
|
|
from typing import Any
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
#: How often a spawned process is looked in on: whether it has finished,
|
|
#: whether Kill has been pressed, whether its deadline has passed. Frequent
|
|
#: enough that a Kill lands in well under a second, rare enough that waiting
|
|
#: an hour for ffmpeg costs a few thousand cheap syscalls.
|
|
POLL = 0.25
|
|
|
|
|
|
class Cancelled(Exception):
|
|
"""Raised by `RunContext.check` when the job has been asked to stop.
|
|
|
|
Caught by the worker and recorded as `cancelled` rather than `failed`. A
|
|
handler should let it travel — catching it is how you build the thing the
|
|
Kill button can't kill.
|
|
"""
|
|
|
|
|
|
class Retry(Exception):
|
|
"""Raised by a handler that wants this same work attempted again later.
|
|
|
|
An ordinary failure is already retried, within whatever bound the job was
|
|
queued with — so this is not how a job asks to be retried at all. It is how
|
|
it says *when*: `after` is the wait the other end named, a `Retry-After`
|
|
header or a rate limit's own answer, and the queue uses it in place of the
|
|
backoff it would otherwise have guessed. Guessing is the thing worth
|
|
avoiding here — a minute is either a minute of doing nothing when the
|
|
service was ready, or another refusal charged against the bound.
|
|
|
|
It does not buy an attempt. A job queued with one attempt raises this and
|
|
fails, exactly as any other exception would: the bound is ours to set and
|
|
the wait is the world's to name, and a handler that could extend its own
|
|
bound would be a policy nobody declared.
|
|
"""
|
|
|
|
def __init__(self, message: str = "", *, after: float | None = None) -> None:
|
|
super().__init__(message)
|
|
#: Seconds to wait, or None to leave it to the job's own backoff.
|
|
self.after = after
|
|
|
|
|
|
def _no_progress(done: int, total: int, message: str) -> None:
|
|
return None
|
|
|
|
|
|
def _no_task(task: Any) -> bool:
|
|
return False
|
|
|
|
|
|
def _no_stop(why: str) -> int:
|
|
"""Where "stop the rest of this run" goes when there is no run to stop.
|
|
|
|
The direct path is one batch after another in the caller's own thread, and
|
|
the caller is standing there: it sees each failure as it happens and can
|
|
stop by not asking again. Nothing to cancel, so nothing is cancelled.
|
|
"""
|
|
return 0
|
|
|
|
|
|
def _no_fan_out(*_args: Any, **_kwargs: Any) -> tuple[int, ...]:
|
|
"""Where a fan-out goes when there is no queue to put it in.
|
|
|
|
Empty rather than an error, and the same answer `task` gives for the same
|
|
reason: a handler called directly — from a terminal, a notebook, a test —
|
|
is being run by somebody who is standing there, and the useful behaviour
|
|
is to do its own work and report that nothing was queued behind it. A
|
|
handler that must know either way checks the return.
|
|
"""
|
|
return ()
|
|
|
|
|
|
def _log_only(notice: Any) -> None:
|
|
"""Where a notice goes when nothing is watching for one.
|
|
|
|
A direct caller — `newsdesk job pull` in a terminal, a notebook, a test — has
|
|
no socket, so the log is the destination. It is not a *lesser* one: it goes
|
|
through the same `notices.render`, so the terminal gets the sentence, the
|
|
command to run and the is-this-a-bug verdict, exactly as the browser does.
|
|
"""
|
|
from pipeline.notices import render
|
|
|
|
log.warning("%s\n%s", notice.code, render(notice))
|
|
|
|
|
|
def _never() -> bool:
|
|
return False
|
|
|
|
|
|
def kill_tree(proc: subprocess.Popen[Any]) -> None:
|
|
"""Kill a spawned process *and* whatever it started.
|
|
|
|
The children are the point. `proc.kill()` reaches the process we launched
|
|
and nothing it launched itself, so killing a shell leaves the ffmpeg it
|
|
ran holding the core — which is the exact failure a deadline exists to
|
|
prevent. Both branches need the process to have been started detached,
|
|
which `spawn` does.
|
|
|
|
Best-effort by construction: between deciding to kill and killing, the
|
|
process may have ended on its own and its id may have been reused. So
|
|
every call is guarded, and a failure to kill something that is already
|
|
gone is not news.
|
|
"""
|
|
if proc.poll() is not None:
|
|
return
|
|
try:
|
|
if os.name == "nt":
|
|
# No process groups to signal, so ask Windows to walk the tree.
|
|
# `/T` is children, `/F` is without asking.
|
|
subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)],
|
|
capture_output=True, check=False, timeout=10)
|
|
else:
|
|
# Guarded by `os.name` above rather than by the type checker,
|
|
# which is reading this file on Windows and knows these three do
|
|
# not exist there.
|
|
os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # type: ignore[attr-defined,unused-ignore]
|
|
except Exception:
|
|
log.debug("could not kill %s", proc.pid, exc_info=True)
|
|
try:
|
|
proc.wait(timeout=5)
|
|
except Exception:
|
|
log.warning("process %s did not go away after being killed", proc.pid)
|
|
|
|
|
|
def spawn(argv: Sequence[str], *, check: Callable[[], None] | None = None,
|
|
timeout: float | None = None, poll: float = POLL,
|
|
**popen: Any) -> subprocess.CompletedProcess[bytes]:
|
|
"""Run a command, watching for Kill and for a deadline while it does.
|
|
|
|
The building block under `RunContext.spawn` and under `isolate=True`, and
|
|
the answer to the one thing a thread cannot do. A handler that blocks in
|
|
`subprocess.run` cannot be stopped, because stopping it would mean tearing
|
|
a thread out of a syscall; a handler that blocks *here* can, because what
|
|
is expensive is the child and a child can be killed. The thread is only
|
|
sleeping, which costs nothing.
|
|
|
|
Output goes to temporary files rather than pipes. `PIPE` plus a poll loop
|
|
is a deadlock: a child that fills the pipe buffer blocks on the write
|
|
while we block on the poll, and neither side is wrong. A file cannot fill
|
|
up in the way that matters, so the loop is free to look away.
|
|
|
|
Started detached — its own process group, or Windows' equivalent — purely
|
|
so `kill_tree` has something to aim at.
|
|
"""
|
|
detach: dict[str, Any] = ({"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
|
|
if os.name == "nt" else {"start_new_session": True})
|
|
with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err:
|
|
popen.setdefault("stdout", out)
|
|
popen.setdefault("stderr", err)
|
|
proc = subprocess.Popen(list(argv), **detach, **popen)
|
|
deadline = time.monotonic() + timeout if timeout else None
|
|
try:
|
|
while proc.poll() is None:
|
|
if check is not None:
|
|
check() # raises Cancelled on Kill
|
|
if deadline is not None and time.monotonic() > deadline:
|
|
kill_tree(proc)
|
|
raise TimeoutError(
|
|
f"{argv[0]} was still running after {timeout:g}s")
|
|
time.sleep(poll)
|
|
except BaseException:
|
|
kill_tree(proc)
|
|
raise
|
|
out.seek(0)
|
|
err.seek(0)
|
|
return subprocess.CompletedProcess(
|
|
list(argv), proc.returncode,
|
|
out.read() if popen["stdout"] is out else b"",
|
|
err.read() if popen["stderr"] is err else b"")
|
|
|
|
|
|
@dataclass(slots=True)
|
|
class RunContext:
|
|
"""The handle a running job has on the run it is part of.
|
|
|
|
Deliberately a handful of plain callables rather than a reference to the
|
|
queue: a handler cannot reach back into the worker, reschedule itself or
|
|
read other jobs, and a test can hand it a lambda without standing anything
|
|
up.
|
|
"""
|
|
|
|
on_progress: Callable[[int, int, str], None] = _no_progress
|
|
is_cancelled: Callable[[], bool] = _never
|
|
#: Whether the *task* this job belongs to was killed — a different
|
|
#: question from `is_cancelled`, which is about this job alone. See
|
|
#: `killed`.
|
|
is_killed: Callable[[], bool] = _never
|
|
on_notice: Callable[[Any], None] = _log_only
|
|
on_task: Callable[[Any], bool] = _no_task
|
|
on_stop: Callable[[str], int] = _no_stop
|
|
#: Where `fan_out` puts the jobs it asks for. Supplied by the queue, which
|
|
#: is the only thing that can write rows; `_no_fan_out` under `NULL`.
|
|
on_fan_out: Callable[..., tuple[int, ...]] = _no_fan_out
|
|
#: How the jobs this one waited for ended, oldest first. Empty for a job
|
|
#: that waited for nothing. Typed loosely for the same reason `notice` is:
|
|
#: nothing in this module may import the rest of desk, and the shape it
|
|
#: carries is `pipeline.models.Job`.
|
|
waited_for: tuple[Any, ...] = ()
|
|
|
|
def upstream(self) -> dict[str, Any]:
|
|
"""What the jobs this one waited for returned, merged into one dict.
|
|
|
|
The read half of a chain. `A -> B -> C` needs A's output to reach B,
|
|
and this is how it arrives: B does not have its payload filled in by
|
|
A, it reads A's result off A's row. Which is the whole difference —
|
|
**a row is complete when it is written.** A payload with a hole in it
|
|
meant for an earlier job to fill would be claimable the instant that
|
|
job landed, so a worker could take it between the write and the
|
|
commit; it could not carry a `dedup_key`, which has to be known at
|
|
queue time; and it could not be re-run from its own row, which is
|
|
what a retry does and what `max_attempts` is.
|
|
|
|
Merged, later parents winning, because the overwhelmingly common case
|
|
is one parent and `upstream()["text"]` is what that should read like.
|
|
A job with several parents that care about the same key wants
|
|
`waited_for` and to decide for itself.
|
|
|
|
Empty under `NULL`, which is what makes the fallback work:
|
|
|
|
def handler(config, store, *, context=NULL, html=""):
|
|
html = html or context.upstream().get("html", "")
|
|
|
|
From the queue nobody passes `html` and it comes from the parent; from
|
|
a terminal you pass it yourself. Without that line a chained handler
|
|
is the one kind in the pipeline you cannot simply call.
|
|
"""
|
|
merged: dict[str, Any] = {}
|
|
for parent in self.waited_for:
|
|
result = getattr(parent, "result", None)
|
|
if isinstance(result, dict):
|
|
merged |= result
|
|
return merged
|
|
|
|
def fan_out(self, kind: str, payloads: Sequence[dict[str, Any]], *,
|
|
then: str = "", settle: str = "", label: str = "",
|
|
carries: dict[str, Any] | None = None,
|
|
max_attempts: int | None = None,
|
|
retry_delay: float | None = None,
|
|
retry_backoff: float | None = None) -> tuple[int, ...]:
|
|
"""Queue one job per payload, and one job behind them that reports.
|
|
|
|
Returns the ids of the children. The reporting job is not among them:
|
|
it is the pipeline's own kind, it is nothing an app writes, and it is
|
|
the reason this method exists rather than a handler assembling a
|
|
`depends_on` list by hand every time.
|
|
|
|
def handler(config, store, *, context=NULL):
|
|
chunks = chunk(needs_a_body(store))
|
|
context.fan_out("body_batch", [{"uids": c} for c in chunks],
|
|
then="newsdesk.jobs.cascade:step")
|
|
|
|
`then` is `module:function`, resolved where it is run, and is handed
|
|
`(config, store, tally, carries)` once every child is over however it
|
|
ended — see `Tally` in `task.py`. A name rather than a function
|
|
because what carries it is a table row, and no row can hold a
|
|
function.
|
|
|
|
One transaction, so a fan-out is never half-queued. The children go
|
|
into a run of their own within this job's cascade, which is what makes
|
|
the callback able to fan out again: that is a cascade, and there is no
|
|
cascade object anywhere — step N's callback queues step N+1.
|
|
|
|
Nothing is queued under `NULL`, and `()` says so.
|
|
"""
|
|
return self.on_fan_out(
|
|
kind, list(payloads), then=then, settle=settle, label=label,
|
|
carries=dict(carries or {}), max_attempts=max_attempts,
|
|
retry_delay=retry_delay, retry_backoff=retry_backoff)
|
|
|
|
def spawn(self, argv: Sequence[str], *, timeout: float | None = None,
|
|
**popen: Any) -> subprocess.CompletedProcess[bytes]:
|
|
"""Run an external command, killable, with this job's Kill wired in.
|
|
|
|
What to reach for when the expensive thing is a binary — ffmpeg,
|
|
stockfish, whisper.cpp. Because the work is in the child and not in
|
|
this thread, Kill is no longer cooperative in the way `check` is: the
|
|
child is killed outright, and its children with it. Blocking a worker
|
|
thread on a process costs nothing, since the thread is only sleeping.
|
|
|
|
`timeout` is a real deadline for the same reason, and the only kind of
|
|
deadline the pipeline offers that can actually end the work. A job
|
|
kind that wants one over Python code rather than over a command wants
|
|
`isolate=True` — see `registry.py`.
|
|
|
|
Raises `Cancelled` on Kill and `TimeoutError` on the deadline, having
|
|
killed the process tree either way. Non-zero exit is *not* an error
|
|
here: the returncode is handed back, because plenty of commands say
|
|
something useful with it and only the handler knows which.
|
|
"""
|
|
return spawn(argv, check=self.check, timeout=timeout, **popen)
|
|
|
|
def progress(self, done: int = 0, total: int = 0, message: str = "") -> None:
|
|
"""Say how far along this is. Units are the handler's own business —
|
|
sources, files, minutes of audio — the UI renders `done/total`.
|
|
|
|
The message has one shape, and every job in this codebase uses it:
|
|
|
|
a lowercase participle, what it is being done to, and — when the
|
|
job is walking a list — `, n of m`.
|
|
|
|
"reading Golem, 3 of 12". "fetching the body from theverge.com, 40 of
|
|
212". "training on your votes" for a job that is one step and has no
|
|
list to be three of. Not an em dash, not a middot, and never a bare
|
|
identifier: this used to say `wordpress`, which named the module that
|
|
reads the feed rather than the outlet, and read as a fragment of a log
|
|
line that had escaped onto the status strip.
|
|
|
|
`n` is the item being worked on now and so runs one ahead of `done`,
|
|
which counts the ones that are finished. They are answering different
|
|
questions — the fraction in the message belongs to the name beside it,
|
|
and the bar belongs to the job — and a line saying "reading Golem, 0 of
|
|
12" while Golem is being read would be wrong about the only thing it
|
|
was there to say. A phase that has nothing left to name ends on the
|
|
collective — "reading every source, 12 of 12" — so the same participle
|
|
carries the line from first item to last, and a full fraction is what
|
|
says it is over. Past tense is not used: it dates a line that is about
|
|
to be replaced by the next phase's.
|
|
|
|
A job's *label* is the same grammar capitalised, because a label names
|
|
the work rather than narrating it: "Tagging 25 items, 2 of 9".
|
|
"""
|
|
self.on_progress(done, total, message)
|
|
|
|
def check(self) -> None:
|
|
"""Stop now if Kill has been pressed.
|
|
|
|
Cancellation is cooperative because the work is synchronous: there is
|
|
no safe way to tear a thread out of the middle of an HTTP fetch or a
|
|
model load, so the job says where stopping is fine. Call it between
|
|
units of work, not inside tight inner loops.
|
|
"""
|
|
if self.is_cancelled():
|
|
raise Cancelled()
|
|
|
|
def killed(self) -> bool:
|
|
"""Whether the task this job belongs to was killed.
|
|
|
|
Not the question `check` asks. That one is about this job, and killing
|
|
one job is dropping one batch. This one is about the run, and killing
|
|
a run is skipping the whole step — which the job that reports on the
|
|
step has to be able to say, since it is spared the kill precisely so
|
|
that somebody hears about it. The counts cannot tell the two apart: a
|
|
batch someone killed by hand and a batch killed by the task going
|
|
down both arrive as `cancelled`.
|
|
|
|
Asked when asked, not read at the start, so a kill landing while the
|
|
reporting job runs is still in what it reports.
|
|
"""
|
|
return bool(self.is_killed())
|
|
|
|
def step(self, done: int = 0, total: int = 0, message: str = "") -> None:
|
|
"""`progress` then `check` — what the top of a loop body always wants."""
|
|
self.progress(done, total, message)
|
|
self.check()
|
|
|
|
def task(self, task: Any) -> bool:
|
|
"""Queue a `Task` once this job has finished well.
|
|
|
|
Recorded rather than queued: the worker starts the task after this job
|
|
lands in `done`, so a task never outlives the failure of the job that
|
|
described it, and a handler still cannot reach the queue. It names what
|
|
it wants to happen and that is all.
|
|
|
|
The task runs in the worker, synchronously, the moment the job is over
|
|
— the same plain code a request would run. It is handed the queue as
|
|
its sink and this job's cascade, so its batches land in the run this
|
|
one belongs to.
|
|
|
|
Returns whether anything took it. `False` under `NULL`, where there is
|
|
no queue to take it: a callback invoked from a notebook or a Claude
|
|
Code session has to do the work itself, and this is how it finds out
|
|
that nobody else will.
|
|
"""
|
|
return bool(self.on_task(task))
|
|
|
|
def stop(self, why: str = "") -> int:
|
|
"""Cancel this task's remaining queued batches. Returns how many.
|
|
|
|
For the failure that is not this batch's: a spend limit, a signed-out
|
|
CLI, no model on the machine at all. The backend has answered, the
|
|
answer is the same one every sibling is about to be given, and the
|
|
only thing left to decide is whether to spend the other twenty-two
|
|
calls finding that out. Twenty-three identical notices saying "looks
|
|
like a bug" is what not deciding looked like.
|
|
|
|
Only what is still *queued* stops. A batch already in flight is a
|
|
request in the air and a bill either way, so it runs to its own end —
|
|
and this one, the one that asked, keeps its own failure. The task's
|
|
final job is spared for the reason `Jobs.cancel_run` spares it:
|
|
reporting is not work, and a run that stopped early still has to say
|
|
so. `why` is written onto each cancelled row, so a batch that never
|
|
ran says on screen why it didn't.
|
|
|
|
Not a kill. Nobody pressed anything, so the task's row is not flagged
|
|
and `killed` stays false: the cascade goes on to the next stage, as it
|
|
does after any batch that did not land. Whether the *next* task should
|
|
stop too is that cascade's question, not this one's.
|
|
"""
|
|
return int(self.on_stop(why))
|
|
|
|
def notice(self, notice: Any) -> None:
|
|
"""Tell the user something went wrong. See `pipeline/notices.py`.
|
|
|
|
The counterpart to `log.exception`, not a replacement for it: log the
|
|
traceback for afterwards, raise a notice for now. Use it for what a
|
|
person has to act on — a login to renew, a feed that has stopped
|
|
resolving — and not for what the program can simply get on with.
|
|
|
|
Under a queue it is written to the notice log and pushed to whoever is
|
|
watching; called directly it is logged and nothing else. Safe to call
|
|
repeatedly: the same `code` and `key` again is one row with a count,
|
|
and only the first interrupts anyone. So give `key` whatever the
|
|
failure is about — the source, the article — or two different
|
|
failures of one kind are told as one.
|
|
"""
|
|
self.on_notice(notice)
|
|
|
|
|
|
#: For calling a handler outside the queue. Reports nowhere, never cancels.
|
|
NULL = RunContext()
|