Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f436104f3
commit
9d666d0937
76 files changed
+15235
No files matched your search
@@ -0,0 +1,446 @@
|
||||
"""What a running job is handed, and the two things it can raise.
|
||||
|
||||
def handler(config, store, *, context=NULL, limit=None) -> dict
|
||||
|
||||
This is the job-side half of the contract; `registry.py` is the other — how
|
||||
a function becomes queueable. A pipeline stage imports this to report
|
||||
progress from inside its own loops, and nothing else: no queue, no worker.
|
||||
Nothing here imports anything else from desk at module level, so it is safe to
|
||||
import from anywhere.
|
||||
|
||||
Everything in here is optional to the job that receives it. Ignore `context`
|
||||
and the job still runs — it just cannot be stopped part-way and shows no
|
||||
progress. Use it and you get both, for two lines.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: How often a spawned process is looked in on: whether it has finished,
|
||||
#: whether Kill has been pressed, whether its deadline has passed. Frequent
|
||||
#: enough that a Kill lands in well under a second, rare enough that waiting
|
||||
#: an hour for ffmpeg costs a few thousand cheap syscalls.
|
||||
POLL = 0.25
|
||||
|
||||
|
||||
class Cancelled(Exception):
|
||||
"""Raised by `RunContext.check` when the job has been asked to stop.
|
||||
|
||||
Caught by the worker and recorded as `cancelled` rather than `failed`. A
|
||||
handler should let it travel — catching it is how you build the thing the
|
||||
Kill button can't kill.
|
||||
"""
|
||||
|
||||
|
||||
class Retry(Exception):
|
||||
"""Raised by a handler that wants this same work attempted again later.
|
||||
|
||||
An ordinary failure is already retried, within whatever bound the job was
|
||||
queued with — so this is not how a job asks to be retried at all. It is how
|
||||
it says *when*: `after` is the wait the other end named, a `Retry-After`
|
||||
header or a rate limit's own answer, and the queue uses it in place of the
|
||||
backoff it would otherwise have guessed. Guessing is the thing worth
|
||||
avoiding here — a minute is either a minute of doing nothing when the
|
||||
service was ready, or another refusal charged against the bound.
|
||||
|
||||
It does not buy an attempt. A job queued with one attempt raises this and
|
||||
fails, exactly as any other exception would: the bound is ours to set and
|
||||
the wait is the world's to name, and a handler that could extend its own
|
||||
bound would be a policy nobody declared.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str = "", *, after: float | None = None) -> None:
|
||||
super().__init__(message)
|
||||
#: Seconds to wait, or None to leave it to the job's own backoff.
|
||||
self.after = after
|
||||
|
||||
|
||||
def _no_progress(done: int, total: int, message: str) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _no_task(task: Any) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _no_stop(why: str) -> int:
|
||||
"""Where "stop the rest of this run" goes when there is no run to stop.
|
||||
|
||||
The direct path is one batch after another in the caller's own thread, and
|
||||
the caller is standing there: it sees each failure as it happens and can
|
||||
stop by not asking again. Nothing to cancel, so nothing is cancelled.
|
||||
"""
|
||||
return 0
|
||||
|
||||
|
||||
def _no_fan_out(*_args: Any, **_kwargs: Any) -> tuple[int, ...]:
|
||||
"""Where a fan-out goes when there is no queue to put it in.
|
||||
|
||||
Empty rather than an error, and the same answer `task` gives for the same
|
||||
reason: a handler called directly — from a terminal, a notebook, a test —
|
||||
is being run by somebody who is standing there, and the useful behaviour
|
||||
is to do its own work and report that nothing was queued behind it. A
|
||||
handler that must know either way checks the return.
|
||||
"""
|
||||
return ()
|
||||
|
||||
|
||||
def _log_only(notice: Any) -> None:
|
||||
"""Where a notice goes when nothing is watching for one.
|
||||
|
||||
A direct caller — `newsdesk job pull` in a terminal, a notebook, a test — has
|
||||
no socket, so the log is the destination. It is not a *lesser* one: it goes
|
||||
through the same `notices.render`, so the terminal gets the sentence, the
|
||||
command to run and the is-this-a-bug verdict, exactly as the browser does.
|
||||
"""
|
||||
from pipeline.notices import render
|
||||
|
||||
log.warning("%s\n%s", notice.code, render(notice))
|
||||
|
||||
|
||||
def _never() -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def kill_tree(proc: subprocess.Popen[Any]) -> None:
|
||||
"""Kill a spawned process *and* whatever it started.
|
||||
|
||||
The children are the point. `proc.kill()` reaches the process we launched
|
||||
and nothing it launched itself, so killing a shell leaves the ffmpeg it
|
||||
ran holding the core — which is the exact failure a deadline exists to
|
||||
prevent. Both branches need the process to have been started detached,
|
||||
which `spawn` does.
|
||||
|
||||
Best-effort by construction: between deciding to kill and killing, the
|
||||
process may have ended on its own and its id may have been reused. So
|
||||
every call is guarded, and a failure to kill something that is already
|
||||
gone is not news.
|
||||
"""
|
||||
if proc.poll() is not None:
|
||||
return
|
||||
try:
|
||||
if os.name == "nt":
|
||||
# No process groups to signal, so ask Windows to walk the tree.
|
||||
# `/T` is children, `/F` is without asking.
|
||||
subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)],
|
||||
capture_output=True, check=False, timeout=10)
|
||||
else:
|
||||
# Guarded by `os.name` above rather than by the type checker,
|
||||
# which is reading this file on Windows and knows these three do
|
||||
# not exist there.
|
||||
os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # type: ignore[attr-defined,unused-ignore]
|
||||
except Exception:
|
||||
log.debug("could not kill %s", proc.pid, exc_info=True)
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except Exception:
|
||||
log.warning("process %s did not go away after being killed", proc.pid)
|
||||
|
||||
|
||||
def spawn(argv: Sequence[str], *, check: Callable[[], None] | None = None,
|
||||
timeout: float | None = None, poll: float = POLL,
|
||||
**popen: Any) -> subprocess.CompletedProcess[bytes]:
|
||||
"""Run a command, watching for Kill and for a deadline while it does.
|
||||
|
||||
The building block under `RunContext.spawn` and under `isolate=True`, and
|
||||
the answer to the one thing a thread cannot do. A handler that blocks in
|
||||
`subprocess.run` cannot be stopped, because stopping it would mean tearing
|
||||
a thread out of a syscall; a handler that blocks *here* can, because what
|
||||
is expensive is the child and a child can be killed. The thread is only
|
||||
sleeping, which costs nothing.
|
||||
|
||||
Output goes to temporary files rather than pipes. `PIPE` plus a poll loop
|
||||
is a deadlock: a child that fills the pipe buffer blocks on the write
|
||||
while we block on the poll, and neither side is wrong. A file cannot fill
|
||||
up in the way that matters, so the loop is free to look away.
|
||||
|
||||
Started detached — its own process group, or Windows' equivalent — purely
|
||||
so `kill_tree` has something to aim at.
|
||||
"""
|
||||
detach: dict[str, Any] = ({"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
|
||||
if os.name == "nt" else {"start_new_session": True})
|
||||
with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err:
|
||||
popen.setdefault("stdout", out)
|
||||
popen.setdefault("stderr", err)
|
||||
proc = subprocess.Popen(list(argv), **detach, **popen)
|
||||
deadline = time.monotonic() + timeout if timeout else None
|
||||
try:
|
||||
while proc.poll() is None:
|
||||
if check is not None:
|
||||
check() # raises Cancelled on Kill
|
||||
if deadline is not None and time.monotonic() > deadline:
|
||||
kill_tree(proc)
|
||||
raise TimeoutError(
|
||||
f"{argv[0]} was still running after {timeout:g}s")
|
||||
time.sleep(poll)
|
||||
except BaseException:
|
||||
kill_tree(proc)
|
||||
raise
|
||||
out.seek(0)
|
||||
err.seek(0)
|
||||
return subprocess.CompletedProcess(
|
||||
list(argv), proc.returncode,
|
||||
out.read() if popen["stdout"] is out else b"",
|
||||
err.read() if popen["stderr"] is err else b"")
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class RunContext:
|
||||
"""The handle a running job has on the run it is part of.
|
||||
|
||||
Deliberately a handful of plain callables rather than a reference to the
|
||||
queue: a handler cannot reach back into the worker, reschedule itself or
|
||||
read other jobs, and a test can hand it a lambda without standing anything
|
||||
up.
|
||||
"""
|
||||
|
||||
on_progress: Callable[[int, int, str], None] = _no_progress
|
||||
is_cancelled: Callable[[], bool] = _never
|
||||
#: Whether the *task* this job belongs to was killed — a different
|
||||
#: question from `is_cancelled`, which is about this job alone. See
|
||||
#: `killed`.
|
||||
is_killed: Callable[[], bool] = _never
|
||||
on_notice: Callable[[Any], None] = _log_only
|
||||
on_task: Callable[[Any], bool] = _no_task
|
||||
on_stop: Callable[[str], int] = _no_stop
|
||||
#: Where `fan_out` puts the jobs it asks for. Supplied by the queue, which
|
||||
#: is the only thing that can write rows; `_no_fan_out` under `NULL`.
|
||||
on_fan_out: Callable[..., tuple[int, ...]] = _no_fan_out
|
||||
#: How the jobs this one waited for ended, oldest first. Empty for a job
|
||||
#: that waited for nothing. Typed loosely for the same reason `notice` is:
|
||||
#: nothing in this module may import the rest of desk, and the shape it
|
||||
#: carries is `pipeline.models.Job`.
|
||||
waited_for: tuple[Any, ...] = ()
|
||||
|
||||
def upstream(self) -> dict[str, Any]:
|
||||
"""What the jobs this one waited for returned, merged into one dict.
|
||||
|
||||
The read half of a chain. `A -> B -> C` needs A's output to reach B,
|
||||
and this is how it arrives: B does not have its payload filled in by
|
||||
A, it reads A's result off A's row. Which is the whole difference —
|
||||
**a row is complete when it is written.** A payload with a hole in it
|
||||
meant for an earlier job to fill would be claimable the instant that
|
||||
job landed, so a worker could take it between the write and the
|
||||
commit; it could not carry a `dedup_key`, which has to be known at
|
||||
queue time; and it could not be re-run from its own row, which is
|
||||
what a retry does and what `max_attempts` is.
|
||||
|
||||
Merged, later parents winning, because the overwhelmingly common case
|
||||
is one parent and `upstream()["text"]` is what that should read like.
|
||||
A job with several parents that care about the same key wants
|
||||
`waited_for` and to decide for itself.
|
||||
|
||||
Empty under `NULL`, which is what makes the fallback work:
|
||||
|
||||
def handler(config, store, *, context=NULL, html=""):
|
||||
html = html or context.upstream().get("html", "")
|
||||
|
||||
From the queue nobody passes `html` and it comes from the parent; from
|
||||
a terminal you pass it yourself. Without that line a chained handler
|
||||
is the one kind in the pipeline you cannot simply call.
|
||||
"""
|
||||
merged: dict[str, Any] = {}
|
||||
for parent in self.waited_for:
|
||||
result = getattr(parent, "result", None)
|
||||
if isinstance(result, dict):
|
||||
merged |= result
|
||||
return merged
|
||||
|
||||
def fan_out(self, kind: str, payloads: Sequence[dict[str, Any]], *,
|
||||
then: str = "", settle: str = "", label: str = "",
|
||||
carries: dict[str, Any] | None = None,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> tuple[int, ...]:
|
||||
"""Queue one job per payload, and one job behind them that reports.
|
||||
|
||||
Returns the ids of the children. The reporting job is not among them:
|
||||
it is the pipeline's own kind, it is nothing an app writes, and it is
|
||||
the reason this method exists rather than a handler assembling a
|
||||
`depends_on` list by hand every time.
|
||||
|
||||
def handler(config, store, *, context=NULL):
|
||||
chunks = chunk(needs_a_body(store))
|
||||
context.fan_out("body_batch", [{"uids": c} for c in chunks],
|
||||
then="newsdesk.jobs.cascade:step")
|
||||
|
||||
`then` is `module:function`, resolved where it is run, and is handed
|
||||
`(config, store, tally, carries)` once every child is over however it
|
||||
ended — see `Tally` in `task.py`. A name rather than a function
|
||||
because what carries it is a table row, and no row can hold a
|
||||
function.
|
||||
|
||||
One transaction, so a fan-out is never half-queued. The children go
|
||||
into a run of their own within this job's cascade, which is what makes
|
||||
the callback able to fan out again: that is a cascade, and there is no
|
||||
cascade object anywhere — step N's callback queues step N+1.
|
||||
|
||||
Nothing is queued under `NULL`, and `()` says so.
|
||||
"""
|
||||
return self.on_fan_out(
|
||||
kind, list(payloads), then=then, settle=settle, label=label,
|
||||
carries=dict(carries or {}), max_attempts=max_attempts,
|
||||
retry_delay=retry_delay, retry_backoff=retry_backoff)
|
||||
|
||||
def spawn(self, argv: Sequence[str], *, timeout: float | None = None,
|
||||
**popen: Any) -> subprocess.CompletedProcess[bytes]:
|
||||
"""Run an external command, killable, with this job's Kill wired in.
|
||||
|
||||
What to reach for when the expensive thing is a binary — ffmpeg,
|
||||
stockfish, whisper.cpp. Because the work is in the child and not in
|
||||
this thread, Kill is no longer cooperative in the way `check` is: the
|
||||
child is killed outright, and its children with it. Blocking a worker
|
||||
thread on a process costs nothing, since the thread is only sleeping.
|
||||
|
||||
`timeout` is a real deadline for the same reason, and the only kind of
|
||||
deadline the pipeline offers that can actually end the work. A job
|
||||
kind that wants one over Python code rather than over a command wants
|
||||
`isolate=True` — see `registry.py`.
|
||||
|
||||
Raises `Cancelled` on Kill and `TimeoutError` on the deadline, having
|
||||
killed the process tree either way. Non-zero exit is *not* an error
|
||||
here: the returncode is handed back, because plenty of commands say
|
||||
something useful with it and only the handler knows which.
|
||||
"""
|
||||
return spawn(argv, check=self.check, timeout=timeout, **popen)
|
||||
|
||||
def progress(self, done: int = 0, total: int = 0, message: str = "") -> None:
|
||||
"""Say how far along this is. Units are the handler's own business —
|
||||
sources, files, minutes of audio — the UI renders `done/total`.
|
||||
|
||||
The message has one shape, and every job in this codebase uses it:
|
||||
|
||||
a lowercase participle, what it is being done to, and — when the
|
||||
job is walking a list — `, n of m`.
|
||||
|
||||
"reading Golem, 3 of 12". "fetching the body from theverge.com, 40 of
|
||||
212". "training on your votes" for a job that is one step and has no
|
||||
list to be three of. Not an em dash, not a middot, and never a bare
|
||||
identifier: this used to say `wordpress`, which named the module that
|
||||
reads the feed rather than the outlet, and read as a fragment of a log
|
||||
line that had escaped onto the status strip.
|
||||
|
||||
`n` is the item being worked on now and so runs one ahead of `done`,
|
||||
which counts the ones that are finished. They are answering different
|
||||
questions — the fraction in the message belongs to the name beside it,
|
||||
and the bar belongs to the job — and a line saying "reading Golem, 0 of
|
||||
12" while Golem is being read would be wrong about the only thing it
|
||||
was there to say. A phase that has nothing left to name ends on the
|
||||
collective — "reading every source, 12 of 12" — so the same participle
|
||||
carries the line from first item to last, and a full fraction is what
|
||||
says it is over. Past tense is not used: it dates a line that is about
|
||||
to be replaced by the next phase's.
|
||||
|
||||
A job's *label* is the same grammar capitalised, because a label names
|
||||
the work rather than narrating it: "Tagging 25 items, 2 of 9".
|
||||
"""
|
||||
self.on_progress(done, total, message)
|
||||
|
||||
def check(self) -> None:
|
||||
"""Stop now if Kill has been pressed.
|
||||
|
||||
Cancellation is cooperative because the work is synchronous: there is
|
||||
no safe way to tear a thread out of the middle of an HTTP fetch or a
|
||||
model load, so the job says where stopping is fine. Call it between
|
||||
units of work, not inside tight inner loops.
|
||||
"""
|
||||
if self.is_cancelled():
|
||||
raise Cancelled()
|
||||
|
||||
def killed(self) -> bool:
|
||||
"""Whether the task this job belongs to was killed.
|
||||
|
||||
Not the question `check` asks. That one is about this job, and killing
|
||||
one job is dropping one batch. This one is about the run, and killing
|
||||
a run is skipping the whole step — which the job that reports on the
|
||||
step has to be able to say, since it is spared the kill precisely so
|
||||
that somebody hears about it. The counts cannot tell the two apart: a
|
||||
batch someone killed by hand and a batch killed by the task going
|
||||
down both arrive as `cancelled`.
|
||||
|
||||
Asked when asked, not read at the start, so a kill landing while the
|
||||
reporting job runs is still in what it reports.
|
||||
"""
|
||||
return bool(self.is_killed())
|
||||
|
||||
def step(self, done: int = 0, total: int = 0, message: str = "") -> None:
|
||||
"""`progress` then `check` — what the top of a loop body always wants."""
|
||||
self.progress(done, total, message)
|
||||
self.check()
|
||||
|
||||
def task(self, task: Any) -> bool:
|
||||
"""Queue a `Task` once this job has finished well.
|
||||
|
||||
Recorded rather than queued: the worker starts the task after this job
|
||||
lands in `done`, so a task never outlives the failure of the job that
|
||||
described it, and a handler still cannot reach the queue. It names what
|
||||
it wants to happen and that is all.
|
||||
|
||||
The task runs in the worker, synchronously, the moment the job is over
|
||||
— the same plain code a request would run. It is handed the queue as
|
||||
its sink and this job's cascade, so its batches land in the run this
|
||||
one belongs to.
|
||||
|
||||
Returns whether anything took it. `False` under `NULL`, where there is
|
||||
no queue to take it: a callback invoked from a notebook or a Claude
|
||||
Code session has to do the work itself, and this is how it finds out
|
||||
that nobody else will.
|
||||
"""
|
||||
return bool(self.on_task(task))
|
||||
|
||||
def stop(self, why: str = "") -> int:
|
||||
"""Cancel this task's remaining queued batches. Returns how many.
|
||||
|
||||
For the failure that is not this batch's: a spend limit, a signed-out
|
||||
CLI, no model on the machine at all. The backend has answered, the
|
||||
answer is the same one every sibling is about to be given, and the
|
||||
only thing left to decide is whether to spend the other twenty-two
|
||||
calls finding that out. Twenty-three identical notices saying "looks
|
||||
like a bug" is what not deciding looked like.
|
||||
|
||||
Only what is still *queued* stops. A batch already in flight is a
|
||||
request in the air and a bill either way, so it runs to its own end —
|
||||
and this one, the one that asked, keeps its own failure. The task's
|
||||
final job is spared for the reason `Jobs.cancel_run` spares it:
|
||||
reporting is not work, and a run that stopped early still has to say
|
||||
so. `why` is written onto each cancelled row, so a batch that never
|
||||
ran says on screen why it didn't.
|
||||
|
||||
Not a kill. Nobody pressed anything, so the task's row is not flagged
|
||||
and `killed` stays false: the cascade goes on to the next stage, as it
|
||||
does after any batch that did not land. Whether the *next* task should
|
||||
stop too is that cascade's question, not this one's.
|
||||
"""
|
||||
return int(self.on_stop(why))
|
||||
|
||||
def notice(self, notice: Any) -> None:
|
||||
"""Tell the user something went wrong. See `pipeline/notices.py`.
|
||||
|
||||
The counterpart to `log.exception`, not a replacement for it: log the
|
||||
traceback for afterwards, raise a notice for now. Use it for what a
|
||||
person has to act on — a login to renew, a feed that has stopped
|
||||
resolving — and not for what the program can simply get on with.
|
||||
|
||||
Under a queue it is written to the notice log and pushed to whoever is
|
||||
watching; called directly it is logged and nothing else. Safe to call
|
||||
repeatedly: the same `code` and `key` again is one row with a count,
|
||||
and only the first interrupts anyone. So give `key` whatever the
|
||||
failure is about — the source, the article — or two different
|
||||
failures of one kind are told as one.
|
||||
"""
|
||||
self.on_notice(notice)
|
||||
|
||||
|
||||
#: For calling a handler outside the queue. Reports nowhere, never cancels.
|
||||
NULL = RunContext()
|
||||
Reference in new issue
Block a user