Files
WolfandClaude Opus 5.5 9d666d0937 Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and
scheduler), auth (one password, sessions, passkeys), the deploy kit and its
docs, plus the coupling and packaging tests. Downstream copies, unedited.

Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:16 +02:00

196 lines
9.2 KiB
Python

"""How a function becomes a job.
The whole contract is one line:
def handler(config, store, *, context=NULL, limit=None) -> dict
`@job` records that function against a name and **returns it unchanged**. That
is the point of this module rather than an accident of it: a handler stays an
ordinary function you can import and call.
run_pull(config, store) # sync, no queue, no event loop
run_pull(config, store, limit=5) # payload keys are just keyword args
Declared keyword arguments, not `**payload`: a key the handler does not name
is reported against the job, and a handler that swallowed everything would
turn a misspelt payload key into work that quietly did the wrong thing.
Which means a job is testable, reusable from a command, and reusable from
another job, without any of them going near the queue. The queue's entire
involvement is calling the same function with a real `context` and the stored
payload spread over it.
What that `context` *is* lives next door in `context.py`. Neither module
imports the handlers: they live in the app (`newsdesk/jobs/` for newsdesk),
import `@job` from here, and fill `JOBS` when they are imported.
"""
from __future__ import annotations
import inspect
from collections.abc import Callable
from dataclasses import dataclass
from typing import Any
#: A handler takes (config, store), keyword-only `context`, and whatever payload it
#: declares. It returns a JSON-serialisable dict, which is stored as the job's
#: result and shown when the job is done.
Handler = Callable[..., dict[str, Any]]
@dataclass(frozen=True, slots=True)
class Spec:
kind: str
label: str
fn: Handler
#: Whether `fn` actually accepts `context`. Worked out once, here, so the
#: worker doesn't inspect a signature per run — and so that "any function
#: with the right shape is queueable" holds for a function that wants
#: neither progress nor cancellation and says so by not taking the
#: argument. Passing `context` to one of those would be a TypeError, which
#: would surface as the job mysteriously failing.
wants_context: bool = True
#: When true, only one job of this kind may be queued or running at a time
#: — enforced in the database, so it holds across every client at once.
#: For the sweeps: one `pull`, one `rank`, however many tabs are open.
#:
#: Off by default, because the two ways of being wrong are not equal. A
#: sweep that forgets to say `solo=True` runs twice and wastes a minute;
#: a batch kind that forgets to say `solo=False` has seven of its eight
#: batches silently handed back the first one, and the work simply never
#: happens. The default belongs on the side that fails loudly.
solo: bool = False
#: What running one spends, and so which kind of worker runs it:
#: `llm` for anything that asks a model, `http` for a fetch, `cpu` for
#: work that wants the machine to itself. Declared here rather than worked
#: out, because the handler is arbitrary code and only its author knows.
resource: str = "cpu"
#: How many attempts this kind of work is worth, counting the first, and
#: how long to wait between them. Copied onto every job queued of this kind
#: — see `pipeline/table.py`, where the row rather than the kind is
#: what a later attempt reads.
#:
#: One by default, which is no retry and is what every job did before this
#: existed. Retrying is off by default for the same reason `solo` is: the
#: two ways of being wrong are not equal. A fetch that should have been
#: retried and was not costs one item until the next sweep finds it again;
#: work retried three times that never should have been is three times the
#: writes, or three times the bill.
max_attempts: int = 1
retry_delay: float = 0.0
retry_backoff: float = 1.0
#: Run this kind in a child process of its own rather than in a worker
#: thread. What it buys is the one thing a thread cannot give: the job can
#: be *ended*. A handler inside `sklearn.fit` or a tight Python loop
#: cannot be stopped, because no signal and no exception reaches a thread
#: in the middle of a C call — only a separate process can be killed.
#:
#: Nothing has to be pickled to make it work, which is why it is cheap
#: here and expensive elsewhere: a job row is already a complete
#: description of the work — a kind, a JSON payload, a database. The child
#: is handed those three and rebuilds the rest. See `pipeline/child.py`.
#:
#: The cost is that the queue needs a `bootstrap` able to build the app's
#: `(config, store)` from nothing, since neither survives a process
#: boundary. One function per app.
#:
#: Not for work that is expensive because of a *command* it runs —
#: ffmpeg, stockfish. That wants `RunContext.spawn`, which kills the same
#: way for none of the setup.
isolate: bool = False
#: Seconds this kind may run before it is ended, or 0 for no deadline.
#: Requires `isolate`, and `job` refuses the pair without it: a deadline
#: over a thread could only mark the row failed and walk away, leaving the
#: work still running and still holding the core it was killed for
#: holding — and leaving two things intending to write the same row. A
#: promise the pipeline cannot keep is worse than no promise.
timeout: float = 0.0
#: Payload keys the handler cannot run without: its keyword arguments
#: that have no default. A kind with any is not a button, because a press
#: sends an empty payload and the job would fail on the TypeError.
needs: tuple[str, ...] = ()
#: name -> spec. Explicit registration, like `routers/ROUTERS`: a new job is
#: a change to this project, and the cost of importing its module once in
#: the app's handlers module is one line. Empty until something imports
#: `newsdesk.jobs`, which every entry point does before it looks here.
JOBS: dict[str, Spec] = {}
def job(kind: str, label: str = "", *, solo: bool = False,
resource: str = "cpu", max_attempts: int = 1,
retry_delay: float = 0.0,
retry_backoff: float = 1.0,
isolate: bool = False,
timeout: float = 0.0) -> Callable[[Handler], Handler]:
"""Register a plain function as queueable work.
Returns the function untouched, so importing it and calling it directly
goes on working exactly as before it was decorated.
`solo=True` for a job there should only ever be one of at a time — see
`Spec.solo`, which is where the default and its reasoning live.
`max_attempts=3, retry_delay=60` for work that is worth another go when
the other end is having a bad minute. The retry is the queue's, not the
handler's: a failed attempt puts the row back in the queue with a time on
it, so nothing sleeps and nothing holds a worker while it waits. A
retrying job must be safe to run twice, which every handler here has to be
anyway — see the pipeline's usage card.
`isolate=True, timeout=600` for work that has to be endable: it runs in a
child process, and ten minutes in it is killed rather than asked. See
`Spec.isolate` for what that costs and `RunContext.spawn` for the cheaper
answer when the expensive part is a command rather than Python.
"""
if timeout and not isolate:
raise ValueError(
f"{kind!r} asks for a {timeout:g}s timeout without isolate=True. "
"A job in a thread cannot be ended — the deadline could only mark "
"the row failed while the work carried on. Either isolate it, or "
"give the handler a RunContext.spawn(timeout=...) to hold."
)
if timeout < 0:
raise ValueError(f"{kind!r} asks for a negative timeout: {timeout!r}")
def register(fn: Handler) -> Handler:
if kind in JOBS and JOBS[kind].fn is not fn:
raise ValueError(
f"two different handlers are both registered as {kind!r}: "
f"{JOBS[kind].fn.__qualname__} and {fn.__qualname__}"
)
JOBS[kind] = Spec(
kind=kind, label=label or kind, fn=fn, solo=solo,
wants_context=_accepts_context(fn), needs=_required(fn),
resource=resource, max_attempts=max_attempts,
retry_delay=retry_delay, retry_backoff=retry_backoff,
isolate=isolate, timeout=timeout,
)
return fn
return register
def _required(fn: Handler) -> tuple[str, ...]:
"""`fn`'s keyword-only parameters without a default, in declared order."""
try:
params = inspect.signature(fn).parameters
except (TypeError, ValueError):
return ()
return tuple(name for name, p in params.items()
if p.kind is inspect.Parameter.KEYWORD_ONLY
and p.default is inspect.Parameter.empty)
def _accepts_context(fn: Handler) -> bool:
"""Whether `fn` has a `context` parameter, or a `**kwargs` that would swallow one."""
try:
params = inspect.signature(fn).parameters
except (TypeError, ValueError):
return True # unintrospectable; assume the contract
return "context" in params or any(
p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()
)