Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f436104f3
commit
9d666d0937
76 files changed
+15235
No files matched your search
@@ -0,0 +1,195 @@
|
||||
"""How a function becomes a job.
|
||||
|
||||
The whole contract is one line:
|
||||
|
||||
def handler(config, store, *, context=NULL, limit=None) -> dict
|
||||
|
||||
`@job` records that function against a name and **returns it unchanged**. That
|
||||
is the point of this module rather than an accident of it: a handler stays an
|
||||
ordinary function you can import and call.
|
||||
|
||||
run_pull(config, store) # sync, no queue, no event loop
|
||||
run_pull(config, store, limit=5) # payload keys are just keyword args
|
||||
|
||||
Declared keyword arguments, not `**payload`: a key the handler does not name
|
||||
is reported against the job, and a handler that swallowed everything would
|
||||
turn a misspelt payload key into work that quietly did the wrong thing.
|
||||
|
||||
Which means a job is testable, reusable from a command, and reusable from
|
||||
another job, without any of them going near the queue. The queue's entire
|
||||
involvement is calling the same function with a real `context` and the stored
|
||||
payload spread over it.
|
||||
|
||||
What that `context` *is* lives next door in `context.py`. Neither module
|
||||
imports the handlers: they live in the app (`newsdesk/jobs/` for newsdesk),
|
||||
import `@job` from here, and fill `JOBS` when they are imported.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
#: A handler takes (config, store), keyword-only `context`, and whatever payload it
|
||||
#: declares. It returns a JSON-serialisable dict, which is stored as the job's
|
||||
#: result and shown when the job is done.
|
||||
Handler = Callable[..., dict[str, Any]]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Spec:
|
||||
kind: str
|
||||
label: str
|
||||
fn: Handler
|
||||
#: Whether `fn` actually accepts `context`. Worked out once, here, so the
|
||||
#: worker doesn't inspect a signature per run — and so that "any function
|
||||
#: with the right shape is queueable" holds for a function that wants
|
||||
#: neither progress nor cancellation and says so by not taking the
|
||||
#: argument. Passing `context` to one of those would be a TypeError, which
|
||||
#: would surface as the job mysteriously failing.
|
||||
wants_context: bool = True
|
||||
#: When true, only one job of this kind may be queued or running at a time
|
||||
#: — enforced in the database, so it holds across every client at once.
|
||||
#: For the sweeps: one `pull`, one `rank`, however many tabs are open.
|
||||
#:
|
||||
#: Off by default, because the two ways of being wrong are not equal. A
|
||||
#: sweep that forgets to say `solo=True` runs twice and wastes a minute;
|
||||
#: a batch kind that forgets to say `solo=False` has seven of its eight
|
||||
#: batches silently handed back the first one, and the work simply never
|
||||
#: happens. The default belongs on the side that fails loudly.
|
||||
solo: bool = False
|
||||
#: What running one spends, and so which kind of worker runs it:
|
||||
#: `llm` for anything that asks a model, `http` for a fetch, `cpu` for
|
||||
#: work that wants the machine to itself. Declared here rather than worked
|
||||
#: out, because the handler is arbitrary code and only its author knows.
|
||||
resource: str = "cpu"
|
||||
#: How many attempts this kind of work is worth, counting the first, and
|
||||
#: how long to wait between them. Copied onto every job queued of this kind
|
||||
#: — see `pipeline/table.py`, where the row rather than the kind is
|
||||
#: what a later attempt reads.
|
||||
#:
|
||||
#: One by default, which is no retry and is what every job did before this
|
||||
#: existed. Retrying is off by default for the same reason `solo` is: the
|
||||
#: two ways of being wrong are not equal. A fetch that should have been
|
||||
#: retried and was not costs one item until the next sweep finds it again;
|
||||
#: work retried three times that never should have been is three times the
|
||||
#: writes, or three times the bill.
|
||||
max_attempts: int = 1
|
||||
retry_delay: float = 0.0
|
||||
retry_backoff: float = 1.0
|
||||
#: Run this kind in a child process of its own rather than in a worker
|
||||
#: thread. What it buys is the one thing a thread cannot give: the job can
|
||||
#: be *ended*. A handler inside `sklearn.fit` or a tight Python loop
|
||||
#: cannot be stopped, because no signal and no exception reaches a thread
|
||||
#: in the middle of a C call — only a separate process can be killed.
|
||||
#:
|
||||
#: Nothing has to be pickled to make it work, which is why it is cheap
|
||||
#: here and expensive elsewhere: a job row is already a complete
|
||||
#: description of the work — a kind, a JSON payload, a database. The child
|
||||
#: is handed those three and rebuilds the rest. See `pipeline/child.py`.
|
||||
#:
|
||||
#: The cost is that the queue needs a `bootstrap` able to build the app's
|
||||
#: `(config, store)` from nothing, since neither survives a process
|
||||
#: boundary. One function per app.
|
||||
#:
|
||||
#: Not for work that is expensive because of a *command* it runs —
|
||||
#: ffmpeg, stockfish. That wants `RunContext.spawn`, which kills the same
|
||||
#: way for none of the setup.
|
||||
isolate: bool = False
|
||||
#: Seconds this kind may run before it is ended, or 0 for no deadline.
|
||||
#: Requires `isolate`, and `job` refuses the pair without it: a deadline
|
||||
#: over a thread could only mark the row failed and walk away, leaving the
|
||||
#: work still running and still holding the core it was killed for
|
||||
#: holding — and leaving two things intending to write the same row. A
|
||||
#: promise the pipeline cannot keep is worse than no promise.
|
||||
timeout: float = 0.0
|
||||
#: Payload keys the handler cannot run without: its keyword arguments
|
||||
#: that have no default. A kind with any is not a button, because a press
|
||||
#: sends an empty payload and the job would fail on the TypeError.
|
||||
needs: tuple[str, ...] = ()
|
||||
|
||||
|
||||
#: name -> spec. Explicit registration, like `routers/ROUTERS`: a new job is
|
||||
#: a change to this project, and the cost of importing its module once in
|
||||
#: the app's handlers module is one line. Empty until something imports
|
||||
#: `newsdesk.jobs`, which every entry point does before it looks here.
|
||||
JOBS: dict[str, Spec] = {}
|
||||
|
||||
|
||||
def job(kind: str, label: str = "", *, solo: bool = False,
|
||||
resource: str = "cpu", max_attempts: int = 1,
|
||||
retry_delay: float = 0.0,
|
||||
retry_backoff: float = 1.0,
|
||||
isolate: bool = False,
|
||||
timeout: float = 0.0) -> Callable[[Handler], Handler]:
|
||||
"""Register a plain function as queueable work.
|
||||
|
||||
Returns the function untouched, so importing it and calling it directly
|
||||
goes on working exactly as before it was decorated.
|
||||
|
||||
`solo=True` for a job there should only ever be one of at a time — see
|
||||
`Spec.solo`, which is where the default and its reasoning live.
|
||||
|
||||
`max_attempts=3, retry_delay=60` for work that is worth another go when
|
||||
the other end is having a bad minute. The retry is the queue's, not the
|
||||
handler's: a failed attempt puts the row back in the queue with a time on
|
||||
it, so nothing sleeps and nothing holds a worker while it waits. A
|
||||
retrying job must be safe to run twice, which every handler here has to be
|
||||
anyway — see the pipeline's usage card.
|
||||
|
||||
`isolate=True, timeout=600` for work that has to be endable: it runs in a
|
||||
child process, and ten minutes in it is killed rather than asked. See
|
||||
`Spec.isolate` for what that costs and `RunContext.spawn` for the cheaper
|
||||
answer when the expensive part is a command rather than Python.
|
||||
"""
|
||||
if timeout and not isolate:
|
||||
raise ValueError(
|
||||
f"{kind!r} asks for a {timeout:g}s timeout without isolate=True. "
|
||||
"A job in a thread cannot be ended — the deadline could only mark "
|
||||
"the row failed while the work carried on. Either isolate it, or "
|
||||
"give the handler a RunContext.spawn(timeout=...) to hold."
|
||||
)
|
||||
if timeout < 0:
|
||||
raise ValueError(f"{kind!r} asks for a negative timeout: {timeout!r}")
|
||||
|
||||
def register(fn: Handler) -> Handler:
|
||||
if kind in JOBS and JOBS[kind].fn is not fn:
|
||||
raise ValueError(
|
||||
f"two different handlers are both registered as {kind!r}: "
|
||||
f"{JOBS[kind].fn.__qualname__} and {fn.__qualname__}"
|
||||
)
|
||||
JOBS[kind] = Spec(
|
||||
kind=kind, label=label or kind, fn=fn, solo=solo,
|
||||
wants_context=_accepts_context(fn), needs=_required(fn),
|
||||
resource=resource, max_attempts=max_attempts,
|
||||
retry_delay=retry_delay, retry_backoff=retry_backoff,
|
||||
isolate=isolate, timeout=timeout,
|
||||
)
|
||||
return fn
|
||||
|
||||
return register
|
||||
|
||||
|
||||
def _required(fn: Handler) -> tuple[str, ...]:
|
||||
"""`fn`'s keyword-only parameters without a default, in declared order."""
|
||||
try:
|
||||
params = inspect.signature(fn).parameters
|
||||
except (TypeError, ValueError):
|
||||
return ()
|
||||
return tuple(name for name, p in params.items()
|
||||
if p.kind is inspect.Parameter.KEYWORD_ONLY
|
||||
and p.default is inspect.Parameter.empty)
|
||||
|
||||
|
||||
def _accepts_context(fn: Handler) -> bool:
|
||||
"""Whether `fn` has a `context` parameter, or a `**kwargs` that would swallow one."""
|
||||
try:
|
||||
params = inspect.signature(fn).parameters
|
||||
except (TypeError, ValueError):
|
||||
return True # unintrospectable; assume the contract
|
||||
return "context" in params or any(
|
||||
p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()
|
||||
)
|
||||
Reference in new issue
Block a user