"""How a function becomes a job. The whole contract is one line: def handler(config, store, *, context=NULL, limit=None) -> dict `@job` records that function against a name and **returns it unchanged**. That is the point of this module rather than an accident of it: a handler stays an ordinary function you can import and call. run_pull(config, store) # sync, no queue, no event loop run_pull(config, store, limit=5) # payload keys are just keyword args Declared keyword arguments, not `**payload`: a key the handler does not name is reported against the job, and a handler that swallowed everything would turn a misspelt payload key into work that quietly did the wrong thing. Which means a job is testable, reusable from a command, and reusable from another job, without any of them going near the queue. The queue's entire involvement is calling the same function with a real `context` and the stored payload spread over it. What that `context` *is* lives next door in `context.py`. Neither module imports the handlers: they live in the app (`newsdesk/jobs/` for newsdesk), import `@job` from here, and fill `JOBS` when they are imported. """ from __future__ import annotations import inspect from collections.abc import Callable from dataclasses import dataclass from typing import Any #: A handler takes (config, store), keyword-only `context`, and whatever payload it #: declares. It returns a JSON-serialisable dict, which is stored as the job's #: result and shown when the job is done. Handler = Callable[..., dict[str, Any]] @dataclass(frozen=True, slots=True) class Spec: kind: str label: str fn: Handler #: Whether `fn` actually accepts `context`. Worked out once, here, so the #: worker doesn't inspect a signature per run — and so that "any function #: with the right shape is queueable" holds for a function that wants #: neither progress nor cancellation and says so by not taking the #: argument. Passing `context` to one of those would be a TypeError, which #: would surface as the job mysteriously failing. wants_context: bool = True #: When true, only one job of this kind may be queued or running at a time #: — enforced in the database, so it holds across every client at once. #: For the sweeps: one `pull`, one `rank`, however many tabs are open. #: #: Off by default, because the two ways of being wrong are not equal. A #: sweep that forgets to say `solo=True` runs twice and wastes a minute; #: a batch kind that forgets to say `solo=False` has seven of its eight #: batches silently handed back the first one, and the work simply never #: happens. The default belongs on the side that fails loudly. solo: bool = False #: What running one spends, and so which kind of worker runs it: #: `llm` for anything that asks a model, `http` for a fetch, `cpu` for #: work that wants the machine to itself. Declared here rather than worked #: out, because the handler is arbitrary code and only its author knows. resource: str = "cpu" #: How many attempts this kind of work is worth, counting the first, and #: how long to wait between them. Copied onto every job queued of this kind #: — see `pipeline/table.py`, where the row rather than the kind is #: what a later attempt reads. #: #: One by default, which is no retry and is what every job did before this #: existed. Retrying is off by default for the same reason `solo` is: the #: two ways of being wrong are not equal. A fetch that should have been #: retried and was not costs one item until the next sweep finds it again; #: work retried three times that never should have been is three times the #: writes, or three times the bill. max_attempts: int = 1 retry_delay: float = 0.0 retry_backoff: float = 1.0 #: Run this kind in a child process of its own rather than in a worker #: thread. What it buys is the one thing a thread cannot give: the job can #: be *ended*. A handler inside `sklearn.fit` or a tight Python loop #: cannot be stopped, because no signal and no exception reaches a thread #: in the middle of a C call — only a separate process can be killed. #: #: Nothing has to be pickled to make it work, which is why it is cheap #: here and expensive elsewhere: a job row is already a complete #: description of the work — a kind, a JSON payload, a database. The child #: is handed those three and rebuilds the rest. See `pipeline/child.py`. #: #: The cost is that the queue needs a `bootstrap` able to build the app's #: `(config, store)` from nothing, since neither survives a process #: boundary. One function per app. #: #: Not for work that is expensive because of a *command* it runs — #: ffmpeg, stockfish. That wants `RunContext.spawn`, which kills the same #: way for none of the setup. isolate: bool = False #: Seconds this kind may run before it is ended, or 0 for no deadline. #: Requires `isolate`, and `job` refuses the pair without it: a deadline #: over a thread could only mark the row failed and walk away, leaving the #: work still running and still holding the core it was killed for #: holding — and leaving two things intending to write the same row. A #: promise the pipeline cannot keep is worse than no promise. timeout: float = 0.0 #: Payload keys the handler cannot run without: its keyword arguments #: that have no default. A kind with any is not a button, because a press #: sends an empty payload and the job would fail on the TypeError. needs: tuple[str, ...] = () #: name -> spec. Explicit registration, like `routers/ROUTERS`: a new job is #: a change to this project, and the cost of importing its module once in #: the app's handlers module is one line. Empty until something imports #: `newsdesk.jobs`, which every entry point does before it looks here. JOBS: dict[str, Spec] = {} def job(kind: str, label: str = "", *, solo: bool = False, resource: str = "cpu", max_attempts: int = 1, retry_delay: float = 0.0, retry_backoff: float = 1.0, isolate: bool = False, timeout: float = 0.0) -> Callable[[Handler], Handler]: """Register a plain function as queueable work. Returns the function untouched, so importing it and calling it directly goes on working exactly as before it was decorated. `solo=True` for a job there should only ever be one of at a time — see `Spec.solo`, which is where the default and its reasoning live. `max_attempts=3, retry_delay=60` for work that is worth another go when the other end is having a bad minute. The retry is the queue's, not the handler's: a failed attempt puts the row back in the queue with a time on it, so nothing sleeps and nothing holds a worker while it waits. A retrying job must be safe to run twice, which every handler here has to be anyway — see the pipeline's usage card. `isolate=True, timeout=600` for work that has to be endable: it runs in a child process, and ten minutes in it is killed rather than asked. See `Spec.isolate` for what that costs and `RunContext.spawn` for the cheaper answer when the expensive part is a command rather than Python. """ if timeout and not isolate: raise ValueError( f"{kind!r} asks for a {timeout:g}s timeout without isolate=True. " "A job in a thread cannot be ended — the deadline could only mark " "the row failed while the work carried on. Either isolate it, or " "give the handler a RunContext.spawn(timeout=...) to hold." ) if timeout < 0: raise ValueError(f"{kind!r} asks for a negative timeout: {timeout!r}") def register(fn: Handler) -> Handler: if kind in JOBS and JOBS[kind].fn is not fn: raise ValueError( f"two different handlers are both registered as {kind!r}: " f"{JOBS[kind].fn.__qualname__} and {fn.__qualname__}" ) JOBS[kind] = Spec( kind=kind, label=label or kind, fn=fn, solo=solo, wants_context=_accepts_context(fn), needs=_required(fn), resource=resource, max_attempts=max_attempts, retry_delay=retry_delay, retry_backoff=retry_backoff, isolate=isolate, timeout=timeout, ) return fn return register def _required(fn: Handler) -> tuple[str, ...]: """`fn`'s keyword-only parameters without a default, in declared order.""" try: params = inspect.signature(fn).parameters except (TypeError, ValueError): return () return tuple(name for name, p in params.items() if p.kind is inspect.Parameter.KEYWORD_ONLY and p.default is inspect.Parameter.empty) def _accepts_context(fn: Handler) -> bool: """Whether `fn` has a `context` parameter, or a `**kwargs` that would swallow one.""" try: params = inspect.signature(fn).parameters except (TypeError, ValueError): return True # unintrospectable; assume the contract return "context" in params or any( p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values() )