Bring in pipeline, auth, deploy and docs from the boilerplate

roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and
scheduler), auth (one password, sessions, passkeys), the deploy kit and its
docs, plus the coupling and packaging tests. Downstream copies, unedited.

Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-23 12:29:16 +02:00
1 parent 1f436104f3
commit 9d666d0937
76 files changed
+15235

No files matched your search

+172
View File
@@ -0,0 +1,172 @@
# pipeline
A job queue that lives in a SQLite table. Workers per resource, batching with a report job, dependency chains, dedupe, retries, cancellation that can kill, a WebSocket job list, and a standalone worker. No broker.
Needs Python 3.12+ and `pydantic`. `pipeline/web/` also needs `fastapi`; an app without a web face never imports it.
The source of truth is `boilerplate/pipeline/`. Copies elsewhere — news, health — are copies: take the whole folder, and run its tests where it lands.
```bash
cp -r boilerplate/pipeline yourproject/pipeline
python -m pytest yourproject/pipeline/tests
```
## A job
A job is a plain function. `@job` registers it and returns it unchanged, so it stays callable from a terminal or a test with no queue.
```python
from pipeline import NULL, JobQueue, Jobs, job
@job("greet", "Saying hello", resource="cpu")
def greet(config, store, *, context=NULL, name: str = "") -> dict:
context.progress(0, 1, f"greeting {name}")
context.check() # stops here if Kill was pressed
return {"said": f"hello {name}"}
greet(config, store, name="you") # direct call, no queue
queue = JobQueue(Jobs("app.db"), config, store)
queue.enqueue("greet", {"name": "you"})
queue.drain() # run everything queued, then stop
```
`config` and `store` are whatever your app uses. The queue passes them through untouched. The payload is the keyword arguments, stored as JSON on the row.
| `@job(...)` | |
|---|---|
| `resource="cpu"` | which worker pool runs it. Pool sizes: `JobQueue(workers={"cpu": 1, "http": 8, "llm": 4})` |
| `solo=True` | at most one queued or running at a time; a second enqueue returns the first |
| `max_attempts=3, retry_delay=60` | retried by the queue, not by a sleeping worker |
| `isolate=True, timeout=600` | runs in its own process, so Kill and the deadline actually end it |
## A web app
```python
from fastapi import FastAPI
from pipeline import Jobs
from pipeline.web import connect
app = FastAPI()
queue = connect(app, Jobs("app.db"), config, store)
```
That mounts `GET /api/jobs`, `POST /api/jobs`, `POST /api/jobs/{id}/cancel`, the `/ws/jobs` socket, and starts the worker with the app. For a job list with progress and Kill buttons, put this in a page:
```html
<script type="module" src="/_jobs/jobs.js"></script>
<desk-jobs></desk-jobs>
```
It themes from CSS custom properties (`--desk-jobs-bg`, `--desk-jobs-accent`, …) and raises a `desk-jobs:notice` DOM event for failures instead of wording them. The wording is the app's.
## Many jobs and one report
```python
@job("fetch_all")
def fetch_all(config, store, *, context=NULL):
urls = store.pending_urls()
context.fan_out("fetch", [{"urls": urls[i:i + 10]} for i in range(0, len(urls), 10)],
then="myapp.jobs:after_fetch")
def after_fetch(config, store, tally, carries, *, context):
# tally.done / .failed / .cancelled, and tally.results: what each child returned
context.fan_out("parse", ...) # the next step; this is a cascade
```
`then` is `module:function` because it is stored on a row. The report job runs whether the children succeeded or not. `Task(kind, finder=...)` is the same thing with the finding done for you; register it in `TASKS` to make it a button.
## Chains
```python
a = queue.enqueue("fetch", {"url": url})
b = queue.enqueue("extract", {}, depends_on=[a.id], after="ok") # "done" = however A ended
@job("extract")
def extract(config, store, *, context=NULL, html: str = ""):
html = html or context.upstream().get("html", "") # A's result
```
B reads A's result off A's row. Do not queue a row you mean to fill in later. The fallback keyword keeps `extract` callable by hand, since `upstream()` is `{}` outside the queue.
Results are reports: counts, ids, errors. A result over 64 KB fails the job with `ResultTooLarge` rather than being cut down. Put data in your store and return its ids, or raise `pipeline.table.MAX_RESULT` if a project really needs more.
## Ending work that does not look up
A thread cannot be interrupted, so `check()` is cooperative. Two cases can be killed:
- **A command** (ffmpeg, stockfish): `context.spawn([...], timeout=600)` kills the process tree on Kill or deadline.
- **Python that blocks** (`sklearn.fit`): `@job(..., isolate=True, timeout=600)`. The job runs as `python -m pipeline.child`, which rebuilds `(config, store)` from a bootstrap you provide:
```python
def bootstrap(config_path: str = ""): # myapp/jobs.py; importing it registers the jobs
config = load_config(config_path)
return config, Store(config.database)
JobQueue(jobs, config, store, bootstrap="myapp.jobs:bootstrap", bootstrap_arg=config_path)
```
`timeout` without `isolate` is refused at registration: a thread's deadline could only mark the row failed while the work carried on.
## Schedules
A schedule is a row that says "enqueue this kind with this payload on this cadence". Same SQLite file as the jobs, so there is no second process to keep alive and a restart between two runs loses nothing.
```python
from pipeline.schedule import declare
declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
declare("hourly count", "count", "every 1h", {"n": 5})
```
`declare` goes next to `@job` in the app's jobs module and fills `SCHEDULES` the way `@job` fills `JOBS`. On `JobQueue.start()` every declared name with no row gets one, and a row that already exists is left exactly as it was edited — cadence, payload and enabled alike. So a declaration is a starting point, not a setting: switching one off on screen survives the next deploy.
Five cadences and nothing else:
| | |
|---|---|
| `every 15m` `every 6h` `every 2d` | elapsed time. Unaffected by a daylight-saving change, because an interval is an interval |
| `daily at 06:00` | 06:00 on the wall clock of `JobQueue(tz=...)`, on both sides of a change |
| `weekly mon 07:00` | the same, on that weekday. Long names (`monday`) are accepted and stored short |
Anything else raises `ValueError` listing all five, which is the sentence the API hands a person who typed something else. There is no seconds unit: the loop looks every `SCHEDULE_TICK` (30 s), and a cadence it cannot keep is one to refuse rather than round.
The tick runs inside the drain loop — no second thread, no second clock — and once on `start()`, so an app that was down over 06:00 runs that job when it comes back. Once: a firing sets `next_run` from *now* rather than stepping through the slots it missed, so there is nowhere for a backlog to accumulate. A schedule whose last job is still queued or running skips its turn and advances instead of piling copies of it up, and so does one whose kind is `solo` while another is under way. A row whose kind is no longer registered never fires, keeps its place, and is listed as `unknown kind` for somebody to delete.
```python
queue = JobQueue(jobs, config, store, tz="Europe/Copenhagen") # or schedules=False
```
Over HTTP, from `pipeline.web.connect`: `GET/POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`, `POST /api/schedules/{id}/run` (fires now without moving the next run). Every refusal is `{"detail": "<sentence>"}` — an unregistered kind 404s naming what there is, a payload missing keys 422s naming them, a bad cadence 422s with all five wordings. Any change pushes a `schedules` message on `/ws/jobs`.
```html
<script type="module" src="/_jobs/schedules.js"></script>
<desk-schedules></desk-schedules>
```
`<desk-schedules>` is the table: name, kind, cadence edited in place, an on/off toggle, next and last run with the last job's state, Run now, Delete, and a row for adding one. Same `--desk-jobs-*` custom properties as `<desk-jobs>`, and just as silent about failures — it raises `desk-schedules:error` carrying the server's `detail` and leaves the wording to the app.
## A worker without the web app
```bash
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
python -m pipeline.worker ... --until-idle # drain and exit, for cron
python -m pipeline.worker ... --timezone Europe/Copenhagen
python -m pipeline.worker ... --no-schedules # every worker after the first
```
Any number of workers may share one file. A claim is one atomic statement and a claimed job is held on a lease; a worker that dies has its jobs re-queued by whichever is still alive.
This worker fires schedules too, which is what makes it a complete replacement for the web app's own rather than half of one. Exactly one process per file should: `--no-schedules` is what the second and third pass, because the pile-up guard reads `last_job_id` and neither of two simultaneous firings has written it when the other looks.
## Housekeeping
Finished jobs older than `JobQueue(retain=30 * 86400)` are purged hourly while idle. Unfinished jobs and the newest ending of each kind are kept regardless. `retain=0` keeps everything.
The table is WAL-mode SQLite. Back it up with `VACUUM INTO`, not a file copy, and keep it off network filesystems.
## More
- `pipeline.md` — the reference card: every entity, the timings, the limits.
- `__init__.py` — what each module is for.
- `tests/` — every claim above, including two worker processes racing on one file and a schedule either side of a daylight-saving change.
+127
View File
@@ -0,0 +1,127 @@
"""The pipeline: how work is queued, run, watched and stopped.
from pipeline import JobQueue, Jobs, job, RunContext, Task, NULL
**This folder is meant to be copied.** It imports nothing from the app around
it and nothing outside the standard library but pydantic — and FastAPI, in
`pipeline/web/`, which an app that has no web face never imports. Drop the
directory into another project, point `Jobs` at a SQLite file, and it works.
`pipeline.md` beside this file is the write-up, and it travels with the folder.
**The source of truth is the boilerplate repo's copy.** news and health hold
copies; a fix lands here first and is copied out with `git archive`.
Nothing here knows what a job *does*. The jobs themselves are the app's —
`app/jobs.py` for the starter — and import `@job` from here. That is the one
rule: the pipeline is imported by the work, never the other way round, so
there is no cycle to break and no lazy import to remember.
models.py `Job`, `Run`, `JobsSnapshot`, `Notice` and the `Model` base
they share — the shapes the queue stores and serves
notices.py a failure turned into one line worth putting on screen.
Only the part the queue needs; classifying a failure is the
app's, because a useful sentence names things only it knows
registry.py `@job`, `Spec`, and `JOBS` — how a function becomes a job,
how many attempts it is worth, and whether it runs in a
thread or in a child process of its own
schedule.py `Cadence`, `declare` and `SCHEDULES` — the five wordings a
standing instruction may be written in, when each next comes
round, and how an app ships a default schedule next to `@job`
context.py `RunContext` — what a running job is handed: progress,
cancellation, notices, its parents' results, the fan-out,
the subprocess helper, and the two things it can raise
table.py `Jobs` — the jobs table, on whatever SQLite file it is given
child.py the other side of `isolate=True`: one job, one process,
reconstructed from its row
queue.py `JobQueue` — the worker loop that drains that table
task.py `Task` — plain code that finds work, cuts it into batches
and queues them, plus the one final job every task ends
with and the tally it hands to the callback. `Bound` is
what a batch carries when how often to try it is the
batch's own business rather than the job kind's
worker.py `python -m pipeline.worker`: the same queue with no web app
around it, for a second machine, a cron entry or a script
web/ the queue over HTTP, the socket that reports on it, and the
`<desk-jobs>` element that renders it. Optional: an app
with no web face never imports it, and so never needs
FastAPI
What the pipeline asks of an app is small and is typed where it is asked:
handlers take `(config, store)` and the queue passes those through untouched,
`Jobs` wants a file path, and `isolate=True` wants a `bootstrap` that can
rebuild that pair in a child process. That is the whole of what a second app
has to provide. `README.md` beside this file is the ten-minute version.
Within the package the order is `models, notices > registry > schedule >
context > table > child > task > queue > worker > web`, and `tests/test_architecture.py` holds
every file to it, and to one door: what is listed below is the whole of what
anything outside may use.
"""
from __future__ import annotations
from pipeline import notices
from pipeline.context import NULL, Cancelled, Retry, RunContext, spawn
from pipeline.models import (
TERMINAL, CascadeProgress, Job, JobState, JobsSnapshot, Notice,
NoticeEntry, NoticeKind, NoticeLevel, Run, Schedule, TaskProgress,
)
from pipeline.queue import (
ChildFailed, JobQueue, Listener, NoticeListener, ScheduleListener,
local_zone,
)
from pipeline.registry import JOBS, Handler, Spec, job
from pipeline.schedule import (
DAYS, FORMS, SCHEDULES, BadCadence, Cadence, Declared, declare,
)
from pipeline.table import MAX_RESULT, Jobs, ResultTooLarge, RunCancelled
from pipeline.task import FINAL, TASKS, Batch, Bound, Tally, Task, fan_out
__all__ = [
"DAYS",
"FINAL",
"FORMS",
"JOBS",
"MAX_RESULT",
"NULL",
"SCHEDULES",
"TASKS",
"TERMINAL",
"BadCadence",
"Batch",
"Bound",
"Cadence",
"Cancelled",
"CascadeProgress",
"ChildFailed",
"Declared",
"Handler",
"Job",
"JobQueue",
"JobState",
"Jobs",
"JobsSnapshot",
"Listener",
"Notice",
"NoticeEntry",
"NoticeKind",
"NoticeLevel",
"NoticeListener",
"ResultTooLarge",
"Retry",
"Run",
"RunCancelled",
"RunContext",
"Schedule",
"ScheduleListener",
"Spec",
"Tally",
"Task",
"TaskProgress",
"declare",
"fan_out",
"job",
"local_zone",
"notices",
"spawn",
]
+140
View File
@@ -0,0 +1,140 @@
"""The other side of `isolate=True`: one job, one process, and then gone.
python -m pipeline.child --db news.db --job 412 \
--bootstrap newsdesk.jobs:bootstrap --out C:/Temp/tmp123.json
Run by the worker, never by a person, and the whole reason it can exist is
that **a job row is already a complete description of the work.** A kind, a
JSON payload, a database file. Nothing is pickled and nothing is sent: the
child is told where to look and reads the same row the worker read.
Which is also the division of labour that keeps the state machine intact.
The child writes *progress* — a non-terminal update, and the snapshot the UI
reads is a `SELECT`, so a child's progress reaches the browser for free. The
child does **not** write the ending. It hands its result back through a file
and exits, and the worker writes `done` or `failed` exactly as it does for a
job that ran in a thread. One writer of the terminal state, as before.
A file rather than stdout because a handler is arbitrary code and arbitrary
code prints. Parsing a result out of a stream something else is also writing
to is a bug waiting for the first `print` in a fetch loop.
The `bootstrap` is the one thing an app has to provide, and it has two jobs:
import the modules that fill `JOBS`, and return the `(config, store)` pair
the handlers take. Neither survives a process boundary, so neither is sent —
it is rebuilt here from the config file the app already knows how to read.
"""
from __future__ import annotations
import argparse
import importlib
import json
import logging
import sys
import traceback
from pathlib import Path
from typing import Any
from pipeline.context import Cancelled, RunContext
from pipeline.registry import JOBS
from pipeline.table import Jobs
log = logging.getLogger(__name__)
#: What the worker reads off the exit code. Anything else is the process
#: dying of something that was never ours to catch — an OOM kill, a segfault
#: in a native library — and the worker reports it as the failure it is.
OK = 0
FAILED = 1
CANCELLED = 2
def resolve(name: str) -> Any:
"""`module:attribute` to the attribute. The same spelling a callback uses."""
module, _, attr = name.partition(":")
if not module or not attr:
raise ValueError(f"a bootstrap is 'module:function', not {name!r}")
return getattr(importlib.import_module(module), attr)
def context_for(jobs: Jobs, job_id: int, task_id: int) -> RunContext:
"""A context that reports to the table and asks it about cancellation.
Every channel the in-process context has, except the two that cannot
cross a process boundary and are not wanted here anyway: `on_task` and
`on_fan_out` both queue work, and an isolated job is isolated precisely
because it is one long piece of work rather than an organiser of other
work. Both fall back to their no-ops, which report that nothing took it.
Cancellation is a read of the row, which is what makes Kill work at all
from here: the button writes `cancel_requested` in the web process, and
this is the only way this process would ever hear about it. The worker
also kills us outright — this is the polite half, so a handler that calls
`check()` stops at a line of its own choosing and gets to clean up.
"""
return RunContext(
on_progress=lambda done, total, message: jobs.progress(
job_id, done, total, message),
is_cancelled=lambda: jobs.cancel_requested(job_id),
is_killed=lambda: bool(task_id) and jobs.cancelled(task_id),
on_notice=lambda notice: print(
json.dumps({"notice": notice.model_dump()}), file=sys.stderr),
)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="pipeline.child", description=__doc__)
parser.add_argument("--db", required=True, help="the SQLite file the jobs table is on")
parser.add_argument("--job", required=True, type=int, help="which row to run")
parser.add_argument("--bootstrap", required=True,
help="module:function returning (config, store)")
parser.add_argument("--bootstrap-arg", default=None,
help="one string handed to the bootstrap, e.g. a config path")
parser.add_argument("--out", required=True, help="where to write the result JSON")
args = parser.parse_args(argv)
out = Path(args.out)
jobs = Jobs(args.db)
job = jobs.get(args.job)
if job is None:
out.write_text(json.dumps({"error": f"no job {args.job}"}), encoding="utf-8")
return FAILED
try:
build = resolve(args.bootstrap)
config, store = (build() if args.bootstrap_arg is None
else build(args.bootstrap_arg))
spec = JOBS.get(job.kind)
if spec is None:
raise KeyError(
f"{job.kind!r} is not a registered job in this process — "
f"does {args.bootstrap} import the module that declares it?"
)
context = context_for(jobs, job.id, job.task_id)
kwargs = dict(job.payload)
if spec.wants_context:
kwargs["context"] = context
result = spec.fn(config, store, **kwargs)
except Cancelled:
out.write_text(json.dumps({}), encoding="utf-8")
return CANCELLED
except BaseException as exc:
# The traceback goes in the file as well as to stderr. stderr is the
# worker's fallback and is captured either way; the file is what
# survives a child that also wrote something confusing to the stream.
out.write_text(json.dumps({
"error": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
}), encoding="utf-8")
traceback.print_exc()
return FAILED
out.write_text(json.dumps({"result": result or {}}, default=str), encoding="utf-8")
return OK
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO,
format="%(levelname)s %(name)s: %(message)s")
raise SystemExit(main())
+446
View File
@@ -0,0 +1,446 @@
"""What a running job is handed, and the two things it can raise.
def handler(config, store, *, context=NULL, limit=None) -> dict
This is the job-side half of the contract; `registry.py` is the other — how
a function becomes queueable. A pipeline stage imports this to report
progress from inside its own loops, and nothing else: no queue, no worker.
Nothing here imports anything else from desk at module level, so it is safe to
import from anywhere.
Everything in here is optional to the job that receives it. Ignore `context`
and the job still runs — it just cannot be stopped part-way and shows no
progress. Use it and you get both, for two lines.
"""
from __future__ import annotations
import logging
import os
import signal
import subprocess
import tempfile
import time
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import Any
log = logging.getLogger(__name__)
#: How often a spawned process is looked in on: whether it has finished,
#: whether Kill has been pressed, whether its deadline has passed. Frequent
#: enough that a Kill lands in well under a second, rare enough that waiting
#: an hour for ffmpeg costs a few thousand cheap syscalls.
POLL = 0.25
class Cancelled(Exception):
"""Raised by `RunContext.check` when the job has been asked to stop.
Caught by the worker and recorded as `cancelled` rather than `failed`. A
handler should let it travel — catching it is how you build the thing the
Kill button can't kill.
"""
class Retry(Exception):
"""Raised by a handler that wants this same work attempted again later.
An ordinary failure is already retried, within whatever bound the job was
queued with — so this is not how a job asks to be retried at all. It is how
it says *when*: `after` is the wait the other end named, a `Retry-After`
header or a rate limit's own answer, and the queue uses it in place of the
backoff it would otherwise have guessed. Guessing is the thing worth
avoiding here — a minute is either a minute of doing nothing when the
service was ready, or another refusal charged against the bound.
It does not buy an attempt. A job queued with one attempt raises this and
fails, exactly as any other exception would: the bound is ours to set and
the wait is the world's to name, and a handler that could extend its own
bound would be a policy nobody declared.
"""
def __init__(self, message: str = "", *, after: float | None = None) -> None:
super().__init__(message)
#: Seconds to wait, or None to leave it to the job's own backoff.
self.after = after
def _no_progress(done: int, total: int, message: str) -> None:
return None
def _no_task(task: Any) -> bool:
return False
def _no_stop(why: str) -> int:
"""Where "stop the rest of this run" goes when there is no run to stop.
The direct path is one batch after another in the caller's own thread, and
the caller is standing there: it sees each failure as it happens and can
stop by not asking again. Nothing to cancel, so nothing is cancelled.
"""
return 0
def _no_fan_out(*_args: Any, **_kwargs: Any) -> tuple[int, ...]:
"""Where a fan-out goes when there is no queue to put it in.
Empty rather than an error, and the same answer `task` gives for the same
reason: a handler called directly — from a terminal, a notebook, a test —
is being run by somebody who is standing there, and the useful behaviour
is to do its own work and report that nothing was queued behind it. A
handler that must know either way checks the return.
"""
return ()
def _log_only(notice: Any) -> None:
"""Where a notice goes when nothing is watching for one.
A direct caller — `newsdesk job pull` in a terminal, a notebook, a test — has
no socket, so the log is the destination. It is not a *lesser* one: it goes
through the same `notices.render`, so the terminal gets the sentence, the
command to run and the is-this-a-bug verdict, exactly as the browser does.
"""
from pipeline.notices import render
log.warning("%s\n%s", notice.code, render(notice))
def _never() -> bool:
return False
def kill_tree(proc: subprocess.Popen[Any]) -> None:
"""Kill a spawned process *and* whatever it started.
The children are the point. `proc.kill()` reaches the process we launched
and nothing it launched itself, so killing a shell leaves the ffmpeg it
ran holding the core — which is the exact failure a deadline exists to
prevent. Both branches need the process to have been started detached,
which `spawn` does.
Best-effort by construction: between deciding to kill and killing, the
process may have ended on its own and its id may have been reused. So
every call is guarded, and a failure to kill something that is already
gone is not news.
"""
if proc.poll() is not None:
return
try:
if os.name == "nt":
# No process groups to signal, so ask Windows to walk the tree.
# `/T` is children, `/F` is without asking.
subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)],
capture_output=True, check=False, timeout=10)
else:
# Guarded by `os.name` above rather than by the type checker,
# which is reading this file on Windows and knows these three do
# not exist there.
os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # type: ignore[attr-defined,unused-ignore]
except Exception:
log.debug("could not kill %s", proc.pid, exc_info=True)
try:
proc.wait(timeout=5)
except Exception:
log.warning("process %s did not go away after being killed", proc.pid)
def spawn(argv: Sequence[str], *, check: Callable[[], None] | None = None,
timeout: float | None = None, poll: float = POLL,
**popen: Any) -> subprocess.CompletedProcess[bytes]:
"""Run a command, watching for Kill and for a deadline while it does.
The building block under `RunContext.spawn` and under `isolate=True`, and
the answer to the one thing a thread cannot do. A handler that blocks in
`subprocess.run` cannot be stopped, because stopping it would mean tearing
a thread out of a syscall; a handler that blocks *here* can, because what
is expensive is the child and a child can be killed. The thread is only
sleeping, which costs nothing.
Output goes to temporary files rather than pipes. `PIPE` plus a poll loop
is a deadlock: a child that fills the pipe buffer blocks on the write
while we block on the poll, and neither side is wrong. A file cannot fill
up in the way that matters, so the loop is free to look away.
Started detached — its own process group, or Windows' equivalent — purely
so `kill_tree` has something to aim at.
"""
detach: dict[str, Any] = ({"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
if os.name == "nt" else {"start_new_session": True})
with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err:
popen.setdefault("stdout", out)
popen.setdefault("stderr", err)
proc = subprocess.Popen(list(argv), **detach, **popen)
deadline = time.monotonic() + timeout if timeout else None
try:
while proc.poll() is None:
if check is not None:
check() # raises Cancelled on Kill
if deadline is not None and time.monotonic() > deadline:
kill_tree(proc)
raise TimeoutError(
f"{argv[0]} was still running after {timeout:g}s")
time.sleep(poll)
except BaseException:
kill_tree(proc)
raise
out.seek(0)
err.seek(0)
return subprocess.CompletedProcess(
list(argv), proc.returncode,
out.read() if popen["stdout"] is out else b"",
err.read() if popen["stderr"] is err else b"")
@dataclass(slots=True)
class RunContext:
"""The handle a running job has on the run it is part of.
Deliberately a handful of plain callables rather than a reference to the
queue: a handler cannot reach back into the worker, reschedule itself or
read other jobs, and a test can hand it a lambda without standing anything
up.
"""
on_progress: Callable[[int, int, str], None] = _no_progress
is_cancelled: Callable[[], bool] = _never
#: Whether the *task* this job belongs to was killed — a different
#: question from `is_cancelled`, which is about this job alone. See
#: `killed`.
is_killed: Callable[[], bool] = _never
on_notice: Callable[[Any], None] = _log_only
on_task: Callable[[Any], bool] = _no_task
on_stop: Callable[[str], int] = _no_stop
#: Where `fan_out` puts the jobs it asks for. Supplied by the queue, which
#: is the only thing that can write rows; `_no_fan_out` under `NULL`.
on_fan_out: Callable[..., tuple[int, ...]] = _no_fan_out
#: How the jobs this one waited for ended, oldest first. Empty for a job
#: that waited for nothing. Typed loosely for the same reason `notice` is:
#: nothing in this module may import the rest of desk, and the shape it
#: carries is `pipeline.models.Job`.
waited_for: tuple[Any, ...] = ()
def upstream(self) -> dict[str, Any]:
"""What the jobs this one waited for returned, merged into one dict.
The read half of a chain. `A -> B -> C` needs A's output to reach B,
and this is how it arrives: B does not have its payload filled in by
A, it reads A's result off A's row. Which is the whole difference —
**a row is complete when it is written.** A payload with a hole in it
meant for an earlier job to fill would be claimable the instant that
job landed, so a worker could take it between the write and the
commit; it could not carry a `dedup_key`, which has to be known at
queue time; and it could not be re-run from its own row, which is
what a retry does and what `max_attempts` is.
Merged, later parents winning, because the overwhelmingly common case
is one parent and `upstream()["text"]` is what that should read like.
A job with several parents that care about the same key wants
`waited_for` and to decide for itself.
Empty under `NULL`, which is what makes the fallback work:
def handler(config, store, *, context=NULL, html=""):
html = html or context.upstream().get("html", "")
From the queue nobody passes `html` and it comes from the parent; from
a terminal you pass it yourself. Without that line a chained handler
is the one kind in the pipeline you cannot simply call.
"""
merged: dict[str, Any] = {}
for parent in self.waited_for:
result = getattr(parent, "result", None)
if isinstance(result, dict):
merged |= result
return merged
def fan_out(self, kind: str, payloads: Sequence[dict[str, Any]], *,
then: str = "", settle: str = "", label: str = "",
carries: dict[str, Any] | None = None,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> tuple[int, ...]:
"""Queue one job per payload, and one job behind them that reports.
Returns the ids of the children. The reporting job is not among them:
it is the pipeline's own kind, it is nothing an app writes, and it is
the reason this method exists rather than a handler assembling a
`depends_on` list by hand every time.
def handler(config, store, *, context=NULL):
chunks = chunk(needs_a_body(store))
context.fan_out("body_batch", [{"uids": c} for c in chunks],
then="newsdesk.jobs.cascade:step")
`then` is `module:function`, resolved where it is run, and is handed
`(config, store, tally, carries)` once every child is over however it
ended — see `Tally` in `task.py`. A name rather than a function
because what carries it is a table row, and no row can hold a
function.
One transaction, so a fan-out is never half-queued. The children go
into a run of their own within this job's cascade, which is what makes
the callback able to fan out again: that is a cascade, and there is no
cascade object anywhere — step N's callback queues step N+1.
Nothing is queued under `NULL`, and `()` says so.
"""
return self.on_fan_out(
kind, list(payloads), then=then, settle=settle, label=label,
carries=dict(carries or {}), max_attempts=max_attempts,
retry_delay=retry_delay, retry_backoff=retry_backoff)
def spawn(self, argv: Sequence[str], *, timeout: float | None = None,
**popen: Any) -> subprocess.CompletedProcess[bytes]:
"""Run an external command, killable, with this job's Kill wired in.
What to reach for when the expensive thing is a binary — ffmpeg,
stockfish, whisper.cpp. Because the work is in the child and not in
this thread, Kill is no longer cooperative in the way `check` is: the
child is killed outright, and its children with it. Blocking a worker
thread on a process costs nothing, since the thread is only sleeping.
`timeout` is a real deadline for the same reason, and the only kind of
deadline the pipeline offers that can actually end the work. A job
kind that wants one over Python code rather than over a command wants
`isolate=True` — see `registry.py`.
Raises `Cancelled` on Kill and `TimeoutError` on the deadline, having
killed the process tree either way. Non-zero exit is *not* an error
here: the returncode is handed back, because plenty of commands say
something useful with it and only the handler knows which.
"""
return spawn(argv, check=self.check, timeout=timeout, **popen)
def progress(self, done: int = 0, total: int = 0, message: str = "") -> None:
"""Say how far along this is. Units are the handler's own business —
sources, files, minutes of audio — the UI renders `done/total`.
The message has one shape, and every job in this codebase uses it:
a lowercase participle, what it is being done to, and — when the
job is walking a list — `, n of m`.
"reading Golem, 3 of 12". "fetching the body from theverge.com, 40 of
212". "training on your votes" for a job that is one step and has no
list to be three of. Not an em dash, not a middot, and never a bare
identifier: this used to say `wordpress`, which named the module that
reads the feed rather than the outlet, and read as a fragment of a log
line that had escaped onto the status strip.
`n` is the item being worked on now and so runs one ahead of `done`,
which counts the ones that are finished. They are answering different
questions — the fraction in the message belongs to the name beside it,
and the bar belongs to the job — and a line saying "reading Golem, 0 of
12" while Golem is being read would be wrong about the only thing it
was there to say. A phase that has nothing left to name ends on the
collective — "reading every source, 12 of 12" — so the same participle
carries the line from first item to last, and a full fraction is what
says it is over. Past tense is not used: it dates a line that is about
to be replaced by the next phase's.
A job's *label* is the same grammar capitalised, because a label names
the work rather than narrating it: "Tagging 25 items, 2 of 9".
"""
self.on_progress(done, total, message)
def check(self) -> None:
"""Stop now if Kill has been pressed.
Cancellation is cooperative because the work is synchronous: there is
no safe way to tear a thread out of the middle of an HTTP fetch or a
model load, so the job says where stopping is fine. Call it between
units of work, not inside tight inner loops.
"""
if self.is_cancelled():
raise Cancelled()
def killed(self) -> bool:
"""Whether the task this job belongs to was killed.
Not the question `check` asks. That one is about this job, and killing
one job is dropping one batch. This one is about the run, and killing
a run is skipping the whole step — which the job that reports on the
step has to be able to say, since it is spared the kill precisely so
that somebody hears about it. The counts cannot tell the two apart: a
batch someone killed by hand and a batch killed by the task going
down both arrive as `cancelled`.
Asked when asked, not read at the start, so a kill landing while the
reporting job runs is still in what it reports.
"""
return bool(self.is_killed())
def step(self, done: int = 0, total: int = 0, message: str = "") -> None:
"""`progress` then `check` — what the top of a loop body always wants."""
self.progress(done, total, message)
self.check()
def task(self, task: Any) -> bool:
"""Queue a `Task` once this job has finished well.
Recorded rather than queued: the worker starts the task after this job
lands in `done`, so a task never outlives the failure of the job that
described it, and a handler still cannot reach the queue. It names what
it wants to happen and that is all.
The task runs in the worker, synchronously, the moment the job is over
— the same plain code a request would run. It is handed the queue as
its sink and this job's cascade, so its batches land in the run this
one belongs to.
Returns whether anything took it. `False` under `NULL`, where there is
no queue to take it: a callback invoked from a notebook or a Claude
Code session has to do the work itself, and this is how it finds out
that nobody else will.
"""
return bool(self.on_task(task))
def stop(self, why: str = "") -> int:
"""Cancel this task's remaining queued batches. Returns how many.
For the failure that is not this batch's: a spend limit, a signed-out
CLI, no model on the machine at all. The backend has answered, the
answer is the same one every sibling is about to be given, and the
only thing left to decide is whether to spend the other twenty-two
calls finding that out. Twenty-three identical notices saying "looks
like a bug" is what not deciding looked like.
Only what is still *queued* stops. A batch already in flight is a
request in the air and a bill either way, so it runs to its own end —
and this one, the one that asked, keeps its own failure. The task's
final job is spared for the reason `Jobs.cancel_run` spares it:
reporting is not work, and a run that stopped early still has to say
so. `why` is written onto each cancelled row, so a batch that never
ran says on screen why it didn't.
Not a kill. Nobody pressed anything, so the task's row is not flagged
and `killed` stays false: the cascade goes on to the next stage, as it
does after any batch that did not land. Whether the *next* task should
stop too is that cascade's question, not this one's.
"""
return int(self.on_stop(why))
def notice(self, notice: Any) -> None:
"""Tell the user something went wrong. See `pipeline/notices.py`.
The counterpart to `log.exception`, not a replacement for it: log the
traceback for afterwards, raise a notice for now. Use it for what a
person has to act on — a login to renew, a feed that has stopped
resolving — and not for what the program can simply get on with.
Under a queue it is written to the notice log and pushed to whoever is
watching; called directly it is logged and nothing else. Safe to call
repeatedly: the same `code` and `key` again is one row with a count,
and only the first interrupts anyone. So give `key` whatever the
failure is about — the source, the article — or two different
failures of one kind are told as one.
"""
self.on_notice(notice)
#: For calling a handler outside the queue. Reports nowhere, never cancels.
NULL = RunContext()
+432
View File
@@ -0,0 +1,432 @@
"""The shapes the queue stores and serves: a job, a run, a snapshot, a notice.
One set of models, not two — there is no ORM here, so the row the table
returns and the object the API serves are the same declaration. `Model` is
the base an app's own shapes are welcome to extend as well, because the one
thing it does (tolerate a `sqlite3.Row`, ignore columns nobody declared) is
wanted by every table anyone will ever `SELECT *` from.
Pydantic is the package's only dependency outside the standard library, and
this is the file that wants it. Everything else here would run without it.
"""
from __future__ import annotations
import json
from datetime import UTC, datetime
from typing import Annotated, Any, Literal, Self
from pydantic import (BaseModel, BeforeValidator, ConfigDict, computed_field)
#: A column that holds JSON. SQLite hands back the text; everything above the
#: storage layer wants the dict. Decoded here so neither side has to remember,
#: and tolerant of a malformed value because a job whose payload failed to
#: parse should still be listable and cancellable rather than un-readable.
JsonDict = Annotated[
dict[str, Any],
BeforeValidator(lambda v: v if isinstance(v, dict) else _loads(v)),
]
def _loads(value: Any) -> dict[str, Any]:
try:
parsed = json.loads(value or "{}")
except (TypeError, ValueError):
return {}
return parsed if isinstance(parsed, dict) else {}
#: The same for a column holding a JSON array of job ids. Tolerant for the same
#: reason: a job whose dependency list will not parse has already been let
#: through by the queue, which reads the column in SQL — so refusing to build
#: the model here would only hide it from the list it is holding up.
JsonIds = Annotated[
list[int],
BeforeValidator(lambda v: v if isinstance(v, list) else _ids(v)),
]
def _ids(value: Any) -> list[int]:
try:
parsed = json.loads(value or "[]")
except (TypeError, ValueError):
return []
return [int(i) for i in parsed] if isinstance(parsed, list) else []
class Model(BaseModel):
"""Base: tolerate sqlite3.Row and reject fields nobody declared."""
model_config = ConfigDict(from_attributes=True, extra="forbid")
@classmethod
def from_row(cls, row: Any) -> Self:
"""Build one from a database row, ignoring columns nobody declared.
`SELECT *` into a model that forbids extras means the next
`ALTER TABLE` is a 500 in whatever route reads that table — raised a
long way from the migration that caused it, over a column the API was
never going to serve anyway. A storage column the API has not heard of
is not something to hand out; it is something to ignore until someone
declares a field for it.
The half of `extra="forbid"` worth keeping still works. A column a
model *requires* going missing — a rename, a typo — is a validation
error exactly as before. Only additions go quiet.
"""
return cls.model_validate(
{k: v for k, v in dict(row).items() if k in cls.model_fields}
)
# ------------------------------------------------------------------- jobs
#: The five states a job can be in. `queued` and `running` are live; the other
#: three are terminal and a job never leaves them.
JobState = Literal["queued", "running", "done", "failed", "cancelled"]
#: Everything that is over, however it ended. Used to decide what the dedup
#: index constrains, what the UI greys out, and when to stop pushing updates.
TERMINAL: frozenset[str] = frozenset({"done", "failed", "cancelled"})
class Job(Model):
"""One unit of queued work, as stored.
`payload` is the handler's keyword arguments and `result` is what it
returned — both JSON in SQLite, both dicts everywhere else, translated
once here rather than at each call site.
`done`/`total` are deliberately untyped as units. A pull counts sources, a
transcription counts minutes of audio; the UI renders "3/12" and a bar and
does not need to know which.
"""
id: int
kind: str
label: str = ""
payload: JsonDict = {}
state: JobState = "queued"
done: int = 0
total: int = 0
message: str = ""
result: JsonDict = {}
error: str = ""
dedup_key: str = ""
#: Which jobs this one waits for, and whether it needs them to have
#: succeeded (`ok`) or merely to be over (`done`).
depends_on: JsonIds = []
after: str = "ok"
#: Which kind of worker runs it: `llm`, `http`, `cpu`.
resource: str = "cpu"
#: Which cascade this belongs to, and which of its tasks. A job queued on
#: its own is its own cascade and its own task. Both name a row in `runs`
#: — a series of their own, not this one's. See `Run`.
cascade_id: int = 0
task_id: int = 0
cancel_requested: bool = False
#: How many times this has been started, counted by `claim`. One after the
#: first run, whether that run ended well, badly, or with the worker dying
#: under it.
attempt: int = 0
#: The bound, counting the first attempt — so `1` is the default and means
#: no second try. Carried on the row rather than looked up from the job
#: kind, because the attempt that decides is the one after a restart, in a
#: worker that may be running a different version of the code.
max_attempts: int = 1
#: The wait before the next attempt, and what multiplies it each time
#: round. `1.0` is a flat wait.
retry_delay: float = 0.0
retry_backoff: float = 1.0
#: Not before this. Empty means now, which is why it is `''` rather than
#: NULL: the column is compared as text and `''` sorts before every stamp,
#: so "is it due?" is one comparison with no special case in it.
run_after: str = ""
created_at: str
started_at: str | None = None
finished_at: str | None = None
@property
def live(self) -> bool:
return self.state not in TERMINAL
@computed_field # type: ignore[prop-decorator]
@property
def waiting(self) -> bool:
"""Queued, but not until its next-attempt time comes round.
Serialised, unlike its two neighbours, because it is the one of the
three a client cannot work out for itself. `live` is a state, and
`exhausted` is `attempt`, `max_attempts` and `state`, all of which go
out on the row; this one is `run_after` against *now*, and the only
clock worth asking is the one the stamp was written by. A browser four
minutes fast would otherwise draw a job as due while the queue goes on
passing over it.
`run_after` travels too and still has to: this says *that* it is
waiting and the stamp says until when, which is the difference between
greying a row out and counting down on it.
The resting state of a failure that is going to be tried again, and the
thing that makes it different from a failure: it is still `queued`, so
nothing is running it, nothing holds a slot for it, and whatever waits
on it goes on waiting. Compared as text, which is safe because
`run_after` is written in the same UTC ISO form as every other stamp
here — see `table.py`.
"""
return self.state == "queued" and self.run_after > datetime.now(UTC).isoformat()
@property
def exhausted(self) -> bool:
"""Failed on the last attempt it was allowed, rather than the only one.
Worth telling apart, because the two want opposite things done about
them. A job that failed with attempts still to come is not this — it is
back in the queue and `waiting`. A job that failed at `max_attempts=1`
is not this either: nobody ever said it should be retried, so nothing
was used up. This is the third case, the one where the policy ran its
course and the work still did not land, and it is the only one where
trying again is a decision a person has to make.
"""
return self.state == "failed" and self.attempt >= self.max_attempts > 1
class Run(Model):
"""A cascade or a task: an identity, and whether it was killed.
No state and no progress: the jobs carrying this id are the record of what
happened, and a second copy of it here could only go out of step. What it
holds is what cannot be worked out from them — that this id is a cascade
rather than a task, what it was called, when it began, and the one fact its
jobs cannot tell you, which is that somebody stopped it.
"""
id: int
kind: str
name: str = ""
#: What to call it on screen. A task is not a job, so this is the only
#: place it can be written down.
label: str = ""
created_at: str
#: When it was killed, or "" for a run nobody stopped. A cascade exists
#: nowhere but this row, so this is where its Kill has to live.
cancelled_at: str = ""
class Schedule(Model):
"""A standing instruction: enqueue this kind, with this payload, this often.
A row rather than a timer, for the reason every other waiting thing here is
a row: nothing is alive between runs, so a process restarted at 05:59 still
fires the 06:00 one, and a second worker on the same file reads the same
schedule rather than keeping a second copy of it.
`name` is unique and is the handle declared defaults are matched on — see
`pipeline/schedule.py`. Everything else about a row is the user's to edit,
the cadence and whether it is on at all included, and redeclaring a default
never overwrites those.
`next_run` and `last_run` are UTC ISO stamps, like every other time in this
table, and are compared as text — so a cadence worded in a local timezone
still sorts correctly against `now` and reads the same from any process.
Empty means never, which for `next_run` also means not due, and is why it
is `''` rather than NULL.
"""
id: int
name: str
kind: str
#: What to call it on screen, from `declare(label=...)`. Empty means the
#: job kind's own label, which is what the queued job then carries.
label: str = ""
payload: JsonDict = {}
cadence: str
enabled: bool = True
next_run: str = ""
last_run: str = ""
#: The job the last firing queued, and so the answer to "is one still
#: going?" — which is what stops a slow job from piling up behind itself.
last_job_id: int = 0
#: How that job ended, or `queued`/`running` while it has not. Joined on in
#: `Jobs.schedules` rather than fetched per row: it is one column on a query
#: that is already running, and the time is no use on screen without it.
last_state: str = ""
created_at: str
class TaskProgress(Model):
"""One task of a live cascade, as a tally of the jobs carrying its id.
Counts and not the jobs themselves. A task is a body fetch shared over
twenty-three jobs and there is nothing to tell them apart — the ones still
going are in `JobsSnapshot.running` with their own `done`/`total`, and the
ones that landed are a number, because a landed job is a full box and full
boxes are all alike.
Derived on read, never stored, for the reason `runs` holds no state: the
job rows are the record of what happened and a second copy of the tally
could only go out of step with them.
"""
#: The `runs` row this task is, and the id `/api/tasks/{id}/cancel` takes.
id: int
#: What the task is — `body`, `tag`. Its own name, minted with the `runs`
#: row before any of its jobs existed, and how a cascade's list of tasks is
#: matched against it.
name: str = ""
#: How to say that on screen: "Fetching what the feeds left out". The
#: task's own, off the `runs` row, because a task is not a job and has no
#: handler to carry one.
label: str = ""
done: int = 0
failed: int = 0
cancelled: int = 0
running: int = 0
queued: int = 0
class CascadeProgress(Model):
"""A cascade that is still going, and how far each of its tasks has got.
Only the tasks that exist. A cascade splits at every boundary — how many
jobs summarising becomes is a query that is only meaningful once the
bodies have landed — so the ones ahead have no jobs, no ids and nothing to
count. Which of them are coming is the app's own business, in its cascade
module and served beside this; here there is nothing to say about them.
"""
#: The `runs` row, and the id `/api/cascades/{id}/cancel` takes.
id: int
#: Which cascade it is a run of — `pull`. For a job or a task queued on
#: its own, which is a cascade of one, that name and a word saying it
#: stands alone: `pull on its own`. The two are different runs and the
#: name is where they are told apart, so it is also what the strip shows.
name: str = ""
#: In the order they began, which for a cascade is the order they run.
tasks: list[TaskProgress] = []
class JobsSnapshot(Model):
"""What the queue looks like right now — the whole WebSocket payload.
Sent on connect and after every state change, rather than diffed. The
whole thing is a few hundred bytes and one shape that is always complete
is worth more here than the bytes a patch protocol would save.
"""
#: Everything under way. A list because the queue has several workers, so
#: "the running job" stopped being a single thing.
running: list[Job] = []
queued: list[Job] = []
recent: list[Job] = []
#: The cascades any of that belongs to, tallied by task. Here rather than
#: behind a URL of its own because a tally of job states changes exactly
#: when this message does: asked for separately it would arrive a beat
#: later and draw a bar that disagreed with the list above it. And it
#: cannot be counted from that list — `recent` is the last thirty jobs, so
#: by the time tagging starts the body batches that landed have dropped
#: off the end and a bar drawn from it would count backwards.
cascades: list[CascadeProgress] = []
# ---------------------------------------------------------------- notices
#: How loud a `Notice` is, and the four colours a reader renders them in.
#: `error` is red — "you have to do something about this". `warning` is amber,
#: the same amber a work panel uses for a row that wants you — "this went
#: wrong and carried on". `info` is grey, worth saying once. `good` is green
#: and is the odd one out: nothing in `notices.py` produces one, because that
#: module is about failure — it exists for a UI to confirm something worked
#: without inventing a second mechanism to say so.
NoticeLevel = Literal["info", "good", "warning", "error"]
#: Whether a failure is the world's or ours. See `Notice.kind`.
NoticeKind = Literal["expected", "bug"]
class Notice(Model):
"""Something went wrong that you, not the log, need to know about *now*.
Broadcast to whatever browsers are open, and written to the `notices`
table so that one opened afterwards can read it — see `NoticeEntry`, and
`pipeline/notices.py` for why it is kept after all.
So the division of labour is:
this "the tagger stopped, your claude login expired" — on
screen, the moment it happens, with enough to tell you
whether to shrug, fix something, or go and read the log.
the table the same sentence, findable once it has faded, with how
often it has been said. Cleared by you, or by a fortnight
of not being said again.
the log everything, with the traceback, still there tomorrow.
Which means a notice does not have to be complete. It has to be *enough to
orient you*, and `where` points at the rest.
"""
#: Stable dotted identifier — `agent.auth`, `source.failed`. What the UI
#: dedupes a burst on, and what a test asserts against, so the prose in
#: `text` can be reworded without breaking either.
code: str
level: NoticeLevel = "warning"
#: The distinction worth drawing when something breaks: is this the world
#: misbehaving in a way it is entitled to, or is this ours?
#:
#: expected an expired login, a feed returning 403, a rate limit. Not a
#: defect. Either you fix it or you wait, and either way there
#: is nothing to debug.
#: bug an exception we did not anticipate. Something to go and read
#: the log about, and probably to fix in the code.
#:
#: Getting this wrong in either direction is expensive — hunting a bug that
#: was a dead feed, or shrugging at a real one — so it is a field rather
#: than something to infer from the wording.
kind: NoticeKind = "expected"
text: str
#: What to do about it, when there is something: `claude auth login`. Empty
#: when there isn't, which is worth saying honestly rather than padding.
fix: str = ""
#: The raw error. Shown folded — it is the bit you would paste into a bug
#: report, and the bit that tells you which log line to look for.
detail: str = ""
#: Where to look for the whole story. Always the log; this names the part.
where: str = ""
#: What this is about within its code — a source name, a job kind. Lets the
#: UI collapse a burst of the same failure into one line with a count.
key: str = ""
class NoticeEntry(Notice):
"""A notice as the log holds it: one row per `(code, key)`.
What `Jobs.record_notice` hands back and what `/api/notices` lists. The
same failure said again is this row with a higher `count` and a later
`last_at`, never a second row — which is also how a client knows whether
to interrupt anyone: `count == 1` is news, anything above it is a repeat
of something already said and still sitting in the log.
"""
id: int
first_at: str
last_at: str
count: int = 1
__all__ = [
"TERMINAL",
"CascadeProgress",
"Job",
"JobState",
"JobsSnapshot",
"JsonDict",
"JsonIds",
"Model",
"Notice",
"NoticeEntry",
"NoticeKind",
"NoticeLevel",
"Run",
"Schedule",
"TaskProgress",
]
+136
View File
@@ -0,0 +1,136 @@
"""Turning a failure into one line worth putting on screen.
The whole of this module's job is a handoff. Something broke; the log has the
traceback and always will; what the log cannot do is *tell you it happened*
while you are looking at the app wondering why nothing appeared. So a notice
is pushed to whoever is watching — and, since nobody usually is, written down.
**It is stored, and it was not always.** The first design kept nothing: a
notices table would be a second record of what the log already records, with
rows that go stale and a dismiss button to maintain. That argument assumed
somebody was watching when the notice was raised, and somebody who could read
a server's log when they were not. Neither holds for a pull that runs on a
schedule and a reader holding a phone. What it cost was three things, each
reported as a defect of its own:
a line that faded was gone twelve seconds, then nowhere to find it
nothing open, nothing heard the usual case, and the notice was lost
every run said it again "already said" was known only to a tab
So `Jobs.record_notice` keeps one row per `(code, key)`, and the three are one
fix. The same failure said again is that row with a higher count, which is how
a client tells news from a repeat: it interrupts for the first and only counts
the rest. A line somebody dismissed is still in the log, so dismissing means
later rather than never. A row goes when somebody clears it, or a fortnight
after it was last said — by which time it has stopped being true, which is the
only staleness the first design was right to worry about.
The table is not the log and does not try to be. No traceback, no history of
each occurrence: a sentence, what to do, the raw error, how often, and when.
This module is only the part the queue itself needs: the notice every failed
job gets, the check that stops it being said twice, and the rendering a
terminal wants. **Classifying a failure is the app's job**, because a useful
sentence names things only the app knows — its model backend, its feeds, its
config keys. An app writes its own vocabulary next to its handlers and passes
notices in through `RunContext.notice`; newsdesk's is `desk/notices.py`.
What a notice has to carry is not the full story but enough to orient you:
text what happened, in a sentence, in terms of what you lost by it.
kind whether this is the world misbehaving — an expired login, a feed
gone 403 — or a bug of ours. That is the fork in what you do next,
and guessing it from an error string is exactly the work you should
not have to do at a glance.
fix the command, when there is one.
"""
from __future__ import annotations
import logging
from collections.abc import Iterable
from pipeline.models import Notice, NoticeKind
log = logging.getLogger(__name__)
#: Words that mean the failure happened at the other end of a wire. An app's
#: own classifier will want more than this; `whose_fault` is the cheap default
#: for the failures the queue meets on its own.
THEIR_FAULT = (
"http", "url", "timeout", "connection", "ssl", "certificate", "socket",
"dns", "resolve", "403", "404", "429", "500", "502", "503", "refused",
"unreachable", "forbidden", "not found", "unavailable", "moved",
)
def job_failed(kind: str, label: str, error: BaseException | str) -> Notice:
"""A job that ended in `failed`.
A handler raising something it did not expect is the definition of ours to
fix, so this is the one place that says `bug` outright rather than working
it out.
"""
return Notice(
code="job.failed", level="error", kind="bug", key=kind,
text=f"{label or kind} failed.",
detail=str(error).strip(),
where=f"the `job … ({kind}) failed` traceback in the log",
)
def already_said(said: Iterable[Notice], error: str) -> bool:
"""Whether one of these notices is about this failure.
The caller is the worker, which announces `job_failed` for every job that
ends in `failed` — and must not, when the handler already announced a
classified notice about the very exception that ended it. "agent.backend
is `off`, so nothing that needs a model runs" followed by "Tagging a batch
failed — looks like a bug" is two contradictory lines about one failure,
and the wrong one is the louder: it sends you to a traceback about a
machine that is doing exactly what it was configured to do.
Matched on `detail` rather than on identity, because what the handler
classified is an exception and what reaches the worker is that same
exception rendered `Type: message` — the same string with a prefix. Which
is also the line in the right place: a notice about something else the job
met on the way, a source that 403'd during a pull that then crashed over
something unrelated, does not match, and that crash still gets its own
line as the unexplained failure it is.
"""
return any(notice.detail.strip() and notice.detail.strip() in error
for notice in said)
def render(notice: Notice) -> str:
"""The same notice, for a terminal.
The browser is the first-class destination, but it is not the only one: a
handler run from a terminal hits every one of these failures and deserves
the same sentence, the same command to run, and the same "is this mine to
fix". Without this the CLI got the code and the text and silently lost the
two fields that make a notice worth having.
Rendered rather than logged in pieces so that the wording lives in one
place. An app changes a rule and both destinations change together.
"""
lines = [notice.text]
if notice.fix:
lines.append(f" -> {notice.fix}")
if notice.kind == "bug":
lines.append(f" this looks like a bug rather than a bad day; see {notice.where}"
if notice.where else " this looks like a bug rather than a bad day")
return "\n".join(lines)
def whose_fault(error: BaseException | str) -> NoticeKind:
"""Whether a failure is the other end's or ours.
A 403 is the outlet's business and there is nothing here to fix. A
`KeyError: 'entries'` is us assuming a shape the other end didn't have, and
it will keep happening until someone changes the code — so it belongs in
front of you as a defect, not as weather.
"""
text = f"{type(error).__name__} {error}".lower() if isinstance(
error, BaseException) else str(error).lower()
return "expected" if any(hint in text for hint in THEIR_FAULT) else "bug"
+395
View File
@@ -0,0 +1,395 @@
# `pipeline` — the job queue
## What it is
A jobs **table in SQLite** with an async worker draining it. Handlers are plain synchronous functions run in threads.
| | |
|---|---|
| Packages | stdlib here — `sqlite3`, `asyncio`, `dataclasses`, `inspect` — plus pydantic, which `pipeline/models.py` wants |
| Not used | no Celery / RQ / Redis / Huey, no `multiprocessing`, no in-memory queue |
| Workers | threads, several at once, one per running job, from a pool of the queue's own |
| Storage | one SQLite file (WAL). Point it at the app's DB or one of its own |
| Concurrency | threads within a worker; any number of workers on one file — `claim` is one atomic `UPDATE`, and a claimed job is held on a lease its worker renews. `python -m pipeline.worker` starts one with no web app; `pipeline/tests/test_housekeeping.py` races two |
| Web | `pipeline.web.connect(app, jobs, config, store)` — routes, socket, the `<desk-jobs>` and `<desk-schedules>` elements, and the worker in the app's lifespan |
| States | `queued → running → done \| failed \| cancelled`, and `running → queued` again for a job with an attempt left |
| Scheduling | five cadences on a `schedules` row in the same file — `every 15m`, `daily at 06:00`, `weekly mon 07:00` — fired from the drain loop, defaults declared next to `@job` |
| Door | `from pipeline import DAYS, FINAL, FORMS, JOBS, MAX_RESULT, NULL, SCHEDULES, TASKS, TERMINAL, BadCadence, Batch, Bound, Cadence, Cancelled, CascadeProgress, ChildFailed, Declared, Handler, Job, JobQueue, JobState, Jobs, JobsSnapshot, Listener, Notice, NoticeEntry, NoticeKind, NoticeLevel, NoticeListener, ResultTooLarge, Retry, Run, RunCancelled, RunContext, Schedule, ScheduleListener, Spec, Tally, Task, TaskProgress, declare, fan_out, job, local_zone, notices, spawn` |
| Copyable | yes, and that is the point — `cp -r pipeline yourproject/`. Nothing here imports `desk` or `newsdesk`, and `tests/test_architecture.py` fails the build if it ever does. See `docs/pipeline-package.md` |
The table *is* the queue, which is why a job survives a restart, why the UI's job list is a `SELECT`, and why a worker can live in another process.
## The Entities
**Job** — one physical unit of work, admitted to the queue and run on one worker. It has a resource class, so the queue knows which worker to run it on. A job is the only thing the queue runs, and the only thing that has a row in `jobs`.
**Handler** — the function that defines the job, aka the code. `@job` registers the handler in the queue. The handler is the piece of code which is called when the job is about to run.
**Task** — **plain code that creates jobs, and is never a job.** A business step comes in — fetch every feed, give every article a body — and is translated into jobs the queue can run, typically by cutting the work into batch payloads. A task mints a run id for itself, queues one job per payload tagged with that id, and queues one **final job** behind them with `after="done"` on all of them, which is how it comes to be reported on. Then it is over: nothing of the task is alive afterwards, only its jobs and the callback the final job carries.
Everything a task queues, it queues in one go. That is what makes it plain synchronous code and not a job: finding the work is a query, and a query is cheap enough to run inside the request that pressed the button, or in the worker thread that has just finished the job before. If the planning itself needs real work done first, that work is a job, and what follows it is the next task of a cascade.
**Cascade** — a series of tasks, for when each can only be planned once the one before it is over, or where there is a natural breaking point between them. Structurally it is the same shape as a task, and the pipeline does not distinguish the two. Both are rows in `runs`, both group jobs by an id, both are killed by the same call, and neither has an object of its own. The difference is a **convention**, and it is worth stating because nothing enforces it:
> A **task** queues everything it wants at once.
> A **cascade** is where the breakpoints are: it queues the next task only once the last one is over.
The final job of a task is what calls back to the cascade to start the next task.
Jobs are tied to a task and a cascade by `task_id` and `cascade_id`.
**Schedule** — **a row that queues a job, and is never a job itself.** "Enqueue this kind with this payload on this cadence": a name, a kind, a payload, a cadence, whether it is on, when it is next due and what it queued last time. It is a row rather than a timer for the same reason a queued job is: nothing is alive between runs, so a process restarted at 05:59 still fires the 06:00 one, and two workers on the file read one schedule instead of each holding a copy of it in memory. It queues *jobs*, not tasks — a schedule that wants a whole cascade queues the job that starts one.
**Cadence** — one of five wordings, and the only five: `every 15m`, `every 6h`, `every 2d`, `daily at 06:00`, `weekly mon 07:00`. Not cron, which is five fields of positional punctuation that nobody reads back correctly. The interval forms are elapsed time and ignore the timezone; the clock forms are wall-clock in `JobQueue(tz=...)` and are 06:00 on both sides of a daylight-saving change. The two mean different things and a person picking one has already chosen which.
**Declaration** — a default schedule an app ships, written next to `@job` with `declare(name, kind, cadence, payload, label)`. `SCHEDULES` is to it what `JOBS` is to `@job`. On start, a declared name with no row gets one and a name that has one is not touched, so the declarations are where a schedule *comes from* and the table is what it *is*.
## How a job has to look
```python
def handler(config, store, *, context=NULL, limit=None) -> dict: ...
```
`config` and `store` are the app's, passed through untouched. Payload keys arrive as keyword arguments — **declare the ones you take**; an undeclared key is a `TypeError` recorded against the job, and `**kwargs` would swallow a typo. `context` is optional: a handler that takes neither progress nor cancellation says so by not declaring the parameter, and `registry.py` works that out once at registration rather than inspecting a signature per run. One payload key is reserved — a job whose payload names `context` fails, because the worker's own is what goes there.
`limit` is a **convention, not a contract**. Nothing in the pipeline reads it. It is an ordinary payload key that most of newsdesk's sweeps happen to accept, so that a person can ask for a short run of one, and a handler is free not to have it.
The return value should be a JSON-serialisable dict, and is stored as the job's result. A handler that returns something else — `None`, a list, a count — is not an error: the worker keeps the ending and stores an empty result, so the job is `done` with `{}`. Which means a result nobody can read is silent, and worth knowing when a handler's tally goes missing rather than its job failing.
**An exception ends the job in `failed`.** The traceback goes to the log, the exception is stored on the row as `Type: message`, and a `job_failed` notice goes to the screen saying it looks like a bug — *unless* the handler already announced a notice about that same failure, in which case the handler's sentence stands alone rather than being contradicted underneath. The `except` is deliberately broad: a handler is a source plugin, a model call, a subprocess, and anything it raises is that one job's failure and nothing else's. Letting it escape would take the worker down and strand every job behind it.
## Minimum example
```python
from pipeline import Jobs, JobQueue, job, RunContext, NULL
@job("greet", "Saying hello", solo=True, resource="cpu")
def run_greet(config, store, *, context: RunContext = NULL, name="world") -> dict:
context.step(0, 1, f"greeting {name}") # progress + cancellation check
return {"greeting": f"hello {name}"}
run_greet(cfg, store, name="wolf") # 1. straight call — no queue at all
queue = JobQueue(Jobs(cfg.database), cfg, store) # 2. queued
await queue.start()
queue.enqueue("greet", {"name": "wolf"})
```
`@job` returns the function unchanged, so every job stays importable and callable on its own. Registration is by import — importing `newsdesk.jobs` fills `JOBS`.
## Workers
`JobQueue.WORKERS = {"llm": 4, "http": 8, "cpu": 1}` — how many workers of each kind there are, and so how many jobs of that resource run at once. An app passes its own; newsdesk has a `workers` section. A resource named in neither gets one. Counted from the table, not from this process, so a second worker on the same file shares the four model workers rather than helping itself to four more.
The count belongs to the worker, not the job kind: tagging and summarising both ask a model, so they share the same four.
Our queue is fanning out threads not processes. Therefore, a worker is a thread, not a program.
## `depends_on`
A job may name jobs it waits for, and say what it wants of them.
| | |
|---|---|
| `after="ok"` | admit only if all of them succeeded; fail with them otherwise. The default, and what one job needing another's output means. |
| `after="done"` | admit once all of them are over, however they ended. |
Waiting costs nothing: `claim` passes the job over and it stays queued. No timer, no listener, nothing holding a place, nothing lost to a restart.
`done` exists for the job that reports on the others — success-only semantics would mean the report never exists when there is something to report. An id naming nothing is satisfied.
A **cancel** and a **failure** are the same thing here: both travel to the `after="ok"` jobs waiting on them, and neither holds back an `after="done"` one. Killing a job is dropping that batch, not stopping the run it belongs to, so the job that reports still runs and counts it among the ones that did not land.
**What a waiting job is handed.** `context.waited_for` is a tuple of `pipeline.models.Job` — the whole row for each job this one named, oldest id first, and empty for a job that waited for nothing. Each carries its `state` and its `result`, which is everything a report needs. Read at claim time off the jobs themselves, never counted up as they landed, so there is no tally to keep in step and nothing to lose to a restart half way through. An id that named a job which no longer exists is simply absent from the tuple, so `len(waited_for)` is what was found, not what was asked for.
## `solo`
**One job of this kind queued or running at a time.** Two tabs pressing Fetch is one pull, and the second caller is handed the job already going rather than an error, because what they asked for is that the work be under way and it is. Enforced by a partial unique index over the unfinished states, so it holds across every tab, every client and every process at once — not by a flag in whichever browser happened to ask.
**Off unless a job says otherwise.** A sweep that forgets to say `solo=True` runs twice and wastes a minute. A batch kind that forgets to say `solo=False` has seven of its eight batches quietly handed the first one, and the work simply never happens. The default belongs on the side that fails loudly.
A job that names `depends_on` never dedups, whatever it declared, since the whole point of a batch is that there are several.
**The job you are handed back may not be yours.** Dedup matches on the kind alone, so a duplicate is answered with whatever live job holds that key — which can belong to another cascade entirely. Press Fetch while an overnight pull is part way through its own `pull` job and you are handed *that* job: the right answer to "is a pull under way", and the wrong one to "is *my* pull under way". Nothing rewrites its `cascade_id`, so a caller that means to follow the run it started has to notice the ids it got back are not the ones it minted.
## Retrying
```python
@job("extract", "Reading the page", resource="http",
max_attempts=3, retry_delay=60, retry_backoff=2.0)
def run_extract(config, store, *, context=NULL, url=""): ...
```
**`max_attempts` counts the first attempt**, so `1` is the default and means what every job here meant before the column existed: run it once, and an exception is the end of it. An attempt that fails with attempts left goes back to `queued` with a `run_after` on it, `retry_delay × retry_backoff ⁿ` seconds hence.
**Waiting costs exactly what `depends_on` costs, which is nothing.** The wait is a column, not a sleeping worker: `claim` passes the row over until its time comes round, so no worker, no thread and no lease is spent on an hour's backoff, and the poll that exists to notice another process's enqueue is what notices the hour is up. Sleeping in a handler would hold a worker for the duration, and is the thing this exists instead of.
**And because it is a column it survives the process that wrote it.** A retry scheduled for 09:40 is taken at 09:40 by whichever worker is alive then, including one started at 09:38 in a different process. There is nothing to hand over, because nothing was alive to hand.
**Three endings, and `Jobs.retry` picks between them.** `cancelled`, if a Kill landed while the attempt ran — a retry that put the job straight back would be a button that stopped nothing and said it had. Back to `queued`, if there are attempts left. `failed`, if there are not, and `Job.exhausted` says so from then on: the policy ran its course and the work still did not land, which is the only one of the three where trying again is a person's decision. A job that failed at `max_attempts=1` is not exhausted — nothing was ever used up.
| | |
|---|---|
| `job.waiting` | queued, with its next attempt still to come |
| `job.exhausted` | failed on the last attempt it was allowed, rather than the only one |
| `job.attempt` | how many times it has been *started*, stamped by `claim` in the statement that takes the row |
| `jobs.waiting()` | every row in the first of those states, oldest first |
| `jobs.blocked()` | everything `claim` would pass over — waiting on a time, or on another job |
**A handler can name the wait but not the bound.** `raise Retry("rate limited", after=120)` puts the other end's own answer in place of the backoff, which is worth having because the alternative is a guess: a minute is either a minute of doing nothing when the service was ready, or another refusal charged against the bound. It buys no extra attempt — a job queued with one attempt that raises `Retry` fails exactly as it would on any other exception. The bound is ours to declare and the wait is the world's to name.
**A retry says nothing to the screen.** A notice is for what a person has to act on, and while attempts remain there is nothing to act on: the queue is already doing what they would have asked for. The log has every attempt either way; only the one that runs out announces.
**Retrying is off by default**, for the reason `solo` is off by default. A fetch that should have been retried and was not costs one item until the next sweep finds it again. Work retried three times that never should have been is three times the writes, or three times the bill.
`queue.enqueue(kind, delay=600)` schedules one job for later with no retry involved at all. `queue.enqueue(kind, max_attempts=n, retry_delay=s, retry_backoff=b)` overrides what the kind declared for one piece of work, and `dedup_key=` overrides the kind's own key — which is how one *item's* acquisition is made unrepeatable while the kind stays freely repeatable, so two presses of a reader's retry button collide in the unique index rather than in one browser tab.
**A batch can carry its own bound.** A task's finder may return `Batch(payload, Bound(max_attempts=3, delay=300, backoff=2.0))` instead of a bare payload, and those numbers go onto that job's row. That is where a plugin's declared retry policy lands: newsdesk cuts its body batches per plugin profile precisely because the row holds one bound, so a batch mixing two outlets could only carry one of their declarations. A finder returning plain dicts — every task that existed before this — gets `Bound()`, which is one attempt and no wait. The direct path (`task.run`) ignores bounds entirely: a bound is a number the queue enforces by putting a row back with a time on it, and there is no row.
## Typical way of dealing with batches
A task queues its batches as independent jobs, and one **final job** behind them. It depends on all of them with `after="done"`, so it runs however they ended, and it is handed each one's state and result in `context.waited_for`. It is the only job of the task that calls back.
A task that finds nothing to do is the ordinary case, not the empty one: it queues no batches and its final job runs immediately, with an empty `waited_for`, and moves the cascade on. Which is why a switch turned off, a backlog already cleared and a stage this pull does not want are all the same thing here — a finder that returns no payloads — and none of them stops the chain.
**The callback is a name in the final job's payload, not a function.** Nothing in a table can hold a function, and a name is what makes the callback survive a restart and be callable from another process; the worker running the final job resolves `module:function` to imported code. `newsdesk.jobs.cascade:step` reads as documentation in the payload, and carries its own import, where a registry would have to be filled before the worker looked in it.
It is called with the task's tally and whatever the task fixed at the time it was made:
```python
def step(config, store, tally: Tally, payload: dict, *, context: RunContext) -> dict | None: ...
```
`payload` is `Task.carries`, stored in the final job's row, so it survives a restart and a worker in another process. Whatever the callback returns is merged into the final job's own result; `None` is fine. A task with no callback stops there, which is what being asked for on its own means.
**`settle` is a second name in the same payload, resolved the same way, and it belongs to the task rather than to what follows.** The final job runs it before the callback, over the same tally. It is for the decision a batch is in no position to make: a batch is one of forty and cannot see what the other thirty-nine are doing, so anything that must hold across the whole run — a cap on what the run may add, a merge of what several batches each asked for — is worked out once, here. Newsdesk's tagger is the one that has one: a batch proposes new tags and adds none, and the settle merges the proposals, takes at most `max_new_tags_per_run` of them and goes back to tag the items that asked. A task started on its own has no callback and still settles, because settling is its own work and not a message to anybody.
**`Tally`** is how the batches ended, read off their rows at the moment the final job runs:
| | |
|---|---|
| `done` `failed` `cancelled` | how many batches ended each way, and `jobs` for the sum |
| `results` | what each batch returned, in the order they were queued |
| `killed` | whether the **task itself** was killed |
`killed` is the one that cannot be inferred from the counts, and is the reason it is there. A task killed while its batches ran leaves every one of them `cancelled`, and so does a person killing every batch by hand — identical numbers meaning opposite things: *skip this step*, or *drop this work*. Only the `runs` row knows, so the final job asks it. What to do about it is the cascade's decision and nothing here presumes it; newsdesk's pull skips the stage and carries on to the next.
## Passing data
Three ways to pass data. Can be mixed and merged.
1. **Store.** A job reads its input or persists its output directly into a storage.
2. **Parent.** A job receives data as a parameter, put in its payload by whatever queued it — a batch's uids, a cascade's options. Fixed at queue time, and the only kind of data a job that runs first can hand to a job that runs later.
3. **Cross job via `context.waited_for`.** The `after="done"` job gets the jobs it waited on: their `state` and their `result`. Read, never pushed — nothing is counted while the work runs, so there is no tally to lose.
The usual arrangement is the third feeding the callback: the final job reads how its batches went and hands the tally to the cascade, which turns it into the next `Task`.
`context.upstream()` is the third one merged into a dict, later parents winning, which is what a chain of one parent wants to read like. It is `{}` under `NULL`, which is what keeps a chained handler callable by hand:
```python
@job("extract")
def extract(config, store, *, context=NULL, html: str = ""):
html = html or context.upstream().get("html", "")
```
From the queue nobody passes `html`; from a terminal you pass it yourself. A handler reading `waited_for[0].result` directly would be the one kind in the pipeline you cannot simply call.
**The rule is that whoever has the data writes the row.** Never insert a row meaning an earlier job to fill its payload in later: a worker claims it the moment the parent is `done`, so it can be claimed between that write and its commit; `dedup_key` has to be known at queue time; and `max_attempts` re-runs it from its own row. Reading a parent has none of that, because the row was complete when written.
Bulk data belongs in the store. `Tally.results` is a query across the children's `result` TEXT columns and those columns never shrink — counts, ids and errors are free, a few hundred large dicts are not.
## `fan_out` — N jobs and the one that reports on them
```python
@job("body")
def body(config, store, *, context=NULL):
context.fan_out("body_batch", [{"uids": c} for c in chunks],
then="newsdesk.jobs.cascade:step")
```
One call, one transaction. The reporting job is `FINAL`, the pipeline's own kind, and an app never writes one. The children go into a run of their own inside the caller's cascade, which is what lets the callback fan out again — step N queues step N+1, and that is the whole of what a cascade is.
Because the thing fanning out is itself a job it has a context, so this works from inside a handler as well as from a request. `Task` is the same three statements with a finder in front, which is why a task and a cascade are one shape and one primitive covers both.
`()` under `NULL`: a handler called directly is being run by somebody standing there, and the useful answer is to do its own work and report that nothing was queued behind it.
## Running it
```python
queue = JobQueue(Jobs("app.db"), config, store) # then, in an app:
connect(app, Jobs("app.db"), config, store) # pipeline.web: routes, socket, element, lifespan
queue.drain(timeout=60) # in a script: run what is queued, stop
```
```bash
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
```
The worker is the same `JobQueue` with no web app, on the same table, and may run beside the web app's own. `--until-idle` drains and exits. The bootstrap is the one `isolate=True` uses.
## Ending work that will not end itself
Cancellation is cooperative because the work is synchronous — but two kinds of work never look up, and for those there is a real kill.
**A command.** `context.spawn(argv, timeout=...)` runs it, polls for Kill and for the deadline, and kills the **process tree** on either — `taskkill /T` on Windows, `killpg` elsewhere, which is why the child is started detached. `proc.kill()` alone would leave the ffmpeg a shell started still holding the core. Blocking a worker thread on this costs nothing: the thread is only sleeping. Output goes to temporary files, not `PIPE`, because a child that fills the pipe buffer while the loop polls is a deadlock. A non-zero exit is handed back rather than raised.
**Python that cannot be interrupted.** `@job(..., isolate=True, timeout=600)` runs the job in a child process, which is the only thing that can be killed mid-`sklearn.fit`. Nothing is pickled: a job row is already a complete description of the work, so the child is handed `--db` and `--job` and reads the same row. It writes *progress* to that table — which is how its progress reaches the browser with nothing forwarding it, the snapshot being a `SELECT` — and hands its result back through a file. The worker still writes the ending, so there is exactly one writer of the terminal state, as before.
`isolate=True` needs `bootstrap="module:function"` on the queue, returning `(config, store)`: neither survives a process boundary, so both are rebuilt in the child, and the same import is what fills `JOBS` there. `bootstrap_arg` is one string passed to it — an app's config path, which the child cannot learn any other way.
**`timeout` without `isolate` raises at registration.** A deadline over a thread could only mark the row failed and walk away — the work would carry on holding the core, and two things would intend to write one row. The refusal names both ways out.
## `context.task` — a job asking for what comes next
Newsdesk's body task fetches each eligible article and passes its HTML to `desk.tools.article.extract_article`. Its storage writer saves structured content, format, provenance and a derived plain-text projection atomically; existing enrichment jobs continue to read the generated `body` view. Empty or late fetch results preserve previously acquired content. The body job still uses the existing wall detector and candidate rechecks, and does not yet execute plugin-declared acquisition *graphs* — one declared step, the one whose tool is `extract_article`, is what it runs. Its declared **retry policy** is executed: a failure is classified with `desk.tools.failures`, the plugin decides whether that category is worth another attempt, and the batch raises so the queue puts it back under the bound its row carries. Batches are therefore cut per plugin profile. A second attempt asks the table again and so re-fetches only what is still missing, and never overwrites text that has since arrived. Structured bodies are retained without the former 60,000-character clipping; downstream model tasks remain responsible for their own input limits.
Hacker News discovery now runs as the `hacker_news` outlet through shared RSS fetching. Its item destination stays the publisher URL, so the subsequent body task extracts that article and records that URL as provenance. Its supplied discussion link is a separate typed input, and feed points/comment statistics stay in source evidence rather than the summary or preview. Its declared tagging policy runs: the destination article is what the tagger reads, and a link whose page would not extract is tagged from its title. Its summary policy stays off — the summariser selects only profiles that ask for an abstract, and nobody has decided that a front-page link earns one.
A handler that discovers work only by doing its own can name the task that should follow it:
```python
context.task(cascade.task("body", cascade="pull"))
```
**Recorded, not queued.** The worker starts it after this job lands in `done`, on the same thread that ran the handler, and hands it this job's `cascade_id` so its batches land in the run this one belongs to. Three consequences worth knowing: a task never outlives the failure of the job that described it, a handler still cannot reach the queue, and a job's row lands a moment *before* the jobs that follow from it exist.
It returns whether anything took it — `False` under `NULL`, where there is no queue. A callback invoked from a notebook or a Claude Code session gets that answer and has to do the work itself.
## `Cancelled` and `RunCancelled`
Two exceptions with similar names, and only one of them is a handler's business.
**`Cancelled`** is raised by `context.check()` inside a handler when this job has been asked to stop. The worker catches it and records `cancelled` rather than `failed`. A handler should let it travel; catching it is how you build the thing the Kill button cannot kill. A handler that never checks is not lost either — the worker asks once more when the handler returns, and a job asked to stop while it ran ends `cancelled`, keeping whatever it made and queueing nothing.
**`RunCancelled`** is raised by `Jobs.enqueue` for a job whose `cascade_id` or `task_id` names a killed run, except for an `after='done'` job under a killed *task* — the same carve-out `cancel_run` and `stop_task` make, so a kill landing while a task is still fanning out its batches cannot take the final job with it. A killed cascade refuses it too, which is what "nothing reports and nothing follows" means. It is not a handler's business and a handler will not see it: `JobQueue.enqueue` catches it, logs it, and returns **`None`** instead of a `Job`. So a caller that queues a job has to expect no job back — `Task.start` skips the id it did not get, and a `depends_on` list is that much shorter. Nothing is told that a run is over; a callback tries to queue the next task and the table says no, which is the same answer across processes, after a restart, and when the kill landed mid-run.
## Ids
Every job has an `id` of its own, and carries a `task_id` and a `cascade_id`. Both are grouping ids: nothing depends on a task, but everything belonging to one can be found by it and killed together. Two of them rather than one because they are two groupings and both have a button.
**Killing a run** is one operation for a task and a cascade alike — `cancel_run`. Three things: flag the row, end its queued jobs, ask its running ones. And a fourth that is not an operation at all: **`enqueue` refuses a job whose `cascade_id` or `task_id` names a killed run**, which is what stops a callback from reviving a killed task or moving a killed cascade on.
Three buttons, three sizes of stop: killing a **job** drops one batch, killing a **task** skips one step, killing a **cascade** ends the run.
**Stopping a run is not killing it.** `Jobs.stop_task` ends a task's queued batches and nothing else: the row is not flagged, so `Tally.killed` stays false and nobody is told a person pressed anything; running batches keep their requests and their own endings; the final job still reports. It is what a handler reaches through `RunContext.stop(why)`, for the failure that is the backend's answer to the whole run rather than to one batch — a spend limit, an expired login, no model on the machine. The reason is written to `error` on each cancelled row, so a batch that never ran says on screen why it didn't. `desk/modeltask.py` is the one caller, and `notices.stops_the_run` is what decides.
The one asymmetry between killing a `cascade` and killing a `task` is what the last two of those mean. Kill a **task** and its final job is spared: it still runs, and what it reports says the *task* was killed rather than only how many of its batches ended cancelled — see `Tally.killed`. Kill the **cascade** and the final job dies with it, so nothing reports and nothing follows — the one stop no cascade gets a say in, because there is nothing left of it to ask.
cascade_id = jobs.new_run("cascade", "pull") # before a single job exists
jobs.of_cascade(n) · jobs.of_task(n) · jobs.run(n) · jobs.cancel_run(n)
jobs.cascades()
POST /api/cascades/{id}/cancel · /api/tasks/{id}/cancel · /api/jobs/{id}/cancel
A job queued with neither begins both: something asked for it on its own, so it is a cascade of one task of one job. A task started with no cascade does the same, for the same reason.
The cascade that comes of it says so in its name — `pull on its own` rather than `pull` — because a chain named `pull` is a different run, and `live_run("cascade", "pull")` is how a caller asks whether that chain is already going. Answering it with a lone job would be a Fetch button that starts nothing and greys itself over a run that is one job long. The task keeps its plain name (`pull`), since the task really is that task however it was started, and the strip draws its box from that.
## Schedules
The tick is in the drain loop, at the top of `_turn`, before the claim — not in a thread of its own and not on the idle path with the reclaim and the purge. Not a second thread, because then two things would race to enqueue one row. Not on the idle path, because reclaim and purge are repairs that can wait for a quiet moment and a schedule is *work*: on a queue that is never idle, 06:00 would never come round. The check itself is a clock comparison, so paying it every pass costs nothing. It also runs once in `start()`, after reconciliation, so a default declared today fires on the start that introduced it.
**Overdue fires once, not once per slot missed.** A firing sets `next_run` from *now*, never by stepping through the times it was down for. There is one `next_run` column and nowhere for a backlog to accumulate even in principle, so an app off for three days runs its hourly job once when it comes back.
**No pile-up.** Before firing, two questions: is the job from the last firing still queued or running (`last_job_id`), and is this kind `solo` with one already under way? Either way `next_run` advances and nothing is queued. A job slower than its own cadence must not grow a queue of itself, and a `solo` kind somebody has already pressed must not be recorded as this schedule's firing.
**A row the code no longer supports keeps its place.** An unregistered kind, or a cadence nothing can parse (only reachable by editing the file by hand), is logged once per process and left exactly as it is — `next_run` included — so registering the handler again runs it rather than starting another cadence. `<desk-schedules>` marks the row `unknown kind`; deleting it is deliberate.
**Reconciliation never overwrites and never deletes.** `Jobs.add_schedule(..., only_if_new=True)` is the whole of it: one statement, so two workers starting at once insert a declared default once between them. A declaration removed from the code leaves its row behind, because a deploy that silently forgot a schedule somebody relied on is worse than an orphan that says so.
**One scheduler per file.** `JobQueue(schedules=False)` and `pipeline.worker --no-schedules` are what every worker after the first passes. The workers themselves race correctly — that is `claim` — but the pile-up guard reads `last_job_id`, which neither of two simultaneous firings has written when the other looks.
declare(name, kind, cadence, payload=None, label="") # next to @job
Cadence.parse(text) · str(cadence) · cadence.next_after(moment, zone)
jobs.schedules() · jobs.schedule(id) · jobs.add_schedule(...)
jobs.update_schedule(id, **fields) · jobs.delete_schedule(id)
jobs.due_schedules(now) · jobs.mark_fired(id, job_id, next_run)
queue.run_schedule(id) · queue.notify_schedules()
GET/POST /api/schedules · PATCH/DELETE /api/schedules/{id}
POST /api/schedules/{id}/run
## The three tables
**`jobs`** is the queue, and holds rather more than the states: the `payload` it was queued with and the `result` it ended with, both JSON; `done`, `total` and `message`, which are the progress bar and are written by `context.progress`; `error`, the exception string of a failure; `dedup_key`, `depends_on` and `after`; `resource`; `cascade_id` and `task_id`; the retry policy and where it has got to, as `attempt`, `max_attempts`, `retry_delay` and `retry_backoff`; and five times — `created_at`, `started_at`, `seen_at` (the lease), `run_after` (the next attempt) and `finished_at`. `cancel_requested` is a flag, because a kill is a request in the table rather than a message to whichever process happens to be running the job.
The policy is on the row rather than read off the job kind when a failure happens, for the same reason the payload is: the attempt that has to decide may be in another process, after a restart, running a version of the code where the kind's numbers are different or the kind is gone.
**`schedules`** is the standing instructions: `id`, `name` (unique, and the handle a declaration is matched on), `kind`, `label`, `payload`, `cadence`, `enabled`, `next_run`, `last_run`, `last_job_id`, `created_at`. `next_run` and `last_run` are UTC ISO stamps like every other time here, and `next_run` is compared as text in `due_schedules` for the reason `run_after` is in `claim`: `''` sorts before every stamp, so "never" and "not due" are one comparison with no coalesce in it. A cadence worded in a local timezone is turned into one of these before it is stored, so `pipeline/schedule.py` is the only file that reads a zone at all.
**`runs`** is where a cascade's id and a task's come from, and it is deliberately almost empty: `id`, `kind` (`cascade` or `task`), `name`, `label`, `created_at`, `cancelled_at`.
What it does **not** have is the point of it. No state, no progress, no counts, no `finished_at`: the jobs carrying the id are the record of what happened, and a second copy of that could only go out of step with them. `cascades()` tallies a run by reading its jobs. So the table exists for exactly two facts its jobs cannot supply — that a number was handed out *before* there was a job to take it from, and that somebody killed this run. `name` carries a third job: it is how `live_run` answers "is one already going", which is why a cascade minted only because a lone job needed an id is named apart from a chain somebody asked for (see `on_its_own`).
## `ModelTask` — next door, not here
`ModelTask` left this package: it imports `desk.config` and `desk.tools.agent`, so it was never the queue's. It is `desk/modeltask.py` now, and this section stays because it is still the thing most jobs in newsdesk are.
`desk/modeltask.py` is one layer above `desk/tools/agent.py`. `agent` answers *how do I reach a model*; this answers *what do I send it and what do I do with what comes back*, once, for every task that needs one. Tagging and summarising are the two, and what differs between them is all content: which rows are waiting, what goes in the batch, what the model is told, what shape it must answer in, what to do with the answer, what the run has to settle once every batch is in, and how a row that never gets one is given up on. That set of callables **is** a `ModelTask`; the module is everything that is not one.
`modeltask.of(TASK)` turns it into an ordinary `Task`, so it is queued, watched and killed like any other stage. `modeltask.one` is what a single batch job does: check for a kill, build the payload from the uids it was given, ask, apply. Two things about it are worth reading before changing:
- **The prompt is built when the batch runs, not when it was handed out.** Tagging's carries the vocabulary as it stands at that moment, so a tag somebody adds by hand mid-run reaches the batches still to come. It does not carry what a sibling batch proposed, because a batch proposes and adds nothing: the run's own additions are worked out once, in `settle`.
- **`pending`, `waiting` and `build_batch` are given the config as well as the store.** Which rows are waiting is no longer a question a table can answer on its own: in newsdesk both backlogs are whatever the installed plugins declare about their own items, and the plugins are found through the config. Both tasks now use it; the argument is still there for one that does not.
- **A kill during the call does not throw the answer away.** `check()` is called before the model is asked, so a Kill pressed while the job sat queued makes no paid call at all. Once the request is in the air the money is spent, so the answer is applied and the worker records the cancellation at return. The asymmetry is deliberate and looks like an oversight otherwise.
**A row is given up on eventually.** "The next run finds the work again" is unbounded, and for a row no model will ever answer usefully — a segment whose only honest abstract is one sentence, a uid dropped from every reply — unbounded means paying for the same call on every pull for as long as the row exists. So a row handed over and left unanswered is charged a turn through `ModelTask.missed`, and one charged three of them stops being offered until a person asks for it deliberately. A batch that *failed* charges nothing: it never reached a model, so it proved nothing about its rows. Every `ModelTask` must supply `missed`, and there is no default, because the default is the bug.
**A refusal is the one failure that is about a row.** The platform runs a classifier over the request and rejects the whole thing at once, naming nothing — and a news archive meets it regularly, because reporting on a worm or a supply-chain compromise is written in the vocabulary it watches for. `one` answers it by asking again in halves (`_narrow`) until the batch is the row, which is then charged and leaves the backlog after three. Halves rather than one at a time: two flagged rows in a batch is ordinary, since they arrive from the same day's news, and stepping through twenty singly would pay eighteen calls for what four find. The notice is `agent.refused` and is deliberately not `standing` — the account is fine and the next batch is about different rows.
**What was given up on is listed, and said once.** A refused row is charged with `why="refused"`, kept in `raw.why.<task>`, so the list can tell a refusal from a row the model left out. The strip interrupts only for given-up rows without `raw.seen.<task>`; its × or opening Jobs & history sets that mark on the server, so a reload does not say them again and a row given up on later still does. Try again there reopens a row with one attempt (`reopen_given_up(..., attempts=1)`) and queues its `_batch` job at once, so a row refused again is back on the list after one call rather than after three pulls.
## `notices`
**Two halves since the extraction.** `pipeline/notices.py` is what the queue itself needs — the notice a failed job gets, the check that stops it being said twice, the rendering a terminal wants. `desk/notices.py` is newsdesk's vocabulary: `Rule`, `RULES`, `classify`, `stops_the_run`, `source_failed`. An app writes its own, because a useful sentence names things only the app knows.
`pipeline/notices.py` turns a failure into one line worth putting on screen. A handler calls `context.notice(...)`, and the queue writes it to the `notices` table and pushes it to whoever is watching — the counterpart to `log.exception`, not a replacement for it: log the traceback for afterwards, raise a notice for now.
**A notice is stored, one row per `(code, key)`.** The first design stored nothing, on the argument that a table would duplicate the log. That assumed somebody was watching when the notice was raised; a scheduled pull and a reader on a phone mean nobody usually is. `Jobs.record_notice` upserts the row and returns it as a `NoticeEntry` with `count`, `first_at` and `last_at`, and that row is what goes over the socket. `count == 1` is news and anything above is a repeat, so a client interrupts for the first and only counts the rest — a refused article is announced on the pull that met it, not on the two retries after. Every notice still reaches `on_notice`; what interrupts a person is the interface's decision.
A row goes when it is cleared (`DELETE /api/notices/{id}`, or `/api/notices` for all), after which the same failure is news again, or when `JobQueue.NOTICE_QUIET` (14 days) has passed since it was *last* said. `GET /api/notices` lists the log, newest first. A write that fails is logged and the notice is pushed anyway: it never costs the job that raised it. The table holds no traceback and no per-occurrence history; those stay in the log.
A notice carries three things beyond its text: `kind`, which says whether this is the world misbehaving or a bug of ours — the fork in what you do next, and exactly the guess you should not have to make from an error string; `fix`, the command when there is one; and `key`, which with `code` names the row in the log — so a feed that has been failing all week is one line with a count rather than one per pull, and two refused articles are two. `classify` is the interface, and `RULES` is the catalogue it reads, which doubles as the list of what this will ever interrupt you about. Everything in the catalogue is marked `expected`; the fallback is marked `bug`, which is the whole point of enumerating them — a failure nobody wrote a rule for is one to go and read about.
For a job author, two rules. Use a notice for what a person has to act on, not for what the program can get on with. And classify your own failure if you can: the worker announces "looks like a bug" for every job that ends `failed`, and `already_said` is what stops it contradicting a handler that had already explained itself.
A rule also says whether the failure it names is `standing` — caused by the account or the machine rather than by this batch, so every sibling of the run would be told the same thing. `stops_the_run(notice)` is that answer, and a task for the model cancels what is still queued rather than paying for twenty-two more refusals. A limit, an expired login and an unreachable model are standing; a timeout, an oversized answer and a reply that was not JSON are not.
## Timing
Seven numbers, all on `JobQueue`, plus one on `Jobs`. None of them is tunable and none is meant to be; they are here so that "why did that take a minute" has an answer.
| | | |
|---|---|---|
| `IDLE_POLL` | 1 s | How long to sleep before looking again when the table was empty. Only reached when nothing woke us — an enqueue in this process is instant, so this is what notices one made by another, and what notices a scheduled attempt whose time has come. |
| `PUSH_EVERY` | 0.25 s | Fastest a running job's progress may reach the browser. The write is not throttled; the push is. |
| `CANCEL_CACHE` | 0.25 s | How long `check()` caches its answer. It reads the table so Kill works across processes, and a handler calling it in a loop would otherwise turn one button press into a query storm. |
| `HEARTBEAT_EVERY` | 5 s | How often a worker restamps `seen_at` on the jobs it holds. |
| `STALE_AFTER` | 30 s | How long a `running` row may go unsaid-about before another worker reads it as abandoned. |
| `STOP_GRACE` | 5 s | How long a shutdown waits for handlers already in a thread. |
| `SCHEDULE_TICK` | 30 s | How often the loop looks for a schedule whose time has come, and the resolution the cadences promise — which is why `pipeline/schedule.py` has no seconds unit. |
| `Jobs.LINGER` | 15 s | How long a finished cascade stays in `cascades()`, so a run you watched is still on screen the moment after it lands. |
`HEARTBEAT_EVERY` and `STALE_AFTER` are far apart on purpose: the gap is how many beats may be missed to a stalled loop or a slow disk before live work is re-queued underneath a worker that is still there, which is the one mistake this must not make. And because reclamation runs at most once per `STALE_AFTER` on the idle path, a crashed worker's job comes back after up to about a minute rather than after thirty seconds.
## Limits
- **Cancellation in a thread is cooperative.** No `context.check()` in the loop, no Kill. A running handler in a thread is never interrupted, not even on `stop()`, because Python cannot kill a thread from outside. Work that might need killing runs in a process: `context.spawn` for a command, `isolate=True` for Python. See "Ending work that will not end itself".
- **A shutdown waits `STOP_GRACE` for its handlers, then walks away.** A thread cannot be cut off, only waited for or abandoned, and both are wrong as an absolute: waiting is a reload held up by a model call, abandoning is a handler still writing to a file the process considers finished with. Most handlers use the moment and end properly; the rest are left running, and land their own ending in the table whenever they reach it.
- **A shutdown hands its leases back, a crash cannot.** `stop()` expires the lease on whatever was still going when the grace ran out, because a process on its way out knows outright what silence can only imply. So a restart re-queues that work at once instead of a window later, and only a real crash pays the wait. Expired rather than re-queued, so a handler thread outliving the shutdown still writes its own ending rather than being overwritten by a second run of itself.
- **A job's ending is written by the thread that ran it.** The handler call and the terminal `done` / `failed` / `cancelled` are one function on one thread; the loop's part is claiming the job and pushing the new state out afterwards. Written back on the loop, the ending would be the first thing a `stop()` costs: cancelling an abandoned job's coroutine cancels the write, while the handler itself carries on to the end regardless — so a reload landing mid-run would leave finished work looking unstarted, and the next start would run it a second time. For a batch that asks a model that is a second bill for something already paid for. The task a job named with `context.task` is queued by that same thread, for the same reason and still only once the job has landed `done`. Which is the one thing to know when reading the table from outside: a job's row lands a moment before the jobs that follow from it exist, so an empty `running` and `queued` is not by itself the end of a run.
- **A kill outlives the crash that interrupted it.** A job asked to stop, whose worker then died, is reclaimed to `cancelled` rather than back to `queued`. The request is in the table exactly so that it survives the process that was meant to act on it.
- **A dead worker is noticed late, on purpose.** A `running` row carries `seen_at`, stamped by `claim` and renewed every `HEARTBEAT_EVERY`. SQLite reads and writes run off the event loop so they do not delay its heartbeat. A lease expires after `STALE_AFTER`; reclamation runs at startup and at most once per `STALE_AFTER` on the idle path, so recovery can take about a minute. A worker excludes jobs it still holds from reclamation, renews their expired leases and warns that its loop stalled. Another process can still reclaim an expired lease.
- **Every handler must be safe to run twice.** Not advice — the rule the rest of this depends on. A reclaimed lease, a shutdown that abandoned a thread, a second process on the same file, an attempt that failed half way through: each of them can put a handler on the same rows again, and the queue does not promise otherwise. So write for it. Read what is still waiting rather than trusting the uids you were handed, upsert rather than insert, and write each unit as it lands so a re-run has less to redo. `max_attempts` bounds how often that happens on a failure; it is not something to lean on, because the paths that re-run a handler without asking it are the ones that were always there.
- **Handlers must be sync.** An async handler is not awaited.
- **A crashed worker holds its slots until it is reclaimed.** `claim` counts `running` rows, so the survivor cannot use a crashed worker's slots until the lease expires and its next reclamation pass runs, about a minute in the worst case with a responsive loop and database. A stalled worker never re-queues its own held jobs. Socket sends run concurrently with a one-second timeout per subscriber, so an unresponsive reader cannot delay drain wake-ups indefinitely.
- **Handlers run in a pool of the queue's own**, sized to the worker counts, because the interpreter's shared pool is sized from the cpu count instead. On a machine smaller than `workers` the shared one would run fewer jobs than the table admits and, worse, queue the loop's own claim and heartbeat behind the handlers, which is a worker starving itself of the thread it needs to hold its leases.
- **Progress is a write per call.** `context.step` is one `UPDATE` in its own transaction, on its own connection. The push to the browser is throttled; the write is not, because the row is the record and a second process reads it. Report per batch rather than per row in anything with thousands of items.
- **One bad turn does not stop the loop.** A locked database is logged and retried on the next pass rather than ending the drain task, because a worker that died silently looks exactly like a healthy one with nothing to do.
- **`depends_on` names jobs, not tasks.** A job waits on ids it was given; a task id is for finding and cancelling, not for waiting.
- **No priorities.** Jobs are claimed oldest first within a resource; there is no way to jump the queue but to be queued earlier.
- **Five cadences, not cron.** `every 15m`, `every 6h`, `every 2d`, `daily at 06:00`, `weekly mon 07:00`, and nothing else — no seconds, no day-of-month, no "every weekday", no ranges. Anything the five do not cover is a job that decides for itself whether today is its day, or cron running `pipeline.worker --until-idle` as before.
- **A schedule is not an alarm.** It is checked every `SCHEDULE_TICK`, so "06:00" means within half a minute of it, and a machine asleep at 06:00 runs the job when it wakes rather than at 06:00. What it will not do is run it once per slot missed.
- **History is purged.** Finished jobs older than `JobQueue(retain=...)`, 30 days by default, are deleted hourly while idle. Kept regardless: unfinished jobs, the newest ending of each kind in each state (what `last(kind)` reads), and runs still pointed at. `retain=0` keeps everything.
- **A result is at most 64 KB of JSON, and a bigger one fails the job.** Past `MAX_RESULT` the job ends `failed` with `ResultTooLarge`, the size and the keys it returned, a notice goes out, and nothing of the result is written. It is not retried, since the same handler returns the same result. Every snapshot and tally reads results back, so data belongs in the app's store; raise `MAX_RESULT` if a project really needs bigger results.
- **A task for the model gives up on a row eventually.** See `ModelTask` above: three unanswered turns and a row stops being offered until a person asks for it deliberately (`reset_tagging` with no slug, `reset_summaries`). `ModelTask.missed` is the seam and `status.py` says how many.
- **Migrations are `_reconcile`.** Columns are added, and dropped, by reading the catalogue on open, because `CREATE TABLE IF NOT EXISTS` leaves an existing table alone and `PRAGMA user_version` on that file belongs to the app.
- **SQLite means one machine.** Not a network queue.
+1135
View File
File diff suppressed because it is too large. Load diff
+195
View File
@@ -0,0 +1,195 @@
"""How a function becomes a job.
The whole contract is one line:
def handler(config, store, *, context=NULL, limit=None) -> dict
`@job` records that function against a name and **returns it unchanged**. That
is the point of this module rather than an accident of it: a handler stays an
ordinary function you can import and call.
run_pull(config, store) # sync, no queue, no event loop
run_pull(config, store, limit=5) # payload keys are just keyword args
Declared keyword arguments, not `**payload`: a key the handler does not name
is reported against the job, and a handler that swallowed everything would
turn a misspelt payload key into work that quietly did the wrong thing.
Which means a job is testable, reusable from a command, and reusable from
another job, without any of them going near the queue. The queue's entire
involvement is calling the same function with a real `context` and the stored
payload spread over it.
What that `context` *is* lives next door in `context.py`. Neither module
imports the handlers: they live in the app (`newsdesk/jobs/` for newsdesk),
import `@job` from here, and fill `JOBS` when they are imported.
"""
from __future__ import annotations
import inspect
from collections.abc import Callable
from dataclasses import dataclass
from typing import Any
#: A handler takes (config, store), keyword-only `context`, and whatever payload it
#: declares. It returns a JSON-serialisable dict, which is stored as the job's
#: result and shown when the job is done.
Handler = Callable[..., dict[str, Any]]
@dataclass(frozen=True, slots=True)
class Spec:
kind: str
label: str
fn: Handler
#: Whether `fn` actually accepts `context`. Worked out once, here, so the
#: worker doesn't inspect a signature per run — and so that "any function
#: with the right shape is queueable" holds for a function that wants
#: neither progress nor cancellation and says so by not taking the
#: argument. Passing `context` to one of those would be a TypeError, which
#: would surface as the job mysteriously failing.
wants_context: bool = True
#: When true, only one job of this kind may be queued or running at a time
#: — enforced in the database, so it holds across every client at once.
#: For the sweeps: one `pull`, one `rank`, however many tabs are open.
#:
#: Off by default, because the two ways of being wrong are not equal. A
#: sweep that forgets to say `solo=True` runs twice and wastes a minute;
#: a batch kind that forgets to say `solo=False` has seven of its eight
#: batches silently handed back the first one, and the work simply never
#: happens. The default belongs on the side that fails loudly.
solo: bool = False
#: What running one spends, and so which kind of worker runs it:
#: `llm` for anything that asks a model, `http` for a fetch, `cpu` for
#: work that wants the machine to itself. Declared here rather than worked
#: out, because the handler is arbitrary code and only its author knows.
resource: str = "cpu"
#: How many attempts this kind of work is worth, counting the first, and
#: how long to wait between them. Copied onto every job queued of this kind
#: — see `pipeline/table.py`, where the row rather than the kind is
#: what a later attempt reads.
#:
#: One by default, which is no retry and is what every job did before this
#: existed. Retrying is off by default for the same reason `solo` is: the
#: two ways of being wrong are not equal. A fetch that should have been
#: retried and was not costs one item until the next sweep finds it again;
#: work retried three times that never should have been is three times the
#: writes, or three times the bill.
max_attempts: int = 1
retry_delay: float = 0.0
retry_backoff: float = 1.0
#: Run this kind in a child process of its own rather than in a worker
#: thread. What it buys is the one thing a thread cannot give: the job can
#: be *ended*. A handler inside `sklearn.fit` or a tight Python loop
#: cannot be stopped, because no signal and no exception reaches a thread
#: in the middle of a C call — only a separate process can be killed.
#:
#: Nothing has to be pickled to make it work, which is why it is cheap
#: here and expensive elsewhere: a job row is already a complete
#: description of the work — a kind, a JSON payload, a database. The child
#: is handed those three and rebuilds the rest. See `pipeline/child.py`.
#:
#: The cost is that the queue needs a `bootstrap` able to build the app's
#: `(config, store)` from nothing, since neither survives a process
#: boundary. One function per app.
#:
#: Not for work that is expensive because of a *command* it runs —
#: ffmpeg, stockfish. That wants `RunContext.spawn`, which kills the same
#: way for none of the setup.
isolate: bool = False
#: Seconds this kind may run before it is ended, or 0 for no deadline.
#: Requires `isolate`, and `job` refuses the pair without it: a deadline
#: over a thread could only mark the row failed and walk away, leaving the
#: work still running and still holding the core it was killed for
#: holding — and leaving two things intending to write the same row. A
#: promise the pipeline cannot keep is worse than no promise.
timeout: float = 0.0
#: Payload keys the handler cannot run without: its keyword arguments
#: that have no default. A kind with any is not a button, because a press
#: sends an empty payload and the job would fail on the TypeError.
needs: tuple[str, ...] = ()
#: name -> spec. Explicit registration, like `routers/ROUTERS`: a new job is
#: a change to this project, and the cost of importing its module once in
#: the app's handlers module is one line. Empty until something imports
#: `newsdesk.jobs`, which every entry point does before it looks here.
JOBS: dict[str, Spec] = {}
def job(kind: str, label: str = "", *, solo: bool = False,
resource: str = "cpu", max_attempts: int = 1,
retry_delay: float = 0.0,
retry_backoff: float = 1.0,
isolate: bool = False,
timeout: float = 0.0) -> Callable[[Handler], Handler]:
"""Register a plain function as queueable work.
Returns the function untouched, so importing it and calling it directly
goes on working exactly as before it was decorated.
`solo=True` for a job there should only ever be one of at a time — see
`Spec.solo`, which is where the default and its reasoning live.
`max_attempts=3, retry_delay=60` for work that is worth another go when
the other end is having a bad minute. The retry is the queue's, not the
handler's: a failed attempt puts the row back in the queue with a time on
it, so nothing sleeps and nothing holds a worker while it waits. A
retrying job must be safe to run twice, which every handler here has to be
anyway — see the pipeline's usage card.
`isolate=True, timeout=600` for work that has to be endable: it runs in a
child process, and ten minutes in it is killed rather than asked. See
`Spec.isolate` for what that costs and `RunContext.spawn` for the cheaper
answer when the expensive part is a command rather than Python.
"""
if timeout and not isolate:
raise ValueError(
f"{kind!r} asks for a {timeout:g}s timeout without isolate=True. "
"A job in a thread cannot be ended — the deadline could only mark "
"the row failed while the work carried on. Either isolate it, or "
"give the handler a RunContext.spawn(timeout=...) to hold."
)
if timeout < 0:
raise ValueError(f"{kind!r} asks for a negative timeout: {timeout!r}")
def register(fn: Handler) -> Handler:
if kind in JOBS and JOBS[kind].fn is not fn:
raise ValueError(
f"two different handlers are both registered as {kind!r}: "
f"{JOBS[kind].fn.__qualname__} and {fn.__qualname__}"
)
JOBS[kind] = Spec(
kind=kind, label=label or kind, fn=fn, solo=solo,
wants_context=_accepts_context(fn), needs=_required(fn),
resource=resource, max_attempts=max_attempts,
retry_delay=retry_delay, retry_backoff=retry_backoff,
isolate=isolate, timeout=timeout,
)
return fn
return register
def _required(fn: Handler) -> tuple[str, ...]:
"""`fn`'s keyword-only parameters without a default, in declared order."""
try:
params = inspect.signature(fn).parameters
except (TypeError, ValueError):
return ()
return tuple(name for name, p in params.items()
if p.kind is inspect.Parameter.KEYWORD_ONLY
and p.default is inspect.Parameter.empty)
def _accepts_context(fn: Handler) -> bool:
"""Whether `fn` has a `context` parameter, or a `**kwargs` that would swallow one."""
try:
params = inspect.signature(fn).parameters
except (TypeError, ValueError):
return True # unintrospectable; assume the contract
return "context" in params or any(
p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()
)
+273
View File
@@ -0,0 +1,273 @@
"""Cadences: the five wordings a schedule may be written in, and nothing else.
A schedule is a row that says "enqueue this kind with this payload on this
cadence". This module is only the *cadence* half: the grammar, the canonical
spelling, and when the next run falls. The row is in `table.py`, the firing is
in `queue.py`, and the buttons are in `web/schedules.py`.
Five wordings, deliberately, rather than cron:
every 15m · every 6h · every 2d · daily at 06:00 · weekly mon 07:00
Cron is five fields of positional punctuation that nobody reads back
correctly, and the schedules a small app actually wants are these. A wording
outside them raises `ValueError` listing what is accepted, which is the
sentence the API hands a person who typed something else. Long weekday names
are taken as well as short ones, because `weekly monday 07:00` is what people
write; `__str__` gives back the short form, so a row stores one spelling
however it was typed.
**The interval forms are intervals and the clock forms are clock times.**
`every 6h` is six hours later, measured in real elapsed time, so it neither
skips nor doubles across a daylight-saving change. `daily at 06:00` is 06:00 on
the wall clock of the queue's timezone, on both sides of one — which is the
whole reason `next_after` takes a zone rather than doing UTC arithmetic. The
two mean different things and a person picking one has already chosen which.
Defaults are declared next to `@job`, in the app's own jobs module:
from pipeline.schedule import declare
declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
`declare` fills `SCHEDULES` the way `@job` fills `JOBS`, and `JobQueue.start()`
inserts a row for any declared name that has none. It parses the cadence there
and then, so a typo is an ImportError at startup rather than a schedule that
quietly never fires.
"""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from datetime import UTC, datetime, time, timedelta, tzinfo
from typing import Any, Literal
#: Short weekday names, Monday first, which is also `datetime.weekday()`'s
#: numbering. The short form is canonical: it is what `__str__` writes and so
#: what the table stores, however the row was typed.
DAYS: tuple[str, ...] = ("mon", "tue", "wed", "thu", "fri", "sat", "sun")
#: The long names, accepted on input for the same index. Spelled out rather
#: than derived from `calendar.day_name`, which is locale-dependent — a cadence
#: that parses on one machine and not on another is the bug this avoids.
LONG_DAYS: tuple[str, ...] = ("monday", "tuesday", "wednesday", "thursday",
"friday", "saturday", "sunday")
#: What each interval unit is worth. Seconds are deliberately absent: a queue
#: that looks for due schedules every thirty seconds cannot honour "every 10s",
#: and a cadence the scheduler cannot keep is worse than one it refuses.
UNITS: dict[str, int] = {"m": 60, "h": 3600, "d": 86400}
#: What a person is told when the wording is not one of the five. One sentence,
#: because it goes out as `{"detail": ...}` and ends up in a toast.
FORMS = ("every 15m", "every 6h", "every 2d", "daily at 06:00", "weekly mon 07:00")
_EVERY = re.compile(r"^every (\d+) ?([mhd])$")
_DAILY = re.compile(r"^daily at (\d{1,2}):(\d{2})$")
_WEEKLY = re.compile(r"^weekly ([a-z]+) (\d{1,2}):(\d{2})$")
class BadCadence(ValueError):
"""A wording that is not one of the five. A `ValueError`, so a caller that
only knows it is parsing something still catches it."""
def _refuse(text: str, because: str = "is not a cadence") -> BadCadence:
"""The one refusal, with the forms always in it.
Always, even when the wording was nearly right and this could have said
only what was wrong with it: the person reading this is typing into a box
and what they need is what to type, not a diagnosis. The specific
complaint goes in front of the list rather than instead of it.
"""
return BadCadence(
f"{text!r} {because}. Write one of: "
+ ", ".join(repr(f) for f in FORMS)
+ " — long weekday names are fine too."
)
@dataclass(frozen=True, slots=True)
class Cadence:
"""One of the five wordings, parsed. Immutable, and its own spelling.
Built through `parse`; the fields are what the wording said and are left
public so a test can assert on them. `str(cadence)` is the canonical form
and is what goes in the table, so two spellings of the same thing —
`weekly monday 07:00` and `weekly mon 07:00` — are one row and compare
equal.
"""
kind: Literal["every", "daily", "weekly"]
#: `every`: how many of `unit`. Unused by the clock forms.
n: int = 0
unit: str = ""
#: `daily` and `weekly`: the wall-clock time, in the queue's zone.
hour: int = 0
minute: int = 0
#: `weekly`: 0 is Monday, matching `datetime.weekday()`. -1 otherwise.
day: int = -1
def __str__(self) -> str:
if self.kind == "every":
return f"every {self.n}{self.unit}"
if self.kind == "daily":
return f"daily at {self.hour:02d}:{self.minute:02d}"
return f"weekly {DAYS[self.day]} {self.hour:02d}:{self.minute:02d}"
@property
def seconds(self) -> int:
"""The interval, for the `every` forms. Zero for the clock ones, which
have no fixed length — a day is 23 or 25 hours twice a year."""
return self.n * UNITS[self.unit] if self.kind == "every" else 0
@classmethod
def parse(cls, text: str) -> Cadence:
"""One of the five wordings, or `BadCadence` naming all five.
Case and inner spacing are forgiven — `Every 15 M` is `every 15m` —
because they are typing, not meaning. Everything else is refused: a
cadence that is nearly right is the one that has to fail loudly, since
the alternative is a row that looks scheduled and never fires.
"""
said = " ".join(str(text or "").lower().split())
found = _EVERY.match(said)
if found:
n = int(found.group(1))
if n < 1:
raise _refuse(text)
return cls(kind="every", n=n, unit=found.group(2))
found = _DAILY.match(said)
if found:
hour, minute = int(found.group(1)), int(found.group(2))
_check_clock(text, hour, minute)
return cls(kind="daily", hour=hour, minute=minute)
found = _WEEKLY.match(said)
if found:
name = found.group(1)
day = (DAYS.index(name) if name in DAYS
else LONG_DAYS.index(name) if name in LONG_DAYS else -1)
if day < 0:
raise _refuse(text)
hour, minute = int(found.group(2)), int(found.group(3))
_check_clock(text, hour, minute)
return cls(kind="weekly", hour=hour, minute=minute, day=day)
raise _refuse(text)
def next_after(self, moment: datetime, zone: tzinfo = UTC) -> datetime:
"""When this next falls, strictly after `moment`. Always UTC-aware.
Strictly after, so a schedule fired at exactly its own time does not
find itself due again on the same instant.
`every` is elapsed time and ignores the zone: six hours is six hours,
so a run neither vanishes nor doubles when the clocks move. The two
clock forms are the opposite and want the zone for exactly that reason
— 06:00 is 06:00 on both sides of a change, which means the interval
across one is 23 or 25 hours, and that is the thing the wording
promised.
Walking forward a day at a time rather than doing arithmetic on the
date: it is at most eight steps, it handles "later today" and "this
weekday next week" with one piece of code, and each candidate is built
from a wall-clock time and then converted, which is what keeps the
promise above.
"""
moment = moment if moment.tzinfo else moment.replace(tzinfo=UTC)
if self.kind == "every":
# Converted to UTC *before* the addition, not after. Adding a
# timedelta to an aware datetime is wall-clock arithmetic within
# its own zone — `06:30 + 24h` on the day the clocks go forward is
# 06:30 the next day, which is 23 real hours. That is exactly the
# promise the interval forms do not make, and the whole reason
# somebody writes `daily` when they want the other one.
return moment.astimezone(UTC) + timedelta(seconds=self.seconds)
local = moment.astimezone(zone)
for step in range(9): # today, then a full week of weekdays
day = local.date() + timedelta(days=step)
if self.kind == "weekly" and day.weekday() != self.day:
continue
when = datetime.combine(day, time(self.hour, self.minute), tzinfo=zone)
if when > local:
return when.astimezone(UTC)
# Unreachable: a daily cadence lands within two steps and a weekly one
# within eight. Raised rather than returned-as-None so that if the
# calendar ever surprises us it says so here.
raise BadCadence(f"{self} never comes round after {moment.isoformat()}")
def _check_clock(text: str, hour: int, minute: int) -> None:
if not (0 <= hour <= 23 and 0 <= minute <= 59):
raise _refuse(text, "is not a time of day — hours run 00-23 and "
"minutes 00-59")
# ------------------------------------------------------------ declared defaults
@dataclass(frozen=True, slots=True)
class Declared:
"""A schedule an app ships with, before any row exists for it.
`name` is the handle and is the whole of the reconciliation: a declared
name with no row gets one, a declared name that already has one is left
exactly as the user edited it. Which means the defaults here are a
*starting point*, not a setting — turning one off in the UI survives the
next restart, and that is the point of the row being the truth.
"""
name: str
kind: str
cadence: str
payload: dict[str, Any] = field(default_factory=dict)
label: str = ""
#: name -> declaration. Filled by `declare` when the app's jobs module is
#: imported, exactly as `JOBS` is filled by `@job`, and read once per
#: `JobQueue.start()`. Empty in a project that declares none, which is fine:
#: every schedule can be added from the UI instead.
SCHEDULES: dict[str, Declared] = {}
def declare(name: str, kind: str, cadence: str,
payload: dict[str, Any] | None = None,
label: str = "") -> Declared:
"""Ship a default schedule. Goes next to `@job`, in the app's jobs module.
declare("nightly tick", "tick", "daily at 06:00")
declare("hourly count", "count", "every 1h", {"n": 5})
The cadence is parsed here, so a typo is an error the moment the module is
imported — at startup, with a traceback pointing at this line — rather than
a schedule that sits in the table looking fine and never fires.
The kind is *not* checked, because it usually cannot be: `declare` and
`@job` are read top to bottom in one file and a declaration above its
handler would be refused for a job that exists a dozen lines later. An
unregistered kind is caught where it matters instead — `JobQueue` never
fires one and `<desk-schedules>` says `unknown kind` against the row.
Declaring the same name twice with different terms is refused, like two
handlers on one `@job` name: the second would silently win, and which
module imported first is not a thing anybody should have to know.
"""
Cadence.parse(cadence)
declared = Declared(name=name, kind=kind, cadence=str(Cadence.parse(cadence)),
payload=dict(payload or {}), label=label)
if name in SCHEDULES and SCHEDULES[name] != declared:
raise ValueError(
f"two different schedules are both declared as {name!r}: "
f"{SCHEDULES[name].kind} {SCHEDULES[name].cadence!r} and "
f"{declared.kind} {declared.cadence!r}"
)
SCHEDULES[name] = declared
return declared
__all__ = ["DAYS", "FORMS", "LONG_DAYS", "SCHEDULES", "UNITS", "BadCadence",
"Cadence", "Declared", "declare"]
+1319
View File
File diff suppressed because it is too large. Load diff
+495
View File
@@ -0,0 +1,495 @@
"""A task: plain code that makes jobs, and is never a job itself.
task = Task(kind="body_batch", name="body", finder=needs_a_body)
task.start(config, store, queue) # queued: batches, then one final job
task.run(config, store) # here and now, no queue at all
A task is a business step — fetch every feed, give every article a body, tag
what has never been tagged. Given the config and the store it finds its own
work, cuts it into batch payloads, mints a run id for itself, queues one job
per payload under that id, and queues one final job behind them with
`after="done"` on all of them. Then it is over. Nothing of the task is alive
afterwards: only its jobs, and the callback the final job carries.
**It runs in whoever calls it.** The HTTP request behind a button, a final
job's handler, a Claude Code session, a test. Finding the work is a query, and
a query is cheap enough to do there — making the preparation asynchronous
would buy a few milliseconds and cost a whole job kind, a payload, a failure
mode and a row in the history that says nothing a person wants to read.
**A cascade is the same shape one level up**, and the pipeline does not tell
them apart: both are runs, both group jobs by an id, both are killed the same
way. The difference is a convention, and it is worth stating because nothing
enforces it:
a task queues everything it wants at once, and only its final job
calls back.
a cascade is where the breakpoints are: it queues the next task only
once the last one is over.
**The callback is a name, not a function.** Nothing in a table can hold a
function, and the final job's payload is a table row — so the name travels
and the worker that runs the job resolves it. `module:function`, imported
where it is used. A registry would work too, and was the other candidate; the
dotted name wins because it carries its own import, where a registry has to be
filled before the worker looks in it, which is exactly the import-order trap
`JOBS` already has to be careful about. It also reads as documentation in the
job's payload: `newsdesk.jobs.cascade:step` says where to go and look.
"""
from __future__ import annotations
import importlib
import logging
from collections.abc import Callable, Iterable, Sequence
from dataclasses import dataclass, field, replace
from typing import Any, Protocol
from pipeline.context import NULL, Cancelled, RunContext
from pipeline.registry import JOBS, job
from pipeline.table import on_its_own
log = logging.getLogger(__name__)
#: The job kind every task ends with. One kind for every task there will ever
#: be, because what it does is the same every time: read how the batches went
#: and call the name in its payload.
FINAL = "advance"
#: A payload for one batch: the keyword arguments its job will be called with.
Payload = dict[str, Any]
@dataclass(frozen=True, slots=True)
class Bound:
"""How many times one batch may be attempted, and how long it waits.
The defaults are one attempt and no wait, which is every batch of every
task that existed before this: a job with no policy fails at its first
exception exactly as it always did.
It exists because a bound is sometimes a property of the *batch* rather
than of the job kind — a plugin declares how long to wait before asking
its publisher again, so two batches of `body_batch` cut from two outlets
carry two different numbers. The values travel onto the row (see
`Jobs.enqueue`), because the attempt that has to decide may be running in
another process, after a restart, under a plugin file that has since
changed.
"""
#: Counting the first attempt, so 1 is "run it once".
max_attempts: int = 1
#: Seconds before the second attempt.
delay: float = 0.0
#: Multiplied into the delay once per attempt already made.
backoff: float = 1.0
@dataclass(frozen=True, slots=True)
class Batch:
"""One batch of a task, when its bound is its own.
A finder may return these instead of plain payloads. Two shapes rather
than one because the second is what nearly every task wants and reads as
what it is — `[{"uids": [...]}, ...]` is a list of batches, and wrapping
each of them to say "and the default bound" would be ceremony on every
task to serve the one that needs it.
"""
payload: Payload
bound: Bound = Bound()
#: What to call this one batch on screen, or "" to be called whatever its
#: job kind is called. The kind's label is the same string on every batch
#: of the task — nine rows reading "Tagging a batch" — because a label is
#: declared once, at registration, by something that cannot see how the
#: work was cut. Only the finder knows that, so this is where a batch that
#: can say which batch it is says so. See `desk/modeltask.py`.
label: str = ""
#: Given the app's config and store, what there is to do — one payload per
#: batch, or one `Batch` where the bound is the batch's own. An empty list is
#: a legitimate answer and the common one: a task with nothing waiting costs
#: one query, queues no batches, and its final job runs immediately and moves
#: the cascade on.
Finder = Callable[[Any, Any], list[Any]]
def _batch(found: Payload | Batch) -> Batch:
"""One thing a finder returned, as a batch. A bare payload gets the
default bound, which is the behaviour it had before bounds existed."""
return found if isinstance(found, Batch) else Batch(found)
#: name -> how to build that task, standing on its own. Filled by the app the
#: way `JOBS` is, and for the same reason: a task is as much a thing to start
#: as a job is, so whatever offers the one — `/api/jobs`, the Jobs view, a
#: script — has to be able to find the other by name.
TASKS: dict[str, Callable[[], Task]] = {}
class Sink(Protocol):
"""Where a task puts the jobs it makes.
Two of them: `JobQueue`, when a task is started from a request or a
handler, and nothing else. It is a protocol rather than the class so that
`Task` does not import the queue — the dependency runs the other way, and a
test can hand it a list.
"""
def new_run(self, kind: str, name: str = "", label: str = "") -> int: ...
def cancelled(self, run_id: int) -> bool: ...
def enqueue(self, kind: str, payload: Payload | None = None, *,
label: str = "", depends_on: Sequence[int] = (), after: str = "ok",
cascade_id: int = 0, task_id: int = 0,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> Any: ...
@dataclass(frozen=True, slots=True)
class Task:
"""One business step: what a batch of it is, how to find them, who hears.
Knows nothing about what the work *is* — that is the batch job's, and the
finder's. This is the shape they are held in.
"""
#: The job kind one batch runs as. A task of one batch names an ordinary
#: job kind (`pull`); a task of many names the batch kind (`body_batch`).
kind: str
#: How the work is found. Called once, in the caller, before anything is
#: queued.
finder: Finder
#: What this task is — `body`, `tag`. Its identity: what a cascade's list
#: matches against, and what the `runs` row is minted under. The job kind
#: when nobody says.
name: str = ""
#: How to say that on screen: "Fetching what the feeds left out". Written
#: onto the `runs` row, because a task is not a job and has no handler to
#: carry a label. The name when nobody says.
label: str = ""
#: `module:function` for the final job to call, or "" for a task that
#: reports to nobody — one started on its own, from a button or a script.
callback: str = ""
#: `module:function` the final job runs first, or "" for a task whose
#: batches finish their own work. A name for the same reason the callback
#: is one: it is a row in a table.
#:
#: The seam exists for a decision no batch can make on its own. A batch is
#: one of forty and cannot see the other thirty-nine, so anything that has
#: to hold across the whole run — a cap on what the run may add, a merge of
#: what several batches each asked for — has exactly one place it can be
#: made correctly, and that is the job that runs when they are all over.
#: It belongs to the task rather than to the callback because it *is* the
#: task's work: a task started on its own has no callback and still has to
#: settle. See `newsdesk/jobs/tagging.py` for the one that does.
settle: str = ""
#: What the callback is handed alongside the tally. Fixed when the task is
#: made, and stored in the final job's payload, so it survives a restart
#: and a worker in another process.
carries: dict[str, Any] = field(default_factory=dict)
#: How many items a payload is worth, for the direct path's progress bar.
#: Only the direct path asks: as a job a batch is on its own and reports
#: its own rows, which is what the boxes on screen are drawn from.
size: Callable[[Payload], int] | None = None
#: How the direct path's batch results become one result. The queue never
#: calls it — there each batch is its own job with its own row.
fold: Callable[[Any, Any, list[Any]], dict[str, Any]] | None = None
@property
def named(self) -> str:
return self.name or self.kind
@property
def called(self) -> str:
return self.label or self.named
# ------------------------------------------------------------- queued
def start(self, config: Any, store: Any, sink: Sink, *,
cascade_id: int = 0) -> int:
"""Find the work, queue it, and be over. Returns the task's run id.
The id is minted before a single job exists, because a task is a thing
before its jobs are — it is what decides how many there will be — and
naming it after whichever job it happened to queue first would be
describing it by one of its own consequences.
With no cascade given it begins one: something asked for this on its
own, so it is a cascade of one task. Named apart from any chain of the
same name — `pull on its own`, not `pull` — because pressing Tag is
not starting the chain that ends in tagging, and whoever asks whether
that chain is going has to be told no. See `on_its_own`.
Returns 0 when the cascade it belongs to has been killed. Nothing is
told; it tries, and the table says no. `Jobs.enqueue` is where that is
actually enforced, across processes and after a restart; this only
saves minting a run for jobs that would all be refused.
"""
task_id, _ = queue_batches(
sink, self.kind, lambda: self.finder(config, store),
name=self.named, label=self.called, then=self.callback,
settle=self.settle, carries=self.carries, cascade_id=cascade_id)
return task_id
# ------------------------------------------------------------- direct
def run(self, config: Any, store: Any, *,
context: RunContext = NULL) -> dict[str, Any]:
"""Do the batches here and now, in order, with no queue at all.
What a Claude Code session, a notebook and a test get. The same batch
handler the worker would call, with the same payloads from the same
finder — there is no second implementation to drift.
A batch that raises is kept and the rest carry on, which is what the
queue does with eight independent jobs of which one fails. Unless
*nothing* got through: then the first exception is raised, because a
run that returned "0 done" with the explanation only in the log would
be a failure reported as a success. A `Cancelled` travels either way —
it is an instruction from the person at the keyboard, not a batch that
did not land.
A batch's `Bound` is ignored here, and that is not an omission: a
bound is a number the *queue* enforces by putting a row back with a
time on it, and there is no queue and no row on this path. Asked for
directly, a batch is attempted once and its failure is returned to the
caller, who is standing right there.
"""
spec = JOBS.get(self.kind)
if spec is None:
raise KeyError(
f"{self.named!r} makes jobs of kind {self.kind!r}, which "
f"nothing registers (have: {', '.join(sorted(JOBS)) or 'none'})")
payloads = [_batch(found).payload for found in self.finder(config, store)]
sizes = [self._size(p) for p in payloads]
total = sum(sizes)
results: list[Any] = []
failure: Exception | None = None
at = 0
for payload, size in zip(payloads, sizes, strict=True):
context.check()
extra = {"context": self._placed(context, at, total, size)} \
if spec.wants_context else {}
try:
results.append(dict(spec.fn(config, store, **extra, **payload) or {}))
except Cancelled:
raise
except Exception as exc:
log.exception("%s: a batch of %s did not land", self.named, self.kind)
results.append(exc)
failure = failure or exc
finally:
# Position through the backlog, not what was answered: a batch
# the model skipped half of still moved the run that far, and a
# bar that goes backwards is worse than one that is approximate.
at += size
if failure is not None and not any(isinstance(r, dict) for r in results):
raise failure
if self.fold is not None:
return self.fold(config, store, results)
return {"batches": sum(1 for r in results if isinstance(r, dict)),
"failed": sum(1 for r in results if isinstance(r, Exception)),
"results": [r for r in results if isinstance(r, dict)]}
def _size(self, payload: Payload) -> int:
return int(self.size(payload)) if self.size is not None else 0
def _placed(self, context: RunContext, at: int, total: int,
size: int) -> RunContext:
"""The batch's context, with its progress shifted into the whole run.
A batch run as a job is on its own and counts its own rows. Run here it
is one of several in a row, and a bar that restarts per batch would be
a bar that goes backwards five times.
"""
if not total:
return context
return replace(context, on_progress=lambda done, _t, message:
context.progress(at + min(done, size), total, message))
# ------------------------------------------------------------- the final job
@dataclass(frozen=True, slots=True)
class Tally:
"""How a task's batches ended, as the callback is handed it.
Read off the jobs at the moment the final job runs, never counted up as
they landed — so there is nothing to increment and nothing to lose to a
restart half way through.
"""
done: int = 0
failed: int = 0
cancelled: int = 0
#: Whether the task itself was killed, as against however many of its
#: batches happen to have ended cancelled. The counts cannot say: one
#: batch killed by hand and every batch killed by the task going down look
#: identical there and mean opposite things — drop this batch, or skip
#: this whole step. Only the run's row knows, which is why it is asked.
killed: bool = False
#: What each batch returned, in the order they were queued.
results: tuple[dict[str, Any], ...] = ()
@property
def jobs(self) -> int:
return self.done + self.failed + self.cancelled
@classmethod
def of(cls, outcomes: Sequence[Any], *, killed: bool = False) -> Tally:
states = [getattr(j, "state", "") for j in outcomes]
return cls(
done=states.count("done"),
failed=states.count("failed"),
cancelled=states.count("cancelled"),
killed=killed,
results=tuple(dict(getattr(j, "result", {}) or {}) for j in outcomes),
)
def as_dict(self) -> dict[str, Any]:
return {"jobs": self.jobs, "done": self.done, "failed": self.failed,
"cancelled": self.cancelled, "killed": self.killed}
def queue_batches(sink: Sink, kind: str,
found: Callable[[], Iterable[Payload | Batch]], *,
name: str = "", label: str = "", then: str = "",
settle: str = "", carries: dict[str, Any] | None = None,
cascade_id: int = 0,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> tuple[int, tuple[int, ...]]:
"""Mint a task, queue one job per batch, and queue the job that reports.
The one implementation behind both `Task.start` and `fan_out`. Returns the
task's run id and the children's job ids; `(0, ())` when the cascade it
would join has been killed.
`found` is called only after the runs exist, which is the order a task has
always had: a task is a thing before its jobs are, since it is what decides
how many there will be.
A plain payload takes the kind's own retry policy, or the uniform one
passed here. A `Batch` carries its own `Bound`, which wins — that is what a
batch cut from one plugin profile needs when the profile declares how long
to leave its publisher alone.
With no cascade given it begins one: something asked for this on its own,
so it is a cascade of one task, named apart from any chain of the same name
(`pull on its own`, not `pull`). See `on_its_own`.
"""
if cascade_id and sink.cancelled(cascade_id):
return 0, ()
name = name or kind
label = label or name
cascade_id = cascade_id or sink.new_run("cascade", on_its_own(name), label)
task_id = sink.new_run("task", name, label)
ids: list[int] = []
for item in found():
if isinstance(item, Batch):
queued = sink.enqueue(kind, dict(item.payload), label=item.label,
cascade_id=cascade_id, task_id=task_id,
max_attempts=item.bound.max_attempts,
retry_delay=item.bound.delay,
retry_backoff=item.bound.backoff)
else:
queued = sink.enqueue(kind, dict(item),
cascade_id=cascade_id, task_id=task_id,
max_attempts=max_attempts,
retry_delay=retry_delay,
retry_backoff=retry_backoff)
if queued is not None:
ids.append(queued.id)
# Behind all of them, and part of the same task — killing the task spares
# it, because reporting is not work and "skip this task" is not "make it
# never end". That holds where the kill landed mid-loop too: `Jobs.enqueue`
# carves `after='done'` out of a task's kill, so this row is written and
# the task ends by saying it was killed. None here is the *cascade* going
# down, which takes the report with it by design: there is nothing left
# to report to. See `Jobs.cancel_run`.
sink.enqueue(FINAL, {"callback": then, "settle": settle,
"payload": dict(carries or {})},
depends_on=ids, after="done",
cascade_id=cascade_id, task_id=task_id)
return task_id, tuple(ids)
def fan_out(sink: Sink, kind: str, payloads: Sequence[Payload | Batch], *,
then: str = "", settle: str = "", label: str = "",
carries: dict[str, Any] | None = None,
cascade_id: int = 0,
max_attempts: int | None = None,
retry_delay: float | None = None,
retry_backoff: float | None = None) -> tuple[int, ...]:
"""Queue these payloads as jobs, and one job behind them that reports.
What `RunContext.fan_out` reaches. `Task` is the same call with a finder in
front, which is why a handler whose whole body is one of these *is* a task.
The children go into a run of their own inside the caller's cascade, so the
callback can fan out again and the result is a chain rather than a nest.
Returns the children's ids. The reporting job is not among them — it is
behind them by construction, and a caller that waited on it would be
holding a worker open to do so.
"""
_, ids = queue_batches(
sink, kind, lambda: payloads, label=label, then=then, settle=settle,
carries=carries, cascade_id=cascade_id, max_attempts=max_attempts,
retry_delay=retry_delay, retry_backoff=retry_backoff)
return ids
@job(FINAL, "Moving on")
def run_final(config: Any, store: Any, *, context: RunContext = NULL,
callback: str = "", settle: str = "",
payload: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""The one job every task ends with: tally the batches, call the name.
It runs for the milliseconds it takes to read how the batches went and
hand that to whoever asked — nothing is alive between two tasks, and
nothing was waiting while the last one ran.
`after="done"`, so it runs however the batches ended. That is the whole
reason the mode exists: a job whose purpose is to report that one of eight
failed cannot be held back by that failure.
A task with no callback stops here, which is what being asked for on its
own means.
The settle runs first and the callback second, because the settle is the
last of this task's own work and the callback is what comes after this
task: a cascade told to move on before the tags of the stage it is leaving
exist would be a stage reporting itself finished early.
It asks whether its own task was killed, as well as counting how the
batches ended, because those are two different facts that arrive looking
the same. A task killed while its batches ran leaves them all `cancelled`,
and so does a person killing every batch by hand; the first is "skip this
step", the second is "drop this work". Only the run's row can tell them
apart, and the callback needs to be told which it is — the whole reason
this job is spared the kill is so that somebody hears.
"""
tally = Tally.of(context.waited_for, killed=context.killed())
out = tally.as_dict()
if settle:
out |= resolve(settle)(config, store, tally, context=context) or {}
if not callback:
return out
return {**out, **(resolve(callback)(config, store, tally,
dict(payload or {}), context=context) or {})}
def resolve(name: str) -> Callable[..., dict[str, Any] | None]:
"""`module:function` to the function. See this module's docstring for why
a callback, or a settle, is a name at all."""
module, _, attr = name.partition(":")
if not module or not attr:
raise ValueError(f"a callback is 'module:function', not {name!r}")
return getattr(importlib.import_module(module), attr)
+2
View File
@@ -0,0 +1,2 @@
"""The pipeline's own tests. They travel with the folder, so a copy can
prove itself where it lands: `python -m pytest pipeline/tests`."""
+74
View File
@@ -0,0 +1,74 @@
"""Handlers for the isolation tests, in a module a child process can import.
Not a test file — pytest does not collect it — and deliberately separate from
the tests that use it, because the point of `isolate=True` is that the work
runs in a *different process*. That process gets nothing from this one: it
imports `pipeline.tests.isolated_handlers` itself, by the name in
`--bootstrap`, which is exactly what an app's own bootstrap has to do.
It also documents the whole of the contract, which is two lines: import the
modules that fill `JOBS`, and return the `(config, store)` pair the handlers
take. Here both are dicts, since nothing in the pipeline reads them.
"""
from __future__ import annotations
import os
import time
from typing import Any
from pipeline import NULL, RunContext, job
@job("iso_ok", "Working in its own process", isolate=True)
def iso_ok(config: Any, store: Any, *, context: RunContext = NULL,
n: int = 0) -> dict[str, Any]:
"""Reports which process it ran in, so a test can prove it was not ours."""
context.progress(n, n, f"counting to {n}")
return {"doubled": n * 2, "pid": os.getpid()}
@job("iso_raises", "Failing in its own process", isolate=True)
def iso_raises(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
raise RuntimeError("the model said no")
@job("iso_slow", "Taking too long on purpose", isolate=True, timeout=1.0)
def iso_slow(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Uninterruptible on purpose: no `check()`, so only a kill ends it.
`time.sleep` stands in for `sklearn.fit` — the difference that matters is
not what it does but that nothing in it ever asks whether to stop, which
is the case a thread has no answer for.
"""
time.sleep(120)
return {"never": True}
def bootstrap() -> tuple[Any, Any]:
"""What `--bootstrap` names. Importing this module registered the jobs."""
return {"config": "rebuilt in the child"}, {"store": "rebuilt in the child"}
# ---------------------------------------------------- two workers, one file
@job("mp_mark", "Leaving a mark")
def mp_mark(config: Any, store: Any, *, context: RunContext = NULL,
i: int = 0) -> dict[str, Any]:
"""Leave a file named after this job and the process that ran it.
A file per run rather than a line in a shared one, because creating a
uniquely named file is atomic on every platform and appending from two
processes is not. A job run twice leaves two files with the same number.
"""
import uuid
from pathlib import Path
(Path(store) / f"{i}-{os.getpid()}-{uuid.uuid4().hex}").touch()
time.sleep(0.02) # long enough that both workers get a share
return {"i": i}
def marking(marks: str) -> tuple[Any, Any]:
"""The bootstrap `pipeline.worker` is given: the marks directory is the store."""
return {}, marks
+39
View File
@@ -0,0 +1,39 @@
"""A job and a declared schedule, in a module a worker process can import.
Not a test file — pytest does not collect it — and separate from the tests that
use it for the same reason `isolated_handlers.py` is: the process under test is
a *different* process, and it gets nothing from this one. It imports this by
the name in `--bootstrap`, which fills `JOBS` and `SCHEDULES` exactly as an
app's own jobs module does.
Separate from `isolated_handlers.py` as well, because importing a module that
calls `declare` fills a global: a test suite that imported this one would find
its own schedule registry non-empty before it had declared anything.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
from pipeline import NULL, RunContext, declare, job
@job("sw_mark", "Leaving a mark")
def sw_mark(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Touch a file in the store directory, so the parent process can see from
outside whether this ran at all."""
(Path(store) / "fired").touch()
return {"fired": True}
#: The declaration itself is what one half of the worker test is about: a
#: worker with no web app around it reconciles declared defaults like any
#: other, so a machine running only `python -m pipeline.worker` still gets
#: them.
declare("marker", "sw_mark", "every 1h", label="Leaving a mark, hourly")
def marking(marks: str) -> tuple[Any, Any]:
"""The bootstrap `pipeline.worker` is given: the marks directory is the store."""
return {}, marks
+113
View File
@@ -0,0 +1,113 @@
"""The web face of the queue, from an app that has written nothing but `connect`.
python -m pytest pipeline/tests/test_web.py -q
"""
from __future__ import annotations
import sys
import tempfile
import time
from contextlib import asynccontextmanager
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
pytest.importorskip("fastapi")
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pipeline import NULL, Jobs, RunContext, job
from pipeline.web import connect, jobs_router
@job("w_hello", "Saying hello")
def w_hello(config, store, *, context: RunContext = NULL, name: str = "") -> dict:
context.progress(1, 1, f"greeting {name}")
return {"said": f"hello {name}", "config": config["app"]}
@pytest.fixture
def jobs():
with tempfile.TemporaryDirectory() as tmp:
yield Jobs(Path(tmp) / "web.db")
def settled(client: TestClient, job_id: int, timeout: float = 10.0) -> dict:
"""The job as the HTTP API reports it, once it has finished."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
snap = client.get("/api/jobs").json()
for row in snap["recent"]:
if row["id"] == job_id:
return row
time.sleep(0.05)
raise AssertionError(f"job {job_id} did not finish")
def test_connect_is_the_whole_of_the_wiring(jobs):
"""Routes, socket and a running worker, from one call.
The app below has nothing else: no lifespan, no broadcaster, no router of
its own. A job enqueued over HTTP runs to `done` because the worker came up
with the app, and the socket hands a new tab the current state on connect.
"""
app = FastAPI()
queue = connect(app, jobs, {"app": "tiny"}, None)
assert app.state.queue is queue
with TestClient(app) as client:
made = client.post("/api/jobs", json={"kind": "w_hello",
"payload": {"name": "you"}})
assert made.status_code == 202, made.text
ended = settled(client, made.json()["id"])
assert ended["state"] == "done"
assert ended["result"] == {"said": "hello you", "config": "tiny"}
with client.websocket_connect("/ws/jobs") as socket:
hello = socket.receive_json()
assert hello["type"] == "jobs"
assert any(row["id"] == ended["id"] for row in hello["recent"])
assert queue._task is None or queue._task.done(), "and it stopped with the app"
def test_connect_keeps_the_apps_own_lifespan(jobs):
"""An app with startup code of its own keeps it, and the worker is already
up when that code runs — so startup code may enqueue."""
seen: list[str] = []
@asynccontextmanager
async def lifespan(app: FastAPI):
seen.append("app started")
app.state.queue.enqueue("w_hello", {"name": "from startup"})
yield
seen.append("app stopped")
app = FastAPI(lifespan=lifespan)
connect(app, jobs, {"app": "tiny"}, None)
with TestClient(app) as client:
# By id: the snapshot reads live and recent jobs separately, so a job
# that finishes in between is listed in both.
snap = client.get("/api/jobs").json()
(job_id,) = {r["id"] for r in snap["queued"] + snap["running"]
+ snap["recent"] if r["kind"] == "w_hello"}
assert settled(client, job_id)["state"] == "done"
assert seen == ["app started", "app stopped"]
def test_the_element_is_served_by_the_queues_own_router():
"""Two lines in a template and a new app has a job list. Served from a
route, because a router cannot mount a static directory and needing the
app to mount one is the setup the element exists to remove."""
app = FastAPI()
app.include_router(jobs_router)
answer = TestClient(app).get("/_jobs/jobs.js")
assert answer.status_code == 200
assert answer.headers["content-type"].startswith("text/javascript")
assert 'customElements.define("desk-jobs"' in answer.text
+209
View File
@@ -0,0 +1,209 @@
"""What keeps a queue healthy for months: retention, bounded results, whose
retry policy wins, draining, and two real processes on one file.
python -m pytest pipeline/tests/test_housekeeping.py -q
"""
from __future__ import annotations
import os
import sqlite3
import subprocess
import sys
import tempfile
from contextlib import closing
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from pipeline import NULL, Batch, Bound, JobQueue, Jobs, Task, job
from pipeline.models import Job
ROOT = Path(__file__).resolve().parents[2]
@job("h_plain", "A job with a policy of its own", max_attempts=3, retry_delay=7)
def h_plain(config, store, *, context=NULL, i: int = 0) -> dict:
return {"i": i}
@job("h_big", "Returning far too much", max_attempts=3)
def h_big(config, store, *, context=NULL) -> dict:
return {"rows": ["x" * 100] * 2000, "count": 2000}
@pytest.fixture
def jobs():
with tempfile.TemporaryDirectory() as tmp:
yield Jobs(Path(tmp) / "h.db")
def got(found: Job | None) -> Job:
assert found is not None
return found
def age(jobs: Jobs, job_id: int, days: float) -> None:
"""Pretend a job finished `days` ago, and its runs were minted then too."""
when = (datetime.now(UTC) - timedelta(days=days)).isoformat()
with closing(sqlite3.connect(jobs.path)) as conn, conn:
conn.execute("UPDATE jobs SET finished_at=? WHERE id=?", (when, job_id))
conn.execute("UPDATE runs SET created_at=? WHERE id IN "
"(SELECT cascade_id FROM jobs WHERE id=? UNION "
" SELECT task_id FROM jobs WHERE id=?)", (when, job_id, job_id))
def runs(jobs: Jobs) -> int:
with closing(sqlite3.connect(jobs.path)) as conn:
return conn.execute("SELECT COUNT(*) FROM runs").fetchone()[0]
# ---------------------------------------------------------------- retention
def test_purge_forgets_old_history_but_not_the_last_of_each_kind(jobs):
"""Without this both tables grow for as long as the app runs.
What survives an old cutoff: anything unfinished, and the newest ending of
each kind in each state — because `last(kind)` is how an app says "last
pulled five weeks ago", and deleting it would make that "never".
"""
old = [got(jobs.enqueue("h_plain")) for _ in range(3)]
for queued in old:
jobs.finish(queued.id, "done")
age(jobs, queued.id, days=40)
waiting = got(jobs.enqueue("h_plain")) # never finished
gone = jobs.purge(30 * 86400)
assert gone == 2, "two of the three old ones went"
assert got(jobs.last("h_plain")).id == old[-1].id, "the newest stayed"
assert got(jobs.get(waiting.id)).state == "queued", "unfinished work is not history"
def test_purge_leaves_recent_history_alone(jobs):
recent = got(jobs.enqueue("h_plain"))
jobs.finish(recent.id, "done")
before = runs(jobs)
assert jobs.purge(30 * 86400) == 0
assert jobs.get(recent.id) is not None
assert runs(jobs) == before
def test_purge_drops_runs_nothing_points_at_any_more(jobs):
first = got(jobs.enqueue("h_plain"))
second = got(jobs.enqueue("h_plain"))
for queued in (first, second):
jobs.finish(queued.id, "done")
age(jobs, queued.id, days=40)
before = runs(jobs)
jobs.purge(30 * 86400)
assert runs(jobs) < before, "the purged job's cascade and task went with it"
assert jobs.run(second.task_id) is not None, "the kept job's runs did not"
# ----------------------------------------------------------- bounded results
def test_a_result_too_large_fails_the_job_loudly_and_once(jobs):
"""Past `MAX_RESULT` the job fails rather than losing data quietly: the row
says why and how to fix it, a notice goes to whoever is watching, and it is
not retried — the same handler would return the same result."""
said: list = []
async def on_notice(notice) -> None:
said.append(notice)
queued = got(jobs.enqueue("h_big", max_attempts=3))
JobQueue(jobs, None, None, on_notice=on_notice).drain(timeout=10)
ended = got(jobs.get(queued.id))
assert ended.state == "failed"
assert "ResultTooLarge" in ended.error and "store" in ended.error, ended.error
assert "count, rows" in ended.error, "it names what was returned"
assert ended.result == {}, "nothing of it was written"
assert ended.attempt == 1, "and it was not tried again"
assert [n.code for n in said] == ["job.failed"], "and somebody was told"
def test_a_result_at_the_limit_is_kept_whole(jobs):
"""The limit is a line, not a tax: everything under it is stored as is."""
queued = got(jobs.enqueue("h_plain", payload={"i": 1}))
JobQueue(jobs, None, None).drain(timeout=10)
assert got(jobs.get(queued.id)).result == {"i": 1}
# -------------------------------------------------------- whose policy wins
def test_a_task_with_plain_payloads_keeps_the_kinds_own_retry_policy(jobs):
"""`@job(max_attempts=3)` has to mean something when a task queues it.
It used not to: a plain payload was given the default bound explicitly,
which silently overrode whatever the kind declared.
"""
queue = JobQueue(jobs, None, None)
Task(kind="h_plain", finder=lambda c, s: [{"i": 1}]).start(None, None, queue)
(child,) = [j for j in jobs.snapshot().queued if j.kind == "h_plain"]
assert (child.max_attempts, child.retry_delay) == (3, 7.0)
def test_a_batch_with_its_own_bound_overrides_the_kind(jobs):
"""The other half: a batch cut from one plugin profile carries that
profile's policy, and that wins, because the outlet knows its publisher."""
queue = JobQueue(jobs, None, None)
Task(kind="h_plain", finder=lambda c, s: [
Batch({"i": 1}, Bound(max_attempts=5, delay=60))]).start(None, None, queue)
(child,) = [j for j in jobs.snapshot().queued if j.kind == "h_plain"]
assert (child.max_attempts, child.retry_delay) == (5, 60.0)
# ------------------------------------------------------------------ draining
def test_drain_runs_everything_queued_and_stops(jobs):
"""The batch shape: a script or a cron entry with no web app around it."""
ids = [got(jobs.enqueue("h_plain", payload={"i": i})).id for i in range(5)]
assert JobQueue(jobs, None, None).drain(timeout=10) is True
assert {got(jobs.get(i)).state for i in ids} == {"done"}
# ------------------------------------------------- two processes, one file
def test_two_worker_processes_never_run_the_same_job_twice(tmp_path):
"""The claim the table-as-queue design rests on, tested the only way that
means anything: two separate `python -m pipeline.worker` processes draining
one file at once. Every job leaves a marker file named after itself and the
process that ran it; each job must have exactly one.
"""
db = tmp_path / "shared.db"
marks = tmp_path / "marks"
marks.mkdir()
jobs = Jobs(db)
for i in range(40):
jobs.enqueue("mp_mark", payload={"i": i})
env = dict(os.environ, PYTHONPATH=str(ROOT))
argv = [sys.executable, "-m", "pipeline.worker", "--db", str(db),
"--bootstrap", "pipeline.tests.isolated_handlers:marking",
"--bootstrap-arg", str(marks), "--workers", "cpu=2", "--until-idle"]
workers = [subprocess.Popen(argv, env=env, cwd=ROOT,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for _ in range(2)]
for worker in workers:
out, _ = worker.communicate(timeout=120)
assert worker.returncode == 0, out.decode(errors="replace")[-2000:]
ran: dict[int, list[str]] = {}
for mark in marks.iterdir():
i, pid = mark.name.split("-")[:2]
ran.setdefault(int(i), []).append(pid)
assert sorted(ran) == list(range(40)), "every job ran"
doubled = {i: pids for i, pids in ran.items() if len(pids) > 1}
assert not doubled, f"and none ran twice: {doubled}"
assert {j.state for j in jobs.snapshot().recent} == {"done"}
+320
View File
@@ -0,0 +1,320 @@
"""Ending work that will not end itself, and how data moves between jobs.
Four claims, and they fail independently:
a deadline is honest `timeout` without `isolate` is refused at
registration rather than promising a kill a
thread cannot deliver.
a process can be killed an isolated job that never calls `check()` is
still over when its deadline passes, and its
row says so.
a chain reads, never writes `upstream()` is how B gets A's output — off A's
row, so B's own row was complete when written.
a fan-out is one call `fan_out` queues the children and the job that
reports on them, and an app never writes the
second one.
python3 -m pytest tests/test_isolation.py -q
"""
from __future__ import annotations
import asyncio
import os
import sys
import tempfile
import time
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from pipeline import NULL, Cancelled, JobQueue, Jobs, RunContext, job, spawn
from pipeline.models import Job
# Imported for the `@job` registrations, which the *parent* needs as much as
# the child does: the worker reads `isolate` and `timeout` off the spec before
# it decides to start a process at all. An app imports its handlers in every
# entry point for this reason; here that import is this line.
from pipeline.tests import isolated_handlers # noqa: F401
BOOTSTRAP = "pipeline.tests.isolated_handlers:bootstrap"
@job("t_child", "A child of a fan-out")
def t_child(config, store, *, context=NULL, i: int = 0) -> dict:
"""Queued by `fan_out` and never run in these tests — what is under test is
the shape of the rows it made, which is the part an app never writes."""
return {"i": i}
@pytest.fixture
def jobs():
with tempfile.TemporaryDirectory() as tmp:
yield Jobs(Path(tmp) / "t.db")
@pytest.fixture
def child_path(monkeypatch):
"""Put the folder holding `pipeline` where a child process can import it.
What an installed app gets for free and a checkout does not: the child is a
fresh interpreter, and the checkout is on *our* path only because pytest
put it there.
"""
root = Path(__file__).resolve().parents[2]
monkeypatch.setenv("PYTHONPATH", str(root))
def drain(queue: JobQueue, *, timeout: float = 20.0) -> None:
"""Run the worker until nothing is queued, running, or held by this one."""
async def go() -> None:
await queue.start()
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
snap = queue.jobs.snapshot()
if not snap.running and not snap.queued and not queue._running:
break
await asyncio.sleep(0.02)
await queue.stop()
asyncio.run(go())
def got(found: Job | None) -> Job:
assert found is not None
return found
# --------------------------------------------------- a deadline is honest
def test_a_timeout_without_isolation_is_refused():
"""The pipeline will not offer a kill it cannot perform.
A deadline over a job in a thread could only mark the row failed and walk
away: the work would carry on holding the core it was killed for holding,
and two things would intend to write one row. So the pair is refused where
it is declared, with the two ways out named.
"""
with pytest.raises(ValueError) as refused:
@job("t_threaded_deadline", timeout=5.0)
def handler(config, store, *, context=NULL):
return {}
said = str(refused.value)
assert "isolate=True" in said, "it names the way to get a real deadline"
assert "RunContext.spawn" in said, "and the cheaper way, for a command"
def test_isolation_without_a_timeout_is_fine():
"""Only the one combination is refused. An isolated job with no deadline
runs until it is done or killed by hand, which is every job the pipeline
has ever had."""
@job("t_isolated_no_deadline", isolate=True)
def handler(config, store, *, context=NULL):
return {}
from pipeline import JOBS
assert JOBS["t_isolated_no_deadline"].isolate
assert JOBS["t_isolated_no_deadline"].timeout == 0.0
def test_a_queue_with_no_bootstrap_says_so(jobs):
"""An isolated job needs its app rebuilt in the child, and a queue that
cannot do it fails the job with the reason rather than with a TypeError
from somewhere inside a subprocess."""
@job("t_needs_bootstrap", isolate=True)
def handler(config, store, *, context=NULL):
return {}
queued = got(jobs.enqueue("t_needs_bootstrap"))
drain(JobQueue(jobs, None, None)) # no bootstrap=
ended = got(jobs.get(queued.id))
assert ended.state == "failed"
assert "bootstrap" in ended.error, f"it names what is missing: {ended.error}"
# ------------------------------------------------ a process can be killed
def test_an_isolated_job_runs_somewhere_else_and_its_result_comes_back(jobs, child_path):
"""The happy path, and the proof that it was a real process.
Nothing was pickled to get there: the child was handed a database and a
row id and rebuilt the rest. Which is also why its progress is on the row
— it wrote it there itself, to the same table the snapshot reads.
"""
queued = got(jobs.enqueue("iso_ok", payload={"n": 21}))
drain(JobQueue(jobs, None, None, bootstrap=BOOTSTRAP))
ended = got(jobs.get(queued.id))
assert ended.state == "done", ended.error
assert ended.result["doubled"] == 42, "the handler's return travelled back"
assert ended.result["pid"] != os.getpid(), "and it ran in another process"
assert ended.message == "counting to 21", "its progress reached our table"
def test_an_isolated_job_that_raises_fails_with_what_it_said(jobs, child_path):
"""A failure across a process boundary reads like any other failure: the
row is `failed`, the error is the exception, and the retry bound applies
as it would to a handler that raised in a thread."""
queued = got(jobs.enqueue("iso_raises"))
drain(JobQueue(jobs, None, None, bootstrap=BOOTSTRAP))
ended = got(jobs.get(queued.id))
assert ended.state == "failed"
assert "the model said no" in ended.error, ended.error
def test_a_deadline_ends_a_job_that_never_looks_up(jobs, child_path):
"""The claim the whole feature exists for.
`iso_slow` sleeps for two minutes and never calls `check()`. In a thread
that is unkillable and the worker could only lie about having stopped it.
In a process of its own it is over in about a second, and the row says
why — which is what a person reading the job list needs.
"""
queued = got(jobs.enqueue("iso_slow"))
started = time.monotonic()
drain(JobQueue(jobs, None, None, bootstrap=BOOTSTRAP))
took = time.monotonic() - started
ended = got(jobs.get(queued.id))
assert ended.state == "failed", f"it ended: {ended.state}"
assert took < 15, f"and on its deadline, not in its own time: {took:.1f}s"
# ------------------------------------------------------ spawn, on its own
def test_spawn_kills_the_process_when_the_job_is_cancelled():
"""Kill, for a handler whose work is a command.
Cooperative in name only from the handler's side: it is blocked in
`spawn`, not in a loop with a `check()` in it, and the child is killed
outright. What makes that safe is that the thread was only sleeping.
"""
asked = [0]
def is_cancelled() -> bool:
asked[0] += 1
return asked[0] > 2 # pressed, a moment in
context = RunContext(is_cancelled=is_cancelled)
started = time.monotonic()
with pytest.raises(Cancelled):
context.spawn([sys.executable, "-c", "import time; time.sleep(120)"])
assert time.monotonic() - started < 10, "it did not wait the two minutes"
def test_spawn_raises_on_its_deadline_and_leaves_nothing_running():
"""The deadline, and that the process is gone after it — not merely
abandoned, which is the failure mode the whole thing is against."""
started = time.monotonic()
with pytest.raises(TimeoutError):
spawn([sys.executable, "-c", "import time; time.sleep(120)"], timeout=0.5)
assert time.monotonic() - started < 10
def test_spawn_hands_back_the_exit_code_rather_than_raising():
"""A non-zero exit is not an error here. Plenty of commands say something
useful with one, and only the handler knows which."""
done = spawn([sys.executable, "-c", "import sys; sys.exit(3)"])
assert done.returncode == 3
def test_spawn_survives_a_command_that_writes_more_than_a_pipe_holds():
"""Output goes to a file, not a `PIPE`, and this is why.
A child that fills the pipe buffer blocks on the write while the poll loop
blocks on the read, and neither side is wrong — the job simply never ends.
A megabyte is far past any platform's buffer.
"""
done = spawn([sys.executable, "-c", "print('x' * 1000000)"], timeout=60)
assert done.returncode == 0
assert len(done.stdout) >= 1000000
# ---------------------------------------------- a chain reads, never writes
def test_upstream_merges_what_the_parents_returned():
"""How B gets A's output: off A's row, not out of B's payload.
Which is the property worth having. B's row was complete when it was
written, so it can carry a dedup key, a retry can re-run it from itself,
and no worker can claim it between an earlier job's write and its commit.
"""
a = Job(id=1, kind="a", created_at="", result={"html": "<p>hi</p>", "code": 200})
b = Job(id=2, kind="b", created_at="", result={"code": 204})
context = RunContext(waited_for=(a, b))
assert context.upstream() == {"html": "<p>hi</p>", "code": 204}, "later parents win"
def test_upstream_is_empty_under_null_so_a_chained_handler_is_still_callable():
"""The one line that keeps a chained handler an ordinary function.
Every other handler in the pipeline can be called from a terminal. One
that read `waited_for[0].result` directly could not — it would raise
IndexError under `NULL` — so `upstream()` answers `{}` and the handler
falls back to the keyword it declared.
"""
assert NULL.upstream() == {}
@job("t_chained")
def handler(config, store, *, context=NULL, html: str = "") -> dict:
return {"html": html or context.upstream().get("html", "")}
assert handler(None, None, html="passed by hand") == {"html": "passed by hand"}
assert handler(None, None) == {"html": ""}, "and it does not blow up with neither"
# ----------------------------------------------------- a fan-out is one call
def test_fan_out_queues_the_children_and_the_job_that_reports_on_them(jobs):
"""One call, and the reporting job is not the app's to write.
The children get a run of their own inside the caller's cascade, and the
report waits on all of them with `after='done'` — the mode that exists
precisely so a job reporting that one of three failed is not held back by
that failure.
"""
queue = JobQueue(jobs, None, None)
parent = got(jobs.enqueue("t_parent"))
context = queue._context(parent, [], [])
children = context.fan_out(
"t_child", [{"i": 1}, {"i": 2}, {"i": 3}], then="mod:fn")
assert len(children) == 3, "it answers with the children's ids"
queued = jobs.snapshot().queued
report = [j for j in queued if j.kind == "advance"]
assert len(report) == 1, "and queued exactly one job to report on them"
assert sorted(report[0].depends_on) == sorted(children)
assert report[0].after == "done"
assert report[0].payload["callback"] == "mod:fn"
assert {j.task_id for j in queued if j.id in children} == {report[0].task_id}, \
"children and report are one run"
def test_fan_out_queues_nothing_under_null_and_says_so():
"""A handler called directly is being run by somebody who is standing
there, and the useful answer is to do its own work and report that nothing
was queued behind it — the same answer `task` gives, for the same reason."""
assert NULL.fan_out("t_child", [{"i": 1}]) == ()
def test_a_fan_out_carries_what_the_caller_puts_on_it(jobs):
"""`carries` goes *around* the children to the callback: static input from
whoever asked, as against `Tally.results`, which is what the children
themselves returned."""
queue = JobQueue(jobs, None, None)
parent = got(jobs.enqueue("t_parent"))
context = queue._context(parent, [], [])
context.fan_out("t_child", [{"i": 1}], then="mod:fn", carries={"since": "monday"})
report = [j for j in jobs.snapshot().queued if j.kind == "advance"]
assert report[0].payload["payload"] == {"since": "monday"}
+134
View File
@@ -0,0 +1,134 @@
"""The notice log: said once, kept until cleared, forgotten when it stops.
A notice used to be a line that faded in twelve seconds in whichever tab was
open, and was raised as often as not while none was. These hold the three
things the table is for: a repeat is the same row and not news, a cleared row
makes the next one news again, and what nobody clears goes when it has stopped
being said rather than when it was first said.
python -m pytest pipeline/tests/test_notice_log.py -q
"""
from __future__ import annotations
import sqlite3
import sys
import tempfile
from contextlib import closing
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from pipeline import NULL, JobQueue, Jobs, Notice, RunContext, job
from pipeline.web import connect
@job("n_refused", "Meeting the same refusal as last time")
def n_refused(config, store, *, context: RunContext = NULL) -> dict:
context.notice(Notice(code="agent.refused", key="Worm hits npm",
text="No abstract: Worm hits npm", fix="Nothing to do."))
return {}
@pytest.fixture
def jobs():
with tempfile.TemporaryDirectory() as tmp:
yield Jobs(Path(tmp) / "n.db")
def refusal(text: str = "No abstract: Worm hits npm") -> Notice:
return Notice(code="agent.refused", key="Worm hits npm", text=text,
where="the tagging lines in the server log")
def test_the_same_notice_again_is_the_same_row_with_a_count(jobs):
first = jobs.record_notice(refusal())
again = jobs.record_notice(refusal("No abstract, again: Worm hits npm"))
other = jobs.record_notice(Notice(code="agent.refused", key="Other", text="x"))
assert (first.count, again.count) == (1, 2), "one is news, two is a repeat"
assert again.id == first.id and other.id != first.id, "keyed by code and key"
assert again.first_at == first.first_at and again.last_at >= first.last_at
assert again.text.startswith("No abstract, again"), "the newer wording is kept"
assert again.where == first.where, "and `where` survives its SQL column name"
assert [n.key for n in jobs.notices()] == ["Other", "Worm hits npm"]
def test_clearing_one_makes_the_next_occurrence_news_again(jobs):
said = jobs.record_notice(refusal())
assert jobs.clear_notice(said.id)
assert not jobs.clear_notice(said.id), "gone already is an answer, not an error"
assert jobs.record_notice(refusal()).count == 1
def test_a_notice_is_forgotten_when_it_stops_being_said_not_when_it_began(jobs):
"""By `last_at`. A feed that has failed nightly for a month is an old row
that was true an hour ago, and dropping it would make tonight's failure
read as the first."""
chronic = jobs.record_notice(Notice(code="source.failed", key="rss", text="x"))
over = jobs.record_notice(Notice(code="source.failed", key="old", text="y"))
month = (datetime.now(UTC) - timedelta(days=30)).isoformat()
with closing(sqlite3.connect(jobs.path)) as conn, conn:
conn.execute("UPDATE notices SET first_at=? WHERE id=?", (month, chronic.id))
conn.execute("UPDATE notices SET first_at=?, last_at=? WHERE id=?",
(month, month, over.id))
assert jobs.purge_notices(14 * 86400) == 1
assert [n.key for n in jobs.notices()] == ["rss"]
def test_a_job_that_repeats_itself_is_heard_as_a_repeat(jobs):
"""The whole path. Every notice still reaches the listener — what
interrupts a person is the interface's business — and the second carries
the count that says it is nothing new."""
heard: list = []
async def on_notice(notice) -> None:
heard.append(notice)
for _ in range(2):
jobs.enqueue("n_refused")
JobQueue(jobs, None, None, on_notice=on_notice).drain(timeout=10)
assert [n.count for n in heard] == [1, 2]
assert len(jobs.notices()) == 1
def test_a_log_that_cannot_be_written_does_not_cost_the_notice(jobs, monkeypatch):
heard: list = []
async def on_notice(notice) -> None:
heard.append(notice)
def broken(notice):
raise sqlite3.OperationalError("database is locked")
monkeypatch.setattr(jobs, "record_notice", broken)
done = jobs.enqueue("n_refused")
JobQueue(jobs, None, None, on_notice=on_notice).drain(timeout=10)
assert [n.code for n in heard] == ["agent.refused"]
assert jobs.get(done.id).state == "done", "and the job that said it is unharmed"
def test_the_log_is_read_and_cleared_over_http(jobs):
app = FastAPI()
connect(app, jobs, None, None)
jobs.record_notice(refusal())
jobs.record_notice(Notice(code="source.failed", key="rss", text="rss failed"))
with TestClient(app) as client:
listed = client.get("/api/notices").json()
assert [n["code"] for n in listed] == ["source.failed", "agent.refused"]
assert listed[1]["where"] == "the tagging lines in the server log"
assert client.delete(f"/api/notices/{listed[0]['id']}").json() == {"cleared": True}
assert client.delete(f"/api/notices/{listed[0]['id']}").json() == {"cleared": False}
assert client.delete("/api/notices").json() == {"cleared": 1}
assert client.get("/api/notices").json() == []
+355
View File
@@ -0,0 +1,355 @@
"""Waiting to be tried again, without anything being alive while it waits.
The claim under test is that a job between attempts costs a row and nothing
else. No worker, no thread, no lease, no timer — so the tests that matter here
are the ones that ask what *else* the queue can do while a job waits, and what
survives the process that scheduled the wait.
Each of the four properties step 12 asks for is pinned on its own, because
they fail independently: a bound that is off by one still waits correctly, a
`run_after` that never reaches the table still retries in one process, and a
Kill that cannot reach a waiting row is invisible until somebody presses it.
python3 -m pytest tests/test_retries.py -q
"""
from __future__ import annotations
import asyncio
import sqlite3
import sys
import tempfile
import time
from contextlib import closing
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from pipeline.models import Job
from pipeline import NULL, JobQueue, Jobs, Retry, job
@pytest.fixture
def jobs():
"""The pipeline's table on a file of its own. No app, no store — nothing
here needs one, which is the point of the table owning its own schema."""
with tempfile.TemporaryDirectory() as tmp:
yield Jobs(Path(tmp) / "t.db")
def made(queued: Job | None) -> Job:
"""The job an enqueue made. `enqueue` answers None for a killed run, which
in these tests nothing is."""
assert queued is not None
return queued
def got(found: Job | None) -> Job:
"""The job a read must have found — a None here is a test failure."""
assert found is not None
return found
def lease(jobs: Jobs, job_id: int) -> tuple[str | None, str | None]:
"""The `started_at` and `seen_at` of one job: whether anybody has it."""
with closing(sqlite3.connect(jobs.path)) as conn:
row = conn.execute(
"SELECT started_at, seen_at FROM jobs WHERE id=?", (job_id,)).fetchone()
return row[0], row[1]
def bring_forward(jobs: Jobs, job_id: int) -> None:
"""Move a waiting job's next attempt into the past, as if its wait were
over. What the clock does, done in one statement so a test of an hour's
backoff does not take an hour."""
with closing(sqlite3.connect(jobs.path)) as conn, conn:
conn.execute("UPDATE jobs SET run_after=? WHERE id=?",
((datetime.now(UTC) - timedelta(hours=1)).isoformat(), job_id))
def drain(queue: JobQueue, *, timeout: float = 5.0) -> None:
"""Run the worker until the table is empty and this worker holds nothing.
Both halves, because a job's last acts belong to its handler thread and
between them the table reads as idle while it is not. A job waiting for its
next attempt keeps this going, since it is still `queued` — which is the
behaviour the retry tests are leaning on.
"""
async def go() -> None:
await queue.start()
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
snap = queue.jobs.snapshot()
if not snap.running and not snap.queued and not queue._running:
break
await asyncio.sleep(0.01)
await queue.stop()
asyncio.run(go())
# ------------------------------------------------------- not holding a place
def test_a_failed_attempt_goes_back_to_the_queue_with_a_time_on_it(jobs):
"""The resting state of a retry, and everything that is true of it."""
queued = made(jobs.enqueue("r_wait", max_attempts=3, retry_delay=600))
assert got(jobs.claim({})).attempt == 1, "claim counts the attempt"
again = got(jobs.retry(queued.id, error="RuntimeError: the feed was down"))
assert again.state == "queued", "it is in the queue, not failed"
assert again.waiting and not again.exhausted
assert again.error == "RuntimeError: the feed was down", "it says why it waits"
assert lease(jobs, queued.id) == (None, None), "nobody holds it"
assert jobs.running() == [], "and nothing is running it"
assert [j.id for j in jobs.waiting()] == [queued.id]
assert [j.id for j in jobs.blocked()] == [queued.id]
def test_a_waiting_job_does_not_occupy_a_worker(jobs):
"""The whole of why the wait is a column. One cpu worker, one job waiting
ten minutes for its next attempt, and the worker is free the entire time —
a handler that slept would have held it and stopped the queue dead."""
waiting = made(jobs.enqueue("r_slot", max_attempts=2, retry_delay=600))
assert jobs.claim({"cpu": 1}) is not None
jobs.retry(waiting.id, error="not this time")
other = made(jobs.enqueue("r_other"))
assert got(jobs.claim({"cpu": 1})).id == other.id, "the slot was free"
assert got(jobs.get(waiting.id)).state == "queued"
def test_a_job_queued_for_later_is_not_claimed_before_its_time(jobs):
"""`delay` on the enqueue itself: the durable form of "in ten minutes"."""
later = made(jobs.enqueue("r_later", delay=600))
assert later.waiting
assert jobs.claim({}) is None
bring_forward(jobs, later.id)
assert got(jobs.claim({})).id == later.id
def test_what_waits_on_a_retrying_job_goes_on_waiting(jobs):
"""A retry is still live work, so a dependent is held rather than settled.
The alternative would be the worse half of both: a batch that is coming
back in a minute counted among the ones that did not land, and a final job
reporting on a task that is not over.
"""
batch = made(jobs.enqueue("r_batch", max_attempts=2, retry_delay=600))
final = made(jobs.enqueue("r_final", depends_on=[batch.id], after="done"))
jobs.claim({})
jobs.retry(batch.id, error="again in a minute")
assert [j.id for j in jobs.blocked()] == [batch.id, final.id]
bring_forward(jobs, batch.id)
assert got(jobs.claim({})).id == batch.id, "the batch goes first"
jobs.finish(batch.id, "done")
assert got(jobs.claim({})).id == final.id
# --------------------------------------------------------------- the bounds
def test_the_bound_counts_the_first_attempt(jobs):
"""`max_attempts=3` is three runs of the handler, not three retries."""
runs = []
@job("r_bounded", "Bounded", max_attempts=3)
def handler(config, store, *, context=NULL):
runs.append(1)
raise RuntimeError("this was never going to work")
queue = JobQueue(jobs, None, None)
queued = made(queue.enqueue("r_bounded"))
drain(queue)
ended = got(jobs.get(queued.id))
assert len(runs) == 3, "it ran exactly as many times as it was allowed"
assert ended.state == "failed" and ended.attempt == 3
assert ended.exhausted, "and says the policy ran out rather than merely failing"
def test_a_job_with_no_policy_fails_at_the_first_exception(jobs):
"""The default is one attempt, which is every job queued before any of
this existed. Exhausted is not what that is: nothing was used up."""
runs = []
@job("r_unbounded", "Unbounded")
def handler(config, store, *, context=NULL):
runs.append(1)
raise RuntimeError("no")
queue = JobQueue(jobs, None, None)
queued = made(queue.enqueue("r_unbounded"))
drain(queue)
ended = got(jobs.get(queued.id))
assert len(runs) == 1
assert ended.state == "failed" and not ended.exhausted
def test_an_attempt_that_lands_stops_the_retrying(jobs):
"""The case the bound exists for: the second go works, and the job is done
rather than carrying a memory of having failed once."""
runs = []
@job("r_second_time", "Second time", max_attempts=4)
def handler(config, store, *, context=NULL):
runs.append(1)
if len(runs) < 2:
raise RuntimeError("the first one never works")
return {"runs": len(runs)}
queue = JobQueue(jobs, None, None)
queued = made(queue.enqueue("r_second_time"))
drain(queue)
ended = got(jobs.get(queued.id))
assert ended.state == "done" and ended.result == {"runs": 2}
assert ended.attempt == 2 and not ended.exhausted
def test_the_wait_lengthens_with_the_backoff(jobs):
"""Doubling, read off the two stamps rather than waited out."""
queued = made(jobs.enqueue("r_backoff", max_attempts=4,
retry_delay=100, retry_backoff=2.0))
waits = []
for _ in range(3):
assert jobs.claim({}) is not None
again = got(jobs.retry(queued.id, error="again"))
waits.append(
datetime.fromisoformat(again.run_after) - datetime.now(UTC))
bring_forward(jobs, queued.id)
seconds = [w.total_seconds() for w in waits]
assert [round(s / 100) for s in seconds] == [1, 2, 4]
def test_a_server_hint_replaces_the_backoff(jobs):
"""Honouring what the other end said, rather than guessing at it.
The bound is not part of the bargain: a hint moves the next attempt, it
does not add one.
"""
queued = made(jobs.enqueue("r_hinted", max_attempts=3, retry_delay=600))
jobs.claim({})
again = got(jobs.retry(queued.id, error="429", delay=5))
waited = datetime.fromisoformat(again.run_after) - datetime.now(UTC)
assert 0 < waited.total_seconds() <= 5
def test_a_handler_can_name_its_own_wait(jobs):
"""`Retry(after=...)` from inside the handler, which is where a
`Retry-After` header is actually read."""
@job("r_asked", "Asked to come back", max_attempts=3, retry_delay=600)
def handler(config, store, *, context=NULL):
raise Retry("rate limited", after=7)
queue = JobQueue(jobs, None, None)
queued = made(queue.enqueue("r_asked"))
async def one_pass() -> None:
await queue.start()
deadline = time.monotonic() + 5
while time.monotonic() < deadline:
if got(jobs.get(queued.id)).waiting:
break
await asyncio.sleep(0.01)
await queue.stop()
asyncio.run(one_pass())
waiting = got(jobs.get(queued.id))
assert waiting.waiting and waiting.attempt == 1
assert "rate limited" in waiting.error
waited = datetime.fromisoformat(waiting.run_after) - datetime.now(UTC)
assert 0 < waited.total_seconds() <= 7, "the handler's number, not the policy's"
def test_asking_to_come_back_does_not_extend_the_bound(jobs):
"""A handler cannot grant itself a policy nobody declared."""
runs = []
@job("r_asked_once", "Asked once")
def handler(config, store, *, context=NULL):
runs.append(1)
raise Retry("come back in a second", after=0)
queue = JobQueue(jobs, None, None)
queued = made(queue.enqueue("r_asked_once"))
drain(queue)
assert len(runs) == 1
assert got(jobs.get(queued.id)).state == "failed"
# ------------------------------------------------------------- across a stop
def test_the_next_attempt_survives_the_process_that_scheduled_it(jobs):
"""The wait is a column, so it outlives everything that is not the file.
A second `Jobs` on the same path is what another process sees, and what
this one sees after a restart: a queued row it will not take yet, and takes
the moment its time comes round. Nothing was handed over, because there
was nothing alive to hand.
"""
queued = made(jobs.enqueue("r_restart", max_attempts=3, retry_delay=600))
jobs.claim({})
jobs.retry(queued.id, error="the feed was down")
restarted = Jobs(jobs.path)
assert restarted.claim({}) is None, "still waiting, on a table that never saw it fail"
assert [j.id for j in restarted.waiting()] == [queued.id]
bring_forward(restarted, queued.id)
taken = got(restarted.claim({}))
assert taken.id == queued.id
assert taken.attempt == 2, "and it is the second attempt, not the first again"
# ------------------------------------------------------------- cancellation
def test_cancelling_a_job_that_is_waiting_to_be_retried_ends_it(jobs):
"""Kill on a waiting row. It is queued, so it ends here and now — there is
no worker to ask and nothing part-way through."""
queued = made(jobs.enqueue("r_killable", max_attempts=3, retry_delay=600))
jobs.claim({})
jobs.retry(queued.id, error="not yet")
ended = got(jobs.request_cancel(queued.id))
assert ended.state == "cancelled"
assert jobs.waiting() == []
def test_killing_the_run_takes_a_job_waiting_to_be_retried_with_it(jobs):
"""The same for the task and cascade buttons: an hour's wait is not an
hour during which a run cannot be stopped."""
cascade = jobs.new_run("cascade", "r_run")
task = jobs.new_run("task", "r_task")
queued = made(jobs.enqueue("r_in_a_run", max_attempts=3, retry_delay=600,
cascade_id=cascade, task_id=task))
jobs.claim({})
jobs.retry(queued.id, error="not yet")
assert jobs.cancel_run(task) == 1
assert got(jobs.get(queued.id)).state == "cancelled"
def test_a_kill_during_the_attempt_is_not_undone_by_the_retry(jobs):
"""The one ordering that would make Kill a lie.
Press it while the attempt runs, the attempt then fails, and a retry that
only looked at the bound would put the job straight back in the queue —
a button that stopped nothing and said it had.
"""
queued = made(jobs.enqueue("r_killed_mid", max_attempts=3, retry_delay=0))
jobs.claim({})
jobs.request_cancel(queued.id)
ended = got(jobs.retry(queued.id, error="it failed as well"))
assert ended.state == "cancelled"
assert not ended.waiting
+701
View File
@@ -0,0 +1,701 @@
"""The scheduler: the five wordings, the table, the firing, and the routes.
python -m pytest pipeline/tests/test_schedule.py -q
Three things here are worth knowing before changing anything.
The daylight-saving tests are the reason `next_after` takes a zone at all, and
they are written around a real transition rather than a synthetic one, because
the bug they guard is precisely that `+ timedelta(days=1)` looks correct.
The firing tests call `JobQueue._fire_due` directly. It is a private, and that
is deliberate: driving it through `start()` would mean asserting on a loop that
ticks on a clock, and a test that sleeps to make a scheduler fire is a test
that fails on a busy machine. One test does start a real queue, to prove the
tick is wired into the loop at all.
`SCHEDULES` is a module-level registry, like `JOBS`, so anything here that
declares into it does so through the `declared` fixture, which puts it back.
"""
from __future__ import annotations
import os
import subprocess
import sys
import tempfile
from datetime import UTC, datetime, timedelta
from pathlib import Path
from zoneinfo import ZoneInfo
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from pipeline import NULL, JobQueue, Jobs, RunContext, job
from pipeline.schedule import SCHEDULES, BadCadence, Cadence, declare
ROOT = Path(__file__).resolve().parents[2]
COPENHAGEN = ZoneInfo("Europe/Copenhagen")
@job("s_tick", "A scheduled tick")
def s_tick(config, store, *, context: RunContext = NULL, note: str = "") -> dict:
store.append(note)
return {"note": note}
@job("s_solo", "One at a time", solo=True)
def s_solo(config, store, *, context: RunContext = NULL) -> dict:
return {}
@job("s_needs", "Needing a payload")
def s_needs(config, store, *, context: RunContext = NULL, target: str) -> dict:
return {"target": target}
@pytest.fixture
def jobs():
with tempfile.TemporaryDirectory() as tmp:
yield Jobs(Path(tmp) / "s.db")
@pytest.fixture
def queue(jobs):
"""A queue that is never started: the tests below fire it by hand.
`tz="UTC"` so a cadence's next run is the same number wherever this runs.
"""
return JobQueue(jobs, {}, [], tz="UTC")
@pytest.fixture
def declared():
"""Borrow the declaration registry and give it back.
`SCHEDULES` is global, like `JOBS`, and a test that left something in it
would give every later test in the session a schedule it never asked for.
"""
saved = dict(SCHEDULES)
SCHEDULES.clear()
yield SCHEDULES
SCHEDULES.clear()
SCHEDULES.update(saved)
def ago(seconds: float) -> str:
return (datetime.now(UTC) - timedelta(seconds=seconds)).isoformat()
# ------------------------------------------------------------ the grammar
WORDINGS = [
("every 15m", "every 15m"),
("every 6h", "every 6h"),
("every 2d", "every 2d"),
("daily at 06:00", "daily at 06:00"),
("weekly mon 07:00", "weekly mon 07:00"),
# Written how people write them, stored how the table stores them.
("weekly monday 07:00", "weekly mon 07:00"),
("Weekly Sunday 23:59", "weekly sun 23:59"),
("EVERY 15 M", "every 15m"),
(" daily at 6:05 ", "daily at 06:05"),
("every 1d", "every 1d"),
]
@pytest.mark.parametrize("typed,canonical", WORDINGS)
def test_every_wording_parses_and_comes_back_canonical(typed, canonical):
"""Five forms, one spelling each.
The round trip is the point: the table stores `str(cadence)`, so two
people writing the same schedule two ways end up with one row that reads
the same to both of them.
"""
cadence = Cadence.parse(typed)
assert str(cadence) == canonical
assert str(Cadence.parse(canonical)) == canonical
assert Cadence.parse(typed) == Cadence.parse(canonical)
def test_the_units_are_what_they_say():
assert Cadence.parse("every 15m").seconds == 900
assert Cadence.parse("every 6h").seconds == 6 * 3600
assert Cadence.parse("every 2d").seconds == 2 * 86400
# A clock cadence has no fixed length, and says so rather than guessing.
assert Cadence.parse("daily at 06:00").seconds == 0
@pytest.mark.parametrize("wording", [
"sometimes", "", "cron 0 6 * * *", "every 15", "every m", "every 0h",
"every 15s", "daily 06:00", "daily at 25:00", "daily at 06:60",
"weekly 07:00", "weekly funday 07:00", "every 15m please",
])
def test_a_wording_outside_the_five_is_refused_with_all_five(wording):
"""And the refusal is the sentence a person is shown.
It goes out of the API as `{"detail": ...}` and into a toast, so it has to
say what to type instead rather than merely that this was wrong.
"""
with pytest.raises(ValueError) as refused:
Cadence.parse(wording)
said = str(refused.value)
assert isinstance(refused.value, BadCadence)
for form in ("every 15m", "every 6h", "every 2d",
"daily at 06:00", "weekly mon 07:00"):
assert form in said
# ------------------------------------------------------------- next_after
def test_next_after_is_strictly_after():
"""A schedule fired at its own time must not find itself due again on the
same instant, which would be one firing per tick for ever."""
at_six = datetime(2025, 6, 1, 6, 0, tzinfo=COPENHAGEN)
when = Cadence.parse("daily at 06:00").next_after(at_six, COPENHAGEN)
assert when.astimezone(COPENHAGEN) == at_six + timedelta(days=1)
def test_next_after_always_comes_back_in_utc():
"""Because it is written straight into the table, where every other time
is UTC and the column is compared as text."""
when = Cadence.parse("daily at 06:00").next_after(
datetime(2025, 6, 1, 12, 0, tzinfo=COPENHAGEN), COPENHAGEN)
assert when.tzinfo is UTC
assert when.isoformat().endswith("+00:00")
def test_weekly_lands_on_the_day_it_names():
monday = Cadence.parse("weekly mon 07:00")
# A Wednesday: the next Monday is five days away, not tomorrow.
from_wed = monday.next_after(datetime(2025, 6, 4, 9, 0, tzinfo=COPENHAGEN),
COPENHAGEN).astimezone(COPENHAGEN)
assert from_wed.weekday() == 0 and (from_wed.hour, from_wed.minute) == (7, 0)
assert from_wed.date() == datetime(2025, 6, 9).date()
# Monday morning before seven: today, not next week.
from_mon = monday.next_after(datetime(2025, 6, 9, 6, 0, tzinfo=COPENHAGEN),
COPENHAGEN).astimezone(COPENHAGEN)
assert from_mon.date() == datetime(2025, 6, 9).date()
def test_daily_keeps_its_wall_clock_across_both_dst_changes():
"""06:00 is 06:00 on both sides, which is the whole reason for the zone.
Spring forward is a 23-hour day and autumn back is a 25-hour one, so the
real elapsed time between two runs is not 24 hours — and it must not be,
because what the wording promised is a time of day.
"""
daily = Cadence.parse("daily at 06:00")
# Europe/Copenhagen, 2025: forward on 30 March, back on 26 October.
forward = daily.next_after(
datetime(2025, 3, 29, 6, 30, tzinfo=COPENHAGEN), COPENHAGEN)
local = forward.astimezone(COPENHAGEN)
assert (local.hour, local.minute) == (6, 0)
assert local.date() == datetime(2025, 3, 30).date()
# 05:30Z to 04:00Z: 22h30m of real time for a 23h30m step of the clock.
assert forward - datetime(2025, 3, 29, 6, 30, tzinfo=COPENHAGEN) \
== timedelta(hours=22, minutes=30)
back = daily.next_after(
datetime(2025, 10, 25, 6, 30, tzinfo=COPENHAGEN), COPENHAGEN)
local = back.astimezone(COPENHAGEN)
assert (local.hour, local.minute) == (6, 0)
assert local.date() == datetime(2025, 10, 26).date()
assert back - datetime(2025, 10, 25, 6, 30, tzinfo=COPENHAGEN) \
== timedelta(hours=24, minutes=30)
def test_weekly_keeps_its_wall_clock_across_a_dst_change():
when = Cadence.parse("weekly sun 07:00").next_after(
datetime(2025, 3, 26, 9, 0, tzinfo=COPENHAGEN), COPENHAGEN)
local = when.astimezone(COPENHAGEN)
assert local.date() == datetime(2025, 3, 30).date() # the change itself
assert (local.hour, local.minute) == (7, 0)
def test_an_interval_is_elapsed_time_and_ignores_the_change():
"""The other half of the promise, and the reason both wordings exist.
`every 24h` across a spring-forward is 24 real hours and therefore lands
an hour later on the clock. A person who wanted 06:00 wrote `daily`.
"""
start = datetime(2025, 3, 29, 6, 30, tzinfo=COPENHAGEN)
when = Cadence.parse("every 24h").next_after(start, COPENHAGEN)
assert when - start == timedelta(hours=24)
assert when.astimezone(COPENHAGEN).hour == 7
def test_a_naive_moment_is_read_as_utc():
"""Rather than raising. Every stamp in the table is UTC, so a caller that
lost the tzinfo on the way is very nearly certainly holding one."""
naive = datetime(2025, 6, 1, 12, 0)
assert Cadence.parse("every 1h").next_after(naive) \
== datetime(2025, 6, 1, 13, 0, tzinfo=UTC)
# ---------------------------------------------------------------- the table
def test_the_table_holds_a_schedule_through_its_life(jobs):
row = jobs.add_schedule("nightly", "s_tick", "daily at 06:00",
payload={"note": "hello"}, label="A nightly tick",
next_run="2030-01-01T06:00:00+00:00")
assert row is not None
assert (row.name, row.kind, row.cadence) == ("nightly", "s_tick", "daily at 06:00")
assert row.payload == {"note": "hello"} and row.enabled is True
assert row.last_job_id == 0 and row.last_run == "" and row.last_state == ""
assert [s.name for s in jobs.schedules()] == ["nightly"]
assert jobs.schedule(row.id) == row
edited = jobs.update_schedule(row.id, cadence="every 15m", enabled=False,
payload={"note": "other"})
assert edited.cadence == "every 15m" and edited.enabled is False
assert edited.payload == {"note": "other"}
# Untouched columns stay as they were.
assert edited.name == "nightly" and edited.label == "A nightly tick"
assert jobs.delete_schedule(row.id) is True
assert jobs.delete_schedule(row.id) is False # two tabs; not an error
assert jobs.schedules() == []
def test_a_name_is_taken_once(jobs):
assert jobs.add_schedule("one", "s_tick", "every 15m") is not None
assert jobs.add_schedule("one", "s_tick", "every 6h", only_if_new=True) is None
assert jobs.schedules()[0].cadence == "every 15m"
def test_schedules_carry_the_state_of_their_last_job(jobs):
row = jobs.add_schedule("one", "s_tick", "every 15m")
queued = jobs.enqueue("s_tick")
jobs.mark_fired(row.id, queued.id, "2030-01-01T00:00:00+00:00")
assert jobs.schedules()[0].last_state == "queued"
jobs.claim({"cpu": 1})
jobs.finish(queued.id, "failed", error="no")
assert jobs.schedules()[0].last_state == "failed"
# And the time it fired, which is not the time it is next due.
listed = jobs.schedules()[0]
assert listed.last_run and listed.next_run == "2030-01-01T00:00:00+00:00"
def test_due_schedules_passes_over_what_is_off_or_not_yet(jobs):
jobs.add_schedule("due", "s_tick", "every 15m", next_run=ago(60))
jobs.add_schedule("off", "s_tick", "every 15m", next_run=ago(60), enabled=False)
jobs.add_schedule("later", "s_tick", "every 15m",
next_run="2099-01-01T00:00:00+00:00")
jobs.add_schedule("never", "s_tick", "every 15m") # no time at all
due = jobs.due_schedules(datetime.now(UTC).isoformat())
assert [s.name for s in due] == ["due"]
# --------------------------------------------------------------- the firing
def test_the_tick_fires_a_due_row_and_moves_it_on(queue, jobs):
row = jobs.add_schedule("regular", "s_tick", "every 15m",
payload={"note": "scheduled"}, next_run=ago(30))
queue._fire_due()
queued = jobs.snapshot().queued
assert [j.kind for j in queued] == ["s_tick"]
assert queued[0].payload == {"note": "scheduled"}
after = jobs.schedule(row.id)
assert after.last_job_id == queued[0].id
assert after.last_run
# Fifteen minutes from now, not from the moment it was due.
assert after.next_run > datetime.now(UTC).isoformat()
assert after.next_run < (datetime.now(UTC) + timedelta(minutes=16)).isoformat()
def test_a_schedule_does_not_pile_up_behind_its_own_job(queue, jobs):
"""The job from the last firing is still queued, so this one is skipped.
Skipped and *moved on*, not left due: a schedule whose job takes longer
than its cadence would otherwise fire on every tick for as long as the job
ran, and the queue would fill with copies of work already under way.
"""
row = jobs.add_schedule("regular", "s_tick", "every 15m", next_run=ago(30))
queue._fire_due()
first = jobs.schedule(row.id).last_job_id
jobs.update_schedule(row.id, next_run=ago(1)) # due again
queue._fire_due()
assert len(jobs.snapshot().queued) == 1
again = jobs.schedule(row.id)
assert again.last_job_id == first # nothing new was queued
assert again.next_run > datetime.now(UTC).isoformat()
def test_a_solo_kind_already_underway_holds_its_schedule_back(queue, jobs):
"""The other way a schedule collides: not with itself, with somebody else.
A button, a second schedule or a cascade started one, and `enqueue` would
hand that job back — so the schedule would record work it did not ask for
as its own firing.
"""
pressed = jobs.enqueue("s_solo", dedup_key="s_solo")
row = jobs.add_schedule("solo", "s_solo", "every 15m", next_run=ago(30))
queue._fire_due()
after = jobs.schedule(row.id)
assert after.last_job_id == 0 # it never fired
assert after.next_run > datetime.now(UTC).isoformat()
assert [j.id for j in jobs.snapshot().queued] == [pressed.id]
def test_downtime_is_caught_up_once_and_not_once_per_slot(queue, jobs):
"""Three days off an hourly schedule is one job, not seventy-two.
The row carries one `next_run`, and a firing sets it from *now* rather
than stepping through what was missed — so there is nowhere for a backlog
to accumulate even in principle.
"""
row = jobs.add_schedule("hourly", "s_tick", "every 1h", next_run=ago(3 * 86400))
queue._fire_due()
assert len(jobs.snapshot().queued) == 1
after = jobs.schedule(row.id)
assert after.next_run > datetime.now(UTC).isoformat()
assert after.next_run < (datetime.now(UTC) + timedelta(minutes=61)).isoformat()
def test_an_unknown_kind_never_fires_and_keeps_its_place(queue, jobs):
"""A handler that went away in a deploy. The row stays exactly as it is —
time included — so reinstalling the plugin runs it rather than starting
another cadence from scratch."""
row = jobs.add_schedule("gone", "s_vanished", "every 15m", next_run=ago(30))
queue._fire_due()
assert jobs.snapshot().queued == []
assert jobs.schedule(row.id).next_run == row.next_run
queue._fire_due() # and it says so once
assert queue._said_about == {(row.id, "kind")}
def test_a_cadence_the_table_cannot_parse_never_fires(queue, jobs):
"""Only reachable by editing the file by hand, since every route parses
first. It must not take the tick down with it."""
jobs.add_schedule("bent", "s_tick", "every 15m", next_run=ago(30))
jobs._write("UPDATE schedules SET cadence='sometimes'")
queue._fire_due()
assert jobs.snapshot().queued == []
assert jobs.schedules()[0].cadence == "sometimes" # left to be fixed
def test_a_disabled_row_is_not_fired(queue, jobs):
jobs.add_schedule("off", "s_tick", "every 15m", next_run=ago(30), enabled=False)
queue._fire_due()
assert jobs.snapshot().queued == []
def test_run_now_fires_without_moving_the_next_run(queue, jobs):
row = jobs.add_schedule("nightly", "s_tick", "daily at 06:00",
next_run="2099-01-01T06:00:00+00:00")
started = queue.run_schedule(row.id)
assert started is not None and started.kind == "s_tick"
after = jobs.schedule(row.id)
assert after.next_run == "2099-01-01T06:00:00+00:00"
# But it did just run, and that is the job it queued — so the next tick
# will not start a second one while this is still going.
assert after.last_job_id == started.id and after.last_run
# ------------------------------------------------------------ reconciliation
def test_declared_defaults_get_a_row_on_start(declared, jobs):
declare("nightly tick", "s_tick", "daily at 06:00", {"note": "declared"},
label="A tick, nightly")
JobQueue(jobs, {}, [], tz="UTC")._reconcile_schedules()
row = jobs.schedules()[0]
assert (row.name, row.kind, row.cadence) == ("nightly tick", "s_tick",
"daily at 06:00")
assert row.payload == {"note": "declared"} and row.label == "A tick, nightly"
assert row.next_run # and it knows when it is next
def test_reconciliation_keeps_what_the_user_edited(declared, jobs):
"""The whole of what a declaration means: a starting point, not a setting.
Switching a default off, or slowing it down, has to survive the next
deploy — otherwise the only way to be rid of one is to edit the code and
redeploy, which is not a thing the person running the app can do.
"""
declare("nightly tick", "s_tick", "daily at 06:00")
queue = JobQueue(jobs, {}, [], tz="UTC")
queue._reconcile_schedules()
row = jobs.schedules()[0]
jobs.update_schedule(row.id, cadence="every 2d", enabled=False,
payload={"note": "mine"})
queue._reconcile_schedules() # a restart
again = jobs.schedules()[0]
assert again.id == row.id
assert again.cadence == "every 2d"
assert again.enabled is False
assert again.payload == {"note": "mine"}
def test_a_declaration_that_goes_away_leaves_its_row(declared, jobs):
"""Nothing here deletes. A deploy that silently forgot a schedule somebody
was relying on is the failure this avoids; an orphan says `unknown kind`
and waits to be deleted deliberately."""
declare("nightly tick", "s_tick", "daily at 06:00")
queue = JobQueue(jobs, {}, [], tz="UTC")
queue._reconcile_schedules()
declared.clear()
queue._reconcile_schedules()
assert [s.name for s in jobs.schedules()] == ["nightly tick"]
def test_declaring_one_name_twice_differently_is_refused(declared):
declare("nightly", "s_tick", "daily at 06:00")
declare("nightly", "s_tick", "daily at 06:00") # the same: fine
with pytest.raises(ValueError) as refused:
declare("nightly", "s_tick", "daily at 07:00")
assert "nightly" in str(refused.value)
def test_a_declared_cadence_is_parsed_where_it_is_written(declared):
"""At import, with a traceback pointing at the line — rather than at 06:00
in a log nobody is reading."""
with pytest.raises(ValueError):
declare("bad", "s_tick", "every so often")
assert "bad" not in declared
# ------------------------------------------------------------- a real queue
def test_a_started_queue_fires_what_is_overdue(jobs, declared):
"""The one test that drives the loop rather than the method.
Everything above proves the firing decides correctly; this proves it is
wired into `start()` at all, which is the wire nobody notices is missing
until a schedule silently stops running.
"""
ran: list[str] = []
jobs.add_schedule("regular", "s_tick", "every 15m",
payload={"note": "from the loop"}, next_run=ago(30))
queue = JobQueue(jobs, {}, ran, tz="UTC", workers={"cpu": 1})
assert queue.drain(timeout=10) is True
assert ran == ["from the loop"]
assert jobs.schedules()[0].last_state == "done"
def test_a_queue_with_schedules_off_fires_nothing(jobs, declared):
ran: list[str] = []
jobs.add_schedule("regular", "s_tick", "every 15m", next_run=ago(30))
queue = JobQueue(jobs, {}, ran, tz="UTC", schedules=False)
assert queue.drain(timeout=10) is True
assert ran == []
assert jobs.schedules()[0].last_job_id == 0
def test_an_unknown_timezone_is_a_warning_and_not_a_refusal(jobs, caplog):
"""A mistyped zone must not be the reason a worker will not start."""
queue = JobQueue(jobs, {}, [], tz="Mars/Olympus_Mons")
assert queue.zone is not None
assert Cadence.parse("daily at 06:00").next_after(datetime.now(UTC), queue.zone)
# ------------------------------------------------------------------ the web
@pytest.fixture
def client(jobs, declared):
pytest.importorskip("fastapi")
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pipeline.web import connect
app = FastAPI()
connect(app, jobs, {}, [], tz="UTC", schedules=False)
with TestClient(app) as client:
yield client
def test_the_element_is_served(client):
answer = client.get("/_jobs/schedules.js")
assert answer.status_code == 200
assert "javascript" in answer.headers["content-type"]
assert "desk-schedules" in answer.text
def test_adding_listing_editing_and_deleting(client):
added = client.post("/api/schedules", json={
"kind": "s_tick", "cadence": "weekly monday 07:00",
"payload": {"note": "over http"}})
assert added.status_code == 201
row = added.json()
# Named after its kind, since none was given, and stored canonically.
assert row["name"] == "s_tick" and row["cadence"] == "weekly mon 07:00"
assert row["next_run"] and row["unknown"] is False
assert [s["id"] for s in client.get("/api/schedules").json()] == [row["id"]]
off = client.patch(f"/api/schedules/{row['id']}", json={"enabled": False})
assert off.status_code == 200 and off.json()["enabled"] is False
# And only that: a PATCH of one field does not resubmit the rest.
assert off.json()["cadence"] == "weekly mon 07:00"
assert off.json()["payload"] == {"note": "over http"}
faster = client.patch(f"/api/schedules/{row['id']}", json={"cadence": "every 15m"})
assert faster.json()["cadence"] == "every 15m"
# A new cadence moves the time with it, rather than leaving the old one.
assert faster.json()["next_run"] != row["next_run"]
gone = client.delete(f"/api/schedules/{row['id']}")
assert gone.status_code == 200 and gone.json() == {"deleted": True}
assert client.delete(f"/api/schedules/{row['id']}").json() == {"deleted": False}
def test_a_second_schedule_of_one_kind_is_named_by_counting(client):
first = client.post("/api/schedules",
json={"kind": "s_tick", "cadence": "every 15m"}).json()
second = client.post("/api/schedules",
json={"kind": "s_tick", "cadence": "every 6h"}).json()
assert (first["name"], second["name"]) == ("s_tick", "s_tick 2")
def test_a_name_somebody_typed_is_refused_rather_than_altered(client):
client.post("/api/schedules",
json={"name": "nightly", "kind": "s_tick", "cadence": "every 15m"})
clash = client.post("/api/schedules",
json={"name": "nightly", "kind": "s_tick", "cadence": "every 6h"})
assert clash.status_code == 409
assert "nightly" in clash.json()["detail"]
def test_an_unregistered_kind_is_a_404_naming_what_there_is(client):
refused = client.post("/api/schedules",
json={"kind": "s_nonesuch", "cadence": "every 15m"})
assert refused.status_code == 404
detail = refused.json()["detail"]
assert "s_nonesuch" in detail and "s_tick" in detail
def test_a_payload_missing_what_the_handler_needs_is_a_422_naming_it(client):
refused = client.post("/api/schedules",
json={"kind": "s_needs", "cadence": "every 15m"})
assert refused.status_code == 422
assert "target" in refused.json()["detail"]
# And with it, the same request is fine.
assert client.post("/api/schedules", json={
"kind": "s_needs", "cadence": "every 15m",
"payload": {"target": "somewhere"}}).status_code == 201
def test_a_bad_cadence_is_a_422_listing_the_accepted_wordings(client):
refused = client.post("/api/schedules",
json={"kind": "s_tick", "cadence": "sometimes"})
assert refused.status_code == 422
detail = refused.json()["detail"]
assert isinstance(detail, str)
for form in ("every 15m", "every 6h", "every 2d",
"daily at 06:00", "weekly mon 07:00"):
assert form in detail
def test_an_edit_is_validated_the_same_way(client):
row = client.post("/api/schedules",
json={"kind": "s_tick", "cadence": "every 15m"}).json()
assert client.patch(f"/api/schedules/{row['id']}",
json={"cadence": "now and then"}).status_code == 422
assert client.patch(f"/api/schedules/{row['id']}",
json={"kind": "s_needs"}).status_code == 422
assert client.patch("/api/schedules/9999",
json={"enabled": False}).status_code == 404
def test_run_now_queues_the_job(client, jobs):
row = client.post("/api/schedules",
json={"kind": "s_tick", "cadence": "daily at 06:00"}).json()
started = client.post(f"/api/schedules/{row['id']}/run")
assert started.status_code == 202
assert started.json()["kind"] == "s_tick"
after = client.get("/api/schedules").json()[0]
assert after["last_job_id"] == started.json()["id"]
assert after["next_run"] == row["next_run"]
assert client.post("/api/schedules/9999/run").status_code == 404
def test_a_row_whose_handler_went_away_is_flagged_not_hidden(client, jobs):
jobs.add_schedule("orphan", "s_vanished", "every 15m")
row = next(s for s in client.get("/api/schedules").json()
if s["name"] == "orphan")
assert row["unknown"] is True
# Listed and deletable, which is the whole of what can be done about it.
assert client.delete(f"/api/schedules/{row['id']}").json() == {"deleted": True}
def test_the_socket_pushes_the_schedule_list_on_every_change(client):
"""So a row edited in one tab redraws in the others without polling.
The opening message is the *job* list — the socket is shared — which is
why `<desk-schedules>` reads `/api/schedules` once on connect and follows
this afterwards.
"""
with client.websocket_connect("/ws/jobs") as socket:
assert socket.receive_json()["type"] == "jobs"
added = client.post("/api/schedules",
json={"kind": "s_tick", "cadence": "every 15m"}).json()
pushed = _next_of(socket, "schedules")
assert [s["id"] for s in pushed["schedules"]] == [added["id"]]
assert pushed["schedules"][0]["cadence"] == "every 15m"
client.patch(f"/api/schedules/{added['id']}", json={"cadence": "every 6h"})
assert _next_of(socket, "schedules")["schedules"][0]["cadence"] == "every 6h"
client.delete(f"/api/schedules/{added['id']}")
assert _next_of(socket, "schedules")["schedules"] == []
def _next_of(socket, type_: str, tries: int = 12) -> dict:
"""The next message of one type, stepping over the job pushes between."""
for _ in range(tries):
message = socket.receive_json()
if message["type"] == type_:
return message
raise AssertionError(f"no {type_} message arrived")
# --------------------------------------------------------------- the worker
def _worker(db: Path, marks: Path, *extra: str) -> subprocess.CompletedProcess:
argv = [sys.executable, "-m", "pipeline.worker", "--db", str(db),
"--bootstrap", "pipeline.tests.scheduled_handlers:marking",
"--bootstrap-arg", str(marks), "--until-idle", *extra]
return subprocess.run(argv, env=dict(os.environ, PYTHONPATH=str(ROOT)),
cwd=ROOT, capture_output=True, timeout=120)
def test_the_standalone_worker_fires_schedules(tmp_path):
"""A machine running only `python -m pipeline.worker` still gets its 06:00
job, which is what makes it a complete replacement for the web app's
worker rather than half of one."""
db, marks = tmp_path / "w.db", tmp_path / "marks"
marks.mkdir()
jobs = Jobs(db)
jobs.add_schedule("marker", "sw_mark", "every 1h", next_run=ago(60))
done = _worker(db, marks)
assert done.returncode == 0, done.stdout.decode("utf-8", "replace")
assert (marks / "fired").exists()
assert jobs.schedules()[0].last_job_id
def test_no_schedules_turns_it_off_for_a_second_worker(tmp_path):
"""One scheduler per file, as many workers as you like: the pile-up guard
reads `last_job_id`, which neither of two simultaneous firings has written
yet when the other looks."""
db, marks = tmp_path / "w.db", tmp_path / "marks"
marks.mkdir()
jobs = Jobs(db)
jobs.add_schedule("marker", "sw_mark", "every 1h", next_run=ago(60))
done = _worker(db, marks, "--no-schedules")
assert done.returncode == 0, done.stdout.decode("utf-8", "replace")
assert not (marks / "fired").exists()
assert jobs.schedules()[0].last_job_id == 0
# Not even the declared default was reconciled: nothing about schedules
# happens in a worker that was told not to do them.
assert [s.name for s in jobs.schedules()] == ["marker"]
+59
View File
@@ -0,0 +1,59 @@
"""The parts of a web app that are the pipeline's, not the application's.
from pipeline.web import connect
queue = connect(app, Jobs("app.db"), config, store)
An app of this shape has a queue, and therefore has the same endpoints and the
same socket over it: list, kinds, enqueue, cancel, and `/ws/jobs`. None of that
knows what the work is *for*, so it is here rather than copied into each app —
and a socket's reconnect and fan-out behaviour is exactly the sort of thing a
second copy gets subtly wrong.
app.py `connect`: all of the below wired into a FastAPI app, and
the worker started and stopped with it, in one call
broadcaster.py one message out to every connected socket
jobs.py the queue over HTTP, the socket that reports on it, and
the one route that serves the element below
schedules.py schedules over HTTP: list, add, edit, delete, run now,
with the readable refusals a thing that runs while nobody
is watching has to have
deps.py how those routes reach the queue and the bus
schemas.py the request bodies they take
static/jobs.js `<desk-jobs>`: the queue on screen. Two lines in a
template and a new app has a job list with progress and a
Kill button, before it has written any UI of its own
static/schedules.js
`<desk-schedules>`: the same two lines for the schedule
table, themed from the same custom properties
<script type="module" src="/_jobs/jobs.js"></script>
<desk-jobs></desk-jobs>
The element is the one thing here with an opinion about how something looks,
and it is deliberate: this is the part nobody wants to write twice. It keeps
to a shadow root and themes from custom properties, so an app overrides it
without forking it, and it says nothing about *failures* — it raises an event
and leaves the wording to the app, which is the same split `notices.py` makes
on the server. newsdesk does not use it: it had a richer strip of its own
before this existed, which is what an app is expected to grow into.
What an app still owns: its own factory, its own store and config
dependencies, its own routers, its own page.
"""
from __future__ import annotations
from pipeline.web.app import connect
from pipeline.web.broadcaster import Broadcaster
from pipeline.web.deps import BroadcasterDep, QueueDep, get_broadcaster, get_queue
from pipeline.web.jobs import STATIC, router as jobs_router
from pipeline.web.schedules import router as schedules_router, served
from pipeline.web.schemas import (
EnqueueBody, JobKind, NewSchedule, ScheduleEdit, ScheduleRow,
)
__all__ = ["STATIC", "Broadcaster", "BroadcasterDep", "EnqueueBody",
"JobKind", "NewSchedule", "QueueDep", "ScheduleEdit", "ScheduleRow",
"connect", "get_broadcaster", "get_queue", "jobs_router",
"schedules_router", "served",
]
+86
View File
@@ -0,0 +1,86 @@
"""The whole web face of the queue, in one call.
app = FastAPI()
queue = connect(app, Jobs("app.db"), config, store)
That is the HTTP routes, the `/ws/jobs` socket, the `<desk-jobs>` and
`<desk-schedules>` elements, the schedule endpoints, the broadcaster that feeds
the socket, and the worker starting and stopping with the app. Before this
every app wrote the same thirty lines of glue — three callbacks, a lifespan,
two `app.state` slots, two `include_router`s — which is the part of "a day of
pipeline per project" that was still left.
Keyword arguments go to `JobQueue`: `workers=`, `bootstrap=`, `retain=`, `tz=`
and the rest. `tz` is the one a deployed app has to pass: a server runs on UTC
and `daily at 06:00` means 06:00 where the person reading it lives. The queue
is returned for the app's own routes and commands to enqueue on; it is also on
`app.state.queue`, which is where `QueueDep` finds it.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from typing import Any
from fastapi import FastAPI
from pipeline import JobQueue, Jobs, JobsSnapshot, Notice, Schedule
from pipeline.web.broadcaster import Broadcaster
from pipeline.web.jobs import router
from pipeline.web.schedules import router as schedules_router, served
def connect(app: FastAPI, jobs: Jobs, config: Any, store: Any,
**queue_options: Any) -> JobQueue:
"""Give `app` a queue: routes, socket, element, and a worker in its lifespan.
The lifespan is wrapped rather than replaced, so an app's own lifespan —
passed to `FastAPI(lifespan=...)` before or after this — still runs, inside
the queue's: the worker is up before the app's startup code and still up
through its shutdown code, so either may enqueue.
"""
bus = Broadcaster()
async def publish(snapshot: JobsSnapshot) -> None:
"""State: the whole queue, replaced wholesale by the next one."""
await bus.broadcast({"type": "jobs", **snapshot.model_dump()})
async def announce(notice: Notice) -> None:
"""An event: one failure, out to every open tab. The row it was
written to, so a tab can tell news from a repeat — see `NoticeEntry`."""
await bus.broadcast({"type": "notice", **notice.model_dump()})
async def scheduled(rows: list[Schedule]) -> None:
"""State, on its own clock: the schedule list, whenever one changes.
Separate from `jobs` rather than folded into it because the two change
at completely different rates — a running job pushes four times a
second and a schedule a few times a day — and because a client that
only draws the table should not have to ignore everything else.
"""
await bus.broadcast({"type": "schedules",
"schedules": [served(row).model_dump() for row in rows]})
queue = JobQueue(jobs, config, store, listener=publish, on_notice=announce,
on_schedules=scheduled, **queue_options)
app.state.queue = queue
app.state.bus = bus
app.include_router(router)
app.include_router(schedules_router)
inner = app.router.lifespan_context
@asynccontextmanager
async def lifespan(this: FastAPI) -> AsyncIterator[Any]:
# The worker needs a running event loop, which exists here and not
# where `connect` was called.
await queue.start()
try:
async with inner(this) as state:
yield state
finally:
await queue.stop()
app.router.lifespan_context = lifespan
return queue
+56
View File
@@ -0,0 +1,56 @@
"""One message out to every connected socket.
Deliberately knows nothing about jobs. It holds sockets, sends dicts, and drops
the ones that have gone away — which is the whole of what a fan-out is, and is
the reason the next thing that wants to push (a tag landing, a transcript
finishing, another reader in another tab) can use this rather than grow a
second copy of it.
A dead socket is discovered by writing to it, not by asking: a browser tab that
was closed, slept or navigated away leaves a connection that looks open until
the send fails. So every failure is a removal, and `broadcast` never raises —
one stale subscriber must not cost the others their update.
"""
from __future__ import annotations
import logging
import asyncio
from typing import Any
from fastapi import WebSocket
log = logging.getLogger(__name__)
class Broadcaster:
def __init__(self) -> None:
self._subscribers: set[WebSocket] = set()
@property
def count(self) -> int:
return len(self._subscribers)
async def subscribe(self, ws: WebSocket, hello: dict[str, Any] | None = None) -> None:
"""Register an accepted socket, and send it the current state.
The immediate send matters: a client that connects mid-job would
otherwise see nothing at all until the next change, which for a long
job is a blank progress bar next to work that is visibly happening.
"""
self._subscribers.add(ws)
if hello is not None:
await self._send(ws, hello)
def unsubscribe(self, ws: WebSocket) -> None:
self._subscribers.discard(ws)
async def broadcast(self, message: dict[str, Any]) -> None:
await asyncio.gather(*(self._send(ws, message)
for ws in tuple(self._subscribers)))
async def _send(self, ws: WebSocket, message: dict[str, Any]) -> None:
try:
await asyncio.wait_for(ws.send_json(message), timeout=1.0)
except Exception:
self._subscribers.discard(ws)
+34
View File
@@ -0,0 +1,34 @@
"""How a route reaches the queue and the socket bus.
Both are built once by the app's factory and parked on `app.state`. Routes ask
for them as dependencies rather than closing over them, so a router module is
importable on its own and a test can replace either through
`app.dependency_overrides`.
Only the two things that are the *pipeline's*. An app's own store and config
are its own business and stay in its own `deps` — the queue and the bus are
here because `pipeline.web.jobs` is here, and it needs them whichever app it is
mounted in.
"""
from __future__ import annotations
from typing import Annotated
from fastapi import Depends
from starlette.requests import HTTPConnection
from pipeline import JobQueue
from pipeline.web.broadcaster import Broadcaster
def get_queue(request: HTTPConnection) -> JobQueue:
return request.app.state.queue
def get_broadcaster(request: HTTPConnection) -> Broadcaster:
return request.app.state.bus
QueueDep = Annotated[JobQueue, Depends(get_queue)]
BroadcasterDep = Annotated[Broadcaster, Depends(get_broadcaster)]
+219
View File
@@ -0,0 +1,219 @@
"""The work queue over HTTP, and the socket that reports on it.
Four operations and one socket. Anything the UI can start, it starts by
enqueueing a job here; anything it needs to know about running work, it learns
from `/ws/jobs`. There is no polling endpoint on purpose — a queue that pushes
is both less code and less latency than one that is asked every second.
"""
from __future__ import annotations
import logging
import asyncio
from pathlib import Path
from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect
from fastapi.responses import FileResponse
from pipeline import FINAL, JOBS, TASKS
from pipeline.models import Job, JobsSnapshot, NoticeEntry, Run
from pipeline.web.deps import BroadcasterDep, QueueDep
from pipeline.web.schemas import EnqueueBody, JobKind
log = logging.getLogger(__name__)
router = APIRouter(tags=["jobs"])
#: The element that renders all this, shipped with the package. Served from a
#: route rather than mounted as a static directory because a router cannot
#: mount one — only an app can — and needing the app to mount something is
#: exactly the line of setup this is meant to remove. One file, so one route.
STATIC = Path(__file__).resolve().parent / "static"
@router.get("/_jobs/jobs.js", include_in_schema=False)
def element() -> FileResponse:
"""`<desk-jobs>`: the queue on screen, for an app with no UI for it yet.
<script type="module" src="/_jobs/jobs.js"></script>
<desk-jobs></desk-jobs>
Cached for an hour rather than forever. It changes when the package is
copied into a project and then hardly ever, and an app that wants it
fingerprinted can serve the file itself from `STATIC`.
"""
return FileResponse(STATIC / "jobs.js", media_type="text/javascript",
headers={"Cache-Control": "public, max-age=3600"})
@router.get("/api/jobs")
def list_jobs(queue: QueueDep) -> JobsSnapshot:
"""What is running and what recently finished.
The same shape the socket pushes, so a client that cannot hold a socket
open — a script, a `curl`, another tab — sees exactly what the
UI sees.
"""
return queue.snapshot()
@router.get("/api/jobs/kinds")
def kinds() -> list[JobKind]:
"""What can be started. Populated by `@job` and by the app's `TASKS`, so a
new handler or a new task appears here without being registered twice.
Tasks and jobs in one list because they are one thing to press: a task is
what a person means by "tag everything", and that it becomes eight jobs is
the pipeline's business rather than the button's.
Which is also why a job some task queues is not listed. `tag_batch` without
a list of uids is a button that does nothing, and the final job without the
batches it reports on is a button that reports nothing. If a task queues
it, the task is the thing to offer — worked out from `TASKS` rather than
declared, so a new batch kind cannot be forgotten here.
Nor is a job that needs a payload, such as `mark_reclassify` without a
`source`: a press sends none, so its button could only ever fail.
"""
made_for_you = {build().kind for build in TASKS.values()} | {FINAL}
listed = [
JobKind(kind=spec.kind, label=spec.label, solo=spec.solo)
for spec in JOBS.values()
if spec.kind not in made_for_you and not spec.needs
]
listed += [JobKind(kind=name, label=build().called, solo=True)
for name, build in TASKS.items()]
return sorted(listed, key=lambda k: k.kind)
@router.post("/api/jobs", status_code=202)
def enqueue(body: EnqueueBody, queue: QueueDep) -> Job | Run:
"""Start work: one job, or one task.
A kind that names a task starts it rather than being refused — one task,
on its own, which is a cascade of one and comes back as that run. Refusing
with "use /api/cascades" was the other reading and is worse: every caller
would have to know which names had become tasks, when what they asked for
is exactly what this does.
A name that is both a job and a task is the job. A task of a single batch
is that batch, so the two are the same work, and one row says it more
plainly than two.
202 rather than 201, because the useful thing has not happened yet — this
only promises that it will. A `solo` job that is already underway comes
back as itself rather than as an error: the caller asked for that work to
be in progress, and it is.
"""
if body.kind not in JOBS and body.kind in TASKS:
task_id = queue.start_task(TASKS[body.kind]())
run = queue.jobs.run(task_id)
if run is None:
raise HTTPException(409, f"{body.kind!r} would not start")
return run
spec = JOBS.get(body.kind)
missing = [k for k in (spec.needs if spec else ()) if k not in body.payload]
if missing:
raise HTTPException(422, f"{body.kind!r} needs {', '.join(missing)} "
"in its payload")
try:
job = queue.enqueue(body.kind, body.payload)
except KeyError as exc:
raise HTTPException(404, str(exc)) from exc
if job is None:
raise HTTPException(409, "the run it belongs to was killed")
return job
@router.post("/api/jobs/{job_id}/cancel")
def cancel(job_id: int, queue: QueueDep) -> Job:
"""Stop a job.
A queued one stops immediately. A running one stops at its next checkpoint,
so this returns while it is still `running` with `cancel_requested` set —
the socket reports the actual end. A job that has already finished is
returned unchanged rather than treated as an error; the button was simply
pressed a moment too late.
"""
job = queue.cancel(job_id)
if job is None:
raise HTTPException(404, "no such job")
return job
@router.post("/api/tasks/{task_id}/cancel")
def cancel_task(task_id: int, queue: QueueDep) -> dict[str, int]:
"""Stop one task: every batch of it.
Not the cascade it belongs to, and not the task's final job: that still
runs, reports the cancels and calls back, so the cascade knows how this
ended — "skip this", rather than "stop". Stopping is the cascade's own
button, below.
"""
return {"ended": queue.cancel_run(task_id)}
@router.post("/api/cascades/{cascade_id}/cancel")
def cancel_cascade(cascade_id: int, queue: QueueDep) -> dict[str, int]:
"""Stop a whole cascade: every task of it, and every job of those.
The same operation as above — a task and a cascade are one thing to the
pipeline — and two endpoints because they are two buttons and mean
different things to whoever presses one.
An id nobody has heard of, or one already over, ends nothing and says so
rather than being an error — the same reading as pressing Kill a moment
too late.
"""
return {"ended": queue.cancel_run(cascade_id)}
@router.get("/api/notices")
def list_notices(queue: QueueDep) -> list[NoticeEntry]:
"""Everything the queue has said that nobody has cleared, newest first.
What a line on screen was always missing: somewhere to be found again. A
notice fades in seconds and is raised, as often as not, while no tab is
open to show it. This is the same sentence with its fix and its raw error,
how many times it has been said, and when it last was.
"""
return queue.jobs.notices()
@router.delete("/api/notices/{notice_id}")
def clear_notice(notice_id: int, queue: QueueDep) -> dict[str, bool]:
"""Done with this one. Not an error when it has already gone: two tabs, or
the fortnight running out between the list and the press."""
return {"cleared": queue.jobs.clear_notice(notice_id)}
@router.delete("/api/notices")
def clear_notices(queue: QueueDep) -> dict[str, int]:
return {"cleared": queue.jobs.clear_notices()}
@router.websocket("/ws/jobs")
async def jobs_socket(ws: WebSocket, queue: QueueDep, bus: BroadcasterDep) -> None:
"""Push queue state on connect and on every change.
Two message types go out here: `jobs`, the whole queue state, replaced by
each new one; and `notice`, a single failure as the log now holds it, its
count included. See `pipeline/notices.py` for what a client does with that.
The receive loop exists only to notice the client leaving — nothing the
browser sends is read. Starting and cancelling go over HTTP, where they get
status codes and error bodies; making the socket bidirectional would mean
inventing a second request protocol inside it for no gain.
"""
await ws.accept()
snapshot = await asyncio.to_thread(queue.snapshot)
await bus.subscribe(ws, {"type": "jobs", **snapshot.model_dump()})
try:
while True:
await ws.receive_text()
except WebSocketDisconnect:
pass
except Exception:
log.debug("jobs socket closed unexpectedly", exc_info=True)
finally:
bus.unsubscribe(ws)
+218
View File
@@ -0,0 +1,218 @@
"""Schedules over HTTP: list, add, edit, delete, run now.
Five routes and no socket of its own. Changes go out on `/ws/jobs` as a
`schedules` message — the same socket the job list already holds open, because
a second one would be a second reconnect loop to get subtly wrong, for a list
that changes a few times a day.
Every refusal here is a sentence a person can act on, which is the whole
reason these are not four lines of `queue.jobs.*` behind a generic validator:
an unregistered kind 404, and it names what is registered
a payload missing keys 422, and it names the keys
a cadence it cannot read 422, and it lists all five accepted wordings
They arrive as `{"detail": "<sentence>"}`, which is FastAPI's own shape, so an
app's `api()` helper turns any of them into one toast carrying the server's
words rather than inventing its own — see `app/static/app.js` in the starter.
Validating the kind and the payload *here*, rather than letting the job fail at
06:00, is the point of the endpoint existing at all. A schedule is the one way
of starting work where nobody is watching when it runs.
"""
from __future__ import annotations
import logging
from datetime import UTC, datetime
from fastapi import APIRouter, HTTPException
from fastapi.responses import FileResponse
from pipeline import JOBS, JobQueue
from pipeline.models import Job, Schedule
from pipeline.schedule import Cadence
from pipeline.web.deps import QueueDep
from pipeline.web.jobs import STATIC
from pipeline.web.schemas import NewSchedule, ScheduleEdit, ScheduleRow
log = logging.getLogger(__name__)
router = APIRouter(tags=["schedules"])
@router.get("/_jobs/schedules.js", include_in_schema=False)
def element() -> FileResponse:
"""`<desk-schedules>`: the schedule table on screen.
<script type="module" src="/_jobs/schedules.js"></script>
<desk-schedules></desk-schedules>
Served and cached exactly as `/_jobs/jobs.js` is, and it reads the same
`--desk-jobs-*` custom properties, so an app that themed one has themed
both.
"""
return FileResponse(STATIC / "schedules.js", media_type="text/javascript",
headers={"Cache-Control": "public, max-age=3600"})
def listed(queue: JobQueue) -> list[ScheduleRow]:
"""The list as it goes out, over HTTP and over the socket alike.
One function for both, so the two cannot drift: a client that reads the
REST list once and then follows the socket must not be handed two shapes of
the same thing.
"""
return [served(row) for row in queue.jobs.schedules()]
def served(row: Schedule) -> ScheduleRow:
"""One schedule, plus the one thing the table cannot know: whether anything
is still registered to run it. Worked out here rather than stored, because
it is a fact about the code that is loaded and not about the row."""
return ScheduleRow(**row.model_dump(), unknown=row.kind not in JOBS)
def _check(kind: str, payload: dict) -> None:
"""Refuse a kind nothing runs, or a payload its handler cannot take."""
spec = JOBS.get(kind)
if spec is None:
raise HTTPException(404, f"no job is registered as {kind!r}. Registered: "
f"{', '.join(sorted(JOBS)) or 'none'}.")
missing = [k for k in spec.needs if k not in payload]
if missing:
raise HTTPException(422, f"{kind!r} cannot run without "
f"{', '.join(missing)} in its payload.")
def _cadence(text: str) -> Cadence:
"""Parse, or 422 with the accepted wordings.
`Cadence` already writes that sentence — see `pipeline/schedule.py` — so it
is passed through rather than reworded here. One description of the grammar
in the codebase, and it is next to the parser that enforces it.
"""
try:
return Cadence.parse(text)
except ValueError as bad:
raise HTTPException(422, str(bad)) from bad
def _free_name(queue: JobQueue, wanted: str) -> str:
"""A name nothing else has: `tick`, then `tick 2`, then `tick 3`.
Only for a request that gave none. A name somebody typed is refused rather
than quietly altered, because the name is the handle, and being handed a
different one is how you edit the wrong row a month later.
"""
taken = {row.name for row in queue.jobs.schedules()}
if wanted not in taken:
return wanted
n = 2
while f"{wanted} {n}" in taken:
n += 1
return f"{wanted} {n}"
@router.get("/api/schedules")
def list_schedules(queue: QueueDep) -> list[ScheduleRow]:
"""Every schedule, by name, with how its last job ended.
The same list the socket pushes, so a client that cannot hold one open — a
script, a `curl`, a second tab — sees exactly what the element sees.
"""
return listed(queue)
@router.post("/api/schedules", status_code=201)
def add_schedule(body: NewSchedule, queue: QueueDep) -> ScheduleRow:
"""Add one. 201, because unlike a job this really does now exist.
`name` is optional and defaults to the kind, made unique by counting. A
name that was given and is taken is a conflict rather than a second row:
two rows answering to one name is exactly the ambiguity the handle exists
to prevent.
`next_run` is worked out here, from now, so a schedule added at 09:00
saying `every 6h` is next at 15:00 rather than at some moment the table
invented.
"""
_check(body.kind, body.payload)
cadence = _cadence(body.cadence)
name = body.name.strip() or _free_name(queue, body.kind)
row = queue.jobs.add_schedule(
name, body.kind, str(cadence), payload=body.payload, label=body.label,
enabled=body.enabled,
next_run=cadence.next_after(datetime.now(UTC), queue.zone).isoformat(),
only_if_new=True)
if row is None:
raise HTTPException(409, f"there is already a schedule called {name!r}.")
queue.notify_schedules()
return served(row)
@router.patch("/api/schedules/{schedule_id}")
def edit_schedule(schedule_id: int, body: ScheduleEdit, queue: QueueDep) -> ScheduleRow:
"""Change part of one. Only what was sent is touched.
A new cadence moves `next_run` with it, because the alternative is a row
that says `every 15m` and does not run for six hours: the old time is an
answer to a question nobody is asking any more. Switching a row back on
does the same. It would come right on its own — a firing advances to the
next slot from now and never walks through the missed ones — but the time
on screen would be a lie until it did.
"""
row = queue.jobs.schedule(schedule_id)
if row is None:
raise HTTPException(404, f"there is no schedule {schedule_id}.")
# Unset and null are one thing here: every field is optional and every
# column is NOT NULL, so `{"cadence": null}` can only be a client leaving
# a key in rather than a request to blank the column.
fields = {k: v for k, v in body.model_dump(exclude_unset=True).items()
if v is not None}
if "kind" in fields or "payload" in fields:
_check(fields.get("kind", row.kind), fields.get("payload", row.payload))
now = datetime.now(UTC)
if "cadence" in fields:
cadence = _cadence(fields["cadence"])
fields["cadence"] = str(cadence)
fields["next_run"] = cadence.next_after(now, queue.zone).isoformat()
elif fields.get("enabled") and not row.enabled:
fields["next_run"] = _cadence(row.cadence).next_after(now, queue.zone).isoformat()
changed = queue.jobs.update_schedule(schedule_id, **fields)
queue.notify_schedules()
return served(changed or row)
@router.delete("/api/schedules/{schedule_id}")
def delete_schedule(schedule_id: int, queue: QueueDep) -> dict[str, bool]:
"""Forget one. Not an error when it has already gone — two tabs.
A declared default deleted here comes back on the next start, which is
what declaring one means. Switching it off is how to be rid of it for
good: that edit is the row's, and reconciliation never overwrites one.
"""
deleted = queue.jobs.delete_schedule(schedule_id)
if deleted:
queue.notify_schedules()
return {"deleted": deleted}
@router.post("/api/schedules/{schedule_id}/run", status_code=202)
def run_now(schedule_id: int, queue: QueueDep) -> Job:
"""Queue this schedule's job now, without moving when it is next due.
202 like `/api/jobs`, and for the same reason: what comes back is the
promise, not the outcome. A `solo` kind already under way comes back as
that job rather than as an error — the caller asked for the work to be in
progress, and it is.
"""
row = queue.jobs.schedule(schedule_id)
if row is None:
raise HTTPException(404, f"there is no schedule {schedule_id}.")
_check(row.kind, row.payload)
job = queue.run_schedule(schedule_id)
if job is None:
raise HTTPException(409, f"{row.kind!r} was not queued: the run it "
"would have belonged to was killed.")
return job
+75
View File
@@ -0,0 +1,75 @@
"""What the queue's endpoints accept and promise back.
Job-shaped, so they live with the router that serves them rather than in an
app's schema module. The `Job`, `JobsSnapshot` and `Schedule` they sit
alongside are in `pipeline/models.py`; these are only ever request bodies, or a
stored shape with one derived field added for the client.
"""
from __future__ import annotations
from typing import Any
from pydantic import BaseModel
from pipeline.models import Schedule
class EnqueueBody(BaseModel):
"""Start a job. `payload` is spread over the handler's keyword arguments,
so what a kind accepts is the signature of its function and nothing else
has to restate it."""
kind: str
payload: dict[str, Any] = {}
class JobKind(BaseModel):
"""One entry from the job registry, for a UI that wants to offer them."""
kind: str
label: str
solo: bool
class NewSchedule(BaseModel):
"""Add a schedule. `name` is optional because the kind is usually answer
enough — one `tick` schedule wants no name of its own, and the second is
named by counting rather than by making somebody think of one."""
kind: str
cadence: str
name: str = ""
label: str = ""
payload: dict[str, Any] = {}
enabled: bool = True
class ScheduleEdit(BaseModel):
"""Change part of a schedule. Every field optional, and read with
`exclude_unset`, so sending `{"enabled": false}` switches one off without
also resubmitting a cadence somebody edited in another tab a second ago.
`name` is not here: it is the handle a declared default is matched on, and
renaming a row would hand its edits to a fresh one inserted under the old
name on the next start.
"""
kind: str | None = None
label: str | None = None
cadence: str | None = None
payload: dict[str, Any] | None = None
enabled: bool | None = None
class ScheduleRow(Schedule):
"""A schedule as it is served: the row, plus whether anything still runs it.
`unknown` is not stored, because it is a fact about the code that is loaded
rather than about the row — a handler removed in a deploy makes it true
without the table changing, and reinstalling the plugin makes it false
again. The element greys such a row and says `unknown kind`; the queue
never fires one.
"""
unknown: bool = False
+280
View File
@@ -0,0 +1,280 @@
/* The queue, on screen, for an app that has not written a UI yet.
*
* <script type="module" src="/_jobs/jobs.js"></script>
* <desk-jobs></desk-jobs>
*
* That is the whole of it. The element finds `/ws/jobs` itself, shows what is
* running with a progress bar and a Kill button each, says how many are
* waiting, and reconnects when the server restarts under you.
*
* One file, and the styles are in it rather than beside it, because the
* shadow root scopes them anyway and a second file is a second route, a
* second cache header and a second thing to forget to copy. Themed from
* outside through the custom properties at the top of `CSS` — set them on
* `desk-jobs` or on `:root` and nothing here has to be forked.
*
* What it deliberately does not do is talk about failures. It raises a
* `desk-jobs:notice` event and leaves the wording to the app, because a
* useful sentence names things only the app knows — its backends, its feeds,
* its config keys. `pipeline/notices.py` is the same split on the server.
*
* Attributes, all optional:
* base="/api" prefix for the HTTP calls, if the router is mounted deeper
* socket="..." the socket's URL, if it is not `/ws/jobs` on this origin
* probe="/api/status"
* an authenticated endpoint to try when the socket closes. A
* refused handshake and a dead server look identical from
* here — the server rejects an unauthenticated upgrade before
* completing it, so the browser reports the same abnormal
* close for "your session expired" as for "the server is
* gone". One cheap request tells them apart. Without this the
* element assumes the server went away, which is the more
* common reason and the less confusing thing to be wrong about.
* idle="hide" with nothing running or queued, take up no space. The
* default, since a chip that is always there is furniture.
*/
const CSS = `
:host {
--desk-jobs-bg: #16181d;
--desk-jobs-fg: #e7e9ee;
--desk-jobs-dim: #9aa1ad;
--desk-jobs-accent: #4c8dff;
--desk-jobs-error: #ff6b6b;
--desk-jobs-radius: 10px;
--desk-jobs-font: system-ui, -apple-system, "Segoe UI", sans-serif;
--desk-jobs-gap: 8px;
display: block;
font-family: var(--desk-jobs-font);
font-size: 13px;
color: var(--desk-jobs-fg);
}
:host([hidden]) { display: none; }
.strip {
display: flex;
flex-direction: column;
gap: var(--desk-jobs-gap);
background: var(--desk-jobs-bg);
border-radius: var(--desk-jobs-radius);
padding: 10px 12px;
}
.job { display: grid; grid-template-columns: 1fr auto; gap: 2px 10px; align-items: center; }
.what { font-weight: 600; }
.said { color: var(--desk-jobs-dim); grid-column: 1; }
.count { color: var(--desk-jobs-dim); font-variant-numeric: tabular-nums; }
/* A bar only when there is a fraction to draw. A handler that reports no
* total is working on one thing, and an empty trough reads as stalled. */
.bar {
grid-column: 1 / -1;
height: 3px;
border-radius: 2px;
background: color-mix(in srgb, var(--desk-jobs-fg) 15%, transparent);
overflow: hidden;
}
.bar > i { display: block; height: 100%; background: var(--desk-jobs-accent); transition: width .2s; }
button {
font: inherit;
color: var(--desk-jobs-dim);
background: none;
border: 1px solid currentColor;
border-radius: 6px;
padding: 1px 8px;
cursor: pointer;
}
button:hover { color: var(--desk-jobs-error); }
button[disabled] { opacity: .4; cursor: default; }
.offline { color: var(--desk-jobs-error); }
`;
/** Backing off rather than hammering. The usual reason a socket closed is the
* server restarting under --reload, which comes back inside a second; the
* ceiling is for when it is not coming back at all. */
const FIRST_WAIT = 500;
const LONGEST_WAIT = 30000;
/** How long down is long enough to say so. A reload blinks the socket every
* time a file is saved, and a warning that flashes on every save is one you
* stop reading — which would cost us the case that matters, where the server
* really is gone and the strip is quietly showing state from a minute ago. */
const QUIET_ABOUT_IT_FOR = 4000;
class DeskJobs extends HTMLElement {
#root;
#socket = null;
#wait = FIRST_WAIT;
#timer = null;
#state = { running: [], queued: [], recent: [] };
#offline = false;
#killing = new Set();
constructor() {
super();
this.#root = this.attachShadow({ mode: "open" });
const style = document.createElement("style");
style.textContent = CSS;
this.#root.append(style, document.createElement("div"));
}
connectedCallback() { this.#connect(); this.#render(); }
disconnectedCallback() {
clearTimeout(this.#timer);
this.#socket?.close();
this.#socket = null;
}
get #base() { return this.getAttribute("base") ?? ""; }
get #socketUrl() {
const given = this.getAttribute("socket");
if (given) return given;
const scheme = location.protocol === "https:" ? "wss" : "ws";
return `${scheme}://${location.host}${this.#base}/ws/jobs`;
}
#connect() {
let socket;
try {
socket = new WebSocket(this.#socketUrl);
} catch {
this.#reconnect(); // a malformed URL, or no network
return;
}
this.#socket = socket;
socket.onopen = () => {
this.#wait = FIRST_WAIT;
this.#offline = false;
this.#render();
};
socket.onmessage = (event) => {
let message;
try { message = JSON.parse(event.data); } catch { return; }
if (message.type === "jobs") {
const { type, ...state } = message;
this.#state = state;
this.#render();
} else if (message.type === "notice") {
// The app's to word. Composed, so a listener on `document` hears it
// through the shadow boundary.
this.dispatchEvent(new CustomEvent("desk-jobs:notice", {
detail: message, bubbles: true, composed: true,
}));
}
};
socket.onerror = () => socket.close();
socket.onclose = async () => {
if (this.#socket !== socket) return; // a later connection owns us now
this.#socket = null;
await this.#whyClosed();
this.#reconnect();
};
}
/** Ask before assuming. See the `probe` attribute. */
async #whyClosed() {
const probe = this.getAttribute("probe");
if (!probe) return;
try {
const answer = await fetch(this.#base + probe, { credentials: "same-origin" });
if (answer.status === 401 || answer.status === 403) {
this.dispatchEvent(new CustomEvent("desk-jobs:signed-out", {
bubbles: true, composed: true,
}));
}
} catch { /* offline; the strip says so below */ }
}
#reconnect() {
if (this.#wait >= QUIET_ABOUT_IT_FOR) {
this.#offline = true;
this.#render();
}
clearTimeout(this.#timer);
this.#timer = setTimeout(() => this.#connect(), this.#wait);
this.#wait = Math.min(this.#wait * 2, LONGEST_WAIT);
}
async #kill(id) {
this.#killing.add(id);
this.#render();
try {
await fetch(`${this.#base}/api/jobs/${id}/cancel`,
{ method: "POST", credentials: "same-origin" });
} finally {
// The socket reports the actual ending — a running job stops at its next
// checkpoint, so the button going quiet is the whole of what happens here.
this.#killing.delete(id);
}
}
#render() {
const { running = [], queued = [] } = this.#state;
const strip = this.#root.lastElementChild;
this.hidden = this.getAttribute("idle") !== "show"
&& !running.length && !queued.length && !this.#offline;
strip.className = "strip";
strip.replaceChildren();
if (this.#offline) {
const said = document.createElement("div");
said.className = "offline";
said.textContent = "Not connected — what you see here may be out of date.";
strip.append(said);
}
for (const job of running) {
strip.append(this.#row(job));
}
if (queued.length) {
const waiting = document.createElement("div");
waiting.className = "count";
waiting.textContent = `${queued.length} waiting`;
strip.append(waiting);
}
}
#row(job) {
const row = document.createElement("div");
row.className = "job";
const what = document.createElement("span");
what.className = "what";
what.textContent = job.label || job.kind;
row.append(what);
const kill = document.createElement("button");
kill.textContent = "Kill";
kill.disabled = this.#killing.has(job.id) || job.cancel_requested;
kill.addEventListener("click", () => this.#kill(job.id));
row.append(kill);
if (job.message) {
const said = document.createElement("span");
said.className = "said";
said.textContent = job.message;
row.append(said);
}
if (job.total > 0) {
const bar = document.createElement("div");
bar.className = "bar";
const fill = document.createElement("i");
fill.style.width = `${Math.min(100, (job.done / job.total) * 100)}%`;
bar.append(fill);
row.append(bar);
}
return row;
}
}
if (!customElements.get("desk-jobs")) {
customElements.define("desk-jobs", DeskJobs);
}
export { DeskJobs };
+543
View File
@@ -0,0 +1,543 @@
/* The schedule table, on screen, for an app that has not written a UI yet.
*
* <script type="module" src="/_jobs/schedules.js"></script>
* <desk-schedules></desk-schedules>
*
* A row per schedule — name, kind, cadence, on or off, next run, last run and
* how that job ended — plus Run now, Delete, and a row at the bottom for
* adding one. The kinds come from `/api/jobs/kinds`, so a new `@job` is a new
* option here without this file being touched.
*
* It reads `/ws/jobs` for `schedules` messages, so a row edited in another tab
* redraws in this one, and a firing moves the times without anybody polling.
* The socket's opening message is the *job* list, though, so the table is
* fetched once on connect: the push says what changed, not what there is. The
* job list is read as it goes past as well, for the last column — a schedule
* is pushed when it fires, which is before its job has ended, so the state it
* ended in can only come from there.
*
* The same one-file, shadow-root, custom-property shape as `<desk-jobs>`, and
* deliberately the same `--desk-jobs-*` names: an app that themed one element
* has themed both, and two schemes for two halves of one page is the thing
* nobody wants to maintain.
*
* Failures are not worded here. A refused change raises `desk-schedules:error`
* carrying the server's own `detail` — which the endpoints promise is one
* readable sentence — and the app toasts it. Notices arriving on the socket
* are re-raised as `desk-jobs:notice`, the same event `<desk-jobs>` raises, so
* an app that listens for one is listening for both.
*
* Attributes, all optional:
* base="/api" prefix for the HTTP calls, if the router is mounted deeper
* socket="..." the socket's URL, if it is not `/ws/jobs` on this origin
*/
const CSS = `
:host {
--desk-jobs-bg: #16181d;
--desk-jobs-fg: #e7e9ee;
--desk-jobs-dim: #9aa1ad;
--desk-jobs-accent: #4c8dff;
--desk-jobs-error: #ff6b6b;
--desk-jobs-radius: 10px;
--desk-jobs-font: system-ui, -apple-system, "Segoe UI", sans-serif;
--desk-jobs-gap: 8px;
display: block;
font-family: var(--desk-jobs-font);
font-size: 13px;
color: var(--desk-jobs-fg);
}
:host([hidden]) { display: none; }
.panel {
background: var(--desk-jobs-bg);
border-radius: var(--desk-jobs-radius);
padding: 10px 12px;
}
table { width: 100%; border-collapse: collapse; }
th, td {
text-align: left;
padding: 6px 8px 6px 0;
vertical-align: baseline;
border-bottom: 1px solid color-mix(in srgb, var(--desk-jobs-fg) 12%, transparent);
}
th { color: var(--desk-jobs-dim); font-weight: 600; white-space: nowrap; }
tr:last-child > td { border-bottom: 0; }
td.actions { text-align: right; white-space: nowrap; }
.name { font-weight: 600; }
.kind, .when { color: var(--desk-jobs-dim); font-variant-numeric: tabular-nums; }
.off { opacity: .55; }
.bad { color: var(--desk-jobs-error); }
/* The cadence is edited in place: a box that looks like text until you are in
* it, because a table of eight input fields reads as a form to fill in rather
* than as a list of what is running. */
input.cadence, input.text {
font: inherit;
color: inherit;
background: none;
border: 1px solid transparent;
border-radius: 6px;
padding: 1px 4px;
min-width: 9em;
}
input.cadence:hover { border-color: color-mix(in srgb, var(--desk-jobs-fg) 20%, transparent); }
input:focus { outline: none; border-color: var(--desk-jobs-accent); }
input::placeholder { color: var(--desk-jobs-dim); }
select {
font: inherit;
color: inherit;
background: var(--desk-jobs-bg);
border: 1px solid color-mix(in srgb, var(--desk-jobs-fg) 20%, transparent);
border-radius: 6px;
padding: 1px 4px;
}
button {
font: inherit;
color: var(--desk-jobs-dim);
background: none;
border: 1px solid currentColor;
border-radius: 6px;
padding: 1px 8px;
margin-left: 6px;
cursor: pointer;
}
button:hover { color: var(--desk-jobs-accent); }
button.remove:hover { color: var(--desk-jobs-error); }
button[disabled] { opacity: .4; cursor: default; }
.state { font-size: 11px; padding: 0 5px; border-radius: 999px;
border: 1px solid currentColor; margin-left: 6px; }
.state.failed, .state.cancelled { color: var(--desk-jobs-error); }
.state.running, .state.queued { color: var(--desk-jobs-accent); }
.empty, .offline { color: var(--desk-jobs-dim); padding: 6px 0; }
.offline { color: var(--desk-jobs-error); }
`;
/** The same backoff as `<desk-jobs>`: the usual reason a socket closed is the
* server restarting under --reload, which comes back inside a second. */
const FIRST_WAIT = 500;
const LONGEST_WAIT = 30000;
/** The grammar, in the one place a person is asked to use it. Two of the five
* as the placeholder, because a box wide enough for all five is a box wide
* enough to unbalance the row it is in — and all five on hover, which is also
* what the server's refusal says if they get it wrong anyway. */
const FORMS = "every 15m · every 6h · every 2d · daily at 06:00 · weekly mon 07:00";
const HINT = "every 15m · daily at 06:00";
/** A stamp as a person reads it. Today is a time, anything else is a date and
* a time, and "never" is said rather than left blank — an empty cell reads as
* a bug, and "never" is a fact. */
function when(stamp) {
if (!stamp) return "never";
const at = new Date(stamp);
if (isNaN(at)) return stamp;
const today = at.toDateString() === new Date().toDateString();
const clock = at.toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" });
return today ? clock : `${at.toLocaleDateString()} ${clock}`;
}
class DeskSchedules extends HTMLElement {
#root;
#socket = null;
#wait = FIRST_WAIT;
#timer = null;
#rows = [];
#kinds = [];
#offline = false;
#busy = new Set();
#loaded = false;
constructor() {
super();
this.#root = this.attachShadow({ mode: "open" });
const style = document.createElement("style");
style.textContent = CSS;
this.#root.append(style, document.createElement("div"));
}
connectedCallback() {
this.#render();
this.#load();
this.#connect();
}
disconnectedCallback() {
clearTimeout(this.#timer);
this.#socket?.close();
this.#socket = null;
}
get #base() { return this.getAttribute("base") ?? ""; }
get #socketUrl() {
const given = this.getAttribute("socket");
if (given) return given;
const scheme = location.protocol === "https:" ? "wss" : "ws";
return `${scheme}://${location.host}${this.#base}/ws/jobs`;
}
/* -- the server ------------------------------------------------------- */
/** Every call goes through here, so one place raises the event and one
* place knows that a refusal carries `{"detail": "<sentence>"}`. */
async #ask(path, options) {
const o = Object.assign({ credentials: "same-origin" }, options);
if (o.body !== undefined && typeof o.body !== "string") {
o.headers = Object.assign({ "Content-Type": "application/json" }, o.headers);
o.body = JSON.stringify(o.body);
}
let answer;
try {
answer = await fetch(this.#base + path, o);
} catch (e) {
this.#failed("Could not reach the server.");
throw e;
}
const text = await answer.text();
let body = null;
if (text) { try { body = JSON.parse(text); } catch (e) { body = null; } }
if (!answer.ok) {
this.#failed((body && body.detail)
|| `The server answered ${answer.status} and said nothing about why.`);
throw new Error(String((body && body.detail) || answer.status));
}
return body;
}
#failed(detail) {
// Composed, so a listener on `document` hears it through the shadow
// boundary. The wording is the server's; this element has no opinion.
this.dispatchEvent(new CustomEvent("desk-schedules:error", {
detail: { detail: String(detail) }, bubbles: true, composed: true,
}));
}
async #load() {
try {
const [rows, kinds] = await Promise.all([
this.#ask("/api/schedules"),
this.#ask("/api/jobs/kinds"),
]);
this.#rows = rows || [];
this.#kinds = kinds || [];
} catch (e) {
/* already raised */
} finally {
this.#loaded = true;
this.#render();
}
}
#connect() {
let socket;
try {
socket = new WebSocket(this.#socketUrl);
} catch {
this.#reconnect();
return;
}
this.#socket = socket;
socket.onopen = () => {
this.#wait = FIRST_WAIT;
if (this.#offline) {
// Down and back: what changed while we were away came as nothing, so
// the list is read again rather than waited for.
this.#offline = false;
this.#load();
}
this.#render();
};
socket.onmessage = (event) => {
let message;
try { message = JSON.parse(event.data); } catch { return; }
if (message.type === "schedules") {
this.#rows = message.schedules || [];
this.#render();
} else if (message.type === "jobs") {
this.#catchUp(message);
} else if (message.type === "notice") {
this.dispatchEvent(new CustomEvent("desk-jobs:notice", {
detail: message, bubbles: true, composed: true,
}));
}
};
socket.onerror = () => socket.close();
socket.onclose = () => {
if (this.#socket !== socket) return; // a later connection owns us
this.#socket = null;
this.#reconnect();
};
}
/* The last column says how the last job ended, and the `schedules` message
* is pushed when a schedule *fires* — at which point that job has not ended
* and the cell reads `running`. Nothing pushes a schedule again when the
* job finishes, so without this the table goes on reporting a job as
* running long after it is done, which is the one thing this column is for.
*
* The job list arrives on this same socket four times a second while
* anything is going, and it carries the state of every job worth knowing
* about, so the answer is already here: take it off the snapshot rather
* than ask the server a second time. A job old enough to have left `recent`
* keeps whatever it was last seen as; its schedule has fired since, or it
* has been on screen for an hour. */
#catchUp(snapshot) {
const states = new Map();
for (const list of ["running", "queued", "recent"]) {
for (const job of snapshot[list] || []) states.set(job.id, job.state);
}
let moved = false;
for (const row of this.#rows) {
const state = states.get(row.last_job_id);
if (state && state !== row.last_state) {
row.last_state = state;
moved = true;
}
}
if (moved) this.#render();
}
#reconnect() {
if (this.#wait >= 4000) {
this.#offline = true;
this.#render();
}
clearTimeout(this.#timer);
this.#timer = setTimeout(() => this.#connect(), this.#wait);
this.#wait = Math.min(this.#wait * 2, LONGEST_WAIT);
}
/* One wrapper for every change: the row is greyed while the request is in
* the air, and the new list arrives on the socket rather than being patched
* in here — so what is on screen is always what the server holds. */
async #change(id, run) {
this.#busy.add(id);
this.#render();
try {
await run();
} catch (e) {
/* already raised */
} finally {
this.#busy.delete(id);
this.#render();
}
}
/* -- drawing ---------------------------------------------------------- */
#render() {
const panel = this.#root.lastElementChild;
panel.className = "panel";
panel.replaceChildren();
if (this.#offline) {
const said = document.createElement("div");
said.className = "offline";
said.textContent = "Not connected — what you see here may be out of date.";
panel.append(said);
}
const table = document.createElement("table");
table.append(this.#head());
const body = document.createElement("tbody");
for (const row of this.#rows) body.append(this.#row(row));
body.append(this.#adder());
table.append(body);
panel.append(table);
if (this.#loaded && !this.#rows.length) {
const empty = document.createElement("div");
empty.className = "empty";
empty.textContent = "Nothing is scheduled. Add one below.";
panel.insertBefore(empty, table);
}
}
#head() {
const head = document.createElement("thead");
const row = document.createElement("tr");
for (const label of ["Name", "Kind", "Cadence", "On", "Next", "Last", ""]) {
const cell = document.createElement("th");
cell.textContent = label;
row.append(cell);
}
head.append(row);
return head;
}
#row(schedule) {
const row = document.createElement("tr");
const busy = this.#busy.has(schedule.id);
if (!schedule.enabled) row.classList.add("off");
row.append(this.#cell("name", schedule.name));
const kind = this.#cell("kind", schedule.kind);
if (schedule.unknown) {
// Never fires, and says so rather than sitting there with a next run it
// is not going to keep. Deleting it is the only thing left to do.
const bad = document.createElement("span");
bad.className = "state bad";
bad.textContent = "unknown kind";
kind.append(bad);
}
row.append(kind);
const cadence = document.createElement("td");
const box = document.createElement("input");
box.className = "cadence";
box.value = schedule.cadence;
box.title = FORMS;
box.disabled = busy;
const save = () => {
const wanted = box.value.trim();
if (!wanted || wanted === schedule.cadence) {
box.value = schedule.cadence;
return;
}
this.#change(schedule.id, () => this.#ask(`/api/schedules/${schedule.id}`,
{ method: "PATCH", body: { cadence: wanted } }));
};
box.addEventListener("change", save);
box.addEventListener("keydown", (e) => { if (e.key === "Enter") box.blur(); });
cadence.append(box);
row.append(cadence);
const onOff = document.createElement("td");
const toggle = document.createElement("input");
toggle.type = "checkbox";
toggle.checked = !!schedule.enabled;
toggle.disabled = busy;
toggle.title = schedule.enabled ? "switch this schedule off" : "switch it on";
toggle.addEventListener("change", () => this.#change(schedule.id,
() => this.#ask(`/api/schedules/${schedule.id}`,
{ method: "PATCH", body: { enabled: toggle.checked } })));
onOff.append(toggle);
row.append(onOff);
row.append(this.#cell("when", schedule.enabled ? when(schedule.next_run) : "—"));
const last = this.#cell("when", when(schedule.last_run));
if (schedule.last_state) {
const state = document.createElement("span");
state.className = `state ${schedule.last_state}`;
state.textContent = schedule.last_state;
last.append(state);
}
row.append(last);
const actions = document.createElement("td");
actions.className = "actions";
const now = document.createElement("button");
now.textContent = "Run now";
now.disabled = busy || schedule.unknown;
now.addEventListener("click", () => this.#change(schedule.id,
() => this.#ask(`/api/schedules/${schedule.id}/run`, { method: "POST" })));
const remove = document.createElement("button");
remove.className = "remove";
remove.textContent = "Delete";
remove.disabled = busy;
remove.addEventListener("click", () => this.#change(schedule.id,
() => this.#ask(`/api/schedules/${schedule.id}`, { method: "DELETE" })));
actions.append(now, remove);
row.append(actions);
return row;
}
#cell(className, text) {
const cell = document.createElement("td");
cell.className = className;
const span = document.createElement("span");
span.textContent = text;
cell.append(span);
return cell;
}
/** The add row. In the table rather than in a form above it, because what
* is being added is another one of these and the columns already say what
* each box is for. */
#adder() {
const row = document.createElement("tr");
const busy = this.#busy.has(0);
const name = document.createElement("input");
name.className = "text";
name.placeholder = "optional";
name.disabled = busy;
row.append(this.#wrap(name));
const kind = document.createElement("select");
kind.disabled = busy || !this.#kinds.length;
for (const k of this.#kinds) {
const option = document.createElement("option");
option.value = k.kind;
option.textContent = k.label || k.kind;
kind.append(option);
}
if (!this.#kinds.length) {
const none = document.createElement("option");
none.textContent = this.#loaded ? "nothing registered" : "loading…";
kind.append(none);
}
row.append(this.#wrap(kind));
const cadence = document.createElement("input");
cadence.className = "text";
cadence.placeholder = HINT;
cadence.title = FORMS;
cadence.disabled = busy;
row.append(this.#wrap(cadence));
row.append(document.createElement("td"), document.createElement("td"),
document.createElement("td"));
const actions = document.createElement("td");
actions.className = "actions";
const add = document.createElement("button");
add.textContent = "Add";
add.disabled = busy;
const submit = () => {
if (!kind.value) return;
this.#change(0, async () => {
await this.#ask("/api/schedules", {
method: "POST",
body: { name: name.value.trim(), kind: kind.value,
cadence: cadence.value.trim() },
});
// Cleared only on success, so a refused cadence is still there to be
// corrected rather than retyped from memory.
name.value = "";
cadence.value = "";
});
};
add.addEventListener("click", submit);
for (const box of [name, cadence]) {
box.addEventListener("keydown", (e) => { if (e.key === "Enter") submit(); });
}
actions.append(add);
row.append(actions);
return row;
}
#wrap(control) {
const cell = document.createElement("td");
cell.append(control);
return cell;
}
}
if (!customElements.get("desk-schedules")) {
customElements.define("desk-schedules", DeskSchedules);
}
export { DeskSchedules };
+113
View File
@@ -0,0 +1,113 @@
"""A worker with no web app around it.
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap \\
--bootstrap-arg config.yaml --workers cpu=2,http=8 --until-idle
What a second machine, a cron entry or a batch script runs. It is the same
`JobQueue` the web app starts, on the same table, so it races correctly against
any other worker on that file: `Jobs.claim` is one atomic statement and every
claimed job is held on a lease. Start as many as the file will bear.
The bootstrap is the one `isolate=True` already asks for — `module:function`
returning `(config, store)` — and it is used for the same two reasons here:
importing it fills `JOBS`, and calling it builds the pair the handlers take.
An app that has written one for isolation has written this worker's too.
`--until-idle` drains what is queued and exits, which is the batch shape. Without
it the worker runs until it is interrupted, and a SIGINT or SIGTERM stops it the
way the web app stops: handlers in flight get `JobQueue.STOP_GRACE` to land,
and what is still running after that goes back to the queue on its lease.
**This worker fires schedules too**, which is what makes it a complete
replacement for the web app's own worker — a machine running only this still
gets its 06:00 job. `--no-schedules` turns that off, and is what every worker
after the first passes: they all share the jobs table safely, but the pile-up
guard reads `last_job_id`, which neither of two simultaneous firings has
written yet when the other looks. One scheduler per file, as many workers as
you like. `--timezone` is what the clock cadences are read in; without it they
are read in the machine's own zone, which on a server is UTC and is usually not
what the person who wrote `daily at 06:00` meant.
"""
from __future__ import annotations
import argparse
import asyncio
import contextlib
import logging
import signal
from pipeline.child import resolve
from pipeline.queue import JobQueue
from pipeline.table import Jobs
log = logging.getLogger(__name__)
def workers(text: str) -> dict[str, int]:
"""`cpu=2,http=8` to `{"cpu": 2, "http": 8}`."""
counts: dict[str, int] = {}
for part in filter(None, (p.strip() for p in text.split(","))):
name, _, n = part.partition("=")
if not name or not n.isdigit():
raise argparse.ArgumentTypeError(f"not resource=count: {part!r}")
counts[name] = int(n)
return counts
async def serve(queue: JobQueue) -> None:
"""Run until SIGINT or SIGTERM."""
stop = asyncio.Event()
loop = asyncio.get_running_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
# Windows has no loop signal handlers; there Ctrl+C arrives as
# KeyboardInterrupt instead, and `main` turns it into the same stop.
with contextlib.suppress(NotImplementedError, RuntimeError):
loop.add_signal_handler(sig, stop.set)
await queue.start()
try:
await stop.wait()
finally:
await queue.stop()
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="pipeline.worker", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--db", required=True, help="the SQLite file the jobs table is on")
parser.add_argument("--bootstrap", required=True,
help="module:function returning (config, store)")
parser.add_argument("--bootstrap-arg", default="",
help="one string handed to the bootstrap, e.g. a config path")
parser.add_argument("--workers", type=workers, default=None,
help="resource=count,... (default: JobQueue.WORKERS)")
parser.add_argument("--until-idle", action="store_true",
help="drain what is queued, then exit")
parser.add_argument("--no-schedules", action="store_true",
help="do not fire schedules here; for a second worker "
"on a file that already has a scheduler")
parser.add_argument("--timezone", default="",
help="what 'daily at 06:00' is 06:00 in "
"(default: this machine's own zone)")
args = parser.parse_args(argv)
build = resolve(args.bootstrap)
config, store = build(args.bootstrap_arg) if args.bootstrap_arg else build()
queue = JobQueue(Jobs(args.db), config, store, workers=args.workers,
bootstrap=args.bootstrap, bootstrap_arg=args.bootstrap_arg,
tz=args.timezone, schedules=not args.no_schedules)
try:
if args.until_idle:
queue.drain()
else:
asyncio.run(serve(queue))
except KeyboardInterrupt:
pass
return 0
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s")
raise SystemExit(main())