Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f436104f3
commit
9d666d0937
76 files changed
+15235
No files matched your search
@@ -0,0 +1,172 @@
|
||||
# pipeline
|
||||
|
||||
A job queue that lives in a SQLite table. Workers per resource, batching with a report job, dependency chains, dedupe, retries, cancellation that can kill, a WebSocket job list, and a standalone worker. No broker.
|
||||
|
||||
Needs Python 3.12+ and `pydantic`. `pipeline/web/` also needs `fastapi`; an app without a web face never imports it.
|
||||
|
||||
The source of truth is `boilerplate/pipeline/`. Copies elsewhere — news, health — are copies: take the whole folder, and run its tests where it lands.
|
||||
|
||||
```bash
|
||||
cp -r boilerplate/pipeline yourproject/pipeline
|
||||
python -m pytest yourproject/pipeline/tests
|
||||
```
|
||||
|
||||
## A job
|
||||
|
||||
A job is a plain function. `@job` registers it and returns it unchanged, so it stays callable from a terminal or a test with no queue.
|
||||
|
||||
```python
|
||||
from pipeline import NULL, JobQueue, Jobs, job
|
||||
|
||||
@job("greet", "Saying hello", resource="cpu")
|
||||
def greet(config, store, *, context=NULL, name: str = "") -> dict:
|
||||
context.progress(0, 1, f"greeting {name}")
|
||||
context.check() # stops here if Kill was pressed
|
||||
return {"said": f"hello {name}"}
|
||||
|
||||
greet(config, store, name="you") # direct call, no queue
|
||||
|
||||
queue = JobQueue(Jobs("app.db"), config, store)
|
||||
queue.enqueue("greet", {"name": "you"})
|
||||
queue.drain() # run everything queued, then stop
|
||||
```
|
||||
|
||||
`config` and `store` are whatever your app uses. The queue passes them through untouched. The payload is the keyword arguments, stored as JSON on the row.
|
||||
|
||||
| `@job(...)` | |
|
||||
|---|---|
|
||||
| `resource="cpu"` | which worker pool runs it. Pool sizes: `JobQueue(workers={"cpu": 1, "http": 8, "llm": 4})` |
|
||||
| `solo=True` | at most one queued or running at a time; a second enqueue returns the first |
|
||||
| `max_attempts=3, retry_delay=60` | retried by the queue, not by a sleeping worker |
|
||||
| `isolate=True, timeout=600` | runs in its own process, so Kill and the deadline actually end it |
|
||||
|
||||
## A web app
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI
|
||||
from pipeline import Jobs
|
||||
from pipeline.web import connect
|
||||
|
||||
app = FastAPI()
|
||||
queue = connect(app, Jobs("app.db"), config, store)
|
||||
```
|
||||
|
||||
That mounts `GET /api/jobs`, `POST /api/jobs`, `POST /api/jobs/{id}/cancel`, the `/ws/jobs` socket, and starts the worker with the app. For a job list with progress and Kill buttons, put this in a page:
|
||||
|
||||
```html
|
||||
<script type="module" src="/_jobs/jobs.js"></script>
|
||||
<desk-jobs></desk-jobs>
|
||||
```
|
||||
|
||||
It themes from CSS custom properties (`--desk-jobs-bg`, `--desk-jobs-accent`, …) and raises a `desk-jobs:notice` DOM event for failures instead of wording them. The wording is the app's.
|
||||
|
||||
## Many jobs and one report
|
||||
|
||||
```python
|
||||
@job("fetch_all")
|
||||
def fetch_all(config, store, *, context=NULL):
|
||||
urls = store.pending_urls()
|
||||
context.fan_out("fetch", [{"urls": urls[i:i + 10]} for i in range(0, len(urls), 10)],
|
||||
then="myapp.jobs:after_fetch")
|
||||
|
||||
def after_fetch(config, store, tally, carries, *, context):
|
||||
# tally.done / .failed / .cancelled, and tally.results: what each child returned
|
||||
context.fan_out("parse", ...) # the next step; this is a cascade
|
||||
```
|
||||
|
||||
`then` is `module:function` because it is stored on a row. The report job runs whether the children succeeded or not. `Task(kind, finder=...)` is the same thing with the finding done for you; register it in `TASKS` to make it a button.
|
||||
|
||||
## Chains
|
||||
|
||||
```python
|
||||
a = queue.enqueue("fetch", {"url": url})
|
||||
b = queue.enqueue("extract", {}, depends_on=[a.id], after="ok") # "done" = however A ended
|
||||
|
||||
@job("extract")
|
||||
def extract(config, store, *, context=NULL, html: str = ""):
|
||||
html = html or context.upstream().get("html", "") # A's result
|
||||
```
|
||||
|
||||
B reads A's result off A's row. Do not queue a row you mean to fill in later. The fallback keyword keeps `extract` callable by hand, since `upstream()` is `{}` outside the queue.
|
||||
|
||||
Results are reports: counts, ids, errors. A result over 64 KB fails the job with `ResultTooLarge` rather than being cut down. Put data in your store and return its ids, or raise `pipeline.table.MAX_RESULT` if a project really needs more.
|
||||
|
||||
## Ending work that does not look up
|
||||
|
||||
A thread cannot be interrupted, so `check()` is cooperative. Two cases can be killed:
|
||||
|
||||
- **A command** (ffmpeg, stockfish): `context.spawn([...], timeout=600)` kills the process tree on Kill or deadline.
|
||||
- **Python that blocks** (`sklearn.fit`): `@job(..., isolate=True, timeout=600)`. The job runs as `python -m pipeline.child`, which rebuilds `(config, store)` from a bootstrap you provide:
|
||||
|
||||
```python
|
||||
def bootstrap(config_path: str = ""): # myapp/jobs.py; importing it registers the jobs
|
||||
config = load_config(config_path)
|
||||
return config, Store(config.database)
|
||||
|
||||
JobQueue(jobs, config, store, bootstrap="myapp.jobs:bootstrap", bootstrap_arg=config_path)
|
||||
```
|
||||
|
||||
`timeout` without `isolate` is refused at registration: a thread's deadline could only mark the row failed while the work carried on.
|
||||
|
||||
## Schedules
|
||||
|
||||
A schedule is a row that says "enqueue this kind with this payload on this cadence". Same SQLite file as the jobs, so there is no second process to keep alive and a restart between two runs loses nothing.
|
||||
|
||||
```python
|
||||
from pipeline.schedule import declare
|
||||
|
||||
declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
|
||||
declare("hourly count", "count", "every 1h", {"n": 5})
|
||||
```
|
||||
|
||||
`declare` goes next to `@job` in the app's jobs module and fills `SCHEDULES` the way `@job` fills `JOBS`. On `JobQueue.start()` every declared name with no row gets one, and a row that already exists is left exactly as it was edited — cadence, payload and enabled alike. So a declaration is a starting point, not a setting: switching one off on screen survives the next deploy.
|
||||
|
||||
Five cadences and nothing else:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `every 15m` `every 6h` `every 2d` | elapsed time. Unaffected by a daylight-saving change, because an interval is an interval |
|
||||
| `daily at 06:00` | 06:00 on the wall clock of `JobQueue(tz=...)`, on both sides of a change |
|
||||
| `weekly mon 07:00` | the same, on that weekday. Long names (`monday`) are accepted and stored short |
|
||||
|
||||
Anything else raises `ValueError` listing all five, which is the sentence the API hands a person who typed something else. There is no seconds unit: the loop looks every `SCHEDULE_TICK` (30 s), and a cadence it cannot keep is one to refuse rather than round.
|
||||
|
||||
The tick runs inside the drain loop — no second thread, no second clock — and once on `start()`, so an app that was down over 06:00 runs that job when it comes back. Once: a firing sets `next_run` from *now* rather than stepping through the slots it missed, so there is nowhere for a backlog to accumulate. A schedule whose last job is still queued or running skips its turn and advances instead of piling copies of it up, and so does one whose kind is `solo` while another is under way. A row whose kind is no longer registered never fires, keeps its place, and is listed as `unknown kind` for somebody to delete.
|
||||
|
||||
```python
|
||||
queue = JobQueue(jobs, config, store, tz="Europe/Copenhagen") # or schedules=False
|
||||
```
|
||||
|
||||
Over HTTP, from `pipeline.web.connect`: `GET/POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`, `POST /api/schedules/{id}/run` (fires now without moving the next run). Every refusal is `{"detail": "<sentence>"}` — an unregistered kind 404s naming what there is, a payload missing keys 422s naming them, a bad cadence 422s with all five wordings. Any change pushes a `schedules` message on `/ws/jobs`.
|
||||
|
||||
```html
|
||||
<script type="module" src="/_jobs/schedules.js"></script>
|
||||
<desk-schedules></desk-schedules>
|
||||
```
|
||||
|
||||
`<desk-schedules>` is the table: name, kind, cadence edited in place, an on/off toggle, next and last run with the last job's state, Run now, Delete, and a row for adding one. Same `--desk-jobs-*` custom properties as `<desk-jobs>`, and just as silent about failures — it raises `desk-schedules:error` carrying the server's `detail` and leaves the wording to the app.
|
||||
|
||||
## A worker without the web app
|
||||
|
||||
```bash
|
||||
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
|
||||
python -m pipeline.worker ... --until-idle # drain and exit, for cron
|
||||
python -m pipeline.worker ... --timezone Europe/Copenhagen
|
||||
python -m pipeline.worker ... --no-schedules # every worker after the first
|
||||
```
|
||||
|
||||
Any number of workers may share one file. A claim is one atomic statement and a claimed job is held on a lease; a worker that dies has its jobs re-queued by whichever is still alive.
|
||||
|
||||
This worker fires schedules too, which is what makes it a complete replacement for the web app's own rather than half of one. Exactly one process per file should: `--no-schedules` is what the second and third pass, because the pile-up guard reads `last_job_id` and neither of two simultaneous firings has written it when the other looks.
|
||||
|
||||
## Housekeeping
|
||||
|
||||
Finished jobs older than `JobQueue(retain=30 * 86400)` are purged hourly while idle. Unfinished jobs and the newest ending of each kind are kept regardless. `retain=0` keeps everything.
|
||||
|
||||
The table is WAL-mode SQLite. Back it up with `VACUUM INTO`, not a file copy, and keep it off network filesystems.
|
||||
|
||||
## More
|
||||
|
||||
- `pipeline.md` — the reference card: every entity, the timings, the limits.
|
||||
- `__init__.py` — what each module is for.
|
||||
- `tests/` — every claim above, including two worker processes racing on one file and a schedule either side of a daylight-saving change.
|
||||
@@ -0,0 +1,127 @@
|
||||
"""The pipeline: how work is queued, run, watched and stopped.
|
||||
|
||||
from pipeline import JobQueue, Jobs, job, RunContext, Task, NULL
|
||||
|
||||
**This folder is meant to be copied.** It imports nothing from the app around
|
||||
it and nothing outside the standard library but pydantic — and FastAPI, in
|
||||
`pipeline/web/`, which an app that has no web face never imports. Drop the
|
||||
directory into another project, point `Jobs` at a SQLite file, and it works.
|
||||
`pipeline.md` beside this file is the write-up, and it travels with the folder.
|
||||
|
||||
**The source of truth is the boilerplate repo's copy.** news and health hold
|
||||
copies; a fix lands here first and is copied out with `git archive`.
|
||||
|
||||
Nothing here knows what a job *does*. The jobs themselves are the app's —
|
||||
`app/jobs.py` for the starter — and import `@job` from here. That is the one
|
||||
rule: the pipeline is imported by the work, never the other way round, so
|
||||
there is no cycle to break and no lazy import to remember.
|
||||
|
||||
models.py `Job`, `Run`, `JobsSnapshot`, `Notice` and the `Model` base
|
||||
they share — the shapes the queue stores and serves
|
||||
notices.py a failure turned into one line worth putting on screen.
|
||||
Only the part the queue needs; classifying a failure is the
|
||||
app's, because a useful sentence names things only it knows
|
||||
registry.py `@job`, `Spec`, and `JOBS` — how a function becomes a job,
|
||||
how many attempts it is worth, and whether it runs in a
|
||||
thread or in a child process of its own
|
||||
schedule.py `Cadence`, `declare` and `SCHEDULES` — the five wordings a
|
||||
standing instruction may be written in, when each next comes
|
||||
round, and how an app ships a default schedule next to `@job`
|
||||
context.py `RunContext` — what a running job is handed: progress,
|
||||
cancellation, notices, its parents' results, the fan-out,
|
||||
the subprocess helper, and the two things it can raise
|
||||
table.py `Jobs` — the jobs table, on whatever SQLite file it is given
|
||||
child.py the other side of `isolate=True`: one job, one process,
|
||||
reconstructed from its row
|
||||
queue.py `JobQueue` — the worker loop that drains that table
|
||||
task.py `Task` — plain code that finds work, cuts it into batches
|
||||
and queues them, plus the one final job every task ends
|
||||
with and the tally it hands to the callback. `Bound` is
|
||||
what a batch carries when how often to try it is the
|
||||
batch's own business rather than the job kind's
|
||||
worker.py `python -m pipeline.worker`: the same queue with no web app
|
||||
around it, for a second machine, a cron entry or a script
|
||||
web/ the queue over HTTP, the socket that reports on it, and the
|
||||
`<desk-jobs>` element that renders it. Optional: an app
|
||||
with no web face never imports it, and so never needs
|
||||
FastAPI
|
||||
|
||||
What the pipeline asks of an app is small and is typed where it is asked:
|
||||
handlers take `(config, store)` and the queue passes those through untouched,
|
||||
`Jobs` wants a file path, and `isolate=True` wants a `bootstrap` that can
|
||||
rebuild that pair in a child process. That is the whole of what a second app
|
||||
has to provide. `README.md` beside this file is the ten-minute version.
|
||||
|
||||
Within the package the order is `models, notices > registry > schedule >
|
||||
context > table > child > task > queue > worker > web`, and `tests/test_architecture.py` holds
|
||||
every file to it, and to one door: what is listed below is the whole of what
|
||||
anything outside may use.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pipeline import notices
|
||||
from pipeline.context import NULL, Cancelled, Retry, RunContext, spawn
|
||||
from pipeline.models import (
|
||||
TERMINAL, CascadeProgress, Job, JobState, JobsSnapshot, Notice,
|
||||
NoticeEntry, NoticeKind, NoticeLevel, Run, Schedule, TaskProgress,
|
||||
)
|
||||
from pipeline.queue import (
|
||||
ChildFailed, JobQueue, Listener, NoticeListener, ScheduleListener,
|
||||
local_zone,
|
||||
)
|
||||
from pipeline.registry import JOBS, Handler, Spec, job
|
||||
from pipeline.schedule import (
|
||||
DAYS, FORMS, SCHEDULES, BadCadence, Cadence, Declared, declare,
|
||||
)
|
||||
from pipeline.table import MAX_RESULT, Jobs, ResultTooLarge, RunCancelled
|
||||
from pipeline.task import FINAL, TASKS, Batch, Bound, Tally, Task, fan_out
|
||||
|
||||
__all__ = [
|
||||
"DAYS",
|
||||
"FINAL",
|
||||
"FORMS",
|
||||
"JOBS",
|
||||
"MAX_RESULT",
|
||||
"NULL",
|
||||
"SCHEDULES",
|
||||
"TASKS",
|
||||
"TERMINAL",
|
||||
"BadCadence",
|
||||
"Batch",
|
||||
"Bound",
|
||||
"Cadence",
|
||||
"Cancelled",
|
||||
"CascadeProgress",
|
||||
"ChildFailed",
|
||||
"Declared",
|
||||
"Handler",
|
||||
"Job",
|
||||
"JobQueue",
|
||||
"JobState",
|
||||
"Jobs",
|
||||
"JobsSnapshot",
|
||||
"Listener",
|
||||
"Notice",
|
||||
"NoticeEntry",
|
||||
"NoticeKind",
|
||||
"NoticeLevel",
|
||||
"NoticeListener",
|
||||
"ResultTooLarge",
|
||||
"Retry",
|
||||
"Run",
|
||||
"RunCancelled",
|
||||
"RunContext",
|
||||
"Schedule",
|
||||
"ScheduleListener",
|
||||
"Spec",
|
||||
"Tally",
|
||||
"Task",
|
||||
"TaskProgress",
|
||||
"declare",
|
||||
"fan_out",
|
||||
"job",
|
||||
"local_zone",
|
||||
"notices",
|
||||
"spawn",
|
||||
]
|
||||
@@ -0,0 +1,140 @@
|
||||
"""The other side of `isolate=True`: one job, one process, and then gone.
|
||||
|
||||
python -m pipeline.child --db news.db --job 412 \
|
||||
--bootstrap newsdesk.jobs:bootstrap --out C:/Temp/tmp123.json
|
||||
|
||||
Run by the worker, never by a person, and the whole reason it can exist is
|
||||
that **a job row is already a complete description of the work.** A kind, a
|
||||
JSON payload, a database file. Nothing is pickled and nothing is sent: the
|
||||
child is told where to look and reads the same row the worker read.
|
||||
|
||||
Which is also the division of labour that keeps the state machine intact.
|
||||
The child writes *progress* — a non-terminal update, and the snapshot the UI
|
||||
reads is a `SELECT`, so a child's progress reaches the browser for free. The
|
||||
child does **not** write the ending. It hands its result back through a file
|
||||
and exits, and the worker writes `done` or `failed` exactly as it does for a
|
||||
job that ran in a thread. One writer of the terminal state, as before.
|
||||
|
||||
A file rather than stdout because a handler is arbitrary code and arbitrary
|
||||
code prints. Parsing a result out of a stream something else is also writing
|
||||
to is a bug waiting for the first `print` in a fetch loop.
|
||||
|
||||
The `bootstrap` is the one thing an app has to provide, and it has two jobs:
|
||||
import the modules that fill `JOBS`, and return the `(config, store)` pair
|
||||
the handlers take. Neither survives a process boundary, so neither is sent —
|
||||
it is rebuilt here from the config file the app already knows how to read.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pipeline.context import Cancelled, RunContext
|
||||
from pipeline.registry import JOBS
|
||||
from pipeline.table import Jobs
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: What the worker reads off the exit code. Anything else is the process
|
||||
#: dying of something that was never ours to catch — an OOM kill, a segfault
|
||||
#: in a native library — and the worker reports it as the failure it is.
|
||||
OK = 0
|
||||
FAILED = 1
|
||||
CANCELLED = 2
|
||||
|
||||
|
||||
def resolve(name: str) -> Any:
|
||||
"""`module:attribute` to the attribute. The same spelling a callback uses."""
|
||||
module, _, attr = name.partition(":")
|
||||
if not module or not attr:
|
||||
raise ValueError(f"a bootstrap is 'module:function', not {name!r}")
|
||||
return getattr(importlib.import_module(module), attr)
|
||||
|
||||
|
||||
def context_for(jobs: Jobs, job_id: int, task_id: int) -> RunContext:
|
||||
"""A context that reports to the table and asks it about cancellation.
|
||||
|
||||
Every channel the in-process context has, except the two that cannot
|
||||
cross a process boundary and are not wanted here anyway: `on_task` and
|
||||
`on_fan_out` both queue work, and an isolated job is isolated precisely
|
||||
because it is one long piece of work rather than an organiser of other
|
||||
work. Both fall back to their no-ops, which report that nothing took it.
|
||||
|
||||
Cancellation is a read of the row, which is what makes Kill work at all
|
||||
from here: the button writes `cancel_requested` in the web process, and
|
||||
this is the only way this process would ever hear about it. The worker
|
||||
also kills us outright — this is the polite half, so a handler that calls
|
||||
`check()` stops at a line of its own choosing and gets to clean up.
|
||||
"""
|
||||
return RunContext(
|
||||
on_progress=lambda done, total, message: jobs.progress(
|
||||
job_id, done, total, message),
|
||||
is_cancelled=lambda: jobs.cancel_requested(job_id),
|
||||
is_killed=lambda: bool(task_id) and jobs.cancelled(task_id),
|
||||
on_notice=lambda notice: print(
|
||||
json.dumps({"notice": notice.model_dump()}), file=sys.stderr),
|
||||
)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(prog="pipeline.child", description=__doc__)
|
||||
parser.add_argument("--db", required=True, help="the SQLite file the jobs table is on")
|
||||
parser.add_argument("--job", required=True, type=int, help="which row to run")
|
||||
parser.add_argument("--bootstrap", required=True,
|
||||
help="module:function returning (config, store)")
|
||||
parser.add_argument("--bootstrap-arg", default=None,
|
||||
help="one string handed to the bootstrap, e.g. a config path")
|
||||
parser.add_argument("--out", required=True, help="where to write the result JSON")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
out = Path(args.out)
|
||||
jobs = Jobs(args.db)
|
||||
job = jobs.get(args.job)
|
||||
if job is None:
|
||||
out.write_text(json.dumps({"error": f"no job {args.job}"}), encoding="utf-8")
|
||||
return FAILED
|
||||
|
||||
try:
|
||||
build = resolve(args.bootstrap)
|
||||
config, store = (build() if args.bootstrap_arg is None
|
||||
else build(args.bootstrap_arg))
|
||||
spec = JOBS.get(job.kind)
|
||||
if spec is None:
|
||||
raise KeyError(
|
||||
f"{job.kind!r} is not a registered job in this process — "
|
||||
f"does {args.bootstrap} import the module that declares it?"
|
||||
)
|
||||
context = context_for(jobs, job.id, job.task_id)
|
||||
kwargs = dict(job.payload)
|
||||
if spec.wants_context:
|
||||
kwargs["context"] = context
|
||||
result = spec.fn(config, store, **kwargs)
|
||||
except Cancelled:
|
||||
out.write_text(json.dumps({}), encoding="utf-8")
|
||||
return CANCELLED
|
||||
except BaseException as exc:
|
||||
# The traceback goes in the file as well as to stderr. stderr is the
|
||||
# worker's fallback and is captured either way; the file is what
|
||||
# survives a child that also wrote something confusing to the stream.
|
||||
out.write_text(json.dumps({
|
||||
"error": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
}), encoding="utf-8")
|
||||
traceback.print_exc()
|
||||
return FAILED
|
||||
|
||||
out.write_text(json.dumps({"result": result or {}}, default=str), encoding="utf-8")
|
||||
return OK
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO,
|
||||
format="%(levelname)s %(name)s: %(message)s")
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,446 @@
|
||||
"""What a running job is handed, and the two things it can raise.
|
||||
|
||||
def handler(config, store, *, context=NULL, limit=None) -> dict
|
||||
|
||||
This is the job-side half of the contract; `registry.py` is the other — how
|
||||
a function becomes queueable. A pipeline stage imports this to report
|
||||
progress from inside its own loops, and nothing else: no queue, no worker.
|
||||
Nothing here imports anything else from desk at module level, so it is safe to
|
||||
import from anywhere.
|
||||
|
||||
Everything in here is optional to the job that receives it. Ignore `context`
|
||||
and the job still runs — it just cannot be stopped part-way and shows no
|
||||
progress. Use it and you get both, for two lines.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: How often a spawned process is looked in on: whether it has finished,
|
||||
#: whether Kill has been pressed, whether its deadline has passed. Frequent
|
||||
#: enough that a Kill lands in well under a second, rare enough that waiting
|
||||
#: an hour for ffmpeg costs a few thousand cheap syscalls.
|
||||
POLL = 0.25
|
||||
|
||||
|
||||
class Cancelled(Exception):
|
||||
"""Raised by `RunContext.check` when the job has been asked to stop.
|
||||
|
||||
Caught by the worker and recorded as `cancelled` rather than `failed`. A
|
||||
handler should let it travel — catching it is how you build the thing the
|
||||
Kill button can't kill.
|
||||
"""
|
||||
|
||||
|
||||
class Retry(Exception):
|
||||
"""Raised by a handler that wants this same work attempted again later.
|
||||
|
||||
An ordinary failure is already retried, within whatever bound the job was
|
||||
queued with — so this is not how a job asks to be retried at all. It is how
|
||||
it says *when*: `after` is the wait the other end named, a `Retry-After`
|
||||
header or a rate limit's own answer, and the queue uses it in place of the
|
||||
backoff it would otherwise have guessed. Guessing is the thing worth
|
||||
avoiding here — a minute is either a minute of doing nothing when the
|
||||
service was ready, or another refusal charged against the bound.
|
||||
|
||||
It does not buy an attempt. A job queued with one attempt raises this and
|
||||
fails, exactly as any other exception would: the bound is ours to set and
|
||||
the wait is the world's to name, and a handler that could extend its own
|
||||
bound would be a policy nobody declared.
|
||||
"""
|
||||
|
||||
def __init__(self, message: str = "", *, after: float | None = None) -> None:
|
||||
super().__init__(message)
|
||||
#: Seconds to wait, or None to leave it to the job's own backoff.
|
||||
self.after = after
|
||||
|
||||
|
||||
def _no_progress(done: int, total: int, message: str) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def _no_task(task: Any) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _no_stop(why: str) -> int:
|
||||
"""Where "stop the rest of this run" goes when there is no run to stop.
|
||||
|
||||
The direct path is one batch after another in the caller's own thread, and
|
||||
the caller is standing there: it sees each failure as it happens and can
|
||||
stop by not asking again. Nothing to cancel, so nothing is cancelled.
|
||||
"""
|
||||
return 0
|
||||
|
||||
|
||||
def _no_fan_out(*_args: Any, **_kwargs: Any) -> tuple[int, ...]:
|
||||
"""Where a fan-out goes when there is no queue to put it in.
|
||||
|
||||
Empty rather than an error, and the same answer `task` gives for the same
|
||||
reason: a handler called directly — from a terminal, a notebook, a test —
|
||||
is being run by somebody who is standing there, and the useful behaviour
|
||||
is to do its own work and report that nothing was queued behind it. A
|
||||
handler that must know either way checks the return.
|
||||
"""
|
||||
return ()
|
||||
|
||||
|
||||
def _log_only(notice: Any) -> None:
|
||||
"""Where a notice goes when nothing is watching for one.
|
||||
|
||||
A direct caller — `newsdesk job pull` in a terminal, a notebook, a test — has
|
||||
no socket, so the log is the destination. It is not a *lesser* one: it goes
|
||||
through the same `notices.render`, so the terminal gets the sentence, the
|
||||
command to run and the is-this-a-bug verdict, exactly as the browser does.
|
||||
"""
|
||||
from pipeline.notices import render
|
||||
|
||||
log.warning("%s\n%s", notice.code, render(notice))
|
||||
|
||||
|
||||
def _never() -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def kill_tree(proc: subprocess.Popen[Any]) -> None:
|
||||
"""Kill a spawned process *and* whatever it started.
|
||||
|
||||
The children are the point. `proc.kill()` reaches the process we launched
|
||||
and nothing it launched itself, so killing a shell leaves the ffmpeg it
|
||||
ran holding the core — which is the exact failure a deadline exists to
|
||||
prevent. Both branches need the process to have been started detached,
|
||||
which `spawn` does.
|
||||
|
||||
Best-effort by construction: between deciding to kill and killing, the
|
||||
process may have ended on its own and its id may have been reused. So
|
||||
every call is guarded, and a failure to kill something that is already
|
||||
gone is not news.
|
||||
"""
|
||||
if proc.poll() is not None:
|
||||
return
|
||||
try:
|
||||
if os.name == "nt":
|
||||
# No process groups to signal, so ask Windows to walk the tree.
|
||||
# `/T` is children, `/F` is without asking.
|
||||
subprocess.run(["taskkill", "/T", "/F", "/PID", str(proc.pid)],
|
||||
capture_output=True, check=False, timeout=10)
|
||||
else:
|
||||
# Guarded by `os.name` above rather than by the type checker,
|
||||
# which is reading this file on Windows and knows these three do
|
||||
# not exist there.
|
||||
os.killpg(os.getpgid(proc.pid), signal.SIGKILL) # type: ignore[attr-defined,unused-ignore]
|
||||
except Exception:
|
||||
log.debug("could not kill %s", proc.pid, exc_info=True)
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except Exception:
|
||||
log.warning("process %s did not go away after being killed", proc.pid)
|
||||
|
||||
|
||||
def spawn(argv: Sequence[str], *, check: Callable[[], None] | None = None,
|
||||
timeout: float | None = None, poll: float = POLL,
|
||||
**popen: Any) -> subprocess.CompletedProcess[bytes]:
|
||||
"""Run a command, watching for Kill and for a deadline while it does.
|
||||
|
||||
The building block under `RunContext.spawn` and under `isolate=True`, and
|
||||
the answer to the one thing a thread cannot do. A handler that blocks in
|
||||
`subprocess.run` cannot be stopped, because stopping it would mean tearing
|
||||
a thread out of a syscall; a handler that blocks *here* can, because what
|
||||
is expensive is the child and a child can be killed. The thread is only
|
||||
sleeping, which costs nothing.
|
||||
|
||||
Output goes to temporary files rather than pipes. `PIPE` plus a poll loop
|
||||
is a deadlock: a child that fills the pipe buffer blocks on the write
|
||||
while we block on the poll, and neither side is wrong. A file cannot fill
|
||||
up in the way that matters, so the loop is free to look away.
|
||||
|
||||
Started detached — its own process group, or Windows' equivalent — purely
|
||||
so `kill_tree` has something to aim at.
|
||||
"""
|
||||
detach: dict[str, Any] = ({"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
|
||||
if os.name == "nt" else {"start_new_session": True})
|
||||
with tempfile.TemporaryFile() as out, tempfile.TemporaryFile() as err:
|
||||
popen.setdefault("stdout", out)
|
||||
popen.setdefault("stderr", err)
|
||||
proc = subprocess.Popen(list(argv), **detach, **popen)
|
||||
deadline = time.monotonic() + timeout if timeout else None
|
||||
try:
|
||||
while proc.poll() is None:
|
||||
if check is not None:
|
||||
check() # raises Cancelled on Kill
|
||||
if deadline is not None and time.monotonic() > deadline:
|
||||
kill_tree(proc)
|
||||
raise TimeoutError(
|
||||
f"{argv[0]} was still running after {timeout:g}s")
|
||||
time.sleep(poll)
|
||||
except BaseException:
|
||||
kill_tree(proc)
|
||||
raise
|
||||
out.seek(0)
|
||||
err.seek(0)
|
||||
return subprocess.CompletedProcess(
|
||||
list(argv), proc.returncode,
|
||||
out.read() if popen["stdout"] is out else b"",
|
||||
err.read() if popen["stderr"] is err else b"")
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class RunContext:
|
||||
"""The handle a running job has on the run it is part of.
|
||||
|
||||
Deliberately a handful of plain callables rather than a reference to the
|
||||
queue: a handler cannot reach back into the worker, reschedule itself or
|
||||
read other jobs, and a test can hand it a lambda without standing anything
|
||||
up.
|
||||
"""
|
||||
|
||||
on_progress: Callable[[int, int, str], None] = _no_progress
|
||||
is_cancelled: Callable[[], bool] = _never
|
||||
#: Whether the *task* this job belongs to was killed — a different
|
||||
#: question from `is_cancelled`, which is about this job alone. See
|
||||
#: `killed`.
|
||||
is_killed: Callable[[], bool] = _never
|
||||
on_notice: Callable[[Any], None] = _log_only
|
||||
on_task: Callable[[Any], bool] = _no_task
|
||||
on_stop: Callable[[str], int] = _no_stop
|
||||
#: Where `fan_out` puts the jobs it asks for. Supplied by the queue, which
|
||||
#: is the only thing that can write rows; `_no_fan_out` under `NULL`.
|
||||
on_fan_out: Callable[..., tuple[int, ...]] = _no_fan_out
|
||||
#: How the jobs this one waited for ended, oldest first. Empty for a job
|
||||
#: that waited for nothing. Typed loosely for the same reason `notice` is:
|
||||
#: nothing in this module may import the rest of desk, and the shape it
|
||||
#: carries is `pipeline.models.Job`.
|
||||
waited_for: tuple[Any, ...] = ()
|
||||
|
||||
def upstream(self) -> dict[str, Any]:
|
||||
"""What the jobs this one waited for returned, merged into one dict.
|
||||
|
||||
The read half of a chain. `A -> B -> C` needs A's output to reach B,
|
||||
and this is how it arrives: B does not have its payload filled in by
|
||||
A, it reads A's result off A's row. Which is the whole difference —
|
||||
**a row is complete when it is written.** A payload with a hole in it
|
||||
meant for an earlier job to fill would be claimable the instant that
|
||||
job landed, so a worker could take it between the write and the
|
||||
commit; it could not carry a `dedup_key`, which has to be known at
|
||||
queue time; and it could not be re-run from its own row, which is
|
||||
what a retry does and what `max_attempts` is.
|
||||
|
||||
Merged, later parents winning, because the overwhelmingly common case
|
||||
is one parent and `upstream()["text"]` is what that should read like.
|
||||
A job with several parents that care about the same key wants
|
||||
`waited_for` and to decide for itself.
|
||||
|
||||
Empty under `NULL`, which is what makes the fallback work:
|
||||
|
||||
def handler(config, store, *, context=NULL, html=""):
|
||||
html = html or context.upstream().get("html", "")
|
||||
|
||||
From the queue nobody passes `html` and it comes from the parent; from
|
||||
a terminal you pass it yourself. Without that line a chained handler
|
||||
is the one kind in the pipeline you cannot simply call.
|
||||
"""
|
||||
merged: dict[str, Any] = {}
|
||||
for parent in self.waited_for:
|
||||
result = getattr(parent, "result", None)
|
||||
if isinstance(result, dict):
|
||||
merged |= result
|
||||
return merged
|
||||
|
||||
def fan_out(self, kind: str, payloads: Sequence[dict[str, Any]], *,
|
||||
then: str = "", settle: str = "", label: str = "",
|
||||
carries: dict[str, Any] | None = None,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> tuple[int, ...]:
|
||||
"""Queue one job per payload, and one job behind them that reports.
|
||||
|
||||
Returns the ids of the children. The reporting job is not among them:
|
||||
it is the pipeline's own kind, it is nothing an app writes, and it is
|
||||
the reason this method exists rather than a handler assembling a
|
||||
`depends_on` list by hand every time.
|
||||
|
||||
def handler(config, store, *, context=NULL):
|
||||
chunks = chunk(needs_a_body(store))
|
||||
context.fan_out("body_batch", [{"uids": c} for c in chunks],
|
||||
then="newsdesk.jobs.cascade:step")
|
||||
|
||||
`then` is `module:function`, resolved where it is run, and is handed
|
||||
`(config, store, tally, carries)` once every child is over however it
|
||||
ended — see `Tally` in `task.py`. A name rather than a function
|
||||
because what carries it is a table row, and no row can hold a
|
||||
function.
|
||||
|
||||
One transaction, so a fan-out is never half-queued. The children go
|
||||
into a run of their own within this job's cascade, which is what makes
|
||||
the callback able to fan out again: that is a cascade, and there is no
|
||||
cascade object anywhere — step N's callback queues step N+1.
|
||||
|
||||
Nothing is queued under `NULL`, and `()` says so.
|
||||
"""
|
||||
return self.on_fan_out(
|
||||
kind, list(payloads), then=then, settle=settle, label=label,
|
||||
carries=dict(carries or {}), max_attempts=max_attempts,
|
||||
retry_delay=retry_delay, retry_backoff=retry_backoff)
|
||||
|
||||
def spawn(self, argv: Sequence[str], *, timeout: float | None = None,
|
||||
**popen: Any) -> subprocess.CompletedProcess[bytes]:
|
||||
"""Run an external command, killable, with this job's Kill wired in.
|
||||
|
||||
What to reach for when the expensive thing is a binary — ffmpeg,
|
||||
stockfish, whisper.cpp. Because the work is in the child and not in
|
||||
this thread, Kill is no longer cooperative in the way `check` is: the
|
||||
child is killed outright, and its children with it. Blocking a worker
|
||||
thread on a process costs nothing, since the thread is only sleeping.
|
||||
|
||||
`timeout` is a real deadline for the same reason, and the only kind of
|
||||
deadline the pipeline offers that can actually end the work. A job
|
||||
kind that wants one over Python code rather than over a command wants
|
||||
`isolate=True` — see `registry.py`.
|
||||
|
||||
Raises `Cancelled` on Kill and `TimeoutError` on the deadline, having
|
||||
killed the process tree either way. Non-zero exit is *not* an error
|
||||
here: the returncode is handed back, because plenty of commands say
|
||||
something useful with it and only the handler knows which.
|
||||
"""
|
||||
return spawn(argv, check=self.check, timeout=timeout, **popen)
|
||||
|
||||
def progress(self, done: int = 0, total: int = 0, message: str = "") -> None:
|
||||
"""Say how far along this is. Units are the handler's own business —
|
||||
sources, files, minutes of audio — the UI renders `done/total`.
|
||||
|
||||
The message has one shape, and every job in this codebase uses it:
|
||||
|
||||
a lowercase participle, what it is being done to, and — when the
|
||||
job is walking a list — `, n of m`.
|
||||
|
||||
"reading Golem, 3 of 12". "fetching the body from theverge.com, 40 of
|
||||
212". "training on your votes" for a job that is one step and has no
|
||||
list to be three of. Not an em dash, not a middot, and never a bare
|
||||
identifier: this used to say `wordpress`, which named the module that
|
||||
reads the feed rather than the outlet, and read as a fragment of a log
|
||||
line that had escaped onto the status strip.
|
||||
|
||||
`n` is the item being worked on now and so runs one ahead of `done`,
|
||||
which counts the ones that are finished. They are answering different
|
||||
questions — the fraction in the message belongs to the name beside it,
|
||||
and the bar belongs to the job — and a line saying "reading Golem, 0 of
|
||||
12" while Golem is being read would be wrong about the only thing it
|
||||
was there to say. A phase that has nothing left to name ends on the
|
||||
collective — "reading every source, 12 of 12" — so the same participle
|
||||
carries the line from first item to last, and a full fraction is what
|
||||
says it is over. Past tense is not used: it dates a line that is about
|
||||
to be replaced by the next phase's.
|
||||
|
||||
A job's *label* is the same grammar capitalised, because a label names
|
||||
the work rather than narrating it: "Tagging 25 items, 2 of 9".
|
||||
"""
|
||||
self.on_progress(done, total, message)
|
||||
|
||||
def check(self) -> None:
|
||||
"""Stop now if Kill has been pressed.
|
||||
|
||||
Cancellation is cooperative because the work is synchronous: there is
|
||||
no safe way to tear a thread out of the middle of an HTTP fetch or a
|
||||
model load, so the job says where stopping is fine. Call it between
|
||||
units of work, not inside tight inner loops.
|
||||
"""
|
||||
if self.is_cancelled():
|
||||
raise Cancelled()
|
||||
|
||||
def killed(self) -> bool:
|
||||
"""Whether the task this job belongs to was killed.
|
||||
|
||||
Not the question `check` asks. That one is about this job, and killing
|
||||
one job is dropping one batch. This one is about the run, and killing
|
||||
a run is skipping the whole step — which the job that reports on the
|
||||
step has to be able to say, since it is spared the kill precisely so
|
||||
that somebody hears about it. The counts cannot tell the two apart: a
|
||||
batch someone killed by hand and a batch killed by the task going
|
||||
down both arrive as `cancelled`.
|
||||
|
||||
Asked when asked, not read at the start, so a kill landing while the
|
||||
reporting job runs is still in what it reports.
|
||||
"""
|
||||
return bool(self.is_killed())
|
||||
|
||||
def step(self, done: int = 0, total: int = 0, message: str = "") -> None:
|
||||
"""`progress` then `check` — what the top of a loop body always wants."""
|
||||
self.progress(done, total, message)
|
||||
self.check()
|
||||
|
||||
def task(self, task: Any) -> bool:
|
||||
"""Queue a `Task` once this job has finished well.
|
||||
|
||||
Recorded rather than queued: the worker starts the task after this job
|
||||
lands in `done`, so a task never outlives the failure of the job that
|
||||
described it, and a handler still cannot reach the queue. It names what
|
||||
it wants to happen and that is all.
|
||||
|
||||
The task runs in the worker, synchronously, the moment the job is over
|
||||
— the same plain code a request would run. It is handed the queue as
|
||||
its sink and this job's cascade, so its batches land in the run this
|
||||
one belongs to.
|
||||
|
||||
Returns whether anything took it. `False` under `NULL`, where there is
|
||||
no queue to take it: a callback invoked from a notebook or a Claude
|
||||
Code session has to do the work itself, and this is how it finds out
|
||||
that nobody else will.
|
||||
"""
|
||||
return bool(self.on_task(task))
|
||||
|
||||
def stop(self, why: str = "") -> int:
|
||||
"""Cancel this task's remaining queued batches. Returns how many.
|
||||
|
||||
For the failure that is not this batch's: a spend limit, a signed-out
|
||||
CLI, no model on the machine at all. The backend has answered, the
|
||||
answer is the same one every sibling is about to be given, and the
|
||||
only thing left to decide is whether to spend the other twenty-two
|
||||
calls finding that out. Twenty-three identical notices saying "looks
|
||||
like a bug" is what not deciding looked like.
|
||||
|
||||
Only what is still *queued* stops. A batch already in flight is a
|
||||
request in the air and a bill either way, so it runs to its own end —
|
||||
and this one, the one that asked, keeps its own failure. The task's
|
||||
final job is spared for the reason `Jobs.cancel_run` spares it:
|
||||
reporting is not work, and a run that stopped early still has to say
|
||||
so. `why` is written onto each cancelled row, so a batch that never
|
||||
ran says on screen why it didn't.
|
||||
|
||||
Not a kill. Nobody pressed anything, so the task's row is not flagged
|
||||
and `killed` stays false: the cascade goes on to the next stage, as it
|
||||
does after any batch that did not land. Whether the *next* task should
|
||||
stop too is that cascade's question, not this one's.
|
||||
"""
|
||||
return int(self.on_stop(why))
|
||||
|
||||
def notice(self, notice: Any) -> None:
|
||||
"""Tell the user something went wrong. See `pipeline/notices.py`.
|
||||
|
||||
The counterpart to `log.exception`, not a replacement for it: log the
|
||||
traceback for afterwards, raise a notice for now. Use it for what a
|
||||
person has to act on — a login to renew, a feed that has stopped
|
||||
resolving — and not for what the program can simply get on with.
|
||||
|
||||
Under a queue it is written to the notice log and pushed to whoever is
|
||||
watching; called directly it is logged and nothing else. Safe to call
|
||||
repeatedly: the same `code` and `key` again is one row with a count,
|
||||
and only the first interrupts anyone. So give `key` whatever the
|
||||
failure is about — the source, the article — or two different
|
||||
failures of one kind are told as one.
|
||||
"""
|
||||
self.on_notice(notice)
|
||||
|
||||
|
||||
#: For calling a handler outside the queue. Reports nowhere, never cancels.
|
||||
NULL = RunContext()
|
||||
@@ -0,0 +1,432 @@
|
||||
"""The shapes the queue stores and serves: a job, a run, a snapshot, a notice.
|
||||
|
||||
One set of models, not two — there is no ORM here, so the row the table
|
||||
returns and the object the API serves are the same declaration. `Model` is
|
||||
the base an app's own shapes are welcome to extend as well, because the one
|
||||
thing it does (tolerate a `sqlite3.Row`, ignore columns nobody declared) is
|
||||
wanted by every table anyone will ever `SELECT *` from.
|
||||
|
||||
Pydantic is the package's only dependency outside the standard library, and
|
||||
this is the file that wants it. Everything else here would run without it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
from typing import Annotated, Any, Literal, Self
|
||||
|
||||
from pydantic import (BaseModel, BeforeValidator, ConfigDict, computed_field)
|
||||
|
||||
#: A column that holds JSON. SQLite hands back the text; everything above the
|
||||
#: storage layer wants the dict. Decoded here so neither side has to remember,
|
||||
#: and tolerant of a malformed value because a job whose payload failed to
|
||||
#: parse should still be listable and cancellable rather than un-readable.
|
||||
JsonDict = Annotated[
|
||||
dict[str, Any],
|
||||
BeforeValidator(lambda v: v if isinstance(v, dict) else _loads(v)),
|
||||
]
|
||||
|
||||
|
||||
def _loads(value: Any) -> dict[str, Any]:
|
||||
try:
|
||||
parsed = json.loads(value or "{}")
|
||||
except (TypeError, ValueError):
|
||||
return {}
|
||||
return parsed if isinstance(parsed, dict) else {}
|
||||
|
||||
|
||||
#: The same for a column holding a JSON array of job ids. Tolerant for the same
|
||||
#: reason: a job whose dependency list will not parse has already been let
|
||||
#: through by the queue, which reads the column in SQL — so refusing to build
|
||||
#: the model here would only hide it from the list it is holding up.
|
||||
JsonIds = Annotated[
|
||||
list[int],
|
||||
BeforeValidator(lambda v: v if isinstance(v, list) else _ids(v)),
|
||||
]
|
||||
|
||||
|
||||
def _ids(value: Any) -> list[int]:
|
||||
try:
|
||||
parsed = json.loads(value or "[]")
|
||||
except (TypeError, ValueError):
|
||||
return []
|
||||
return [int(i) for i in parsed] if isinstance(parsed, list) else []
|
||||
|
||||
|
||||
class Model(BaseModel):
|
||||
"""Base: tolerate sqlite3.Row and reject fields nobody declared."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True, extra="forbid")
|
||||
|
||||
@classmethod
|
||||
def from_row(cls, row: Any) -> Self:
|
||||
"""Build one from a database row, ignoring columns nobody declared.
|
||||
|
||||
`SELECT *` into a model that forbids extras means the next
|
||||
`ALTER TABLE` is a 500 in whatever route reads that table — raised a
|
||||
long way from the migration that caused it, over a column the API was
|
||||
never going to serve anyway. A storage column the API has not heard of
|
||||
is not something to hand out; it is something to ignore until someone
|
||||
declares a field for it.
|
||||
|
||||
The half of `extra="forbid"` worth keeping still works. A column a
|
||||
model *requires* going missing — a rename, a typo — is a validation
|
||||
error exactly as before. Only additions go quiet.
|
||||
"""
|
||||
return cls.model_validate(
|
||||
{k: v for k, v in dict(row).items() if k in cls.model_fields}
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- jobs
|
||||
|
||||
#: The five states a job can be in. `queued` and `running` are live; the other
|
||||
#: three are terminal and a job never leaves them.
|
||||
JobState = Literal["queued", "running", "done", "failed", "cancelled"]
|
||||
|
||||
#: Everything that is over, however it ended. Used to decide what the dedup
|
||||
#: index constrains, what the UI greys out, and when to stop pushing updates.
|
||||
TERMINAL: frozenset[str] = frozenset({"done", "failed", "cancelled"})
|
||||
|
||||
|
||||
class Job(Model):
|
||||
"""One unit of queued work, as stored.
|
||||
|
||||
`payload` is the handler's keyword arguments and `result` is what it
|
||||
returned — both JSON in SQLite, both dicts everywhere else, translated
|
||||
once here rather than at each call site.
|
||||
|
||||
`done`/`total` are deliberately untyped as units. A pull counts sources, a
|
||||
transcription counts minutes of audio; the UI renders "3/12" and a bar and
|
||||
does not need to know which.
|
||||
"""
|
||||
|
||||
id: int
|
||||
kind: str
|
||||
label: str = ""
|
||||
payload: JsonDict = {}
|
||||
state: JobState = "queued"
|
||||
done: int = 0
|
||||
total: int = 0
|
||||
message: str = ""
|
||||
result: JsonDict = {}
|
||||
error: str = ""
|
||||
dedup_key: str = ""
|
||||
#: Which jobs this one waits for, and whether it needs them to have
|
||||
#: succeeded (`ok`) or merely to be over (`done`).
|
||||
depends_on: JsonIds = []
|
||||
after: str = "ok"
|
||||
#: Which kind of worker runs it: `llm`, `http`, `cpu`.
|
||||
resource: str = "cpu"
|
||||
#: Which cascade this belongs to, and which of its tasks. A job queued on
|
||||
#: its own is its own cascade and its own task. Both name a row in `runs`
|
||||
#: — a series of their own, not this one's. See `Run`.
|
||||
cascade_id: int = 0
|
||||
task_id: int = 0
|
||||
cancel_requested: bool = False
|
||||
#: How many times this has been started, counted by `claim`. One after the
|
||||
#: first run, whether that run ended well, badly, or with the worker dying
|
||||
#: under it.
|
||||
attempt: int = 0
|
||||
#: The bound, counting the first attempt — so `1` is the default and means
|
||||
#: no second try. Carried on the row rather than looked up from the job
|
||||
#: kind, because the attempt that decides is the one after a restart, in a
|
||||
#: worker that may be running a different version of the code.
|
||||
max_attempts: int = 1
|
||||
#: The wait before the next attempt, and what multiplies it each time
|
||||
#: round. `1.0` is a flat wait.
|
||||
retry_delay: float = 0.0
|
||||
retry_backoff: float = 1.0
|
||||
#: Not before this. Empty means now, which is why it is `''` rather than
|
||||
#: NULL: the column is compared as text and `''` sorts before every stamp,
|
||||
#: so "is it due?" is one comparison with no special case in it.
|
||||
run_after: str = ""
|
||||
created_at: str
|
||||
started_at: str | None = None
|
||||
finished_at: str | None = None
|
||||
|
||||
@property
|
||||
def live(self) -> bool:
|
||||
return self.state not in TERMINAL
|
||||
|
||||
@computed_field # type: ignore[prop-decorator]
|
||||
@property
|
||||
def waiting(self) -> bool:
|
||||
"""Queued, but not until its next-attempt time comes round.
|
||||
|
||||
Serialised, unlike its two neighbours, because it is the one of the
|
||||
three a client cannot work out for itself. `live` is a state, and
|
||||
`exhausted` is `attempt`, `max_attempts` and `state`, all of which go
|
||||
out on the row; this one is `run_after` against *now*, and the only
|
||||
clock worth asking is the one the stamp was written by. A browser four
|
||||
minutes fast would otherwise draw a job as due while the queue goes on
|
||||
passing over it.
|
||||
|
||||
`run_after` travels too and still has to: this says *that* it is
|
||||
waiting and the stamp says until when, which is the difference between
|
||||
greying a row out and counting down on it.
|
||||
|
||||
The resting state of a failure that is going to be tried again, and the
|
||||
thing that makes it different from a failure: it is still `queued`, so
|
||||
nothing is running it, nothing holds a slot for it, and whatever waits
|
||||
on it goes on waiting. Compared as text, which is safe because
|
||||
`run_after` is written in the same UTC ISO form as every other stamp
|
||||
here — see `table.py`.
|
||||
"""
|
||||
return self.state == "queued" and self.run_after > datetime.now(UTC).isoformat()
|
||||
|
||||
@property
|
||||
def exhausted(self) -> bool:
|
||||
"""Failed on the last attempt it was allowed, rather than the only one.
|
||||
|
||||
Worth telling apart, because the two want opposite things done about
|
||||
them. A job that failed with attempts still to come is not this — it is
|
||||
back in the queue and `waiting`. A job that failed at `max_attempts=1`
|
||||
is not this either: nobody ever said it should be retried, so nothing
|
||||
was used up. This is the third case, the one where the policy ran its
|
||||
course and the work still did not land, and it is the only one where
|
||||
trying again is a decision a person has to make.
|
||||
"""
|
||||
return self.state == "failed" and self.attempt >= self.max_attempts > 1
|
||||
|
||||
|
||||
class Run(Model):
|
||||
"""A cascade or a task: an identity, and whether it was killed.
|
||||
|
||||
No state and no progress: the jobs carrying this id are the record of what
|
||||
happened, and a second copy of it here could only go out of step. What it
|
||||
holds is what cannot be worked out from them — that this id is a cascade
|
||||
rather than a task, what it was called, when it began, and the one fact its
|
||||
jobs cannot tell you, which is that somebody stopped it.
|
||||
"""
|
||||
|
||||
id: int
|
||||
kind: str
|
||||
name: str = ""
|
||||
#: What to call it on screen. A task is not a job, so this is the only
|
||||
#: place it can be written down.
|
||||
label: str = ""
|
||||
created_at: str
|
||||
#: When it was killed, or "" for a run nobody stopped. A cascade exists
|
||||
#: nowhere but this row, so this is where its Kill has to live.
|
||||
cancelled_at: str = ""
|
||||
|
||||
|
||||
class Schedule(Model):
|
||||
"""A standing instruction: enqueue this kind, with this payload, this often.
|
||||
|
||||
A row rather than a timer, for the reason every other waiting thing here is
|
||||
a row: nothing is alive between runs, so a process restarted at 05:59 still
|
||||
fires the 06:00 one, and a second worker on the same file reads the same
|
||||
schedule rather than keeping a second copy of it.
|
||||
|
||||
`name` is unique and is the handle declared defaults are matched on — see
|
||||
`pipeline/schedule.py`. Everything else about a row is the user's to edit,
|
||||
the cadence and whether it is on at all included, and redeclaring a default
|
||||
never overwrites those.
|
||||
|
||||
`next_run` and `last_run` are UTC ISO stamps, like every other time in this
|
||||
table, and are compared as text — so a cadence worded in a local timezone
|
||||
still sorts correctly against `now` and reads the same from any process.
|
||||
Empty means never, which for `next_run` also means not due, and is why it
|
||||
is `''` rather than NULL.
|
||||
"""
|
||||
|
||||
id: int
|
||||
name: str
|
||||
kind: str
|
||||
#: What to call it on screen, from `declare(label=...)`. Empty means the
|
||||
#: job kind's own label, which is what the queued job then carries.
|
||||
label: str = ""
|
||||
payload: JsonDict = {}
|
||||
cadence: str
|
||||
enabled: bool = True
|
||||
next_run: str = ""
|
||||
last_run: str = ""
|
||||
#: The job the last firing queued, and so the answer to "is one still
|
||||
#: going?" — which is what stops a slow job from piling up behind itself.
|
||||
last_job_id: int = 0
|
||||
#: How that job ended, or `queued`/`running` while it has not. Joined on in
|
||||
#: `Jobs.schedules` rather than fetched per row: it is one column on a query
|
||||
#: that is already running, and the time is no use on screen without it.
|
||||
last_state: str = ""
|
||||
created_at: str
|
||||
|
||||
|
||||
class TaskProgress(Model):
|
||||
"""One task of a live cascade, as a tally of the jobs carrying its id.
|
||||
|
||||
Counts and not the jobs themselves. A task is a body fetch shared over
|
||||
twenty-three jobs and there is nothing to tell them apart — the ones still
|
||||
going are in `JobsSnapshot.running` with their own `done`/`total`, and the
|
||||
ones that landed are a number, because a landed job is a full box and full
|
||||
boxes are all alike.
|
||||
|
||||
Derived on read, never stored, for the reason `runs` holds no state: the
|
||||
job rows are the record of what happened and a second copy of the tally
|
||||
could only go out of step with them.
|
||||
"""
|
||||
|
||||
#: The `runs` row this task is, and the id `/api/tasks/{id}/cancel` takes.
|
||||
id: int
|
||||
#: What the task is — `body`, `tag`. Its own name, minted with the `runs`
|
||||
#: row before any of its jobs existed, and how a cascade's list of tasks is
|
||||
#: matched against it.
|
||||
name: str = ""
|
||||
#: How to say that on screen: "Fetching what the feeds left out". The
|
||||
#: task's own, off the `runs` row, because a task is not a job and has no
|
||||
#: handler to carry one.
|
||||
label: str = ""
|
||||
done: int = 0
|
||||
failed: int = 0
|
||||
cancelled: int = 0
|
||||
running: int = 0
|
||||
queued: int = 0
|
||||
|
||||
|
||||
class CascadeProgress(Model):
|
||||
"""A cascade that is still going, and how far each of its tasks has got.
|
||||
|
||||
Only the tasks that exist. A cascade splits at every boundary — how many
|
||||
jobs summarising becomes is a query that is only meaningful once the
|
||||
bodies have landed — so the ones ahead have no jobs, no ids and nothing to
|
||||
count. Which of them are coming is the app's own business, in its cascade
|
||||
module and served beside this; here there is nothing to say about them.
|
||||
"""
|
||||
|
||||
#: The `runs` row, and the id `/api/cascades/{id}/cancel` takes.
|
||||
id: int
|
||||
#: Which cascade it is a run of — `pull`. For a job or a task queued on
|
||||
#: its own, which is a cascade of one, that name and a word saying it
|
||||
#: stands alone: `pull on its own`. The two are different runs and the
|
||||
#: name is where they are told apart, so it is also what the strip shows.
|
||||
name: str = ""
|
||||
#: In the order they began, which for a cascade is the order they run.
|
||||
tasks: list[TaskProgress] = []
|
||||
|
||||
|
||||
class JobsSnapshot(Model):
|
||||
"""What the queue looks like right now — the whole WebSocket payload.
|
||||
|
||||
Sent on connect and after every state change, rather than diffed. The
|
||||
whole thing is a few hundred bytes and one shape that is always complete
|
||||
is worth more here than the bytes a patch protocol would save.
|
||||
"""
|
||||
|
||||
#: Everything under way. A list because the queue has several workers, so
|
||||
#: "the running job" stopped being a single thing.
|
||||
running: list[Job] = []
|
||||
queued: list[Job] = []
|
||||
recent: list[Job] = []
|
||||
#: The cascades any of that belongs to, tallied by task. Here rather than
|
||||
#: behind a URL of its own because a tally of job states changes exactly
|
||||
#: when this message does: asked for separately it would arrive a beat
|
||||
#: later and draw a bar that disagreed with the list above it. And it
|
||||
#: cannot be counted from that list — `recent` is the last thirty jobs, so
|
||||
#: by the time tagging starts the body batches that landed have dropped
|
||||
#: off the end and a bar drawn from it would count backwards.
|
||||
cascades: list[CascadeProgress] = []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- notices
|
||||
|
||||
#: How loud a `Notice` is, and the four colours a reader renders them in.
|
||||
#: `error` is red — "you have to do something about this". `warning` is amber,
|
||||
#: the same amber a work panel uses for a row that wants you — "this went
|
||||
#: wrong and carried on". `info` is grey, worth saying once. `good` is green
|
||||
#: and is the odd one out: nothing in `notices.py` produces one, because that
|
||||
#: module is about failure — it exists for a UI to confirm something worked
|
||||
#: without inventing a second mechanism to say so.
|
||||
NoticeLevel = Literal["info", "good", "warning", "error"]
|
||||
|
||||
#: Whether a failure is the world's or ours. See `Notice.kind`.
|
||||
NoticeKind = Literal["expected", "bug"]
|
||||
|
||||
|
||||
class Notice(Model):
|
||||
"""Something went wrong that you, not the log, need to know about *now*.
|
||||
|
||||
Broadcast to whatever browsers are open, and written to the `notices`
|
||||
table so that one opened afterwards can read it — see `NoticeEntry`, and
|
||||
`pipeline/notices.py` for why it is kept after all.
|
||||
|
||||
So the division of labour is:
|
||||
|
||||
this "the tagger stopped, your claude login expired" — on
|
||||
screen, the moment it happens, with enough to tell you
|
||||
whether to shrug, fix something, or go and read the log.
|
||||
the table the same sentence, findable once it has faded, with how
|
||||
often it has been said. Cleared by you, or by a fortnight
|
||||
of not being said again.
|
||||
the log everything, with the traceback, still there tomorrow.
|
||||
|
||||
Which means a notice does not have to be complete. It has to be *enough to
|
||||
orient you*, and `where` points at the rest.
|
||||
"""
|
||||
|
||||
#: Stable dotted identifier — `agent.auth`, `source.failed`. What the UI
|
||||
#: dedupes a burst on, and what a test asserts against, so the prose in
|
||||
#: `text` can be reworded without breaking either.
|
||||
code: str
|
||||
level: NoticeLevel = "warning"
|
||||
#: The distinction worth drawing when something breaks: is this the world
|
||||
#: misbehaving in a way it is entitled to, or is this ours?
|
||||
#:
|
||||
#: expected an expired login, a feed returning 403, a rate limit. Not a
|
||||
#: defect. Either you fix it or you wait, and either way there
|
||||
#: is nothing to debug.
|
||||
#: bug an exception we did not anticipate. Something to go and read
|
||||
#: the log about, and probably to fix in the code.
|
||||
#:
|
||||
#: Getting this wrong in either direction is expensive — hunting a bug that
|
||||
#: was a dead feed, or shrugging at a real one — so it is a field rather
|
||||
#: than something to infer from the wording.
|
||||
kind: NoticeKind = "expected"
|
||||
text: str
|
||||
#: What to do about it, when there is something: `claude auth login`. Empty
|
||||
#: when there isn't, which is worth saying honestly rather than padding.
|
||||
fix: str = ""
|
||||
#: The raw error. Shown folded — it is the bit you would paste into a bug
|
||||
#: report, and the bit that tells you which log line to look for.
|
||||
detail: str = ""
|
||||
#: Where to look for the whole story. Always the log; this names the part.
|
||||
where: str = ""
|
||||
#: What this is about within its code — a source name, a job kind. Lets the
|
||||
#: UI collapse a burst of the same failure into one line with a count.
|
||||
key: str = ""
|
||||
|
||||
|
||||
class NoticeEntry(Notice):
|
||||
"""A notice as the log holds it: one row per `(code, key)`.
|
||||
|
||||
What `Jobs.record_notice` hands back and what `/api/notices` lists. The
|
||||
same failure said again is this row with a higher `count` and a later
|
||||
`last_at`, never a second row — which is also how a client knows whether
|
||||
to interrupt anyone: `count == 1` is news, anything above it is a repeat
|
||||
of something already said and still sitting in the log.
|
||||
"""
|
||||
|
||||
id: int
|
||||
first_at: str
|
||||
last_at: str
|
||||
count: int = 1
|
||||
|
||||
|
||||
__all__ = [
|
||||
"TERMINAL",
|
||||
"CascadeProgress",
|
||||
"Job",
|
||||
"JobState",
|
||||
"JobsSnapshot",
|
||||
"JsonDict",
|
||||
"JsonIds",
|
||||
"Model",
|
||||
"Notice",
|
||||
"NoticeEntry",
|
||||
"NoticeKind",
|
||||
"NoticeLevel",
|
||||
"Run",
|
||||
"Schedule",
|
||||
"TaskProgress",
|
||||
]
|
||||
@@ -0,0 +1,136 @@
|
||||
"""Turning a failure into one line worth putting on screen.
|
||||
|
||||
The whole of this module's job is a handoff. Something broke; the log has the
|
||||
traceback and always will; what the log cannot do is *tell you it happened*
|
||||
while you are looking at the app wondering why nothing appeared. So a notice
|
||||
is pushed to whoever is watching — and, since nobody usually is, written down.
|
||||
|
||||
**It is stored, and it was not always.** The first design kept nothing: a
|
||||
notices table would be a second record of what the log already records, with
|
||||
rows that go stale and a dismiss button to maintain. That argument assumed
|
||||
somebody was watching when the notice was raised, and somebody who could read
|
||||
a server's log when they were not. Neither holds for a pull that runs on a
|
||||
schedule and a reader holding a phone. What it cost was three things, each
|
||||
reported as a defect of its own:
|
||||
|
||||
a line that faded was gone twelve seconds, then nowhere to find it
|
||||
nothing open, nothing heard the usual case, and the notice was lost
|
||||
every run said it again "already said" was known only to a tab
|
||||
|
||||
So `Jobs.record_notice` keeps one row per `(code, key)`, and the three are one
|
||||
fix. The same failure said again is that row with a higher count, which is how
|
||||
a client tells news from a repeat: it interrupts for the first and only counts
|
||||
the rest. A line somebody dismissed is still in the log, so dismissing means
|
||||
later rather than never. A row goes when somebody clears it, or a fortnight
|
||||
after it was last said — by which time it has stopped being true, which is the
|
||||
only staleness the first design was right to worry about.
|
||||
|
||||
The table is not the log and does not try to be. No traceback, no history of
|
||||
each occurrence: a sentence, what to do, the raw error, how often, and when.
|
||||
|
||||
This module is only the part the queue itself needs: the notice every failed
|
||||
job gets, the check that stops it being said twice, and the rendering a
|
||||
terminal wants. **Classifying a failure is the app's job**, because a useful
|
||||
sentence names things only the app knows — its model backend, its feeds, its
|
||||
config keys. An app writes its own vocabulary next to its handlers and passes
|
||||
notices in through `RunContext.notice`; newsdesk's is `desk/notices.py`.
|
||||
|
||||
What a notice has to carry is not the full story but enough to orient you:
|
||||
|
||||
text what happened, in a sentence, in terms of what you lost by it.
|
||||
kind whether this is the world misbehaving — an expired login, a feed
|
||||
gone 403 — or a bug of ours. That is the fork in what you do next,
|
||||
and guessing it from an error string is exactly the work you should
|
||||
not have to do at a glance.
|
||||
fix the command, when there is one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterable
|
||||
|
||||
from pipeline.models import Notice, NoticeKind
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: Words that mean the failure happened at the other end of a wire. An app's
|
||||
#: own classifier will want more than this; `whose_fault` is the cheap default
|
||||
#: for the failures the queue meets on its own.
|
||||
THEIR_FAULT = (
|
||||
"http", "url", "timeout", "connection", "ssl", "certificate", "socket",
|
||||
"dns", "resolve", "403", "404", "429", "500", "502", "503", "refused",
|
||||
"unreachable", "forbidden", "not found", "unavailable", "moved",
|
||||
)
|
||||
|
||||
|
||||
def job_failed(kind: str, label: str, error: BaseException | str) -> Notice:
|
||||
"""A job that ended in `failed`.
|
||||
|
||||
A handler raising something it did not expect is the definition of ours to
|
||||
fix, so this is the one place that says `bug` outright rather than working
|
||||
it out.
|
||||
"""
|
||||
return Notice(
|
||||
code="job.failed", level="error", kind="bug", key=kind,
|
||||
text=f"{label or kind} failed.",
|
||||
detail=str(error).strip(),
|
||||
where=f"the `job … ({kind}) failed` traceback in the log",
|
||||
)
|
||||
|
||||
|
||||
def already_said(said: Iterable[Notice], error: str) -> bool:
|
||||
"""Whether one of these notices is about this failure.
|
||||
|
||||
The caller is the worker, which announces `job_failed` for every job that
|
||||
ends in `failed` — and must not, when the handler already announced a
|
||||
classified notice about the very exception that ended it. "agent.backend
|
||||
is `off`, so nothing that needs a model runs" followed by "Tagging a batch
|
||||
failed — looks like a bug" is two contradictory lines about one failure,
|
||||
and the wrong one is the louder: it sends you to a traceback about a
|
||||
machine that is doing exactly what it was configured to do.
|
||||
|
||||
Matched on `detail` rather than on identity, because what the handler
|
||||
classified is an exception and what reaches the worker is that same
|
||||
exception rendered `Type: message` — the same string with a prefix. Which
|
||||
is also the line in the right place: a notice about something else the job
|
||||
met on the way, a source that 403'd during a pull that then crashed over
|
||||
something unrelated, does not match, and that crash still gets its own
|
||||
line as the unexplained failure it is.
|
||||
"""
|
||||
return any(notice.detail.strip() and notice.detail.strip() in error
|
||||
for notice in said)
|
||||
|
||||
|
||||
def render(notice: Notice) -> str:
|
||||
"""The same notice, for a terminal.
|
||||
|
||||
The browser is the first-class destination, but it is not the only one: a
|
||||
handler run from a terminal hits every one of these failures and deserves
|
||||
the same sentence, the same command to run, and the same "is this mine to
|
||||
fix". Without this the CLI got the code and the text and silently lost the
|
||||
two fields that make a notice worth having.
|
||||
|
||||
Rendered rather than logged in pieces so that the wording lives in one
|
||||
place. An app changes a rule and both destinations change together.
|
||||
"""
|
||||
lines = [notice.text]
|
||||
if notice.fix:
|
||||
lines.append(f" -> {notice.fix}")
|
||||
if notice.kind == "bug":
|
||||
lines.append(f" this looks like a bug rather than a bad day; see {notice.where}"
|
||||
if notice.where else " this looks like a bug rather than a bad day")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def whose_fault(error: BaseException | str) -> NoticeKind:
|
||||
"""Whether a failure is the other end's or ours.
|
||||
|
||||
A 403 is the outlet's business and there is nothing here to fix. A
|
||||
`KeyError: 'entries'` is us assuming a shape the other end didn't have, and
|
||||
it will keep happening until someone changes the code — so it belongs in
|
||||
front of you as a defect, not as weather.
|
||||
"""
|
||||
text = f"{type(error).__name__} {error}".lower() if isinstance(
|
||||
error, BaseException) else str(error).lower()
|
||||
return "expected" if any(hint in text for hint in THEIR_FAULT) else "bug"
|
||||
@@ -0,0 +1,395 @@
|
||||
# `pipeline` — the job queue
|
||||
|
||||
## What it is
|
||||
|
||||
A jobs **table in SQLite** with an async worker draining it. Handlers are plain synchronous functions run in threads.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Packages | stdlib here — `sqlite3`, `asyncio`, `dataclasses`, `inspect` — plus pydantic, which `pipeline/models.py` wants |
|
||||
| Not used | no Celery / RQ / Redis / Huey, no `multiprocessing`, no in-memory queue |
|
||||
| Workers | threads, several at once, one per running job, from a pool of the queue's own |
|
||||
| Storage | one SQLite file (WAL). Point it at the app's DB or one of its own |
|
||||
| Concurrency | threads within a worker; any number of workers on one file — `claim` is one atomic `UPDATE`, and a claimed job is held on a lease its worker renews. `python -m pipeline.worker` starts one with no web app; `pipeline/tests/test_housekeeping.py` races two |
|
||||
| Web | `pipeline.web.connect(app, jobs, config, store)` — routes, socket, the `<desk-jobs>` and `<desk-schedules>` elements, and the worker in the app's lifespan |
|
||||
| States | `queued → running → done \| failed \| cancelled`, and `running → queued` again for a job with an attempt left |
|
||||
| Scheduling | five cadences on a `schedules` row in the same file — `every 15m`, `daily at 06:00`, `weekly mon 07:00` — fired from the drain loop, defaults declared next to `@job` |
|
||||
| Door | `from pipeline import DAYS, FINAL, FORMS, JOBS, MAX_RESULT, NULL, SCHEDULES, TASKS, TERMINAL, BadCadence, Batch, Bound, Cadence, Cancelled, CascadeProgress, ChildFailed, Declared, Handler, Job, JobQueue, JobState, Jobs, JobsSnapshot, Listener, Notice, NoticeEntry, NoticeKind, NoticeLevel, NoticeListener, ResultTooLarge, Retry, Run, RunCancelled, RunContext, Schedule, ScheduleListener, Spec, Tally, Task, TaskProgress, declare, fan_out, job, local_zone, notices, spawn` |
|
||||
| Copyable | yes, and that is the point — `cp -r pipeline yourproject/`. Nothing here imports `desk` or `newsdesk`, and `tests/test_architecture.py` fails the build if it ever does. See `docs/pipeline-package.md` |
|
||||
|
||||
The table *is* the queue, which is why a job survives a restart, why the UI's job list is a `SELECT`, and why a worker can live in another process.
|
||||
|
||||
## The Entities
|
||||
|
||||
**Job** — one physical unit of work, admitted to the queue and run on one worker. It has a resource class, so the queue knows which worker to run it on. A job is the only thing the queue runs, and the only thing that has a row in `jobs`.
|
||||
|
||||
**Handler** — the function that defines the job, aka the code. `@job` registers the handler in the queue. The handler is the piece of code which is called when the job is about to run.
|
||||
|
||||
**Task** — **plain code that creates jobs, and is never a job.** A business step comes in — fetch every feed, give every article a body — and is translated into jobs the queue can run, typically by cutting the work into batch payloads. A task mints a run id for itself, queues one job per payload tagged with that id, and queues one **final job** behind them with `after="done"` on all of them, which is how it comes to be reported on. Then it is over: nothing of the task is alive afterwards, only its jobs and the callback the final job carries.
|
||||
|
||||
Everything a task queues, it queues in one go. That is what makes it plain synchronous code and not a job: finding the work is a query, and a query is cheap enough to run inside the request that pressed the button, or in the worker thread that has just finished the job before. If the planning itself needs real work done first, that work is a job, and what follows it is the next task of a cascade.
|
||||
|
||||
**Cascade** — a series of tasks, for when each can only be planned once the one before it is over, or where there is a natural breaking point between them. Structurally it is the same shape as a task, and the pipeline does not distinguish the two. Both are rows in `runs`, both group jobs by an id, both are killed by the same call, and neither has an object of its own. The difference is a **convention**, and it is worth stating because nothing enforces it:
|
||||
|
||||
> A **task** queues everything it wants at once.
|
||||
> A **cascade** is where the breakpoints are: it queues the next task only once the last one is over.
|
||||
|
||||
The final job of a task is what calls back to the cascade to start the next task.
|
||||
|
||||
Jobs are tied to a task and a cascade by `task_id` and `cascade_id`.
|
||||
|
||||
**Schedule** — **a row that queues a job, and is never a job itself.** "Enqueue this kind with this payload on this cadence": a name, a kind, a payload, a cadence, whether it is on, when it is next due and what it queued last time. It is a row rather than a timer for the same reason a queued job is: nothing is alive between runs, so a process restarted at 05:59 still fires the 06:00 one, and two workers on the file read one schedule instead of each holding a copy of it in memory. It queues *jobs*, not tasks — a schedule that wants a whole cascade queues the job that starts one.
|
||||
|
||||
**Cadence** — one of five wordings, and the only five: `every 15m`, `every 6h`, `every 2d`, `daily at 06:00`, `weekly mon 07:00`. Not cron, which is five fields of positional punctuation that nobody reads back correctly. The interval forms are elapsed time and ignore the timezone; the clock forms are wall-clock in `JobQueue(tz=...)` and are 06:00 on both sides of a daylight-saving change. The two mean different things and a person picking one has already chosen which.
|
||||
|
||||
**Declaration** — a default schedule an app ships, written next to `@job` with `declare(name, kind, cadence, payload, label)`. `SCHEDULES` is to it what `JOBS` is to `@job`. On start, a declared name with no row gets one and a name that has one is not touched, so the declarations are where a schedule *comes from* and the table is what it *is*.
|
||||
|
||||
## How a job has to look
|
||||
|
||||
```python
|
||||
def handler(config, store, *, context=NULL, limit=None) -> dict: ...
|
||||
```
|
||||
|
||||
`config` and `store` are the app's, passed through untouched. Payload keys arrive as keyword arguments — **declare the ones you take**; an undeclared key is a `TypeError` recorded against the job, and `**kwargs` would swallow a typo. `context` is optional: a handler that takes neither progress nor cancellation says so by not declaring the parameter, and `registry.py` works that out once at registration rather than inspecting a signature per run. One payload key is reserved — a job whose payload names `context` fails, because the worker's own is what goes there.
|
||||
|
||||
`limit` is a **convention, not a contract**. Nothing in the pipeline reads it. It is an ordinary payload key that most of newsdesk's sweeps happen to accept, so that a person can ask for a short run of one, and a handler is free not to have it.
|
||||
|
||||
The return value should be a JSON-serialisable dict, and is stored as the job's result. A handler that returns something else — `None`, a list, a count — is not an error: the worker keeps the ending and stores an empty result, so the job is `done` with `{}`. Which means a result nobody can read is silent, and worth knowing when a handler's tally goes missing rather than its job failing.
|
||||
|
||||
**An exception ends the job in `failed`.** The traceback goes to the log, the exception is stored on the row as `Type: message`, and a `job_failed` notice goes to the screen saying it looks like a bug — *unless* the handler already announced a notice about that same failure, in which case the handler's sentence stands alone rather than being contradicted underneath. The `except` is deliberately broad: a handler is a source plugin, a model call, a subprocess, and anything it raises is that one job's failure and nothing else's. Letting it escape would take the worker down and strand every job behind it.
|
||||
|
||||
## Minimum example
|
||||
|
||||
```python
|
||||
from pipeline import Jobs, JobQueue, job, RunContext, NULL
|
||||
|
||||
@job("greet", "Saying hello", solo=True, resource="cpu")
|
||||
def run_greet(config, store, *, context: RunContext = NULL, name="world") -> dict:
|
||||
context.step(0, 1, f"greeting {name}") # progress + cancellation check
|
||||
return {"greeting": f"hello {name}"}
|
||||
|
||||
run_greet(cfg, store, name="wolf") # 1. straight call — no queue at all
|
||||
|
||||
queue = JobQueue(Jobs(cfg.database), cfg, store) # 2. queued
|
||||
await queue.start()
|
||||
queue.enqueue("greet", {"name": "wolf"})
|
||||
```
|
||||
|
||||
`@job` returns the function unchanged, so every job stays importable and callable on its own. Registration is by import — importing `newsdesk.jobs` fills `JOBS`.
|
||||
|
||||
## Workers
|
||||
|
||||
`JobQueue.WORKERS = {"llm": 4, "http": 8, "cpu": 1}` — how many workers of each kind there are, and so how many jobs of that resource run at once. An app passes its own; newsdesk has a `workers` section. A resource named in neither gets one. Counted from the table, not from this process, so a second worker on the same file shares the four model workers rather than helping itself to four more.
|
||||
|
||||
The count belongs to the worker, not the job kind: tagging and summarising both ask a model, so they share the same four.
|
||||
|
||||
Our queue is fanning out threads not processes. Therefore, a worker is a thread, not a program.
|
||||
|
||||
## `depends_on`
|
||||
|
||||
A job may name jobs it waits for, and say what it wants of them.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `after="ok"` | admit only if all of them succeeded; fail with them otherwise. The default, and what one job needing another's output means. |
|
||||
| `after="done"` | admit once all of them are over, however they ended. |
|
||||
|
||||
Waiting costs nothing: `claim` passes the job over and it stays queued. No timer, no listener, nothing holding a place, nothing lost to a restart.
|
||||
|
||||
`done` exists for the job that reports on the others — success-only semantics would mean the report never exists when there is something to report. An id naming nothing is satisfied.
|
||||
|
||||
A **cancel** and a **failure** are the same thing here: both travel to the `after="ok"` jobs waiting on them, and neither holds back an `after="done"` one. Killing a job is dropping that batch, not stopping the run it belongs to, so the job that reports still runs and counts it among the ones that did not land.
|
||||
|
||||
**What a waiting job is handed.** `context.waited_for` is a tuple of `pipeline.models.Job` — the whole row for each job this one named, oldest id first, and empty for a job that waited for nothing. Each carries its `state` and its `result`, which is everything a report needs. Read at claim time off the jobs themselves, never counted up as they landed, so there is no tally to keep in step and nothing to lose to a restart half way through. An id that named a job which no longer exists is simply absent from the tuple, so `len(waited_for)` is what was found, not what was asked for.
|
||||
|
||||
## `solo`
|
||||
|
||||
**One job of this kind queued or running at a time.** Two tabs pressing Fetch is one pull, and the second caller is handed the job already going rather than an error, because what they asked for is that the work be under way and it is. Enforced by a partial unique index over the unfinished states, so it holds across every tab, every client and every process at once — not by a flag in whichever browser happened to ask.
|
||||
|
||||
**Off unless a job says otherwise.** A sweep that forgets to say `solo=True` runs twice and wastes a minute. A batch kind that forgets to say `solo=False` has seven of its eight batches quietly handed the first one, and the work simply never happens. The default belongs on the side that fails loudly.
|
||||
|
||||
A job that names `depends_on` never dedups, whatever it declared, since the whole point of a batch is that there are several.
|
||||
|
||||
**The job you are handed back may not be yours.** Dedup matches on the kind alone, so a duplicate is answered with whatever live job holds that key — which can belong to another cascade entirely. Press Fetch while an overnight pull is part way through its own `pull` job and you are handed *that* job: the right answer to "is a pull under way", and the wrong one to "is *my* pull under way". Nothing rewrites its `cascade_id`, so a caller that means to follow the run it started has to notice the ids it got back are not the ones it minted.
|
||||
|
||||
## Retrying
|
||||
|
||||
```python
|
||||
@job("extract", "Reading the page", resource="http",
|
||||
max_attempts=3, retry_delay=60, retry_backoff=2.0)
|
||||
def run_extract(config, store, *, context=NULL, url=""): ...
|
||||
```
|
||||
|
||||
**`max_attempts` counts the first attempt**, so `1` is the default and means what every job here meant before the column existed: run it once, and an exception is the end of it. An attempt that fails with attempts left goes back to `queued` with a `run_after` on it, `retry_delay × retry_backoff ⁿ` seconds hence.
|
||||
|
||||
**Waiting costs exactly what `depends_on` costs, which is nothing.** The wait is a column, not a sleeping worker: `claim` passes the row over until its time comes round, so no worker, no thread and no lease is spent on an hour's backoff, and the poll that exists to notice another process's enqueue is what notices the hour is up. Sleeping in a handler would hold a worker for the duration, and is the thing this exists instead of.
|
||||
|
||||
**And because it is a column it survives the process that wrote it.** A retry scheduled for 09:40 is taken at 09:40 by whichever worker is alive then, including one started at 09:38 in a different process. There is nothing to hand over, because nothing was alive to hand.
|
||||
|
||||
**Three endings, and `Jobs.retry` picks between them.** `cancelled`, if a Kill landed while the attempt ran — a retry that put the job straight back would be a button that stopped nothing and said it had. Back to `queued`, if there are attempts left. `failed`, if there are not, and `Job.exhausted` says so from then on: the policy ran its course and the work still did not land, which is the only one of the three where trying again is a person's decision. A job that failed at `max_attempts=1` is not exhausted — nothing was ever used up.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `job.waiting` | queued, with its next attempt still to come |
|
||||
| `job.exhausted` | failed on the last attempt it was allowed, rather than the only one |
|
||||
| `job.attempt` | how many times it has been *started*, stamped by `claim` in the statement that takes the row |
|
||||
| `jobs.waiting()` | every row in the first of those states, oldest first |
|
||||
| `jobs.blocked()` | everything `claim` would pass over — waiting on a time, or on another job |
|
||||
|
||||
**A handler can name the wait but not the bound.** `raise Retry("rate limited", after=120)` puts the other end's own answer in place of the backoff, which is worth having because the alternative is a guess: a minute is either a minute of doing nothing when the service was ready, or another refusal charged against the bound. It buys no extra attempt — a job queued with one attempt that raises `Retry` fails exactly as it would on any other exception. The bound is ours to declare and the wait is the world's to name.
|
||||
|
||||
**A retry says nothing to the screen.** A notice is for what a person has to act on, and while attempts remain there is nothing to act on: the queue is already doing what they would have asked for. The log has every attempt either way; only the one that runs out announces.
|
||||
|
||||
**Retrying is off by default**, for the reason `solo` is off by default. A fetch that should have been retried and was not costs one item until the next sweep finds it again. Work retried three times that never should have been is three times the writes, or three times the bill.
|
||||
|
||||
`queue.enqueue(kind, delay=600)` schedules one job for later with no retry involved at all. `queue.enqueue(kind, max_attempts=n, retry_delay=s, retry_backoff=b)` overrides what the kind declared for one piece of work, and `dedup_key=` overrides the kind's own key — which is how one *item's* acquisition is made unrepeatable while the kind stays freely repeatable, so two presses of a reader's retry button collide in the unique index rather than in one browser tab.
|
||||
|
||||
**A batch can carry its own bound.** A task's finder may return `Batch(payload, Bound(max_attempts=3, delay=300, backoff=2.0))` instead of a bare payload, and those numbers go onto that job's row. That is where a plugin's declared retry policy lands: newsdesk cuts its body batches per plugin profile precisely because the row holds one bound, so a batch mixing two outlets could only carry one of their declarations. A finder returning plain dicts — every task that existed before this — gets `Bound()`, which is one attempt and no wait. The direct path (`task.run`) ignores bounds entirely: a bound is a number the queue enforces by putting a row back with a time on it, and there is no row.
|
||||
|
||||
## Typical way of dealing with batches
|
||||
|
||||
A task queues its batches as independent jobs, and one **final job** behind them. It depends on all of them with `after="done"`, so it runs however they ended, and it is handed each one's state and result in `context.waited_for`. It is the only job of the task that calls back.
|
||||
|
||||
A task that finds nothing to do is the ordinary case, not the empty one: it queues no batches and its final job runs immediately, with an empty `waited_for`, and moves the cascade on. Which is why a switch turned off, a backlog already cleared and a stage this pull does not want are all the same thing here — a finder that returns no payloads — and none of them stops the chain.
|
||||
|
||||
**The callback is a name in the final job's payload, not a function.** Nothing in a table can hold a function, and a name is what makes the callback survive a restart and be callable from another process; the worker running the final job resolves `module:function` to imported code. `newsdesk.jobs.cascade:step` reads as documentation in the payload, and carries its own import, where a registry would have to be filled before the worker looked in it.
|
||||
|
||||
It is called with the task's tally and whatever the task fixed at the time it was made:
|
||||
|
||||
```python
|
||||
def step(config, store, tally: Tally, payload: dict, *, context: RunContext) -> dict | None: ...
|
||||
```
|
||||
|
||||
`payload` is `Task.carries`, stored in the final job's row, so it survives a restart and a worker in another process. Whatever the callback returns is merged into the final job's own result; `None` is fine. A task with no callback stops there, which is what being asked for on its own means.
|
||||
|
||||
**`settle` is a second name in the same payload, resolved the same way, and it belongs to the task rather than to what follows.** The final job runs it before the callback, over the same tally. It is for the decision a batch is in no position to make: a batch is one of forty and cannot see what the other thirty-nine are doing, so anything that must hold across the whole run — a cap on what the run may add, a merge of what several batches each asked for — is worked out once, here. Newsdesk's tagger is the one that has one: a batch proposes new tags and adds none, and the settle merges the proposals, takes at most `max_new_tags_per_run` of them and goes back to tag the items that asked. A task started on its own has no callback and still settles, because settling is its own work and not a message to anybody.
|
||||
|
||||
**`Tally`** is how the batches ended, read off their rows at the moment the final job runs:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `done` `failed` `cancelled` | how many batches ended each way, and `jobs` for the sum |
|
||||
| `results` | what each batch returned, in the order they were queued |
|
||||
| `killed` | whether the **task itself** was killed |
|
||||
|
||||
`killed` is the one that cannot be inferred from the counts, and is the reason it is there. A task killed while its batches ran leaves every one of them `cancelled`, and so does a person killing every batch by hand — identical numbers meaning opposite things: *skip this step*, or *drop this work*. Only the `runs` row knows, so the final job asks it. What to do about it is the cascade's decision and nothing here presumes it; newsdesk's pull skips the stage and carries on to the next.
|
||||
|
||||
## Passing data
|
||||
|
||||
Three ways to pass data. Can be mixed and merged.
|
||||
|
||||
1. **Store.** A job reads its input or persists its output directly into a storage.
|
||||
2. **Parent.** A job receives data as a parameter, put in its payload by whatever queued it — a batch's uids, a cascade's options. Fixed at queue time, and the only kind of data a job that runs first can hand to a job that runs later.
|
||||
3. **Cross job via `context.waited_for`.** The `after="done"` job gets the jobs it waited on: their `state` and their `result`. Read, never pushed — nothing is counted while the work runs, so there is no tally to lose.
|
||||
|
||||
The usual arrangement is the third feeding the callback: the final job reads how its batches went and hands the tally to the cascade, which turns it into the next `Task`.
|
||||
|
||||
`context.upstream()` is the third one merged into a dict, later parents winning, which is what a chain of one parent wants to read like. It is `{}` under `NULL`, which is what keeps a chained handler callable by hand:
|
||||
|
||||
```python
|
||||
@job("extract")
|
||||
def extract(config, store, *, context=NULL, html: str = ""):
|
||||
html = html or context.upstream().get("html", "")
|
||||
```
|
||||
|
||||
From the queue nobody passes `html`; from a terminal you pass it yourself. A handler reading `waited_for[0].result` directly would be the one kind in the pipeline you cannot simply call.
|
||||
|
||||
**The rule is that whoever has the data writes the row.** Never insert a row meaning an earlier job to fill its payload in later: a worker claims it the moment the parent is `done`, so it can be claimed between that write and its commit; `dedup_key` has to be known at queue time; and `max_attempts` re-runs it from its own row. Reading a parent has none of that, because the row was complete when written.
|
||||
|
||||
Bulk data belongs in the store. `Tally.results` is a query across the children's `result` TEXT columns and those columns never shrink — counts, ids and errors are free, a few hundred large dicts are not.
|
||||
|
||||
## `fan_out` — N jobs and the one that reports on them
|
||||
|
||||
```python
|
||||
@job("body")
|
||||
def body(config, store, *, context=NULL):
|
||||
context.fan_out("body_batch", [{"uids": c} for c in chunks],
|
||||
then="newsdesk.jobs.cascade:step")
|
||||
```
|
||||
|
||||
One call, one transaction. The reporting job is `FINAL`, the pipeline's own kind, and an app never writes one. The children go into a run of their own inside the caller's cascade, which is what lets the callback fan out again — step N queues step N+1, and that is the whole of what a cascade is.
|
||||
|
||||
Because the thing fanning out is itself a job it has a context, so this works from inside a handler as well as from a request. `Task` is the same three statements with a finder in front, which is why a task and a cascade are one shape and one primitive covers both.
|
||||
|
||||
`()` under `NULL`: a handler called directly is being run by somebody standing there, and the useful answer is to do its own work and report that nothing was queued behind it.
|
||||
|
||||
## Running it
|
||||
|
||||
```python
|
||||
queue = JobQueue(Jobs("app.db"), config, store) # then, in an app:
|
||||
connect(app, Jobs("app.db"), config, store) # pipeline.web: routes, socket, element, lifespan
|
||||
queue.drain(timeout=60) # in a script: run what is queued, stop
|
||||
```
|
||||
|
||||
```bash
|
||||
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
|
||||
```
|
||||
|
||||
The worker is the same `JobQueue` with no web app, on the same table, and may run beside the web app's own. `--until-idle` drains and exits. The bootstrap is the one `isolate=True` uses.
|
||||
|
||||
## Ending work that will not end itself
|
||||
|
||||
Cancellation is cooperative because the work is synchronous — but two kinds of work never look up, and for those there is a real kill.
|
||||
|
||||
**A command.** `context.spawn(argv, timeout=...)` runs it, polls for Kill and for the deadline, and kills the **process tree** on either — `taskkill /T` on Windows, `killpg` elsewhere, which is why the child is started detached. `proc.kill()` alone would leave the ffmpeg a shell started still holding the core. Blocking a worker thread on this costs nothing: the thread is only sleeping. Output goes to temporary files, not `PIPE`, because a child that fills the pipe buffer while the loop polls is a deadlock. A non-zero exit is handed back rather than raised.
|
||||
|
||||
**Python that cannot be interrupted.** `@job(..., isolate=True, timeout=600)` runs the job in a child process, which is the only thing that can be killed mid-`sklearn.fit`. Nothing is pickled: a job row is already a complete description of the work, so the child is handed `--db` and `--job` and reads the same row. It writes *progress* to that table — which is how its progress reaches the browser with nothing forwarding it, the snapshot being a `SELECT` — and hands its result back through a file. The worker still writes the ending, so there is exactly one writer of the terminal state, as before.
|
||||
|
||||
`isolate=True` needs `bootstrap="module:function"` on the queue, returning `(config, store)`: neither survives a process boundary, so both are rebuilt in the child, and the same import is what fills `JOBS` there. `bootstrap_arg` is one string passed to it — an app's config path, which the child cannot learn any other way.
|
||||
|
||||
**`timeout` without `isolate` raises at registration.** A deadline over a thread could only mark the row failed and walk away — the work would carry on holding the core, and two things would intend to write one row. The refusal names both ways out.
|
||||
|
||||
## `context.task` — a job asking for what comes next
|
||||
|
||||
Newsdesk's body task fetches each eligible article and passes its HTML to `desk.tools.article.extract_article`. Its storage writer saves structured content, format, provenance and a derived plain-text projection atomically; existing enrichment jobs continue to read the generated `body` view. Empty or late fetch results preserve previously acquired content. The body job still uses the existing wall detector and candidate rechecks, and does not yet execute plugin-declared acquisition *graphs* — one declared step, the one whose tool is `extract_article`, is what it runs. Its declared **retry policy** is executed: a failure is classified with `desk.tools.failures`, the plugin decides whether that category is worth another attempt, and the batch raises so the queue puts it back under the bound its row carries. Batches are therefore cut per plugin profile. A second attempt asks the table again and so re-fetches only what is still missing, and never overwrites text that has since arrived. Structured bodies are retained without the former 60,000-character clipping; downstream model tasks remain responsible for their own input limits.
|
||||
|
||||
Hacker News discovery now runs as the `hacker_news` outlet through shared RSS fetching. Its item destination stays the publisher URL, so the subsequent body task extracts that article and records that URL as provenance. Its supplied discussion link is a separate typed input, and feed points/comment statistics stay in source evidence rather than the summary or preview. Its declared tagging policy runs: the destination article is what the tagger reads, and a link whose page would not extract is tagged from its title. Its summary policy stays off — the summariser selects only profiles that ask for an abstract, and nobody has decided that a front-page link earns one.
|
||||
|
||||
A handler that discovers work only by doing its own can name the task that should follow it:
|
||||
|
||||
```python
|
||||
context.task(cascade.task("body", cascade="pull"))
|
||||
```
|
||||
|
||||
**Recorded, not queued.** The worker starts it after this job lands in `done`, on the same thread that ran the handler, and hands it this job's `cascade_id` so its batches land in the run this one belongs to. Three consequences worth knowing: a task never outlives the failure of the job that described it, a handler still cannot reach the queue, and a job's row lands a moment *before* the jobs that follow from it exist.
|
||||
|
||||
It returns whether anything took it — `False` under `NULL`, where there is no queue. A callback invoked from a notebook or a Claude Code session gets that answer and has to do the work itself.
|
||||
|
||||
## `Cancelled` and `RunCancelled`
|
||||
|
||||
Two exceptions with similar names, and only one of them is a handler's business.
|
||||
|
||||
**`Cancelled`** is raised by `context.check()` inside a handler when this job has been asked to stop. The worker catches it and records `cancelled` rather than `failed`. A handler should let it travel; catching it is how you build the thing the Kill button cannot kill. A handler that never checks is not lost either — the worker asks once more when the handler returns, and a job asked to stop while it ran ends `cancelled`, keeping whatever it made and queueing nothing.
|
||||
|
||||
**`RunCancelled`** is raised by `Jobs.enqueue` for a job whose `cascade_id` or `task_id` names a killed run, except for an `after='done'` job under a killed *task* — the same carve-out `cancel_run` and `stop_task` make, so a kill landing while a task is still fanning out its batches cannot take the final job with it. A killed cascade refuses it too, which is what "nothing reports and nothing follows" means. It is not a handler's business and a handler will not see it: `JobQueue.enqueue` catches it, logs it, and returns **`None`** instead of a `Job`. So a caller that queues a job has to expect no job back — `Task.start` skips the id it did not get, and a `depends_on` list is that much shorter. Nothing is told that a run is over; a callback tries to queue the next task and the table says no, which is the same answer across processes, after a restart, and when the kill landed mid-run.
|
||||
|
||||
## Ids
|
||||
|
||||
Every job has an `id` of its own, and carries a `task_id` and a `cascade_id`. Both are grouping ids: nothing depends on a task, but everything belonging to one can be found by it and killed together. Two of them rather than one because they are two groupings and both have a button.
|
||||
|
||||
**Killing a run** is one operation for a task and a cascade alike — `cancel_run`. Three things: flag the row, end its queued jobs, ask its running ones. And a fourth that is not an operation at all: **`enqueue` refuses a job whose `cascade_id` or `task_id` names a killed run**, which is what stops a callback from reviving a killed task or moving a killed cascade on.
|
||||
|
||||
Three buttons, three sizes of stop: killing a **job** drops one batch, killing a **task** skips one step, killing a **cascade** ends the run.
|
||||
|
||||
**Stopping a run is not killing it.** `Jobs.stop_task` ends a task's queued batches and nothing else: the row is not flagged, so `Tally.killed` stays false and nobody is told a person pressed anything; running batches keep their requests and their own endings; the final job still reports. It is what a handler reaches through `RunContext.stop(why)`, for the failure that is the backend's answer to the whole run rather than to one batch — a spend limit, an expired login, no model on the machine. The reason is written to `error` on each cancelled row, so a batch that never ran says on screen why it didn't. `desk/modeltask.py` is the one caller, and `notices.stops_the_run` is what decides.
|
||||
|
||||
The one asymmetry between killing a `cascade` and killing a `task` is what the last two of those mean. Kill a **task** and its final job is spared: it still runs, and what it reports says the *task* was killed rather than only how many of its batches ended cancelled — see `Tally.killed`. Kill the **cascade** and the final job dies with it, so nothing reports and nothing follows — the one stop no cascade gets a say in, because there is nothing left of it to ask.
|
||||
|
||||
cascade_id = jobs.new_run("cascade", "pull") # before a single job exists
|
||||
jobs.of_cascade(n) · jobs.of_task(n) · jobs.run(n) · jobs.cancel_run(n)
|
||||
jobs.cascades()
|
||||
POST /api/cascades/{id}/cancel · /api/tasks/{id}/cancel · /api/jobs/{id}/cancel
|
||||
|
||||
A job queued with neither begins both: something asked for it on its own, so it is a cascade of one task of one job. A task started with no cascade does the same, for the same reason.
|
||||
|
||||
The cascade that comes of it says so in its name — `pull on its own` rather than `pull` — because a chain named `pull` is a different run, and `live_run("cascade", "pull")` is how a caller asks whether that chain is already going. Answering it with a lone job would be a Fetch button that starts nothing and greys itself over a run that is one job long. The task keeps its plain name (`pull`), since the task really is that task however it was started, and the strip draws its box from that.
|
||||
|
||||
## Schedules
|
||||
|
||||
The tick is in the drain loop, at the top of `_turn`, before the claim — not in a thread of its own and not on the idle path with the reclaim and the purge. Not a second thread, because then two things would race to enqueue one row. Not on the idle path, because reclaim and purge are repairs that can wait for a quiet moment and a schedule is *work*: on a queue that is never idle, 06:00 would never come round. The check itself is a clock comparison, so paying it every pass costs nothing. It also runs once in `start()`, after reconciliation, so a default declared today fires on the start that introduced it.
|
||||
|
||||
**Overdue fires once, not once per slot missed.** A firing sets `next_run` from *now*, never by stepping through the times it was down for. There is one `next_run` column and nowhere for a backlog to accumulate even in principle, so an app off for three days runs its hourly job once when it comes back.
|
||||
|
||||
**No pile-up.** Before firing, two questions: is the job from the last firing still queued or running (`last_job_id`), and is this kind `solo` with one already under way? Either way `next_run` advances and nothing is queued. A job slower than its own cadence must not grow a queue of itself, and a `solo` kind somebody has already pressed must not be recorded as this schedule's firing.
|
||||
|
||||
**A row the code no longer supports keeps its place.** An unregistered kind, or a cadence nothing can parse (only reachable by editing the file by hand), is logged once per process and left exactly as it is — `next_run` included — so registering the handler again runs it rather than starting another cadence. `<desk-schedules>` marks the row `unknown kind`; deleting it is deliberate.
|
||||
|
||||
**Reconciliation never overwrites and never deletes.** `Jobs.add_schedule(..., only_if_new=True)` is the whole of it: one statement, so two workers starting at once insert a declared default once between them. A declaration removed from the code leaves its row behind, because a deploy that silently forgot a schedule somebody relied on is worse than an orphan that says so.
|
||||
|
||||
**One scheduler per file.** `JobQueue(schedules=False)` and `pipeline.worker --no-schedules` are what every worker after the first passes. The workers themselves race correctly — that is `claim` — but the pile-up guard reads `last_job_id`, which neither of two simultaneous firings has written when the other looks.
|
||||
|
||||
declare(name, kind, cadence, payload=None, label="") # next to @job
|
||||
Cadence.parse(text) · str(cadence) · cadence.next_after(moment, zone)
|
||||
jobs.schedules() · jobs.schedule(id) · jobs.add_schedule(...)
|
||||
jobs.update_schedule(id, **fields) · jobs.delete_schedule(id)
|
||||
jobs.due_schedules(now) · jobs.mark_fired(id, job_id, next_run)
|
||||
queue.run_schedule(id) · queue.notify_schedules()
|
||||
GET/POST /api/schedules · PATCH/DELETE /api/schedules/{id}
|
||||
POST /api/schedules/{id}/run
|
||||
|
||||
## The three tables
|
||||
|
||||
**`jobs`** is the queue, and holds rather more than the states: the `payload` it was queued with and the `result` it ended with, both JSON; `done`, `total` and `message`, which are the progress bar and are written by `context.progress`; `error`, the exception string of a failure; `dedup_key`, `depends_on` and `after`; `resource`; `cascade_id` and `task_id`; the retry policy and where it has got to, as `attempt`, `max_attempts`, `retry_delay` and `retry_backoff`; and five times — `created_at`, `started_at`, `seen_at` (the lease), `run_after` (the next attempt) and `finished_at`. `cancel_requested` is a flag, because a kill is a request in the table rather than a message to whichever process happens to be running the job.
|
||||
|
||||
The policy is on the row rather than read off the job kind when a failure happens, for the same reason the payload is: the attempt that has to decide may be in another process, after a restart, running a version of the code where the kind's numbers are different or the kind is gone.
|
||||
|
||||
**`schedules`** is the standing instructions: `id`, `name` (unique, and the handle a declaration is matched on), `kind`, `label`, `payload`, `cadence`, `enabled`, `next_run`, `last_run`, `last_job_id`, `created_at`. `next_run` and `last_run` are UTC ISO stamps like every other time here, and `next_run` is compared as text in `due_schedules` for the reason `run_after` is in `claim`: `''` sorts before every stamp, so "never" and "not due" are one comparison with no coalesce in it. A cadence worded in a local timezone is turned into one of these before it is stored, so `pipeline/schedule.py` is the only file that reads a zone at all.
|
||||
|
||||
**`runs`** is where a cascade's id and a task's come from, and it is deliberately almost empty: `id`, `kind` (`cascade` or `task`), `name`, `label`, `created_at`, `cancelled_at`.
|
||||
|
||||
What it does **not** have is the point of it. No state, no progress, no counts, no `finished_at`: the jobs carrying the id are the record of what happened, and a second copy of that could only go out of step with them. `cascades()` tallies a run by reading its jobs. So the table exists for exactly two facts its jobs cannot supply — that a number was handed out *before* there was a job to take it from, and that somebody killed this run. `name` carries a third job: it is how `live_run` answers "is one already going", which is why a cascade minted only because a lone job needed an id is named apart from a chain somebody asked for (see `on_its_own`).
|
||||
|
||||
## `ModelTask` — next door, not here
|
||||
|
||||
`ModelTask` left this package: it imports `desk.config` and `desk.tools.agent`, so it was never the queue's. It is `desk/modeltask.py` now, and this section stays because it is still the thing most jobs in newsdesk are.
|
||||
|
||||
`desk/modeltask.py` is one layer above `desk/tools/agent.py`. `agent` answers *how do I reach a model*; this answers *what do I send it and what do I do with what comes back*, once, for every task that needs one. Tagging and summarising are the two, and what differs between them is all content: which rows are waiting, what goes in the batch, what the model is told, what shape it must answer in, what to do with the answer, what the run has to settle once every batch is in, and how a row that never gets one is given up on. That set of callables **is** a `ModelTask`; the module is everything that is not one.
|
||||
|
||||
`modeltask.of(TASK)` turns it into an ordinary `Task`, so it is queued, watched and killed like any other stage. `modeltask.one` is what a single batch job does: check for a kill, build the payload from the uids it was given, ask, apply. Two things about it are worth reading before changing:
|
||||
|
||||
- **The prompt is built when the batch runs, not when it was handed out.** Tagging's carries the vocabulary as it stands at that moment, so a tag somebody adds by hand mid-run reaches the batches still to come. It does not carry what a sibling batch proposed, because a batch proposes and adds nothing: the run's own additions are worked out once, in `settle`.
|
||||
- **`pending`, `waiting` and `build_batch` are given the config as well as the store.** Which rows are waiting is no longer a question a table can answer on its own: in newsdesk both backlogs are whatever the installed plugins declare about their own items, and the plugins are found through the config. Both tasks now use it; the argument is still there for one that does not.
|
||||
- **A kill during the call does not throw the answer away.** `check()` is called before the model is asked, so a Kill pressed while the job sat queued makes no paid call at all. Once the request is in the air the money is spent, so the answer is applied and the worker records the cancellation at return. The asymmetry is deliberate and looks like an oversight otherwise.
|
||||
|
||||
**A row is given up on eventually.** "The next run finds the work again" is unbounded, and for a row no model will ever answer usefully — a segment whose only honest abstract is one sentence, a uid dropped from every reply — unbounded means paying for the same call on every pull for as long as the row exists. So a row handed over and left unanswered is charged a turn through `ModelTask.missed`, and one charged three of them stops being offered until a person asks for it deliberately. A batch that *failed* charges nothing: it never reached a model, so it proved nothing about its rows. Every `ModelTask` must supply `missed`, and there is no default, because the default is the bug.
|
||||
|
||||
**A refusal is the one failure that is about a row.** The platform runs a classifier over the request and rejects the whole thing at once, naming nothing — and a news archive meets it regularly, because reporting on a worm or a supply-chain compromise is written in the vocabulary it watches for. `one` answers it by asking again in halves (`_narrow`) until the batch is the row, which is then charged and leaves the backlog after three. Halves rather than one at a time: two flagged rows in a batch is ordinary, since they arrive from the same day's news, and stepping through twenty singly would pay eighteen calls for what four find. The notice is `agent.refused` and is deliberately not `standing` — the account is fine and the next batch is about different rows.
|
||||
|
||||
**What was given up on is listed, and said once.** A refused row is charged with `why="refused"`, kept in `raw.why.<task>`, so the list can tell a refusal from a row the model left out. The strip interrupts only for given-up rows without `raw.seen.<task>`; its × or opening Jobs & history sets that mark on the server, so a reload does not say them again and a row given up on later still does. Try again there reopens a row with one attempt (`reopen_given_up(..., attempts=1)`) and queues its `_batch` job at once, so a row refused again is back on the list after one call rather than after three pulls.
|
||||
|
||||
## `notices`
|
||||
|
||||
**Two halves since the extraction.** `pipeline/notices.py` is what the queue itself needs — the notice a failed job gets, the check that stops it being said twice, the rendering a terminal wants. `desk/notices.py` is newsdesk's vocabulary: `Rule`, `RULES`, `classify`, `stops_the_run`, `source_failed`. An app writes its own, because a useful sentence names things only the app knows.
|
||||
|
||||
`pipeline/notices.py` turns a failure into one line worth putting on screen. A handler calls `context.notice(...)`, and the queue writes it to the `notices` table and pushes it to whoever is watching — the counterpart to `log.exception`, not a replacement for it: log the traceback for afterwards, raise a notice for now.
|
||||
|
||||
**A notice is stored, one row per `(code, key)`.** The first design stored nothing, on the argument that a table would duplicate the log. That assumed somebody was watching when the notice was raised; a scheduled pull and a reader on a phone mean nobody usually is. `Jobs.record_notice` upserts the row and returns it as a `NoticeEntry` with `count`, `first_at` and `last_at`, and that row is what goes over the socket. `count == 1` is news and anything above is a repeat, so a client interrupts for the first and only counts the rest — a refused article is announced on the pull that met it, not on the two retries after. Every notice still reaches `on_notice`; what interrupts a person is the interface's decision.
|
||||
|
||||
A row goes when it is cleared (`DELETE /api/notices/{id}`, or `/api/notices` for all), after which the same failure is news again, or when `JobQueue.NOTICE_QUIET` (14 days) has passed since it was *last* said. `GET /api/notices` lists the log, newest first. A write that fails is logged and the notice is pushed anyway: it never costs the job that raised it. The table holds no traceback and no per-occurrence history; those stay in the log.
|
||||
|
||||
A notice carries three things beyond its text: `kind`, which says whether this is the world misbehaving or a bug of ours — the fork in what you do next, and exactly the guess you should not have to make from an error string; `fix`, the command when there is one; and `key`, which with `code` names the row in the log — so a feed that has been failing all week is one line with a count rather than one per pull, and two refused articles are two. `classify` is the interface, and `RULES` is the catalogue it reads, which doubles as the list of what this will ever interrupt you about. Everything in the catalogue is marked `expected`; the fallback is marked `bug`, which is the whole point of enumerating them — a failure nobody wrote a rule for is one to go and read about.
|
||||
|
||||
For a job author, two rules. Use a notice for what a person has to act on, not for what the program can get on with. And classify your own failure if you can: the worker announces "looks like a bug" for every job that ends `failed`, and `already_said` is what stops it contradicting a handler that had already explained itself.
|
||||
|
||||
A rule also says whether the failure it names is `standing` — caused by the account or the machine rather than by this batch, so every sibling of the run would be told the same thing. `stops_the_run(notice)` is that answer, and a task for the model cancels what is still queued rather than paying for twenty-two more refusals. A limit, an expired login and an unreachable model are standing; a timeout, an oversized answer and a reply that was not JSON are not.
|
||||
|
||||
## Timing
|
||||
|
||||
Seven numbers, all on `JobQueue`, plus one on `Jobs`. None of them is tunable and none is meant to be; they are here so that "why did that take a minute" has an answer.
|
||||
|
||||
| | | |
|
||||
|---|---|---|
|
||||
| `IDLE_POLL` | 1 s | How long to sleep before looking again when the table was empty. Only reached when nothing woke us — an enqueue in this process is instant, so this is what notices one made by another, and what notices a scheduled attempt whose time has come. |
|
||||
| `PUSH_EVERY` | 0.25 s | Fastest a running job's progress may reach the browser. The write is not throttled; the push is. |
|
||||
| `CANCEL_CACHE` | 0.25 s | How long `check()` caches its answer. It reads the table so Kill works across processes, and a handler calling it in a loop would otherwise turn one button press into a query storm. |
|
||||
| `HEARTBEAT_EVERY` | 5 s | How often a worker restamps `seen_at` on the jobs it holds. |
|
||||
| `STALE_AFTER` | 30 s | How long a `running` row may go unsaid-about before another worker reads it as abandoned. |
|
||||
| `STOP_GRACE` | 5 s | How long a shutdown waits for handlers already in a thread. |
|
||||
| `SCHEDULE_TICK` | 30 s | How often the loop looks for a schedule whose time has come, and the resolution the cadences promise — which is why `pipeline/schedule.py` has no seconds unit. |
|
||||
| `Jobs.LINGER` | 15 s | How long a finished cascade stays in `cascades()`, so a run you watched is still on screen the moment after it lands. |
|
||||
|
||||
`HEARTBEAT_EVERY` and `STALE_AFTER` are far apart on purpose: the gap is how many beats may be missed to a stalled loop or a slow disk before live work is re-queued underneath a worker that is still there, which is the one mistake this must not make. And because reclamation runs at most once per `STALE_AFTER` on the idle path, a crashed worker's job comes back after up to about a minute rather than after thirty seconds.
|
||||
|
||||
## Limits
|
||||
|
||||
- **Cancellation in a thread is cooperative.** No `context.check()` in the loop, no Kill. A running handler in a thread is never interrupted, not even on `stop()`, because Python cannot kill a thread from outside. Work that might need killing runs in a process: `context.spawn` for a command, `isolate=True` for Python. See "Ending work that will not end itself".
|
||||
- **A shutdown waits `STOP_GRACE` for its handlers, then walks away.** A thread cannot be cut off, only waited for or abandoned, and both are wrong as an absolute: waiting is a reload held up by a model call, abandoning is a handler still writing to a file the process considers finished with. Most handlers use the moment and end properly; the rest are left running, and land their own ending in the table whenever they reach it.
|
||||
- **A shutdown hands its leases back, a crash cannot.** `stop()` expires the lease on whatever was still going when the grace ran out, because a process on its way out knows outright what silence can only imply. So a restart re-queues that work at once instead of a window later, and only a real crash pays the wait. Expired rather than re-queued, so a handler thread outliving the shutdown still writes its own ending rather than being overwritten by a second run of itself.
|
||||
- **A job's ending is written by the thread that ran it.** The handler call and the terminal `done` / `failed` / `cancelled` are one function on one thread; the loop's part is claiming the job and pushing the new state out afterwards. Written back on the loop, the ending would be the first thing a `stop()` costs: cancelling an abandoned job's coroutine cancels the write, while the handler itself carries on to the end regardless — so a reload landing mid-run would leave finished work looking unstarted, and the next start would run it a second time. For a batch that asks a model that is a second bill for something already paid for. The task a job named with `context.task` is queued by that same thread, for the same reason and still only once the job has landed `done`. Which is the one thing to know when reading the table from outside: a job's row lands a moment before the jobs that follow from it exist, so an empty `running` and `queued` is not by itself the end of a run.
|
||||
- **A kill outlives the crash that interrupted it.** A job asked to stop, whose worker then died, is reclaimed to `cancelled` rather than back to `queued`. The request is in the table exactly so that it survives the process that was meant to act on it.
|
||||
- **A dead worker is noticed late, on purpose.** A `running` row carries `seen_at`, stamped by `claim` and renewed every `HEARTBEAT_EVERY`. SQLite reads and writes run off the event loop so they do not delay its heartbeat. A lease expires after `STALE_AFTER`; reclamation runs at startup and at most once per `STALE_AFTER` on the idle path, so recovery can take about a minute. A worker excludes jobs it still holds from reclamation, renews their expired leases and warns that its loop stalled. Another process can still reclaim an expired lease.
|
||||
- **Every handler must be safe to run twice.** Not advice — the rule the rest of this depends on. A reclaimed lease, a shutdown that abandoned a thread, a second process on the same file, an attempt that failed half way through: each of them can put a handler on the same rows again, and the queue does not promise otherwise. So write for it. Read what is still waiting rather than trusting the uids you were handed, upsert rather than insert, and write each unit as it lands so a re-run has less to redo. `max_attempts` bounds how often that happens on a failure; it is not something to lean on, because the paths that re-run a handler without asking it are the ones that were always there.
|
||||
- **Handlers must be sync.** An async handler is not awaited.
|
||||
- **A crashed worker holds its slots until it is reclaimed.** `claim` counts `running` rows, so the survivor cannot use a crashed worker's slots until the lease expires and its next reclamation pass runs, about a minute in the worst case with a responsive loop and database. A stalled worker never re-queues its own held jobs. Socket sends run concurrently with a one-second timeout per subscriber, so an unresponsive reader cannot delay drain wake-ups indefinitely.
|
||||
- **Handlers run in a pool of the queue's own**, sized to the worker counts, because the interpreter's shared pool is sized from the cpu count instead. On a machine smaller than `workers` the shared one would run fewer jobs than the table admits and, worse, queue the loop's own claim and heartbeat behind the handlers, which is a worker starving itself of the thread it needs to hold its leases.
|
||||
- **Progress is a write per call.** `context.step` is one `UPDATE` in its own transaction, on its own connection. The push to the browser is throttled; the write is not, because the row is the record and a second process reads it. Report per batch rather than per row in anything with thousands of items.
|
||||
- **One bad turn does not stop the loop.** A locked database is logged and retried on the next pass rather than ending the drain task, because a worker that died silently looks exactly like a healthy one with nothing to do.
|
||||
- **`depends_on` names jobs, not tasks.** A job waits on ids it was given; a task id is for finding and cancelling, not for waiting.
|
||||
- **No priorities.** Jobs are claimed oldest first within a resource; there is no way to jump the queue but to be queued earlier.
|
||||
- **Five cadences, not cron.** `every 15m`, `every 6h`, `every 2d`, `daily at 06:00`, `weekly mon 07:00`, and nothing else — no seconds, no day-of-month, no "every weekday", no ranges. Anything the five do not cover is a job that decides for itself whether today is its day, or cron running `pipeline.worker --until-idle` as before.
|
||||
- **A schedule is not an alarm.** It is checked every `SCHEDULE_TICK`, so "06:00" means within half a minute of it, and a machine asleep at 06:00 runs the job when it wakes rather than at 06:00. What it will not do is run it once per slot missed.
|
||||
- **History is purged.** Finished jobs older than `JobQueue(retain=...)`, 30 days by default, are deleted hourly while idle. Kept regardless: unfinished jobs, the newest ending of each kind in each state (what `last(kind)` reads), and runs still pointed at. `retain=0` keeps everything.
|
||||
- **A result is at most 64 KB of JSON, and a bigger one fails the job.** Past `MAX_RESULT` the job ends `failed` with `ResultTooLarge`, the size and the keys it returned, a notice goes out, and nothing of the result is written. It is not retried, since the same handler returns the same result. Every snapshot and tally reads results back, so data belongs in the app's store; raise `MAX_RESULT` if a project really needs bigger results.
|
||||
- **A task for the model gives up on a row eventually.** See `ModelTask` above: three unanswered turns and a row stops being offered until a person asks for it deliberately (`reset_tagging` with no slug, `reset_summaries`). `ModelTask.missed` is the seam and `status.py` says how many.
|
||||
- **Migrations are `_reconcile`.** Columns are added, and dropped, by reading the catalogue on open, because `CREATE TABLE IF NOT EXISTS` leaves an existing table alone and `PRAGMA user_version` on that file belongs to the app.
|
||||
- **SQLite means one machine.** Not a network queue.
|
||||
+1135
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,195 @@
|
||||
"""How a function becomes a job.
|
||||
|
||||
The whole contract is one line:
|
||||
|
||||
def handler(config, store, *, context=NULL, limit=None) -> dict
|
||||
|
||||
`@job` records that function against a name and **returns it unchanged**. That
|
||||
is the point of this module rather than an accident of it: a handler stays an
|
||||
ordinary function you can import and call.
|
||||
|
||||
run_pull(config, store) # sync, no queue, no event loop
|
||||
run_pull(config, store, limit=5) # payload keys are just keyword args
|
||||
|
||||
Declared keyword arguments, not `**payload`: a key the handler does not name
|
||||
is reported against the job, and a handler that swallowed everything would
|
||||
turn a misspelt payload key into work that quietly did the wrong thing.
|
||||
|
||||
Which means a job is testable, reusable from a command, and reusable from
|
||||
another job, without any of them going near the queue. The queue's entire
|
||||
involvement is calling the same function with a real `context` and the stored
|
||||
payload spread over it.
|
||||
|
||||
What that `context` *is* lives next door in `context.py`. Neither module
|
||||
imports the handlers: they live in the app (`newsdesk/jobs/` for newsdesk),
|
||||
import `@job` from here, and fill `JOBS` when they are imported.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
#: A handler takes (config, store), keyword-only `context`, and whatever payload it
|
||||
#: declares. It returns a JSON-serialisable dict, which is stored as the job's
|
||||
#: result and shown when the job is done.
|
||||
Handler = Callable[..., dict[str, Any]]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Spec:
|
||||
kind: str
|
||||
label: str
|
||||
fn: Handler
|
||||
#: Whether `fn` actually accepts `context`. Worked out once, here, so the
|
||||
#: worker doesn't inspect a signature per run — and so that "any function
|
||||
#: with the right shape is queueable" holds for a function that wants
|
||||
#: neither progress nor cancellation and says so by not taking the
|
||||
#: argument. Passing `context` to one of those would be a TypeError, which
|
||||
#: would surface as the job mysteriously failing.
|
||||
wants_context: bool = True
|
||||
#: When true, only one job of this kind may be queued or running at a time
|
||||
#: — enforced in the database, so it holds across every client at once.
|
||||
#: For the sweeps: one `pull`, one `rank`, however many tabs are open.
|
||||
#:
|
||||
#: Off by default, because the two ways of being wrong are not equal. A
|
||||
#: sweep that forgets to say `solo=True` runs twice and wastes a minute;
|
||||
#: a batch kind that forgets to say `solo=False` has seven of its eight
|
||||
#: batches silently handed back the first one, and the work simply never
|
||||
#: happens. The default belongs on the side that fails loudly.
|
||||
solo: bool = False
|
||||
#: What running one spends, and so which kind of worker runs it:
|
||||
#: `llm` for anything that asks a model, `http` for a fetch, `cpu` for
|
||||
#: work that wants the machine to itself. Declared here rather than worked
|
||||
#: out, because the handler is arbitrary code and only its author knows.
|
||||
resource: str = "cpu"
|
||||
#: How many attempts this kind of work is worth, counting the first, and
|
||||
#: how long to wait between them. Copied onto every job queued of this kind
|
||||
#: — see `pipeline/table.py`, where the row rather than the kind is
|
||||
#: what a later attempt reads.
|
||||
#:
|
||||
#: One by default, which is no retry and is what every job did before this
|
||||
#: existed. Retrying is off by default for the same reason `solo` is: the
|
||||
#: two ways of being wrong are not equal. A fetch that should have been
|
||||
#: retried and was not costs one item until the next sweep finds it again;
|
||||
#: work retried three times that never should have been is three times the
|
||||
#: writes, or three times the bill.
|
||||
max_attempts: int = 1
|
||||
retry_delay: float = 0.0
|
||||
retry_backoff: float = 1.0
|
||||
#: Run this kind in a child process of its own rather than in a worker
|
||||
#: thread. What it buys is the one thing a thread cannot give: the job can
|
||||
#: be *ended*. A handler inside `sklearn.fit` or a tight Python loop
|
||||
#: cannot be stopped, because no signal and no exception reaches a thread
|
||||
#: in the middle of a C call — only a separate process can be killed.
|
||||
#:
|
||||
#: Nothing has to be pickled to make it work, which is why it is cheap
|
||||
#: here and expensive elsewhere: a job row is already a complete
|
||||
#: description of the work — a kind, a JSON payload, a database. The child
|
||||
#: is handed those three and rebuilds the rest. See `pipeline/child.py`.
|
||||
#:
|
||||
#: The cost is that the queue needs a `bootstrap` able to build the app's
|
||||
#: `(config, store)` from nothing, since neither survives a process
|
||||
#: boundary. One function per app.
|
||||
#:
|
||||
#: Not for work that is expensive because of a *command* it runs —
|
||||
#: ffmpeg, stockfish. That wants `RunContext.spawn`, which kills the same
|
||||
#: way for none of the setup.
|
||||
isolate: bool = False
|
||||
#: Seconds this kind may run before it is ended, or 0 for no deadline.
|
||||
#: Requires `isolate`, and `job` refuses the pair without it: a deadline
|
||||
#: over a thread could only mark the row failed and walk away, leaving the
|
||||
#: work still running and still holding the core it was killed for
|
||||
#: holding — and leaving two things intending to write the same row. A
|
||||
#: promise the pipeline cannot keep is worse than no promise.
|
||||
timeout: float = 0.0
|
||||
#: Payload keys the handler cannot run without: its keyword arguments
|
||||
#: that have no default. A kind with any is not a button, because a press
|
||||
#: sends an empty payload and the job would fail on the TypeError.
|
||||
needs: tuple[str, ...] = ()
|
||||
|
||||
|
||||
#: name -> spec. Explicit registration, like `routers/ROUTERS`: a new job is
|
||||
#: a change to this project, and the cost of importing its module once in
|
||||
#: the app's handlers module is one line. Empty until something imports
|
||||
#: `newsdesk.jobs`, which every entry point does before it looks here.
|
||||
JOBS: dict[str, Spec] = {}
|
||||
|
||||
|
||||
def job(kind: str, label: str = "", *, solo: bool = False,
|
||||
resource: str = "cpu", max_attempts: int = 1,
|
||||
retry_delay: float = 0.0,
|
||||
retry_backoff: float = 1.0,
|
||||
isolate: bool = False,
|
||||
timeout: float = 0.0) -> Callable[[Handler], Handler]:
|
||||
"""Register a plain function as queueable work.
|
||||
|
||||
Returns the function untouched, so importing it and calling it directly
|
||||
goes on working exactly as before it was decorated.
|
||||
|
||||
`solo=True` for a job there should only ever be one of at a time — see
|
||||
`Spec.solo`, which is where the default and its reasoning live.
|
||||
|
||||
`max_attempts=3, retry_delay=60` for work that is worth another go when
|
||||
the other end is having a bad minute. The retry is the queue's, not the
|
||||
handler's: a failed attempt puts the row back in the queue with a time on
|
||||
it, so nothing sleeps and nothing holds a worker while it waits. A
|
||||
retrying job must be safe to run twice, which every handler here has to be
|
||||
anyway — see the pipeline's usage card.
|
||||
|
||||
`isolate=True, timeout=600` for work that has to be endable: it runs in a
|
||||
child process, and ten minutes in it is killed rather than asked. See
|
||||
`Spec.isolate` for what that costs and `RunContext.spawn` for the cheaper
|
||||
answer when the expensive part is a command rather than Python.
|
||||
"""
|
||||
if timeout and not isolate:
|
||||
raise ValueError(
|
||||
f"{kind!r} asks for a {timeout:g}s timeout without isolate=True. "
|
||||
"A job in a thread cannot be ended — the deadline could only mark "
|
||||
"the row failed while the work carried on. Either isolate it, or "
|
||||
"give the handler a RunContext.spawn(timeout=...) to hold."
|
||||
)
|
||||
if timeout < 0:
|
||||
raise ValueError(f"{kind!r} asks for a negative timeout: {timeout!r}")
|
||||
|
||||
def register(fn: Handler) -> Handler:
|
||||
if kind in JOBS and JOBS[kind].fn is not fn:
|
||||
raise ValueError(
|
||||
f"two different handlers are both registered as {kind!r}: "
|
||||
f"{JOBS[kind].fn.__qualname__} and {fn.__qualname__}"
|
||||
)
|
||||
JOBS[kind] = Spec(
|
||||
kind=kind, label=label or kind, fn=fn, solo=solo,
|
||||
wants_context=_accepts_context(fn), needs=_required(fn),
|
||||
resource=resource, max_attempts=max_attempts,
|
||||
retry_delay=retry_delay, retry_backoff=retry_backoff,
|
||||
isolate=isolate, timeout=timeout,
|
||||
)
|
||||
return fn
|
||||
|
||||
return register
|
||||
|
||||
|
||||
def _required(fn: Handler) -> tuple[str, ...]:
|
||||
"""`fn`'s keyword-only parameters without a default, in declared order."""
|
||||
try:
|
||||
params = inspect.signature(fn).parameters
|
||||
except (TypeError, ValueError):
|
||||
return ()
|
||||
return tuple(name for name, p in params.items()
|
||||
if p.kind is inspect.Parameter.KEYWORD_ONLY
|
||||
and p.default is inspect.Parameter.empty)
|
||||
|
||||
|
||||
def _accepts_context(fn: Handler) -> bool:
|
||||
"""Whether `fn` has a `context` parameter, or a `**kwargs` that would swallow one."""
|
||||
try:
|
||||
params = inspect.signature(fn).parameters
|
||||
except (TypeError, ValueError):
|
||||
return True # unintrospectable; assume the contract
|
||||
return "context" in params or any(
|
||||
p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()
|
||||
)
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Cadences: the five wordings a schedule may be written in, and nothing else.
|
||||
|
||||
A schedule is a row that says "enqueue this kind with this payload on this
|
||||
cadence". This module is only the *cadence* half: the grammar, the canonical
|
||||
spelling, and when the next run falls. The row is in `table.py`, the firing is
|
||||
in `queue.py`, and the buttons are in `web/schedules.py`.
|
||||
|
||||
Five wordings, deliberately, rather than cron:
|
||||
|
||||
every 15m · every 6h · every 2d · daily at 06:00 · weekly mon 07:00
|
||||
|
||||
Cron is five fields of positional punctuation that nobody reads back
|
||||
correctly, and the schedules a small app actually wants are these. A wording
|
||||
outside them raises `ValueError` listing what is accepted, which is the
|
||||
sentence the API hands a person who typed something else. Long weekday names
|
||||
are taken as well as short ones, because `weekly monday 07:00` is what people
|
||||
write; `__str__` gives back the short form, so a row stores one spelling
|
||||
however it was typed.
|
||||
|
||||
**The interval forms are intervals and the clock forms are clock times.**
|
||||
`every 6h` is six hours later, measured in real elapsed time, so it neither
|
||||
skips nor doubles across a daylight-saving change. `daily at 06:00` is 06:00 on
|
||||
the wall clock of the queue's timezone, on both sides of one — which is the
|
||||
whole reason `next_after` takes a zone rather than doing UTC arithmetic. The
|
||||
two mean different things and a person picking one has already chosen which.
|
||||
|
||||
Defaults are declared next to `@job`, in the app's own jobs module:
|
||||
|
||||
from pipeline.schedule import declare
|
||||
declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
|
||||
|
||||
`declare` fills `SCHEDULES` the way `@job` fills `JOBS`, and `JobQueue.start()`
|
||||
inserts a row for any declared name that has none. It parses the cadence there
|
||||
and then, so a typo is an ImportError at startup rather than a schedule that
|
||||
quietly never fires.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, time, timedelta, tzinfo
|
||||
from typing import Any, Literal
|
||||
|
||||
#: Short weekday names, Monday first, which is also `datetime.weekday()`'s
|
||||
#: numbering. The short form is canonical: it is what `__str__` writes and so
|
||||
#: what the table stores, however the row was typed.
|
||||
DAYS: tuple[str, ...] = ("mon", "tue", "wed", "thu", "fri", "sat", "sun")
|
||||
|
||||
#: The long names, accepted on input for the same index. Spelled out rather
|
||||
#: than derived from `calendar.day_name`, which is locale-dependent — a cadence
|
||||
#: that parses on one machine and not on another is the bug this avoids.
|
||||
LONG_DAYS: tuple[str, ...] = ("monday", "tuesday", "wednesday", "thursday",
|
||||
"friday", "saturday", "sunday")
|
||||
|
||||
#: What each interval unit is worth. Seconds are deliberately absent: a queue
|
||||
#: that looks for due schedules every thirty seconds cannot honour "every 10s",
|
||||
#: and a cadence the scheduler cannot keep is worse than one it refuses.
|
||||
UNITS: dict[str, int] = {"m": 60, "h": 3600, "d": 86400}
|
||||
|
||||
#: What a person is told when the wording is not one of the five. One sentence,
|
||||
#: because it goes out as `{"detail": ...}` and ends up in a toast.
|
||||
FORMS = ("every 15m", "every 6h", "every 2d", "daily at 06:00", "weekly mon 07:00")
|
||||
|
||||
_EVERY = re.compile(r"^every (\d+) ?([mhd])$")
|
||||
_DAILY = re.compile(r"^daily at (\d{1,2}):(\d{2})$")
|
||||
_WEEKLY = re.compile(r"^weekly ([a-z]+) (\d{1,2}):(\d{2})$")
|
||||
|
||||
|
||||
class BadCadence(ValueError):
|
||||
"""A wording that is not one of the five. A `ValueError`, so a caller that
|
||||
only knows it is parsing something still catches it."""
|
||||
|
||||
|
||||
def _refuse(text: str, because: str = "is not a cadence") -> BadCadence:
|
||||
"""The one refusal, with the forms always in it.
|
||||
|
||||
Always, even when the wording was nearly right and this could have said
|
||||
only what was wrong with it: the person reading this is typing into a box
|
||||
and what they need is what to type, not a diagnosis. The specific
|
||||
complaint goes in front of the list rather than instead of it.
|
||||
"""
|
||||
return BadCadence(
|
||||
f"{text!r} {because}. Write one of: "
|
||||
+ ", ".join(repr(f) for f in FORMS)
|
||||
+ " — long weekday names are fine too."
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Cadence:
|
||||
"""One of the five wordings, parsed. Immutable, and its own spelling.
|
||||
|
||||
Built through `parse`; the fields are what the wording said and are left
|
||||
public so a test can assert on them. `str(cadence)` is the canonical form
|
||||
and is what goes in the table, so two spellings of the same thing —
|
||||
`weekly monday 07:00` and `weekly mon 07:00` — are one row and compare
|
||||
equal.
|
||||
"""
|
||||
|
||||
kind: Literal["every", "daily", "weekly"]
|
||||
#: `every`: how many of `unit`. Unused by the clock forms.
|
||||
n: int = 0
|
||||
unit: str = ""
|
||||
#: `daily` and `weekly`: the wall-clock time, in the queue's zone.
|
||||
hour: int = 0
|
||||
minute: int = 0
|
||||
#: `weekly`: 0 is Monday, matching `datetime.weekday()`. -1 otherwise.
|
||||
day: int = -1
|
||||
|
||||
def __str__(self) -> str:
|
||||
if self.kind == "every":
|
||||
return f"every {self.n}{self.unit}"
|
||||
if self.kind == "daily":
|
||||
return f"daily at {self.hour:02d}:{self.minute:02d}"
|
||||
return f"weekly {DAYS[self.day]} {self.hour:02d}:{self.minute:02d}"
|
||||
|
||||
@property
|
||||
def seconds(self) -> int:
|
||||
"""The interval, for the `every` forms. Zero for the clock ones, which
|
||||
have no fixed length — a day is 23 or 25 hours twice a year."""
|
||||
return self.n * UNITS[self.unit] if self.kind == "every" else 0
|
||||
|
||||
@classmethod
|
||||
def parse(cls, text: str) -> Cadence:
|
||||
"""One of the five wordings, or `BadCadence` naming all five.
|
||||
|
||||
Case and inner spacing are forgiven — `Every 15 M` is `every 15m` —
|
||||
because they are typing, not meaning. Everything else is refused: a
|
||||
cadence that is nearly right is the one that has to fail loudly, since
|
||||
the alternative is a row that looks scheduled and never fires.
|
||||
"""
|
||||
said = " ".join(str(text or "").lower().split())
|
||||
|
||||
found = _EVERY.match(said)
|
||||
if found:
|
||||
n = int(found.group(1))
|
||||
if n < 1:
|
||||
raise _refuse(text)
|
||||
return cls(kind="every", n=n, unit=found.group(2))
|
||||
|
||||
found = _DAILY.match(said)
|
||||
if found:
|
||||
hour, minute = int(found.group(1)), int(found.group(2))
|
||||
_check_clock(text, hour, minute)
|
||||
return cls(kind="daily", hour=hour, minute=minute)
|
||||
|
||||
found = _WEEKLY.match(said)
|
||||
if found:
|
||||
name = found.group(1)
|
||||
day = (DAYS.index(name) if name in DAYS
|
||||
else LONG_DAYS.index(name) if name in LONG_DAYS else -1)
|
||||
if day < 0:
|
||||
raise _refuse(text)
|
||||
hour, minute = int(found.group(2)), int(found.group(3))
|
||||
_check_clock(text, hour, minute)
|
||||
return cls(kind="weekly", hour=hour, minute=minute, day=day)
|
||||
|
||||
raise _refuse(text)
|
||||
|
||||
def next_after(self, moment: datetime, zone: tzinfo = UTC) -> datetime:
|
||||
"""When this next falls, strictly after `moment`. Always UTC-aware.
|
||||
|
||||
Strictly after, so a schedule fired at exactly its own time does not
|
||||
find itself due again on the same instant.
|
||||
|
||||
`every` is elapsed time and ignores the zone: six hours is six hours,
|
||||
so a run neither vanishes nor doubles when the clocks move. The two
|
||||
clock forms are the opposite and want the zone for exactly that reason
|
||||
— 06:00 is 06:00 on both sides of a change, which means the interval
|
||||
across one is 23 or 25 hours, and that is the thing the wording
|
||||
promised.
|
||||
|
||||
Walking forward a day at a time rather than doing arithmetic on the
|
||||
date: it is at most eight steps, it handles "later today" and "this
|
||||
weekday next week" with one piece of code, and each candidate is built
|
||||
from a wall-clock time and then converted, which is what keeps the
|
||||
promise above.
|
||||
"""
|
||||
moment = moment if moment.tzinfo else moment.replace(tzinfo=UTC)
|
||||
if self.kind == "every":
|
||||
# Converted to UTC *before* the addition, not after. Adding a
|
||||
# timedelta to an aware datetime is wall-clock arithmetic within
|
||||
# its own zone — `06:30 + 24h` on the day the clocks go forward is
|
||||
# 06:30 the next day, which is 23 real hours. That is exactly the
|
||||
# promise the interval forms do not make, and the whole reason
|
||||
# somebody writes `daily` when they want the other one.
|
||||
return moment.astimezone(UTC) + timedelta(seconds=self.seconds)
|
||||
|
||||
local = moment.astimezone(zone)
|
||||
for step in range(9): # today, then a full week of weekdays
|
||||
day = local.date() + timedelta(days=step)
|
||||
if self.kind == "weekly" and day.weekday() != self.day:
|
||||
continue
|
||||
when = datetime.combine(day, time(self.hour, self.minute), tzinfo=zone)
|
||||
if when > local:
|
||||
return when.astimezone(UTC)
|
||||
# Unreachable: a daily cadence lands within two steps and a weekly one
|
||||
# within eight. Raised rather than returned-as-None so that if the
|
||||
# calendar ever surprises us it says so here.
|
||||
raise BadCadence(f"{self} never comes round after {moment.isoformat()}")
|
||||
|
||||
|
||||
def _check_clock(text: str, hour: int, minute: int) -> None:
|
||||
if not (0 <= hour <= 23 and 0 <= minute <= 59):
|
||||
raise _refuse(text, "is not a time of day — hours run 00-23 and "
|
||||
"minutes 00-59")
|
||||
|
||||
|
||||
# ------------------------------------------------------------ declared defaults
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Declared:
|
||||
"""A schedule an app ships with, before any row exists for it.
|
||||
|
||||
`name` is the handle and is the whole of the reconciliation: a declared
|
||||
name with no row gets one, a declared name that already has one is left
|
||||
exactly as the user edited it. Which means the defaults here are a
|
||||
*starting point*, not a setting — turning one off in the UI survives the
|
||||
next restart, and that is the point of the row being the truth.
|
||||
"""
|
||||
|
||||
name: str
|
||||
kind: str
|
||||
cadence: str
|
||||
payload: dict[str, Any] = field(default_factory=dict)
|
||||
label: str = ""
|
||||
|
||||
|
||||
#: name -> declaration. Filled by `declare` when the app's jobs module is
|
||||
#: imported, exactly as `JOBS` is filled by `@job`, and read once per
|
||||
#: `JobQueue.start()`. Empty in a project that declares none, which is fine:
|
||||
#: every schedule can be added from the UI instead.
|
||||
SCHEDULES: dict[str, Declared] = {}
|
||||
|
||||
|
||||
def declare(name: str, kind: str, cadence: str,
|
||||
payload: dict[str, Any] | None = None,
|
||||
label: str = "") -> Declared:
|
||||
"""Ship a default schedule. Goes next to `@job`, in the app's jobs module.
|
||||
|
||||
declare("nightly tick", "tick", "daily at 06:00")
|
||||
declare("hourly count", "count", "every 1h", {"n": 5})
|
||||
|
||||
The cadence is parsed here, so a typo is an error the moment the module is
|
||||
imported — at startup, with a traceback pointing at this line — rather than
|
||||
a schedule that sits in the table looking fine and never fires.
|
||||
|
||||
The kind is *not* checked, because it usually cannot be: `declare` and
|
||||
`@job` are read top to bottom in one file and a declaration above its
|
||||
handler would be refused for a job that exists a dozen lines later. An
|
||||
unregistered kind is caught where it matters instead — `JobQueue` never
|
||||
fires one and `<desk-schedules>` says `unknown kind` against the row.
|
||||
|
||||
Declaring the same name twice with different terms is refused, like two
|
||||
handlers on one `@job` name: the second would silently win, and which
|
||||
module imported first is not a thing anybody should have to know.
|
||||
"""
|
||||
Cadence.parse(cadence)
|
||||
declared = Declared(name=name, kind=kind, cadence=str(Cadence.parse(cadence)),
|
||||
payload=dict(payload or {}), label=label)
|
||||
if name in SCHEDULES and SCHEDULES[name] != declared:
|
||||
raise ValueError(
|
||||
f"two different schedules are both declared as {name!r}: "
|
||||
f"{SCHEDULES[name].kind} {SCHEDULES[name].cadence!r} and "
|
||||
f"{declared.kind} {declared.cadence!r}"
|
||||
)
|
||||
SCHEDULES[name] = declared
|
||||
return declared
|
||||
|
||||
|
||||
__all__ = ["DAYS", "FORMS", "LONG_DAYS", "SCHEDULES", "UNITS", "BadCadence",
|
||||
"Cadence", "Declared", "declare"]
|
||||
+1319
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,495 @@
|
||||
"""A task: plain code that makes jobs, and is never a job itself.
|
||||
|
||||
task = Task(kind="body_batch", name="body", finder=needs_a_body)
|
||||
task.start(config, store, queue) # queued: batches, then one final job
|
||||
task.run(config, store) # here and now, no queue at all
|
||||
|
||||
A task is a business step — fetch every feed, give every article a body, tag
|
||||
what has never been tagged. Given the config and the store it finds its own
|
||||
work, cuts it into batch payloads, mints a run id for itself, queues one job
|
||||
per payload under that id, and queues one final job behind them with
|
||||
`after="done"` on all of them. Then it is over. Nothing of the task is alive
|
||||
afterwards: only its jobs, and the callback the final job carries.
|
||||
|
||||
**It runs in whoever calls it.** The HTTP request behind a button, a final
|
||||
job's handler, a Claude Code session, a test. Finding the work is a query, and
|
||||
a query is cheap enough to do there — making the preparation asynchronous
|
||||
would buy a few milliseconds and cost a whole job kind, a payload, a failure
|
||||
mode and a row in the history that says nothing a person wants to read.
|
||||
|
||||
**A cascade is the same shape one level up**, and the pipeline does not tell
|
||||
them apart: both are runs, both group jobs by an id, both are killed the same
|
||||
way. The difference is a convention, and it is worth stating because nothing
|
||||
enforces it:
|
||||
|
||||
a task queues everything it wants at once, and only its final job
|
||||
calls back.
|
||||
a cascade is where the breakpoints are: it queues the next task only
|
||||
once the last one is over.
|
||||
|
||||
**The callback is a name, not a function.** Nothing in a table can hold a
|
||||
function, and the final job's payload is a table row — so the name travels
|
||||
and the worker that runs the job resolves it. `module:function`, imported
|
||||
where it is used. A registry would work too, and was the other candidate; the
|
||||
dotted name wins because it carries its own import, where a registry has to be
|
||||
filled before the worker looks in it, which is exactly the import-order trap
|
||||
`JOBS` already has to be careful about. It also reads as documentation in the
|
||||
job's payload: `newsdesk.jobs.cascade:step` says where to go and look.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
from collections.abc import Callable, Iterable, Sequence
|
||||
from dataclasses import dataclass, field, replace
|
||||
from typing import Any, Protocol
|
||||
|
||||
from pipeline.context import NULL, Cancelled, RunContext
|
||||
from pipeline.registry import JOBS, job
|
||||
from pipeline.table import on_its_own
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: The job kind every task ends with. One kind for every task there will ever
|
||||
#: be, because what it does is the same every time: read how the batches went
|
||||
#: and call the name in its payload.
|
||||
FINAL = "advance"
|
||||
|
||||
#: A payload for one batch: the keyword arguments its job will be called with.
|
||||
Payload = dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Bound:
|
||||
"""How many times one batch may be attempted, and how long it waits.
|
||||
|
||||
The defaults are one attempt and no wait, which is every batch of every
|
||||
task that existed before this: a job with no policy fails at its first
|
||||
exception exactly as it always did.
|
||||
|
||||
It exists because a bound is sometimes a property of the *batch* rather
|
||||
than of the job kind — a plugin declares how long to wait before asking
|
||||
its publisher again, so two batches of `body_batch` cut from two outlets
|
||||
carry two different numbers. The values travel onto the row (see
|
||||
`Jobs.enqueue`), because the attempt that has to decide may be running in
|
||||
another process, after a restart, under a plugin file that has since
|
||||
changed.
|
||||
"""
|
||||
|
||||
#: Counting the first attempt, so 1 is "run it once".
|
||||
max_attempts: int = 1
|
||||
#: Seconds before the second attempt.
|
||||
delay: float = 0.0
|
||||
#: Multiplied into the delay once per attempt already made.
|
||||
backoff: float = 1.0
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Batch:
|
||||
"""One batch of a task, when its bound is its own.
|
||||
|
||||
A finder may return these instead of plain payloads. Two shapes rather
|
||||
than one because the second is what nearly every task wants and reads as
|
||||
what it is — `[{"uids": [...]}, ...]` is a list of batches, and wrapping
|
||||
each of them to say "and the default bound" would be ceremony on every
|
||||
task to serve the one that needs it.
|
||||
"""
|
||||
|
||||
payload: Payload
|
||||
bound: Bound = Bound()
|
||||
#: What to call this one batch on screen, or "" to be called whatever its
|
||||
#: job kind is called. The kind's label is the same string on every batch
|
||||
#: of the task — nine rows reading "Tagging a batch" — because a label is
|
||||
#: declared once, at registration, by something that cannot see how the
|
||||
#: work was cut. Only the finder knows that, so this is where a batch that
|
||||
#: can say which batch it is says so. See `desk/modeltask.py`.
|
||||
label: str = ""
|
||||
|
||||
|
||||
#: Given the app's config and store, what there is to do — one payload per
|
||||
#: batch, or one `Batch` where the bound is the batch's own. An empty list is
|
||||
#: a legitimate answer and the common one: a task with nothing waiting costs
|
||||
#: one query, queues no batches, and its final job runs immediately and moves
|
||||
#: the cascade on.
|
||||
Finder = Callable[[Any, Any], list[Any]]
|
||||
|
||||
|
||||
def _batch(found: Payload | Batch) -> Batch:
|
||||
"""One thing a finder returned, as a batch. A bare payload gets the
|
||||
default bound, which is the behaviour it had before bounds existed."""
|
||||
return found if isinstance(found, Batch) else Batch(found)
|
||||
|
||||
|
||||
#: name -> how to build that task, standing on its own. Filled by the app the
|
||||
#: way `JOBS` is, and for the same reason: a task is as much a thing to start
|
||||
#: as a job is, so whatever offers the one — `/api/jobs`, the Jobs view, a
|
||||
#: script — has to be able to find the other by name.
|
||||
TASKS: dict[str, Callable[[], Task]] = {}
|
||||
|
||||
|
||||
class Sink(Protocol):
|
||||
"""Where a task puts the jobs it makes.
|
||||
|
||||
Two of them: `JobQueue`, when a task is started from a request or a
|
||||
handler, and nothing else. It is a protocol rather than the class so that
|
||||
`Task` does not import the queue — the dependency runs the other way, and a
|
||||
test can hand it a list.
|
||||
"""
|
||||
|
||||
def new_run(self, kind: str, name: str = "", label: str = "") -> int: ...
|
||||
|
||||
def cancelled(self, run_id: int) -> bool: ...
|
||||
|
||||
def enqueue(self, kind: str, payload: Payload | None = None, *,
|
||||
label: str = "", depends_on: Sequence[int] = (), after: str = "ok",
|
||||
cascade_id: int = 0, task_id: int = 0,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> Any: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Task:
|
||||
"""One business step: what a batch of it is, how to find them, who hears.
|
||||
|
||||
Knows nothing about what the work *is* — that is the batch job's, and the
|
||||
finder's. This is the shape they are held in.
|
||||
"""
|
||||
|
||||
#: The job kind one batch runs as. A task of one batch names an ordinary
|
||||
#: job kind (`pull`); a task of many names the batch kind (`body_batch`).
|
||||
kind: str
|
||||
#: How the work is found. Called once, in the caller, before anything is
|
||||
#: queued.
|
||||
finder: Finder
|
||||
#: What this task is — `body`, `tag`. Its identity: what a cascade's list
|
||||
#: matches against, and what the `runs` row is minted under. The job kind
|
||||
#: when nobody says.
|
||||
name: str = ""
|
||||
#: How to say that on screen: "Fetching what the feeds left out". Written
|
||||
#: onto the `runs` row, because a task is not a job and has no handler to
|
||||
#: carry a label. The name when nobody says.
|
||||
label: str = ""
|
||||
#: `module:function` for the final job to call, or "" for a task that
|
||||
#: reports to nobody — one started on its own, from a button or a script.
|
||||
callback: str = ""
|
||||
#: `module:function` the final job runs first, or "" for a task whose
|
||||
#: batches finish their own work. A name for the same reason the callback
|
||||
#: is one: it is a row in a table.
|
||||
#:
|
||||
#: The seam exists for a decision no batch can make on its own. A batch is
|
||||
#: one of forty and cannot see the other thirty-nine, so anything that has
|
||||
#: to hold across the whole run — a cap on what the run may add, a merge of
|
||||
#: what several batches each asked for — has exactly one place it can be
|
||||
#: made correctly, and that is the job that runs when they are all over.
|
||||
#: It belongs to the task rather than to the callback because it *is* the
|
||||
#: task's work: a task started on its own has no callback and still has to
|
||||
#: settle. See `newsdesk/jobs/tagging.py` for the one that does.
|
||||
settle: str = ""
|
||||
#: What the callback is handed alongside the tally. Fixed when the task is
|
||||
#: made, and stored in the final job's payload, so it survives a restart
|
||||
#: and a worker in another process.
|
||||
carries: dict[str, Any] = field(default_factory=dict)
|
||||
#: How many items a payload is worth, for the direct path's progress bar.
|
||||
#: Only the direct path asks: as a job a batch is on its own and reports
|
||||
#: its own rows, which is what the boxes on screen are drawn from.
|
||||
size: Callable[[Payload], int] | None = None
|
||||
#: How the direct path's batch results become one result. The queue never
|
||||
#: calls it — there each batch is its own job with its own row.
|
||||
fold: Callable[[Any, Any, list[Any]], dict[str, Any]] | None = None
|
||||
|
||||
@property
|
||||
def named(self) -> str:
|
||||
return self.name or self.kind
|
||||
|
||||
@property
|
||||
def called(self) -> str:
|
||||
return self.label or self.named
|
||||
|
||||
# ------------------------------------------------------------- queued
|
||||
|
||||
def start(self, config: Any, store: Any, sink: Sink, *,
|
||||
cascade_id: int = 0) -> int:
|
||||
"""Find the work, queue it, and be over. Returns the task's run id.
|
||||
|
||||
The id is minted before a single job exists, because a task is a thing
|
||||
before its jobs are — it is what decides how many there will be — and
|
||||
naming it after whichever job it happened to queue first would be
|
||||
describing it by one of its own consequences.
|
||||
|
||||
With no cascade given it begins one: something asked for this on its
|
||||
own, so it is a cascade of one task. Named apart from any chain of the
|
||||
same name — `pull on its own`, not `pull` — because pressing Tag is
|
||||
not starting the chain that ends in tagging, and whoever asks whether
|
||||
that chain is going has to be told no. See `on_its_own`.
|
||||
|
||||
Returns 0 when the cascade it belongs to has been killed. Nothing is
|
||||
told; it tries, and the table says no. `Jobs.enqueue` is where that is
|
||||
actually enforced, across processes and after a restart; this only
|
||||
saves minting a run for jobs that would all be refused.
|
||||
"""
|
||||
task_id, _ = queue_batches(
|
||||
sink, self.kind, lambda: self.finder(config, store),
|
||||
name=self.named, label=self.called, then=self.callback,
|
||||
settle=self.settle, carries=self.carries, cascade_id=cascade_id)
|
||||
return task_id
|
||||
|
||||
# ------------------------------------------------------------- direct
|
||||
|
||||
def run(self, config: Any, store: Any, *,
|
||||
context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Do the batches here and now, in order, with no queue at all.
|
||||
|
||||
What a Claude Code session, a notebook and a test get. The same batch
|
||||
handler the worker would call, with the same payloads from the same
|
||||
finder — there is no second implementation to drift.
|
||||
|
||||
A batch that raises is kept and the rest carry on, which is what the
|
||||
queue does with eight independent jobs of which one fails. Unless
|
||||
*nothing* got through: then the first exception is raised, because a
|
||||
run that returned "0 done" with the explanation only in the log would
|
||||
be a failure reported as a success. A `Cancelled` travels either way —
|
||||
it is an instruction from the person at the keyboard, not a batch that
|
||||
did not land.
|
||||
|
||||
A batch's `Bound` is ignored here, and that is not an omission: a
|
||||
bound is a number the *queue* enforces by putting a row back with a
|
||||
time on it, and there is no queue and no row on this path. Asked for
|
||||
directly, a batch is attempted once and its failure is returned to the
|
||||
caller, who is standing right there.
|
||||
"""
|
||||
spec = JOBS.get(self.kind)
|
||||
if spec is None:
|
||||
raise KeyError(
|
||||
f"{self.named!r} makes jobs of kind {self.kind!r}, which "
|
||||
f"nothing registers (have: {', '.join(sorted(JOBS)) or 'none'})")
|
||||
payloads = [_batch(found).payload for found in self.finder(config, store)]
|
||||
sizes = [self._size(p) for p in payloads]
|
||||
total = sum(sizes)
|
||||
|
||||
results: list[Any] = []
|
||||
failure: Exception | None = None
|
||||
at = 0
|
||||
for payload, size in zip(payloads, sizes, strict=True):
|
||||
context.check()
|
||||
extra = {"context": self._placed(context, at, total, size)} \
|
||||
if spec.wants_context else {}
|
||||
try:
|
||||
results.append(dict(spec.fn(config, store, **extra, **payload) or {}))
|
||||
except Cancelled:
|
||||
raise
|
||||
except Exception as exc:
|
||||
log.exception("%s: a batch of %s did not land", self.named, self.kind)
|
||||
results.append(exc)
|
||||
failure = failure or exc
|
||||
finally:
|
||||
# Position through the backlog, not what was answered: a batch
|
||||
# the model skipped half of still moved the run that far, and a
|
||||
# bar that goes backwards is worse than one that is approximate.
|
||||
at += size
|
||||
if failure is not None and not any(isinstance(r, dict) for r in results):
|
||||
raise failure
|
||||
if self.fold is not None:
|
||||
return self.fold(config, store, results)
|
||||
return {"batches": sum(1 for r in results if isinstance(r, dict)),
|
||||
"failed": sum(1 for r in results if isinstance(r, Exception)),
|
||||
"results": [r for r in results if isinstance(r, dict)]}
|
||||
|
||||
def _size(self, payload: Payload) -> int:
|
||||
return int(self.size(payload)) if self.size is not None else 0
|
||||
|
||||
def _placed(self, context: RunContext, at: int, total: int,
|
||||
size: int) -> RunContext:
|
||||
"""The batch's context, with its progress shifted into the whole run.
|
||||
|
||||
A batch run as a job is on its own and counts its own rows. Run here it
|
||||
is one of several in a row, and a bar that restarts per batch would be
|
||||
a bar that goes backwards five times.
|
||||
"""
|
||||
if not total:
|
||||
return context
|
||||
return replace(context, on_progress=lambda done, _t, message:
|
||||
context.progress(at + min(done, size), total, message))
|
||||
|
||||
|
||||
# ------------------------------------------------------------- the final job
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Tally:
|
||||
"""How a task's batches ended, as the callback is handed it.
|
||||
|
||||
Read off the jobs at the moment the final job runs, never counted up as
|
||||
they landed — so there is nothing to increment and nothing to lose to a
|
||||
restart half way through.
|
||||
"""
|
||||
|
||||
done: int = 0
|
||||
failed: int = 0
|
||||
cancelled: int = 0
|
||||
#: Whether the task itself was killed, as against however many of its
|
||||
#: batches happen to have ended cancelled. The counts cannot say: one
|
||||
#: batch killed by hand and every batch killed by the task going down look
|
||||
#: identical there and mean opposite things — drop this batch, or skip
|
||||
#: this whole step. Only the run's row knows, which is why it is asked.
|
||||
killed: bool = False
|
||||
#: What each batch returned, in the order they were queued.
|
||||
results: tuple[dict[str, Any], ...] = ()
|
||||
|
||||
@property
|
||||
def jobs(self) -> int:
|
||||
return self.done + self.failed + self.cancelled
|
||||
|
||||
@classmethod
|
||||
def of(cls, outcomes: Sequence[Any], *, killed: bool = False) -> Tally:
|
||||
states = [getattr(j, "state", "") for j in outcomes]
|
||||
return cls(
|
||||
done=states.count("done"),
|
||||
failed=states.count("failed"),
|
||||
cancelled=states.count("cancelled"),
|
||||
killed=killed,
|
||||
results=tuple(dict(getattr(j, "result", {}) or {}) for j in outcomes),
|
||||
)
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {"jobs": self.jobs, "done": self.done, "failed": self.failed,
|
||||
"cancelled": self.cancelled, "killed": self.killed}
|
||||
|
||||
|
||||
def queue_batches(sink: Sink, kind: str,
|
||||
found: Callable[[], Iterable[Payload | Batch]], *,
|
||||
name: str = "", label: str = "", then: str = "",
|
||||
settle: str = "", carries: dict[str, Any] | None = None,
|
||||
cascade_id: int = 0,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> tuple[int, tuple[int, ...]]:
|
||||
"""Mint a task, queue one job per batch, and queue the job that reports.
|
||||
|
||||
The one implementation behind both `Task.start` and `fan_out`. Returns the
|
||||
task's run id and the children's job ids; `(0, ())` when the cascade it
|
||||
would join has been killed.
|
||||
|
||||
`found` is called only after the runs exist, which is the order a task has
|
||||
always had: a task is a thing before its jobs are, since it is what decides
|
||||
how many there will be.
|
||||
|
||||
A plain payload takes the kind's own retry policy, or the uniform one
|
||||
passed here. A `Batch` carries its own `Bound`, which wins — that is what a
|
||||
batch cut from one plugin profile needs when the profile declares how long
|
||||
to leave its publisher alone.
|
||||
|
||||
With no cascade given it begins one: something asked for this on its own,
|
||||
so it is a cascade of one task, named apart from any chain of the same name
|
||||
(`pull on its own`, not `pull`). See `on_its_own`.
|
||||
"""
|
||||
if cascade_id and sink.cancelled(cascade_id):
|
||||
return 0, ()
|
||||
name = name or kind
|
||||
label = label or name
|
||||
cascade_id = cascade_id or sink.new_run("cascade", on_its_own(name), label)
|
||||
task_id = sink.new_run("task", name, label)
|
||||
|
||||
ids: list[int] = []
|
||||
for item in found():
|
||||
if isinstance(item, Batch):
|
||||
queued = sink.enqueue(kind, dict(item.payload), label=item.label,
|
||||
cascade_id=cascade_id, task_id=task_id,
|
||||
max_attempts=item.bound.max_attempts,
|
||||
retry_delay=item.bound.delay,
|
||||
retry_backoff=item.bound.backoff)
|
||||
else:
|
||||
queued = sink.enqueue(kind, dict(item),
|
||||
cascade_id=cascade_id, task_id=task_id,
|
||||
max_attempts=max_attempts,
|
||||
retry_delay=retry_delay,
|
||||
retry_backoff=retry_backoff)
|
||||
if queued is not None:
|
||||
ids.append(queued.id)
|
||||
# Behind all of them, and part of the same task — killing the task spares
|
||||
# it, because reporting is not work and "skip this task" is not "make it
|
||||
# never end". That holds where the kill landed mid-loop too: `Jobs.enqueue`
|
||||
# carves `after='done'` out of a task's kill, so this row is written and
|
||||
# the task ends by saying it was killed. None here is the *cascade* going
|
||||
# down, which takes the report with it by design: there is nothing left
|
||||
# to report to. See `Jobs.cancel_run`.
|
||||
sink.enqueue(FINAL, {"callback": then, "settle": settle,
|
||||
"payload": dict(carries or {})},
|
||||
depends_on=ids, after="done",
|
||||
cascade_id=cascade_id, task_id=task_id)
|
||||
return task_id, tuple(ids)
|
||||
|
||||
|
||||
def fan_out(sink: Sink, kind: str, payloads: Sequence[Payload | Batch], *,
|
||||
then: str = "", settle: str = "", label: str = "",
|
||||
carries: dict[str, Any] | None = None,
|
||||
cascade_id: int = 0,
|
||||
max_attempts: int | None = None,
|
||||
retry_delay: float | None = None,
|
||||
retry_backoff: float | None = None) -> tuple[int, ...]:
|
||||
"""Queue these payloads as jobs, and one job behind them that reports.
|
||||
|
||||
What `RunContext.fan_out` reaches. `Task` is the same call with a finder in
|
||||
front, which is why a handler whose whole body is one of these *is* a task.
|
||||
The children go into a run of their own inside the caller's cascade, so the
|
||||
callback can fan out again and the result is a chain rather than a nest.
|
||||
|
||||
Returns the children's ids. The reporting job is not among them — it is
|
||||
behind them by construction, and a caller that waited on it would be
|
||||
holding a worker open to do so.
|
||||
"""
|
||||
_, ids = queue_batches(
|
||||
sink, kind, lambda: payloads, label=label, then=then, settle=settle,
|
||||
carries=carries, cascade_id=cascade_id, max_attempts=max_attempts,
|
||||
retry_delay=retry_delay, retry_backoff=retry_backoff)
|
||||
return ids
|
||||
|
||||
|
||||
@job(FINAL, "Moving on")
|
||||
def run_final(config: Any, store: Any, *, context: RunContext = NULL,
|
||||
callback: str = "", settle: str = "",
|
||||
payload: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""The one job every task ends with: tally the batches, call the name.
|
||||
|
||||
It runs for the milliseconds it takes to read how the batches went and
|
||||
hand that to whoever asked — nothing is alive between two tasks, and
|
||||
nothing was waiting while the last one ran.
|
||||
|
||||
`after="done"`, so it runs however the batches ended. That is the whole
|
||||
reason the mode exists: a job whose purpose is to report that one of eight
|
||||
failed cannot be held back by that failure.
|
||||
|
||||
A task with no callback stops here, which is what being asked for on its
|
||||
own means.
|
||||
|
||||
The settle runs first and the callback second, because the settle is the
|
||||
last of this task's own work and the callback is what comes after this
|
||||
task: a cascade told to move on before the tags of the stage it is leaving
|
||||
exist would be a stage reporting itself finished early.
|
||||
|
||||
It asks whether its own task was killed, as well as counting how the
|
||||
batches ended, because those are two different facts that arrive looking
|
||||
the same. A task killed while its batches ran leaves them all `cancelled`,
|
||||
and so does a person killing every batch by hand; the first is "skip this
|
||||
step", the second is "drop this work". Only the run's row can tell them
|
||||
apart, and the callback needs to be told which it is — the whole reason
|
||||
this job is spared the kill is so that somebody hears.
|
||||
"""
|
||||
tally = Tally.of(context.waited_for, killed=context.killed())
|
||||
out = tally.as_dict()
|
||||
if settle:
|
||||
out |= resolve(settle)(config, store, tally, context=context) or {}
|
||||
if not callback:
|
||||
return out
|
||||
return {**out, **(resolve(callback)(config, store, tally,
|
||||
dict(payload or {}), context=context) or {})}
|
||||
|
||||
|
||||
def resolve(name: str) -> Callable[..., dict[str, Any] | None]:
|
||||
"""`module:function` to the function. See this module's docstring for why
|
||||
a callback, or a settle, is a name at all."""
|
||||
module, _, attr = name.partition(":")
|
||||
if not module or not attr:
|
||||
raise ValueError(f"a callback is 'module:function', not {name!r}")
|
||||
return getattr(importlib.import_module(module), attr)
|
||||
@@ -0,0 +1,2 @@
|
||||
"""The pipeline's own tests. They travel with the folder, so a copy can
|
||||
prove itself where it lands: `python -m pytest pipeline/tests`."""
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Handlers for the isolation tests, in a module a child process can import.
|
||||
|
||||
Not a test file — pytest does not collect it — and deliberately separate from
|
||||
the tests that use it, because the point of `isolate=True` is that the work
|
||||
runs in a *different process*. That process gets nothing from this one: it
|
||||
imports `pipeline.tests.isolated_handlers` itself, by the name in
|
||||
`--bootstrap`, which is exactly what an app's own bootstrap has to do.
|
||||
|
||||
It also documents the whole of the contract, which is two lines: import the
|
||||
modules that fill `JOBS`, and return the `(config, store)` pair the handlers
|
||||
take. Here both are dicts, since nothing in the pipeline reads them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from pipeline import NULL, RunContext, job
|
||||
|
||||
|
||||
@job("iso_ok", "Working in its own process", isolate=True)
|
||||
def iso_ok(config: Any, store: Any, *, context: RunContext = NULL,
|
||||
n: int = 0) -> dict[str, Any]:
|
||||
"""Reports which process it ran in, so a test can prove it was not ours."""
|
||||
context.progress(n, n, f"counting to {n}")
|
||||
return {"doubled": n * 2, "pid": os.getpid()}
|
||||
|
||||
|
||||
@job("iso_raises", "Failing in its own process", isolate=True)
|
||||
def iso_raises(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
raise RuntimeError("the model said no")
|
||||
|
||||
|
||||
@job("iso_slow", "Taking too long on purpose", isolate=True, timeout=1.0)
|
||||
def iso_slow(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Uninterruptible on purpose: no `check()`, so only a kill ends it.
|
||||
|
||||
`time.sleep` stands in for `sklearn.fit` — the difference that matters is
|
||||
not what it does but that nothing in it ever asks whether to stop, which
|
||||
is the case a thread has no answer for.
|
||||
"""
|
||||
time.sleep(120)
|
||||
return {"never": True}
|
||||
|
||||
|
||||
def bootstrap() -> tuple[Any, Any]:
|
||||
"""What `--bootstrap` names. Importing this module registered the jobs."""
|
||||
return {"config": "rebuilt in the child"}, {"store": "rebuilt in the child"}
|
||||
|
||||
|
||||
# ---------------------------------------------------- two workers, one file
|
||||
|
||||
@job("mp_mark", "Leaving a mark")
|
||||
def mp_mark(config: Any, store: Any, *, context: RunContext = NULL,
|
||||
i: int = 0) -> dict[str, Any]:
|
||||
"""Leave a file named after this job and the process that ran it.
|
||||
|
||||
A file per run rather than a line in a shared one, because creating a
|
||||
uniquely named file is atomic on every platform and appending from two
|
||||
processes is not. A job run twice leaves two files with the same number.
|
||||
"""
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
(Path(store) / f"{i}-{os.getpid()}-{uuid.uuid4().hex}").touch()
|
||||
time.sleep(0.02) # long enough that both workers get a share
|
||||
return {"i": i}
|
||||
|
||||
|
||||
def marking(marks: str) -> tuple[Any, Any]:
|
||||
"""The bootstrap `pipeline.worker` is given: the marks directory is the store."""
|
||||
return {}, marks
|
||||
@@ -0,0 +1,39 @@
|
||||
"""A job and a declared schedule, in a module a worker process can import.
|
||||
|
||||
Not a test file — pytest does not collect it — and separate from the tests that
|
||||
use it for the same reason `isolated_handlers.py` is: the process under test is
|
||||
a *different* process, and it gets nothing from this one. It imports this by
|
||||
the name in `--bootstrap`, which fills `JOBS` and `SCHEDULES` exactly as an
|
||||
app's own jobs module does.
|
||||
|
||||
Separate from `isolated_handlers.py` as well, because importing a module that
|
||||
calls `declare` fills a global: a test suite that imported this one would find
|
||||
its own schedule registry non-empty before it had declared anything.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pipeline import NULL, RunContext, declare, job
|
||||
|
||||
|
||||
@job("sw_mark", "Leaving a mark")
|
||||
def sw_mark(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Touch a file in the store directory, so the parent process can see from
|
||||
outside whether this ran at all."""
|
||||
(Path(store) / "fired").touch()
|
||||
return {"fired": True}
|
||||
|
||||
|
||||
#: The declaration itself is what one half of the worker test is about: a
|
||||
#: worker with no web app around it reconciles declared defaults like any
|
||||
#: other, so a machine running only `python -m pipeline.worker` still gets
|
||||
#: them.
|
||||
declare("marker", "sw_mark", "every 1h", label="Leaving a mark, hourly")
|
||||
|
||||
|
||||
def marking(marks: str) -> tuple[Any, Any]:
|
||||
"""The bootstrap `pipeline.worker` is given: the marks directory is the store."""
|
||||
return {}, marks
|
||||
@@ -0,0 +1,113 @@
|
||||
"""The web face of the queue, from an app that has written nothing but `connect`.
|
||||
|
||||
python -m pytest pipeline/tests/test_web.py -q
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from pipeline import NULL, Jobs, RunContext, job
|
||||
from pipeline.web import connect, jobs_router
|
||||
|
||||
|
||||
@job("w_hello", "Saying hello")
|
||||
def w_hello(config, store, *, context: RunContext = NULL, name: str = "") -> dict:
|
||||
context.progress(1, 1, f"greeting {name}")
|
||||
return {"said": f"hello {name}", "config": config["app"]}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "web.db")
|
||||
|
||||
|
||||
def settled(client: TestClient, job_id: int, timeout: float = 10.0) -> dict:
|
||||
"""The job as the HTTP API reports it, once it has finished."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
snap = client.get("/api/jobs").json()
|
||||
for row in snap["recent"]:
|
||||
if row["id"] == job_id:
|
||||
return row
|
||||
time.sleep(0.05)
|
||||
raise AssertionError(f"job {job_id} did not finish")
|
||||
|
||||
|
||||
def test_connect_is_the_whole_of_the_wiring(jobs):
|
||||
"""Routes, socket and a running worker, from one call.
|
||||
|
||||
The app below has nothing else: no lifespan, no broadcaster, no router of
|
||||
its own. A job enqueued over HTTP runs to `done` because the worker came up
|
||||
with the app, and the socket hands a new tab the current state on connect.
|
||||
"""
|
||||
app = FastAPI()
|
||||
queue = connect(app, jobs, {"app": "tiny"}, None)
|
||||
assert app.state.queue is queue
|
||||
|
||||
with TestClient(app) as client:
|
||||
made = client.post("/api/jobs", json={"kind": "w_hello",
|
||||
"payload": {"name": "you"}})
|
||||
assert made.status_code == 202, made.text
|
||||
ended = settled(client, made.json()["id"])
|
||||
assert ended["state"] == "done"
|
||||
assert ended["result"] == {"said": "hello you", "config": "tiny"}
|
||||
|
||||
with client.websocket_connect("/ws/jobs") as socket:
|
||||
hello = socket.receive_json()
|
||||
assert hello["type"] == "jobs"
|
||||
assert any(row["id"] == ended["id"] for row in hello["recent"])
|
||||
|
||||
assert queue._task is None or queue._task.done(), "and it stopped with the app"
|
||||
|
||||
|
||||
def test_connect_keeps_the_apps_own_lifespan(jobs):
|
||||
"""An app with startup code of its own keeps it, and the worker is already
|
||||
up when that code runs — so startup code may enqueue."""
|
||||
seen: list[str] = []
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
seen.append("app started")
|
||||
app.state.queue.enqueue("w_hello", {"name": "from startup"})
|
||||
yield
|
||||
seen.append("app stopped")
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
connect(app, jobs, {"app": "tiny"}, None)
|
||||
|
||||
with TestClient(app) as client:
|
||||
# By id: the snapshot reads live and recent jobs separately, so a job
|
||||
# that finishes in between is listed in both.
|
||||
snap = client.get("/api/jobs").json()
|
||||
(job_id,) = {r["id"] for r in snap["queued"] + snap["running"]
|
||||
+ snap["recent"] if r["kind"] == "w_hello"}
|
||||
assert settled(client, job_id)["state"] == "done"
|
||||
assert seen == ["app started", "app stopped"]
|
||||
|
||||
|
||||
def test_the_element_is_served_by_the_queues_own_router():
|
||||
"""Two lines in a template and a new app has a job list. Served from a
|
||||
route, because a router cannot mount a static directory and needing the
|
||||
app to mount one is the setup the element exists to remove."""
|
||||
app = FastAPI()
|
||||
app.include_router(jobs_router)
|
||||
answer = TestClient(app).get("/_jobs/jobs.js")
|
||||
|
||||
assert answer.status_code == 200
|
||||
assert answer.headers["content-type"].startswith("text/javascript")
|
||||
assert 'customElements.define("desk-jobs"' in answer.text
|
||||
@@ -0,0 +1,209 @@
|
||||
"""What keeps a queue healthy for months: retention, bounded results, whose
|
||||
retry policy wins, draining, and two real processes on one file.
|
||||
|
||||
python -m pytest pipeline/tests/test_housekeeping.py -q
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from pipeline import NULL, Batch, Bound, JobQueue, Jobs, Task, job
|
||||
from pipeline.models import Job
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
@job("h_plain", "A job with a policy of its own", max_attempts=3, retry_delay=7)
|
||||
def h_plain(config, store, *, context=NULL, i: int = 0) -> dict:
|
||||
return {"i": i}
|
||||
|
||||
|
||||
@job("h_big", "Returning far too much", max_attempts=3)
|
||||
def h_big(config, store, *, context=NULL) -> dict:
|
||||
return {"rows": ["x" * 100] * 2000, "count": 2000}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "h.db")
|
||||
|
||||
|
||||
def got(found: Job | None) -> Job:
|
||||
assert found is not None
|
||||
return found
|
||||
|
||||
|
||||
def age(jobs: Jobs, job_id: int, days: float) -> None:
|
||||
"""Pretend a job finished `days` ago, and its runs were minted then too."""
|
||||
when = (datetime.now(UTC) - timedelta(days=days)).isoformat()
|
||||
with closing(sqlite3.connect(jobs.path)) as conn, conn:
|
||||
conn.execute("UPDATE jobs SET finished_at=? WHERE id=?", (when, job_id))
|
||||
conn.execute("UPDATE runs SET created_at=? WHERE id IN "
|
||||
"(SELECT cascade_id FROM jobs WHERE id=? UNION "
|
||||
" SELECT task_id FROM jobs WHERE id=?)", (when, job_id, job_id))
|
||||
|
||||
|
||||
def runs(jobs: Jobs) -> int:
|
||||
with closing(sqlite3.connect(jobs.path)) as conn:
|
||||
return conn.execute("SELECT COUNT(*) FROM runs").fetchone()[0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- retention
|
||||
|
||||
def test_purge_forgets_old_history_but_not_the_last_of_each_kind(jobs):
|
||||
"""Without this both tables grow for as long as the app runs.
|
||||
|
||||
What survives an old cutoff: anything unfinished, and the newest ending of
|
||||
each kind in each state — because `last(kind)` is how an app says "last
|
||||
pulled five weeks ago", and deleting it would make that "never".
|
||||
"""
|
||||
old = [got(jobs.enqueue("h_plain")) for _ in range(3)]
|
||||
for queued in old:
|
||||
jobs.finish(queued.id, "done")
|
||||
age(jobs, queued.id, days=40)
|
||||
waiting = got(jobs.enqueue("h_plain")) # never finished
|
||||
|
||||
gone = jobs.purge(30 * 86400)
|
||||
|
||||
assert gone == 2, "two of the three old ones went"
|
||||
assert got(jobs.last("h_plain")).id == old[-1].id, "the newest stayed"
|
||||
assert got(jobs.get(waiting.id)).state == "queued", "unfinished work is not history"
|
||||
|
||||
|
||||
def test_purge_leaves_recent_history_alone(jobs):
|
||||
recent = got(jobs.enqueue("h_plain"))
|
||||
jobs.finish(recent.id, "done")
|
||||
before = runs(jobs)
|
||||
|
||||
assert jobs.purge(30 * 86400) == 0
|
||||
assert jobs.get(recent.id) is not None
|
||||
assert runs(jobs) == before
|
||||
|
||||
|
||||
def test_purge_drops_runs_nothing_points_at_any_more(jobs):
|
||||
first = got(jobs.enqueue("h_plain"))
|
||||
second = got(jobs.enqueue("h_plain"))
|
||||
for queued in (first, second):
|
||||
jobs.finish(queued.id, "done")
|
||||
age(jobs, queued.id, days=40)
|
||||
before = runs(jobs)
|
||||
|
||||
jobs.purge(30 * 86400)
|
||||
|
||||
assert runs(jobs) < before, "the purged job's cascade and task went with it"
|
||||
assert jobs.run(second.task_id) is not None, "the kept job's runs did not"
|
||||
|
||||
|
||||
# ----------------------------------------------------------- bounded results
|
||||
|
||||
def test_a_result_too_large_fails_the_job_loudly_and_once(jobs):
|
||||
"""Past `MAX_RESULT` the job fails rather than losing data quietly: the row
|
||||
says why and how to fix it, a notice goes to whoever is watching, and it is
|
||||
not retried — the same handler would return the same result."""
|
||||
said: list = []
|
||||
|
||||
async def on_notice(notice) -> None:
|
||||
said.append(notice)
|
||||
|
||||
queued = got(jobs.enqueue("h_big", max_attempts=3))
|
||||
JobQueue(jobs, None, None, on_notice=on_notice).drain(timeout=10)
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert ended.state == "failed"
|
||||
assert "ResultTooLarge" in ended.error and "store" in ended.error, ended.error
|
||||
assert "count, rows" in ended.error, "it names what was returned"
|
||||
assert ended.result == {}, "nothing of it was written"
|
||||
assert ended.attempt == 1, "and it was not tried again"
|
||||
assert [n.code for n in said] == ["job.failed"], "and somebody was told"
|
||||
|
||||
|
||||
def test_a_result_at_the_limit_is_kept_whole(jobs):
|
||||
"""The limit is a line, not a tax: everything under it is stored as is."""
|
||||
queued = got(jobs.enqueue("h_plain", payload={"i": 1}))
|
||||
JobQueue(jobs, None, None).drain(timeout=10)
|
||||
assert got(jobs.get(queued.id)).result == {"i": 1}
|
||||
|
||||
|
||||
# -------------------------------------------------------- whose policy wins
|
||||
|
||||
def test_a_task_with_plain_payloads_keeps_the_kinds_own_retry_policy(jobs):
|
||||
"""`@job(max_attempts=3)` has to mean something when a task queues it.
|
||||
|
||||
It used not to: a plain payload was given the default bound explicitly,
|
||||
which silently overrode whatever the kind declared.
|
||||
"""
|
||||
queue = JobQueue(jobs, None, None)
|
||||
Task(kind="h_plain", finder=lambda c, s: [{"i": 1}]).start(None, None, queue)
|
||||
|
||||
(child,) = [j for j in jobs.snapshot().queued if j.kind == "h_plain"]
|
||||
assert (child.max_attempts, child.retry_delay) == (3, 7.0)
|
||||
|
||||
|
||||
def test_a_batch_with_its_own_bound_overrides_the_kind(jobs):
|
||||
"""The other half: a batch cut from one plugin profile carries that
|
||||
profile's policy, and that wins, because the outlet knows its publisher."""
|
||||
queue = JobQueue(jobs, None, None)
|
||||
Task(kind="h_plain", finder=lambda c, s: [
|
||||
Batch({"i": 1}, Bound(max_attempts=5, delay=60))]).start(None, None, queue)
|
||||
|
||||
(child,) = [j for j in jobs.snapshot().queued if j.kind == "h_plain"]
|
||||
assert (child.max_attempts, child.retry_delay) == (5, 60.0)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ draining
|
||||
|
||||
def test_drain_runs_everything_queued_and_stops(jobs):
|
||||
"""The batch shape: a script or a cron entry with no web app around it."""
|
||||
ids = [got(jobs.enqueue("h_plain", payload={"i": i})).id for i in range(5)]
|
||||
|
||||
assert JobQueue(jobs, None, None).drain(timeout=10) is True
|
||||
assert {got(jobs.get(i)).state for i in ids} == {"done"}
|
||||
|
||||
|
||||
# ------------------------------------------------- two processes, one file
|
||||
|
||||
def test_two_worker_processes_never_run_the_same_job_twice(tmp_path):
|
||||
"""The claim the table-as-queue design rests on, tested the only way that
|
||||
means anything: two separate `python -m pipeline.worker` processes draining
|
||||
one file at once. Every job leaves a marker file named after itself and the
|
||||
process that ran it; each job must have exactly one.
|
||||
"""
|
||||
db = tmp_path / "shared.db"
|
||||
marks = tmp_path / "marks"
|
||||
marks.mkdir()
|
||||
jobs = Jobs(db)
|
||||
for i in range(40):
|
||||
jobs.enqueue("mp_mark", payload={"i": i})
|
||||
|
||||
env = dict(os.environ, PYTHONPATH=str(ROOT))
|
||||
argv = [sys.executable, "-m", "pipeline.worker", "--db", str(db),
|
||||
"--bootstrap", "pipeline.tests.isolated_handlers:marking",
|
||||
"--bootstrap-arg", str(marks), "--workers", "cpu=2", "--until-idle"]
|
||||
workers = [subprocess.Popen(argv, env=env, cwd=ROOT,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
for _ in range(2)]
|
||||
for worker in workers:
|
||||
out, _ = worker.communicate(timeout=120)
|
||||
assert worker.returncode == 0, out.decode(errors="replace")[-2000:]
|
||||
|
||||
ran: dict[int, list[str]] = {}
|
||||
for mark in marks.iterdir():
|
||||
i, pid = mark.name.split("-")[:2]
|
||||
ran.setdefault(int(i), []).append(pid)
|
||||
assert sorted(ran) == list(range(40)), "every job ran"
|
||||
doubled = {i: pids for i, pids in ran.items() if len(pids) > 1}
|
||||
assert not doubled, f"and none ran twice: {doubled}"
|
||||
assert {j.state for j in jobs.snapshot().recent} == {"done"}
|
||||
@@ -0,0 +1,320 @@
|
||||
"""Ending work that will not end itself, and how data moves between jobs.
|
||||
|
||||
Four claims, and they fail independently:
|
||||
|
||||
a deadline is honest `timeout` without `isolate` is refused at
|
||||
registration rather than promising a kill a
|
||||
thread cannot deliver.
|
||||
a process can be killed an isolated job that never calls `check()` is
|
||||
still over when its deadline passes, and its
|
||||
row says so.
|
||||
a chain reads, never writes `upstream()` is how B gets A's output — off A's
|
||||
row, so B's own row was complete when written.
|
||||
a fan-out is one call `fan_out` queues the children and the job that
|
||||
reports on them, and an app never writes the
|
||||
second one.
|
||||
|
||||
python3 -m pytest tests/test_isolation.py -q
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from pipeline import NULL, Cancelled, JobQueue, Jobs, RunContext, job, spawn
|
||||
from pipeline.models import Job
|
||||
|
||||
# Imported for the `@job` registrations, which the *parent* needs as much as
|
||||
# the child does: the worker reads `isolate` and `timeout` off the spec before
|
||||
# it decides to start a process at all. An app imports its handlers in every
|
||||
# entry point for this reason; here that import is this line.
|
||||
from pipeline.tests import isolated_handlers # noqa: F401
|
||||
|
||||
BOOTSTRAP = "pipeline.tests.isolated_handlers:bootstrap"
|
||||
|
||||
|
||||
@job("t_child", "A child of a fan-out")
|
||||
def t_child(config, store, *, context=NULL, i: int = 0) -> dict:
|
||||
"""Queued by `fan_out` and never run in these tests — what is under test is
|
||||
the shape of the rows it made, which is the part an app never writes."""
|
||||
return {"i": i}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "t.db")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def child_path(monkeypatch):
|
||||
"""Put the folder holding `pipeline` where a child process can import it.
|
||||
|
||||
What an installed app gets for free and a checkout does not: the child is a
|
||||
fresh interpreter, and the checkout is on *our* path only because pytest
|
||||
put it there.
|
||||
"""
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
monkeypatch.setenv("PYTHONPATH", str(root))
|
||||
|
||||
|
||||
def drain(queue: JobQueue, *, timeout: float = 20.0) -> None:
|
||||
"""Run the worker until nothing is queued, running, or held by this one."""
|
||||
|
||||
async def go() -> None:
|
||||
await queue.start()
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
snap = queue.jobs.snapshot()
|
||||
if not snap.running and not snap.queued and not queue._running:
|
||||
break
|
||||
await asyncio.sleep(0.02)
|
||||
await queue.stop()
|
||||
|
||||
asyncio.run(go())
|
||||
|
||||
|
||||
def got(found: Job | None) -> Job:
|
||||
assert found is not None
|
||||
return found
|
||||
|
||||
|
||||
# --------------------------------------------------- a deadline is honest
|
||||
|
||||
def test_a_timeout_without_isolation_is_refused():
|
||||
"""The pipeline will not offer a kill it cannot perform.
|
||||
|
||||
A deadline over a job in a thread could only mark the row failed and walk
|
||||
away: the work would carry on holding the core it was killed for holding,
|
||||
and two things would intend to write one row. So the pair is refused where
|
||||
it is declared, with the two ways out named.
|
||||
"""
|
||||
with pytest.raises(ValueError) as refused:
|
||||
@job("t_threaded_deadline", timeout=5.0)
|
||||
def handler(config, store, *, context=NULL):
|
||||
return {}
|
||||
|
||||
said = str(refused.value)
|
||||
assert "isolate=True" in said, "it names the way to get a real deadline"
|
||||
assert "RunContext.spawn" in said, "and the cheaper way, for a command"
|
||||
|
||||
|
||||
def test_isolation_without_a_timeout_is_fine():
|
||||
"""Only the one combination is refused. An isolated job with no deadline
|
||||
runs until it is done or killed by hand, which is every job the pipeline
|
||||
has ever had."""
|
||||
|
||||
@job("t_isolated_no_deadline", isolate=True)
|
||||
def handler(config, store, *, context=NULL):
|
||||
return {}
|
||||
|
||||
from pipeline import JOBS
|
||||
assert JOBS["t_isolated_no_deadline"].isolate
|
||||
assert JOBS["t_isolated_no_deadline"].timeout == 0.0
|
||||
|
||||
|
||||
def test_a_queue_with_no_bootstrap_says_so(jobs):
|
||||
"""An isolated job needs its app rebuilt in the child, and a queue that
|
||||
cannot do it fails the job with the reason rather than with a TypeError
|
||||
from somewhere inside a subprocess."""
|
||||
|
||||
@job("t_needs_bootstrap", isolate=True)
|
||||
def handler(config, store, *, context=NULL):
|
||||
return {}
|
||||
|
||||
queued = got(jobs.enqueue("t_needs_bootstrap"))
|
||||
drain(JobQueue(jobs, None, None)) # no bootstrap=
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert ended.state == "failed"
|
||||
assert "bootstrap" in ended.error, f"it names what is missing: {ended.error}"
|
||||
|
||||
|
||||
# ------------------------------------------------ a process can be killed
|
||||
|
||||
def test_an_isolated_job_runs_somewhere_else_and_its_result_comes_back(jobs, child_path):
|
||||
"""The happy path, and the proof that it was a real process.
|
||||
|
||||
Nothing was pickled to get there: the child was handed a database and a
|
||||
row id and rebuilt the rest. Which is also why its progress is on the row
|
||||
— it wrote it there itself, to the same table the snapshot reads.
|
||||
"""
|
||||
queued = got(jobs.enqueue("iso_ok", payload={"n": 21}))
|
||||
drain(JobQueue(jobs, None, None, bootstrap=BOOTSTRAP))
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert ended.state == "done", ended.error
|
||||
assert ended.result["doubled"] == 42, "the handler's return travelled back"
|
||||
assert ended.result["pid"] != os.getpid(), "and it ran in another process"
|
||||
assert ended.message == "counting to 21", "its progress reached our table"
|
||||
|
||||
|
||||
def test_an_isolated_job_that_raises_fails_with_what_it_said(jobs, child_path):
|
||||
"""A failure across a process boundary reads like any other failure: the
|
||||
row is `failed`, the error is the exception, and the retry bound applies
|
||||
as it would to a handler that raised in a thread."""
|
||||
queued = got(jobs.enqueue("iso_raises"))
|
||||
drain(JobQueue(jobs, None, None, bootstrap=BOOTSTRAP))
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert ended.state == "failed"
|
||||
assert "the model said no" in ended.error, ended.error
|
||||
|
||||
|
||||
def test_a_deadline_ends_a_job_that_never_looks_up(jobs, child_path):
|
||||
"""The claim the whole feature exists for.
|
||||
|
||||
`iso_slow` sleeps for two minutes and never calls `check()`. In a thread
|
||||
that is unkillable and the worker could only lie about having stopped it.
|
||||
In a process of its own it is over in about a second, and the row says
|
||||
why — which is what a person reading the job list needs.
|
||||
"""
|
||||
queued = got(jobs.enqueue("iso_slow"))
|
||||
started = time.monotonic()
|
||||
drain(JobQueue(jobs, None, None, bootstrap=BOOTSTRAP))
|
||||
took = time.monotonic() - started
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert ended.state == "failed", f"it ended: {ended.state}"
|
||||
assert took < 15, f"and on its deadline, not in its own time: {took:.1f}s"
|
||||
|
||||
|
||||
# ------------------------------------------------------ spawn, on its own
|
||||
|
||||
def test_spawn_kills_the_process_when_the_job_is_cancelled():
|
||||
"""Kill, for a handler whose work is a command.
|
||||
|
||||
Cooperative in name only from the handler's side: it is blocked in
|
||||
`spawn`, not in a loop with a `check()` in it, and the child is killed
|
||||
outright. What makes that safe is that the thread was only sleeping.
|
||||
"""
|
||||
asked = [0]
|
||||
|
||||
def is_cancelled() -> bool:
|
||||
asked[0] += 1
|
||||
return asked[0] > 2 # pressed, a moment in
|
||||
|
||||
context = RunContext(is_cancelled=is_cancelled)
|
||||
started = time.monotonic()
|
||||
with pytest.raises(Cancelled):
|
||||
context.spawn([sys.executable, "-c", "import time; time.sleep(120)"])
|
||||
assert time.monotonic() - started < 10, "it did not wait the two minutes"
|
||||
|
||||
|
||||
def test_spawn_raises_on_its_deadline_and_leaves_nothing_running():
|
||||
"""The deadline, and that the process is gone after it — not merely
|
||||
abandoned, which is the failure mode the whole thing is against."""
|
||||
started = time.monotonic()
|
||||
with pytest.raises(TimeoutError):
|
||||
spawn([sys.executable, "-c", "import time; time.sleep(120)"], timeout=0.5)
|
||||
assert time.monotonic() - started < 10
|
||||
|
||||
|
||||
def test_spawn_hands_back_the_exit_code_rather_than_raising():
|
||||
"""A non-zero exit is not an error here. Plenty of commands say something
|
||||
useful with one, and only the handler knows which."""
|
||||
done = spawn([sys.executable, "-c", "import sys; sys.exit(3)"])
|
||||
assert done.returncode == 3
|
||||
|
||||
|
||||
def test_spawn_survives_a_command_that_writes_more_than_a_pipe_holds():
|
||||
"""Output goes to a file, not a `PIPE`, and this is why.
|
||||
|
||||
A child that fills the pipe buffer blocks on the write while the poll loop
|
||||
blocks on the read, and neither side is wrong — the job simply never ends.
|
||||
A megabyte is far past any platform's buffer.
|
||||
"""
|
||||
done = spawn([sys.executable, "-c", "print('x' * 1000000)"], timeout=60)
|
||||
assert done.returncode == 0
|
||||
assert len(done.stdout) >= 1000000
|
||||
|
||||
|
||||
# ---------------------------------------------- a chain reads, never writes
|
||||
|
||||
def test_upstream_merges_what_the_parents_returned():
|
||||
"""How B gets A's output: off A's row, not out of B's payload.
|
||||
|
||||
Which is the property worth having. B's row was complete when it was
|
||||
written, so it can carry a dedup key, a retry can re-run it from itself,
|
||||
and no worker can claim it between an earlier job's write and its commit.
|
||||
"""
|
||||
a = Job(id=1, kind="a", created_at="", result={"html": "<p>hi</p>", "code": 200})
|
||||
b = Job(id=2, kind="b", created_at="", result={"code": 204})
|
||||
context = RunContext(waited_for=(a, b))
|
||||
|
||||
assert context.upstream() == {"html": "<p>hi</p>", "code": 204}, "later parents win"
|
||||
|
||||
|
||||
def test_upstream_is_empty_under_null_so_a_chained_handler_is_still_callable():
|
||||
"""The one line that keeps a chained handler an ordinary function.
|
||||
|
||||
Every other handler in the pipeline can be called from a terminal. One
|
||||
that read `waited_for[0].result` directly could not — it would raise
|
||||
IndexError under `NULL` — so `upstream()` answers `{}` and the handler
|
||||
falls back to the keyword it declared.
|
||||
"""
|
||||
assert NULL.upstream() == {}
|
||||
|
||||
@job("t_chained")
|
||||
def handler(config, store, *, context=NULL, html: str = "") -> dict:
|
||||
return {"html": html or context.upstream().get("html", "")}
|
||||
|
||||
assert handler(None, None, html="passed by hand") == {"html": "passed by hand"}
|
||||
assert handler(None, None) == {"html": ""}, "and it does not blow up with neither"
|
||||
|
||||
|
||||
# ----------------------------------------------------- a fan-out is one call
|
||||
|
||||
def test_fan_out_queues_the_children_and_the_job_that_reports_on_them(jobs):
|
||||
"""One call, and the reporting job is not the app's to write.
|
||||
|
||||
The children get a run of their own inside the caller's cascade, and the
|
||||
report waits on all of them with `after='done'` — the mode that exists
|
||||
precisely so a job reporting that one of three failed is not held back by
|
||||
that failure.
|
||||
"""
|
||||
queue = JobQueue(jobs, None, None)
|
||||
parent = got(jobs.enqueue("t_parent"))
|
||||
context = queue._context(parent, [], [])
|
||||
|
||||
children = context.fan_out(
|
||||
"t_child", [{"i": 1}, {"i": 2}, {"i": 3}], then="mod:fn")
|
||||
|
||||
assert len(children) == 3, "it answers with the children's ids"
|
||||
queued = jobs.snapshot().queued
|
||||
report = [j for j in queued if j.kind == "advance"]
|
||||
assert len(report) == 1, "and queued exactly one job to report on them"
|
||||
assert sorted(report[0].depends_on) == sorted(children)
|
||||
assert report[0].after == "done"
|
||||
assert report[0].payload["callback"] == "mod:fn"
|
||||
assert {j.task_id for j in queued if j.id in children} == {report[0].task_id}, \
|
||||
"children and report are one run"
|
||||
|
||||
|
||||
def test_fan_out_queues_nothing_under_null_and_says_so():
|
||||
"""A handler called directly is being run by somebody who is standing
|
||||
there, and the useful answer is to do its own work and report that nothing
|
||||
was queued behind it — the same answer `task` gives, for the same reason."""
|
||||
assert NULL.fan_out("t_child", [{"i": 1}]) == ()
|
||||
|
||||
|
||||
def test_a_fan_out_carries_what_the_caller_puts_on_it(jobs):
|
||||
"""`carries` goes *around* the children to the callback: static input from
|
||||
whoever asked, as against `Tally.results`, which is what the children
|
||||
themselves returned."""
|
||||
queue = JobQueue(jobs, None, None)
|
||||
parent = got(jobs.enqueue("t_parent"))
|
||||
context = queue._context(parent, [], [])
|
||||
|
||||
context.fan_out("t_child", [{"i": 1}], then="mod:fn", carries={"since": "monday"})
|
||||
|
||||
report = [j for j in jobs.snapshot().queued if j.kind == "advance"]
|
||||
assert report[0].payload["payload"] == {"since": "monday"}
|
||||
@@ -0,0 +1,134 @@
|
||||
"""The notice log: said once, kept until cleared, forgotten when it stops.
|
||||
|
||||
A notice used to be a line that faded in twelve seconds in whichever tab was
|
||||
open, and was raised as often as not while none was. These hold the three
|
||||
things the table is for: a repeat is the same row and not news, a cleared row
|
||||
makes the next one news again, and what nobody clears goes when it has stopped
|
||||
being said rather than when it was first said.
|
||||
|
||||
python -m pytest pipeline/tests/test_notice_log.py -q
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from pipeline import NULL, JobQueue, Jobs, Notice, RunContext, job
|
||||
from pipeline.web import connect
|
||||
|
||||
|
||||
@job("n_refused", "Meeting the same refusal as last time")
|
||||
def n_refused(config, store, *, context: RunContext = NULL) -> dict:
|
||||
context.notice(Notice(code="agent.refused", key="Worm hits npm",
|
||||
text="No abstract: Worm hits npm", fix="Nothing to do."))
|
||||
return {}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "n.db")
|
||||
|
||||
|
||||
def refusal(text: str = "No abstract: Worm hits npm") -> Notice:
|
||||
return Notice(code="agent.refused", key="Worm hits npm", text=text,
|
||||
where="the tagging lines in the server log")
|
||||
|
||||
|
||||
def test_the_same_notice_again_is_the_same_row_with_a_count(jobs):
|
||||
first = jobs.record_notice(refusal())
|
||||
again = jobs.record_notice(refusal("No abstract, again: Worm hits npm"))
|
||||
other = jobs.record_notice(Notice(code="agent.refused", key="Other", text="x"))
|
||||
|
||||
assert (first.count, again.count) == (1, 2), "one is news, two is a repeat"
|
||||
assert again.id == first.id and other.id != first.id, "keyed by code and key"
|
||||
assert again.first_at == first.first_at and again.last_at >= first.last_at
|
||||
assert again.text.startswith("No abstract, again"), "the newer wording is kept"
|
||||
assert again.where == first.where, "and `where` survives its SQL column name"
|
||||
assert [n.key for n in jobs.notices()] == ["Other", "Worm hits npm"]
|
||||
|
||||
|
||||
def test_clearing_one_makes_the_next_occurrence_news_again(jobs):
|
||||
said = jobs.record_notice(refusal())
|
||||
|
||||
assert jobs.clear_notice(said.id)
|
||||
assert not jobs.clear_notice(said.id), "gone already is an answer, not an error"
|
||||
assert jobs.record_notice(refusal()).count == 1
|
||||
|
||||
|
||||
def test_a_notice_is_forgotten_when_it_stops_being_said_not_when_it_began(jobs):
|
||||
"""By `last_at`. A feed that has failed nightly for a month is an old row
|
||||
that was true an hour ago, and dropping it would make tonight's failure
|
||||
read as the first."""
|
||||
chronic = jobs.record_notice(Notice(code="source.failed", key="rss", text="x"))
|
||||
over = jobs.record_notice(Notice(code="source.failed", key="old", text="y"))
|
||||
month = (datetime.now(UTC) - timedelta(days=30)).isoformat()
|
||||
with closing(sqlite3.connect(jobs.path)) as conn, conn:
|
||||
conn.execute("UPDATE notices SET first_at=? WHERE id=?", (month, chronic.id))
|
||||
conn.execute("UPDATE notices SET first_at=?, last_at=? WHERE id=?",
|
||||
(month, month, over.id))
|
||||
|
||||
assert jobs.purge_notices(14 * 86400) == 1
|
||||
assert [n.key for n in jobs.notices()] == ["rss"]
|
||||
|
||||
|
||||
def test_a_job_that_repeats_itself_is_heard_as_a_repeat(jobs):
|
||||
"""The whole path. Every notice still reaches the listener — what
|
||||
interrupts a person is the interface's business — and the second carries
|
||||
the count that says it is nothing new."""
|
||||
heard: list = []
|
||||
|
||||
async def on_notice(notice) -> None:
|
||||
heard.append(notice)
|
||||
|
||||
for _ in range(2):
|
||||
jobs.enqueue("n_refused")
|
||||
JobQueue(jobs, None, None, on_notice=on_notice).drain(timeout=10)
|
||||
|
||||
assert [n.count for n in heard] == [1, 2]
|
||||
assert len(jobs.notices()) == 1
|
||||
|
||||
|
||||
def test_a_log_that_cannot_be_written_does_not_cost_the_notice(jobs, monkeypatch):
|
||||
heard: list = []
|
||||
|
||||
async def on_notice(notice) -> None:
|
||||
heard.append(notice)
|
||||
|
||||
def broken(notice):
|
||||
raise sqlite3.OperationalError("database is locked")
|
||||
|
||||
monkeypatch.setattr(jobs, "record_notice", broken)
|
||||
done = jobs.enqueue("n_refused")
|
||||
JobQueue(jobs, None, None, on_notice=on_notice).drain(timeout=10)
|
||||
|
||||
assert [n.code for n in heard] == ["agent.refused"]
|
||||
assert jobs.get(done.id).state == "done", "and the job that said it is unharmed"
|
||||
|
||||
|
||||
def test_the_log_is_read_and_cleared_over_http(jobs):
|
||||
app = FastAPI()
|
||||
connect(app, jobs, None, None)
|
||||
jobs.record_notice(refusal())
|
||||
jobs.record_notice(Notice(code="source.failed", key="rss", text="rss failed"))
|
||||
|
||||
with TestClient(app) as client:
|
||||
listed = client.get("/api/notices").json()
|
||||
assert [n["code"] for n in listed] == ["source.failed", "agent.refused"]
|
||||
assert listed[1]["where"] == "the tagging lines in the server log"
|
||||
|
||||
assert client.delete(f"/api/notices/{listed[0]['id']}").json() == {"cleared": True}
|
||||
assert client.delete(f"/api/notices/{listed[0]['id']}").json() == {"cleared": False}
|
||||
assert client.delete("/api/notices").json() == {"cleared": 1}
|
||||
assert client.get("/api/notices").json() == []
|
||||
@@ -0,0 +1,355 @@
|
||||
"""Waiting to be tried again, without anything being alive while it waits.
|
||||
|
||||
The claim under test is that a job between attempts costs a row and nothing
|
||||
else. No worker, no thread, no lease, no timer — so the tests that matter here
|
||||
are the ones that ask what *else* the queue can do while a job waits, and what
|
||||
survives the process that scheduled the wait.
|
||||
|
||||
Each of the four properties step 12 asks for is pinned on its own, because
|
||||
they fail independently: a bound that is off by one still waits correctly, a
|
||||
`run_after` that never reaches the table still retries in one process, and a
|
||||
Kill that cannot reach a waiting row is invisible until somebody presses it.
|
||||
|
||||
python3 -m pytest tests/test_retries.py -q
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from pipeline.models import Job
|
||||
from pipeline import NULL, JobQueue, Jobs, Retry, job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
"""The pipeline's table on a file of its own. No app, no store — nothing
|
||||
here needs one, which is the point of the table owning its own schema."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "t.db")
|
||||
|
||||
|
||||
def made(queued: Job | None) -> Job:
|
||||
"""The job an enqueue made. `enqueue` answers None for a killed run, which
|
||||
in these tests nothing is."""
|
||||
assert queued is not None
|
||||
return queued
|
||||
|
||||
|
||||
def got(found: Job | None) -> Job:
|
||||
"""The job a read must have found — a None here is a test failure."""
|
||||
assert found is not None
|
||||
return found
|
||||
|
||||
|
||||
def lease(jobs: Jobs, job_id: int) -> tuple[str | None, str | None]:
|
||||
"""The `started_at` and `seen_at` of one job: whether anybody has it."""
|
||||
with closing(sqlite3.connect(jobs.path)) as conn:
|
||||
row = conn.execute(
|
||||
"SELECT started_at, seen_at FROM jobs WHERE id=?", (job_id,)).fetchone()
|
||||
return row[0], row[1]
|
||||
|
||||
|
||||
def bring_forward(jobs: Jobs, job_id: int) -> None:
|
||||
"""Move a waiting job's next attempt into the past, as if its wait were
|
||||
over. What the clock does, done in one statement so a test of an hour's
|
||||
backoff does not take an hour."""
|
||||
with closing(sqlite3.connect(jobs.path)) as conn, conn:
|
||||
conn.execute("UPDATE jobs SET run_after=? WHERE id=?",
|
||||
((datetime.now(UTC) - timedelta(hours=1)).isoformat(), job_id))
|
||||
|
||||
|
||||
def drain(queue: JobQueue, *, timeout: float = 5.0) -> None:
|
||||
"""Run the worker until the table is empty and this worker holds nothing.
|
||||
|
||||
Both halves, because a job's last acts belong to its handler thread and
|
||||
between them the table reads as idle while it is not. A job waiting for its
|
||||
next attempt keeps this going, since it is still `queued` — which is the
|
||||
behaviour the retry tests are leaning on.
|
||||
"""
|
||||
|
||||
async def go() -> None:
|
||||
await queue.start()
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
snap = queue.jobs.snapshot()
|
||||
if not snap.running and not snap.queued and not queue._running:
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
await queue.stop()
|
||||
|
||||
asyncio.run(go())
|
||||
|
||||
|
||||
# ------------------------------------------------------- not holding a place
|
||||
|
||||
def test_a_failed_attempt_goes_back_to_the_queue_with_a_time_on_it(jobs):
|
||||
"""The resting state of a retry, and everything that is true of it."""
|
||||
queued = made(jobs.enqueue("r_wait", max_attempts=3, retry_delay=600))
|
||||
assert got(jobs.claim({})).attempt == 1, "claim counts the attempt"
|
||||
|
||||
again = got(jobs.retry(queued.id, error="RuntimeError: the feed was down"))
|
||||
assert again.state == "queued", "it is in the queue, not failed"
|
||||
assert again.waiting and not again.exhausted
|
||||
assert again.error == "RuntimeError: the feed was down", "it says why it waits"
|
||||
assert lease(jobs, queued.id) == (None, None), "nobody holds it"
|
||||
assert jobs.running() == [], "and nothing is running it"
|
||||
assert [j.id for j in jobs.waiting()] == [queued.id]
|
||||
assert [j.id for j in jobs.blocked()] == [queued.id]
|
||||
|
||||
|
||||
def test_a_waiting_job_does_not_occupy_a_worker(jobs):
|
||||
"""The whole of why the wait is a column. One cpu worker, one job waiting
|
||||
ten minutes for its next attempt, and the worker is free the entire time —
|
||||
a handler that slept would have held it and stopped the queue dead."""
|
||||
waiting = made(jobs.enqueue("r_slot", max_attempts=2, retry_delay=600))
|
||||
assert jobs.claim({"cpu": 1}) is not None
|
||||
jobs.retry(waiting.id, error="not this time")
|
||||
|
||||
other = made(jobs.enqueue("r_other"))
|
||||
assert got(jobs.claim({"cpu": 1})).id == other.id, "the slot was free"
|
||||
assert got(jobs.get(waiting.id)).state == "queued"
|
||||
|
||||
|
||||
def test_a_job_queued_for_later_is_not_claimed_before_its_time(jobs):
|
||||
"""`delay` on the enqueue itself: the durable form of "in ten minutes"."""
|
||||
later = made(jobs.enqueue("r_later", delay=600))
|
||||
assert later.waiting
|
||||
assert jobs.claim({}) is None
|
||||
bring_forward(jobs, later.id)
|
||||
assert got(jobs.claim({})).id == later.id
|
||||
|
||||
|
||||
def test_what_waits_on_a_retrying_job_goes_on_waiting(jobs):
|
||||
"""A retry is still live work, so a dependent is held rather than settled.
|
||||
|
||||
The alternative would be the worse half of both: a batch that is coming
|
||||
back in a minute counted among the ones that did not land, and a final job
|
||||
reporting on a task that is not over.
|
||||
"""
|
||||
batch = made(jobs.enqueue("r_batch", max_attempts=2, retry_delay=600))
|
||||
final = made(jobs.enqueue("r_final", depends_on=[batch.id], after="done"))
|
||||
jobs.claim({})
|
||||
jobs.retry(batch.id, error="again in a minute")
|
||||
|
||||
assert [j.id for j in jobs.blocked()] == [batch.id, final.id]
|
||||
bring_forward(jobs, batch.id)
|
||||
assert got(jobs.claim({})).id == batch.id, "the batch goes first"
|
||||
jobs.finish(batch.id, "done")
|
||||
assert got(jobs.claim({})).id == final.id
|
||||
|
||||
|
||||
# --------------------------------------------------------------- the bounds
|
||||
|
||||
def test_the_bound_counts_the_first_attempt(jobs):
|
||||
"""`max_attempts=3` is three runs of the handler, not three retries."""
|
||||
runs = []
|
||||
|
||||
@job("r_bounded", "Bounded", max_attempts=3)
|
||||
def handler(config, store, *, context=NULL):
|
||||
runs.append(1)
|
||||
raise RuntimeError("this was never going to work")
|
||||
|
||||
queue = JobQueue(jobs, None, None)
|
||||
queued = made(queue.enqueue("r_bounded"))
|
||||
drain(queue)
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert len(runs) == 3, "it ran exactly as many times as it was allowed"
|
||||
assert ended.state == "failed" and ended.attempt == 3
|
||||
assert ended.exhausted, "and says the policy ran out rather than merely failing"
|
||||
|
||||
|
||||
def test_a_job_with_no_policy_fails_at_the_first_exception(jobs):
|
||||
"""The default is one attempt, which is every job queued before any of
|
||||
this existed. Exhausted is not what that is: nothing was used up."""
|
||||
runs = []
|
||||
|
||||
@job("r_unbounded", "Unbounded")
|
||||
def handler(config, store, *, context=NULL):
|
||||
runs.append(1)
|
||||
raise RuntimeError("no")
|
||||
|
||||
queue = JobQueue(jobs, None, None)
|
||||
queued = made(queue.enqueue("r_unbounded"))
|
||||
drain(queue)
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert len(runs) == 1
|
||||
assert ended.state == "failed" and not ended.exhausted
|
||||
|
||||
|
||||
def test_an_attempt_that_lands_stops_the_retrying(jobs):
|
||||
"""The case the bound exists for: the second go works, and the job is done
|
||||
rather than carrying a memory of having failed once."""
|
||||
runs = []
|
||||
|
||||
@job("r_second_time", "Second time", max_attempts=4)
|
||||
def handler(config, store, *, context=NULL):
|
||||
runs.append(1)
|
||||
if len(runs) < 2:
|
||||
raise RuntimeError("the first one never works")
|
||||
return {"runs": len(runs)}
|
||||
|
||||
queue = JobQueue(jobs, None, None)
|
||||
queued = made(queue.enqueue("r_second_time"))
|
||||
drain(queue)
|
||||
|
||||
ended = got(jobs.get(queued.id))
|
||||
assert ended.state == "done" and ended.result == {"runs": 2}
|
||||
assert ended.attempt == 2 and not ended.exhausted
|
||||
|
||||
|
||||
def test_the_wait_lengthens_with_the_backoff(jobs):
|
||||
"""Doubling, read off the two stamps rather than waited out."""
|
||||
queued = made(jobs.enqueue("r_backoff", max_attempts=4,
|
||||
retry_delay=100, retry_backoff=2.0))
|
||||
|
||||
waits = []
|
||||
for _ in range(3):
|
||||
assert jobs.claim({}) is not None
|
||||
again = got(jobs.retry(queued.id, error="again"))
|
||||
waits.append(
|
||||
datetime.fromisoformat(again.run_after) - datetime.now(UTC))
|
||||
bring_forward(jobs, queued.id)
|
||||
|
||||
seconds = [w.total_seconds() for w in waits]
|
||||
assert [round(s / 100) for s in seconds] == [1, 2, 4]
|
||||
|
||||
|
||||
def test_a_server_hint_replaces_the_backoff(jobs):
|
||||
"""Honouring what the other end said, rather than guessing at it.
|
||||
|
||||
The bound is not part of the bargain: a hint moves the next attempt, it
|
||||
does not add one.
|
||||
"""
|
||||
queued = made(jobs.enqueue("r_hinted", max_attempts=3, retry_delay=600))
|
||||
jobs.claim({})
|
||||
again = got(jobs.retry(queued.id, error="429", delay=5))
|
||||
waited = datetime.fromisoformat(again.run_after) - datetime.now(UTC)
|
||||
assert 0 < waited.total_seconds() <= 5
|
||||
|
||||
|
||||
def test_a_handler_can_name_its_own_wait(jobs):
|
||||
"""`Retry(after=...)` from inside the handler, which is where a
|
||||
`Retry-After` header is actually read."""
|
||||
|
||||
@job("r_asked", "Asked to come back", max_attempts=3, retry_delay=600)
|
||||
def handler(config, store, *, context=NULL):
|
||||
raise Retry("rate limited", after=7)
|
||||
|
||||
queue = JobQueue(jobs, None, None)
|
||||
queued = made(queue.enqueue("r_asked"))
|
||||
|
||||
async def one_pass() -> None:
|
||||
await queue.start()
|
||||
deadline = time.monotonic() + 5
|
||||
while time.monotonic() < deadline:
|
||||
if got(jobs.get(queued.id)).waiting:
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
await queue.stop()
|
||||
|
||||
asyncio.run(one_pass())
|
||||
|
||||
waiting = got(jobs.get(queued.id))
|
||||
assert waiting.waiting and waiting.attempt == 1
|
||||
assert "rate limited" in waiting.error
|
||||
waited = datetime.fromisoformat(waiting.run_after) - datetime.now(UTC)
|
||||
assert 0 < waited.total_seconds() <= 7, "the handler's number, not the policy's"
|
||||
|
||||
|
||||
def test_asking_to_come_back_does_not_extend_the_bound(jobs):
|
||||
"""A handler cannot grant itself a policy nobody declared."""
|
||||
runs = []
|
||||
|
||||
@job("r_asked_once", "Asked once")
|
||||
def handler(config, store, *, context=NULL):
|
||||
runs.append(1)
|
||||
raise Retry("come back in a second", after=0)
|
||||
|
||||
queue = JobQueue(jobs, None, None)
|
||||
queued = made(queue.enqueue("r_asked_once"))
|
||||
drain(queue)
|
||||
|
||||
assert len(runs) == 1
|
||||
assert got(jobs.get(queued.id)).state == "failed"
|
||||
|
||||
|
||||
# ------------------------------------------------------------- across a stop
|
||||
|
||||
def test_the_next_attempt_survives_the_process_that_scheduled_it(jobs):
|
||||
"""The wait is a column, so it outlives everything that is not the file.
|
||||
|
||||
A second `Jobs` on the same path is what another process sees, and what
|
||||
this one sees after a restart: a queued row it will not take yet, and takes
|
||||
the moment its time comes round. Nothing was handed over, because there
|
||||
was nothing alive to hand.
|
||||
"""
|
||||
queued = made(jobs.enqueue("r_restart", max_attempts=3, retry_delay=600))
|
||||
jobs.claim({})
|
||||
jobs.retry(queued.id, error="the feed was down")
|
||||
|
||||
restarted = Jobs(jobs.path)
|
||||
assert restarted.claim({}) is None, "still waiting, on a table that never saw it fail"
|
||||
assert [j.id for j in restarted.waiting()] == [queued.id]
|
||||
|
||||
bring_forward(restarted, queued.id)
|
||||
taken = got(restarted.claim({}))
|
||||
assert taken.id == queued.id
|
||||
assert taken.attempt == 2, "and it is the second attempt, not the first again"
|
||||
|
||||
|
||||
# ------------------------------------------------------------- cancellation
|
||||
|
||||
def test_cancelling_a_job_that_is_waiting_to_be_retried_ends_it(jobs):
|
||||
"""Kill on a waiting row. It is queued, so it ends here and now — there is
|
||||
no worker to ask and nothing part-way through."""
|
||||
queued = made(jobs.enqueue("r_killable", max_attempts=3, retry_delay=600))
|
||||
jobs.claim({})
|
||||
jobs.retry(queued.id, error="not yet")
|
||||
|
||||
ended = got(jobs.request_cancel(queued.id))
|
||||
assert ended.state == "cancelled"
|
||||
assert jobs.waiting() == []
|
||||
|
||||
|
||||
def test_killing_the_run_takes_a_job_waiting_to_be_retried_with_it(jobs):
|
||||
"""The same for the task and cascade buttons: an hour's wait is not an
|
||||
hour during which a run cannot be stopped."""
|
||||
cascade = jobs.new_run("cascade", "r_run")
|
||||
task = jobs.new_run("task", "r_task")
|
||||
queued = made(jobs.enqueue("r_in_a_run", max_attempts=3, retry_delay=600,
|
||||
cascade_id=cascade, task_id=task))
|
||||
jobs.claim({})
|
||||
jobs.retry(queued.id, error="not yet")
|
||||
|
||||
assert jobs.cancel_run(task) == 1
|
||||
assert got(jobs.get(queued.id)).state == "cancelled"
|
||||
|
||||
|
||||
def test_a_kill_during_the_attempt_is_not_undone_by_the_retry(jobs):
|
||||
"""The one ordering that would make Kill a lie.
|
||||
|
||||
Press it while the attempt runs, the attempt then fails, and a retry that
|
||||
only looked at the bound would put the job straight back in the queue —
|
||||
a button that stopped nothing and said it had.
|
||||
"""
|
||||
queued = made(jobs.enqueue("r_killed_mid", max_attempts=3, retry_delay=0))
|
||||
jobs.claim({})
|
||||
jobs.request_cancel(queued.id)
|
||||
|
||||
ended = got(jobs.retry(queued.id, error="it failed as well"))
|
||||
assert ended.state == "cancelled"
|
||||
assert not ended.waiting
|
||||
@@ -0,0 +1,701 @@
|
||||
"""The scheduler: the five wordings, the table, the firing, and the routes.
|
||||
|
||||
python -m pytest pipeline/tests/test_schedule.py -q
|
||||
|
||||
Three things here are worth knowing before changing anything.
|
||||
|
||||
The daylight-saving tests are the reason `next_after` takes a zone at all, and
|
||||
they are written around a real transition rather than a synthetic one, because
|
||||
the bug they guard is precisely that `+ timedelta(days=1)` looks correct.
|
||||
|
||||
The firing tests call `JobQueue._fire_due` directly. It is a private, and that
|
||||
is deliberate: driving it through `start()` would mean asserting on a loop that
|
||||
ticks on a clock, and a test that sleeps to make a scheduler fire is a test
|
||||
that fails on a busy machine. One test does start a real queue, to prove the
|
||||
tick is wired into the loop at all.
|
||||
|
||||
`SCHEDULES` is a module-level registry, like `JOBS`, so anything here that
|
||||
declares into it does so through the `declared` fixture, which puts it back.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from pipeline import NULL, JobQueue, Jobs, RunContext, job
|
||||
from pipeline.schedule import SCHEDULES, BadCadence, Cadence, declare
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
COPENHAGEN = ZoneInfo("Europe/Copenhagen")
|
||||
|
||||
|
||||
@job("s_tick", "A scheduled tick")
|
||||
def s_tick(config, store, *, context: RunContext = NULL, note: str = "") -> dict:
|
||||
store.append(note)
|
||||
return {"note": note}
|
||||
|
||||
|
||||
@job("s_solo", "One at a time", solo=True)
|
||||
def s_solo(config, store, *, context: RunContext = NULL) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
@job("s_needs", "Needing a payload")
|
||||
def s_needs(config, store, *, context: RunContext = NULL, target: str) -> dict:
|
||||
return {"target": target}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "s.db")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def queue(jobs):
|
||||
"""A queue that is never started: the tests below fire it by hand.
|
||||
|
||||
`tz="UTC"` so a cadence's next run is the same number wherever this runs.
|
||||
"""
|
||||
return JobQueue(jobs, {}, [], tz="UTC")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def declared():
|
||||
"""Borrow the declaration registry and give it back.
|
||||
|
||||
`SCHEDULES` is global, like `JOBS`, and a test that left something in it
|
||||
would give every later test in the session a schedule it never asked for.
|
||||
"""
|
||||
saved = dict(SCHEDULES)
|
||||
SCHEDULES.clear()
|
||||
yield SCHEDULES
|
||||
SCHEDULES.clear()
|
||||
SCHEDULES.update(saved)
|
||||
|
||||
|
||||
def ago(seconds: float) -> str:
|
||||
return (datetime.now(UTC) - timedelta(seconds=seconds)).isoformat()
|
||||
|
||||
|
||||
# ------------------------------------------------------------ the grammar
|
||||
|
||||
WORDINGS = [
|
||||
("every 15m", "every 15m"),
|
||||
("every 6h", "every 6h"),
|
||||
("every 2d", "every 2d"),
|
||||
("daily at 06:00", "daily at 06:00"),
|
||||
("weekly mon 07:00", "weekly mon 07:00"),
|
||||
# Written how people write them, stored how the table stores them.
|
||||
("weekly monday 07:00", "weekly mon 07:00"),
|
||||
("Weekly Sunday 23:59", "weekly sun 23:59"),
|
||||
("EVERY 15 M", "every 15m"),
|
||||
(" daily at 6:05 ", "daily at 06:05"),
|
||||
("every 1d", "every 1d"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("typed,canonical", WORDINGS)
|
||||
def test_every_wording_parses_and_comes_back_canonical(typed, canonical):
|
||||
"""Five forms, one spelling each.
|
||||
|
||||
The round trip is the point: the table stores `str(cadence)`, so two
|
||||
people writing the same schedule two ways end up with one row that reads
|
||||
the same to both of them.
|
||||
"""
|
||||
cadence = Cadence.parse(typed)
|
||||
assert str(cadence) == canonical
|
||||
assert str(Cadence.parse(canonical)) == canonical
|
||||
assert Cadence.parse(typed) == Cadence.parse(canonical)
|
||||
|
||||
|
||||
def test_the_units_are_what_they_say():
|
||||
assert Cadence.parse("every 15m").seconds == 900
|
||||
assert Cadence.parse("every 6h").seconds == 6 * 3600
|
||||
assert Cadence.parse("every 2d").seconds == 2 * 86400
|
||||
# A clock cadence has no fixed length, and says so rather than guessing.
|
||||
assert Cadence.parse("daily at 06:00").seconds == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("wording", [
|
||||
"sometimes", "", "cron 0 6 * * *", "every 15", "every m", "every 0h",
|
||||
"every 15s", "daily 06:00", "daily at 25:00", "daily at 06:60",
|
||||
"weekly 07:00", "weekly funday 07:00", "every 15m please",
|
||||
])
|
||||
def test_a_wording_outside_the_five_is_refused_with_all_five(wording):
|
||||
"""And the refusal is the sentence a person is shown.
|
||||
|
||||
It goes out of the API as `{"detail": ...}` and into a toast, so it has to
|
||||
say what to type instead rather than merely that this was wrong.
|
||||
"""
|
||||
with pytest.raises(ValueError) as refused:
|
||||
Cadence.parse(wording)
|
||||
said = str(refused.value)
|
||||
assert isinstance(refused.value, BadCadence)
|
||||
for form in ("every 15m", "every 6h", "every 2d",
|
||||
"daily at 06:00", "weekly mon 07:00"):
|
||||
assert form in said
|
||||
|
||||
|
||||
# ------------------------------------------------------------- next_after
|
||||
|
||||
def test_next_after_is_strictly_after():
|
||||
"""A schedule fired at its own time must not find itself due again on the
|
||||
same instant, which would be one firing per tick for ever."""
|
||||
at_six = datetime(2025, 6, 1, 6, 0, tzinfo=COPENHAGEN)
|
||||
when = Cadence.parse("daily at 06:00").next_after(at_six, COPENHAGEN)
|
||||
assert when.astimezone(COPENHAGEN) == at_six + timedelta(days=1)
|
||||
|
||||
|
||||
def test_next_after_always_comes_back_in_utc():
|
||||
"""Because it is written straight into the table, where every other time
|
||||
is UTC and the column is compared as text."""
|
||||
when = Cadence.parse("daily at 06:00").next_after(
|
||||
datetime(2025, 6, 1, 12, 0, tzinfo=COPENHAGEN), COPENHAGEN)
|
||||
assert when.tzinfo is UTC
|
||||
assert when.isoformat().endswith("+00:00")
|
||||
|
||||
|
||||
def test_weekly_lands_on_the_day_it_names():
|
||||
monday = Cadence.parse("weekly mon 07:00")
|
||||
# A Wednesday: the next Monday is five days away, not tomorrow.
|
||||
from_wed = monday.next_after(datetime(2025, 6, 4, 9, 0, tzinfo=COPENHAGEN),
|
||||
COPENHAGEN).astimezone(COPENHAGEN)
|
||||
assert from_wed.weekday() == 0 and (from_wed.hour, from_wed.minute) == (7, 0)
|
||||
assert from_wed.date() == datetime(2025, 6, 9).date()
|
||||
# Monday morning before seven: today, not next week.
|
||||
from_mon = monday.next_after(datetime(2025, 6, 9, 6, 0, tzinfo=COPENHAGEN),
|
||||
COPENHAGEN).astimezone(COPENHAGEN)
|
||||
assert from_mon.date() == datetime(2025, 6, 9).date()
|
||||
|
||||
|
||||
def test_daily_keeps_its_wall_clock_across_both_dst_changes():
|
||||
"""06:00 is 06:00 on both sides, which is the whole reason for the zone.
|
||||
|
||||
Spring forward is a 23-hour day and autumn back is a 25-hour one, so the
|
||||
real elapsed time between two runs is not 24 hours — and it must not be,
|
||||
because what the wording promised is a time of day.
|
||||
"""
|
||||
daily = Cadence.parse("daily at 06:00")
|
||||
|
||||
# Europe/Copenhagen, 2025: forward on 30 March, back on 26 October.
|
||||
forward = daily.next_after(
|
||||
datetime(2025, 3, 29, 6, 30, tzinfo=COPENHAGEN), COPENHAGEN)
|
||||
local = forward.astimezone(COPENHAGEN)
|
||||
assert (local.hour, local.minute) == (6, 0)
|
||||
assert local.date() == datetime(2025, 3, 30).date()
|
||||
# 05:30Z to 04:00Z: 22h30m of real time for a 23h30m step of the clock.
|
||||
assert forward - datetime(2025, 3, 29, 6, 30, tzinfo=COPENHAGEN) \
|
||||
== timedelta(hours=22, minutes=30)
|
||||
|
||||
back = daily.next_after(
|
||||
datetime(2025, 10, 25, 6, 30, tzinfo=COPENHAGEN), COPENHAGEN)
|
||||
local = back.astimezone(COPENHAGEN)
|
||||
assert (local.hour, local.minute) == (6, 0)
|
||||
assert local.date() == datetime(2025, 10, 26).date()
|
||||
assert back - datetime(2025, 10, 25, 6, 30, tzinfo=COPENHAGEN) \
|
||||
== timedelta(hours=24, minutes=30)
|
||||
|
||||
|
||||
def test_weekly_keeps_its_wall_clock_across_a_dst_change():
|
||||
when = Cadence.parse("weekly sun 07:00").next_after(
|
||||
datetime(2025, 3, 26, 9, 0, tzinfo=COPENHAGEN), COPENHAGEN)
|
||||
local = when.astimezone(COPENHAGEN)
|
||||
assert local.date() == datetime(2025, 3, 30).date() # the change itself
|
||||
assert (local.hour, local.minute) == (7, 0)
|
||||
|
||||
|
||||
def test_an_interval_is_elapsed_time_and_ignores_the_change():
|
||||
"""The other half of the promise, and the reason both wordings exist.
|
||||
|
||||
`every 24h` across a spring-forward is 24 real hours and therefore lands
|
||||
an hour later on the clock. A person who wanted 06:00 wrote `daily`.
|
||||
"""
|
||||
start = datetime(2025, 3, 29, 6, 30, tzinfo=COPENHAGEN)
|
||||
when = Cadence.parse("every 24h").next_after(start, COPENHAGEN)
|
||||
assert when - start == timedelta(hours=24)
|
||||
assert when.astimezone(COPENHAGEN).hour == 7
|
||||
|
||||
|
||||
def test_a_naive_moment_is_read_as_utc():
|
||||
"""Rather than raising. Every stamp in the table is UTC, so a caller that
|
||||
lost the tzinfo on the way is very nearly certainly holding one."""
|
||||
naive = datetime(2025, 6, 1, 12, 0)
|
||||
assert Cadence.parse("every 1h").next_after(naive) \
|
||||
== datetime(2025, 6, 1, 13, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the table
|
||||
|
||||
def test_the_table_holds_a_schedule_through_its_life(jobs):
|
||||
row = jobs.add_schedule("nightly", "s_tick", "daily at 06:00",
|
||||
payload={"note": "hello"}, label="A nightly tick",
|
||||
next_run="2030-01-01T06:00:00+00:00")
|
||||
assert row is not None
|
||||
assert (row.name, row.kind, row.cadence) == ("nightly", "s_tick", "daily at 06:00")
|
||||
assert row.payload == {"note": "hello"} and row.enabled is True
|
||||
assert row.last_job_id == 0 and row.last_run == "" and row.last_state == ""
|
||||
|
||||
assert [s.name for s in jobs.schedules()] == ["nightly"]
|
||||
assert jobs.schedule(row.id) == row
|
||||
|
||||
edited = jobs.update_schedule(row.id, cadence="every 15m", enabled=False,
|
||||
payload={"note": "other"})
|
||||
assert edited.cadence == "every 15m" and edited.enabled is False
|
||||
assert edited.payload == {"note": "other"}
|
||||
# Untouched columns stay as they were.
|
||||
assert edited.name == "nightly" and edited.label == "A nightly tick"
|
||||
|
||||
assert jobs.delete_schedule(row.id) is True
|
||||
assert jobs.delete_schedule(row.id) is False # two tabs; not an error
|
||||
assert jobs.schedules() == []
|
||||
|
||||
|
||||
def test_a_name_is_taken_once(jobs):
|
||||
assert jobs.add_schedule("one", "s_tick", "every 15m") is not None
|
||||
assert jobs.add_schedule("one", "s_tick", "every 6h", only_if_new=True) is None
|
||||
assert jobs.schedules()[0].cadence == "every 15m"
|
||||
|
||||
|
||||
def test_schedules_carry_the_state_of_their_last_job(jobs):
|
||||
row = jobs.add_schedule("one", "s_tick", "every 15m")
|
||||
queued = jobs.enqueue("s_tick")
|
||||
jobs.mark_fired(row.id, queued.id, "2030-01-01T00:00:00+00:00")
|
||||
assert jobs.schedules()[0].last_state == "queued"
|
||||
jobs.claim({"cpu": 1})
|
||||
jobs.finish(queued.id, "failed", error="no")
|
||||
assert jobs.schedules()[0].last_state == "failed"
|
||||
# And the time it fired, which is not the time it is next due.
|
||||
listed = jobs.schedules()[0]
|
||||
assert listed.last_run and listed.next_run == "2030-01-01T00:00:00+00:00"
|
||||
|
||||
|
||||
def test_due_schedules_passes_over_what_is_off_or_not_yet(jobs):
|
||||
jobs.add_schedule("due", "s_tick", "every 15m", next_run=ago(60))
|
||||
jobs.add_schedule("off", "s_tick", "every 15m", next_run=ago(60), enabled=False)
|
||||
jobs.add_schedule("later", "s_tick", "every 15m",
|
||||
next_run="2099-01-01T00:00:00+00:00")
|
||||
jobs.add_schedule("never", "s_tick", "every 15m") # no time at all
|
||||
due = jobs.due_schedules(datetime.now(UTC).isoformat())
|
||||
assert [s.name for s in due] == ["due"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------- the firing
|
||||
|
||||
def test_the_tick_fires_a_due_row_and_moves_it_on(queue, jobs):
|
||||
row = jobs.add_schedule("regular", "s_tick", "every 15m",
|
||||
payload={"note": "scheduled"}, next_run=ago(30))
|
||||
queue._fire_due()
|
||||
|
||||
queued = jobs.snapshot().queued
|
||||
assert [j.kind for j in queued] == ["s_tick"]
|
||||
assert queued[0].payload == {"note": "scheduled"}
|
||||
|
||||
after = jobs.schedule(row.id)
|
||||
assert after.last_job_id == queued[0].id
|
||||
assert after.last_run
|
||||
# Fifteen minutes from now, not from the moment it was due.
|
||||
assert after.next_run > datetime.now(UTC).isoformat()
|
||||
assert after.next_run < (datetime.now(UTC) + timedelta(minutes=16)).isoformat()
|
||||
|
||||
|
||||
def test_a_schedule_does_not_pile_up_behind_its_own_job(queue, jobs):
|
||||
"""The job from the last firing is still queued, so this one is skipped.
|
||||
|
||||
Skipped and *moved on*, not left due: a schedule whose job takes longer
|
||||
than its cadence would otherwise fire on every tick for as long as the job
|
||||
ran, and the queue would fill with copies of work already under way.
|
||||
"""
|
||||
row = jobs.add_schedule("regular", "s_tick", "every 15m", next_run=ago(30))
|
||||
queue._fire_due()
|
||||
first = jobs.schedule(row.id).last_job_id
|
||||
|
||||
jobs.update_schedule(row.id, next_run=ago(1)) # due again
|
||||
queue._fire_due()
|
||||
|
||||
assert len(jobs.snapshot().queued) == 1
|
||||
again = jobs.schedule(row.id)
|
||||
assert again.last_job_id == first # nothing new was queued
|
||||
assert again.next_run > datetime.now(UTC).isoformat()
|
||||
|
||||
|
||||
def test_a_solo_kind_already_underway_holds_its_schedule_back(queue, jobs):
|
||||
"""The other way a schedule collides: not with itself, with somebody else.
|
||||
|
||||
A button, a second schedule or a cascade started one, and `enqueue` would
|
||||
hand that job back — so the schedule would record work it did not ask for
|
||||
as its own firing.
|
||||
"""
|
||||
pressed = jobs.enqueue("s_solo", dedup_key="s_solo")
|
||||
row = jobs.add_schedule("solo", "s_solo", "every 15m", next_run=ago(30))
|
||||
queue._fire_due()
|
||||
after = jobs.schedule(row.id)
|
||||
assert after.last_job_id == 0 # it never fired
|
||||
assert after.next_run > datetime.now(UTC).isoformat()
|
||||
assert [j.id for j in jobs.snapshot().queued] == [pressed.id]
|
||||
|
||||
|
||||
def test_downtime_is_caught_up_once_and_not_once_per_slot(queue, jobs):
|
||||
"""Three days off an hourly schedule is one job, not seventy-two.
|
||||
|
||||
The row carries one `next_run`, and a firing sets it from *now* rather
|
||||
than stepping through what was missed — so there is nowhere for a backlog
|
||||
to accumulate even in principle.
|
||||
"""
|
||||
row = jobs.add_schedule("hourly", "s_tick", "every 1h", next_run=ago(3 * 86400))
|
||||
queue._fire_due()
|
||||
assert len(jobs.snapshot().queued) == 1
|
||||
after = jobs.schedule(row.id)
|
||||
assert after.next_run > datetime.now(UTC).isoformat()
|
||||
assert after.next_run < (datetime.now(UTC) + timedelta(minutes=61)).isoformat()
|
||||
|
||||
|
||||
def test_an_unknown_kind_never_fires_and_keeps_its_place(queue, jobs):
|
||||
"""A handler that went away in a deploy. The row stays exactly as it is —
|
||||
time included — so reinstalling the plugin runs it rather than starting
|
||||
another cadence from scratch."""
|
||||
row = jobs.add_schedule("gone", "s_vanished", "every 15m", next_run=ago(30))
|
||||
queue._fire_due()
|
||||
assert jobs.snapshot().queued == []
|
||||
assert jobs.schedule(row.id).next_run == row.next_run
|
||||
queue._fire_due() # and it says so once
|
||||
assert queue._said_about == {(row.id, "kind")}
|
||||
|
||||
|
||||
def test_a_cadence_the_table_cannot_parse_never_fires(queue, jobs):
|
||||
"""Only reachable by editing the file by hand, since every route parses
|
||||
first. It must not take the tick down with it."""
|
||||
jobs.add_schedule("bent", "s_tick", "every 15m", next_run=ago(30))
|
||||
jobs._write("UPDATE schedules SET cadence='sometimes'")
|
||||
queue._fire_due()
|
||||
assert jobs.snapshot().queued == []
|
||||
assert jobs.schedules()[0].cadence == "sometimes" # left to be fixed
|
||||
|
||||
|
||||
def test_a_disabled_row_is_not_fired(queue, jobs):
|
||||
jobs.add_schedule("off", "s_tick", "every 15m", next_run=ago(30), enabled=False)
|
||||
queue._fire_due()
|
||||
assert jobs.snapshot().queued == []
|
||||
|
||||
|
||||
def test_run_now_fires_without_moving_the_next_run(queue, jobs):
|
||||
row = jobs.add_schedule("nightly", "s_tick", "daily at 06:00",
|
||||
next_run="2099-01-01T06:00:00+00:00")
|
||||
started = queue.run_schedule(row.id)
|
||||
assert started is not None and started.kind == "s_tick"
|
||||
after = jobs.schedule(row.id)
|
||||
assert after.next_run == "2099-01-01T06:00:00+00:00"
|
||||
# But it did just run, and that is the job it queued — so the next tick
|
||||
# will not start a second one while this is still going.
|
||||
assert after.last_job_id == started.id and after.last_run
|
||||
|
||||
|
||||
# ------------------------------------------------------------ reconciliation
|
||||
|
||||
def test_declared_defaults_get_a_row_on_start(declared, jobs):
|
||||
declare("nightly tick", "s_tick", "daily at 06:00", {"note": "declared"},
|
||||
label="A tick, nightly")
|
||||
JobQueue(jobs, {}, [], tz="UTC")._reconcile_schedules()
|
||||
row = jobs.schedules()[0]
|
||||
assert (row.name, row.kind, row.cadence) == ("nightly tick", "s_tick",
|
||||
"daily at 06:00")
|
||||
assert row.payload == {"note": "declared"} and row.label == "A tick, nightly"
|
||||
assert row.next_run # and it knows when it is next
|
||||
|
||||
|
||||
def test_reconciliation_keeps_what_the_user_edited(declared, jobs):
|
||||
"""The whole of what a declaration means: a starting point, not a setting.
|
||||
|
||||
Switching a default off, or slowing it down, has to survive the next
|
||||
deploy — otherwise the only way to be rid of one is to edit the code and
|
||||
redeploy, which is not a thing the person running the app can do.
|
||||
"""
|
||||
declare("nightly tick", "s_tick", "daily at 06:00")
|
||||
queue = JobQueue(jobs, {}, [], tz="UTC")
|
||||
queue._reconcile_schedules()
|
||||
row = jobs.schedules()[0]
|
||||
jobs.update_schedule(row.id, cadence="every 2d", enabled=False,
|
||||
payload={"note": "mine"})
|
||||
|
||||
queue._reconcile_schedules() # a restart
|
||||
|
||||
again = jobs.schedules()[0]
|
||||
assert again.id == row.id
|
||||
assert again.cadence == "every 2d"
|
||||
assert again.enabled is False
|
||||
assert again.payload == {"note": "mine"}
|
||||
|
||||
|
||||
def test_a_declaration_that_goes_away_leaves_its_row(declared, jobs):
|
||||
"""Nothing here deletes. A deploy that silently forgot a schedule somebody
|
||||
was relying on is the failure this avoids; an orphan says `unknown kind`
|
||||
and waits to be deleted deliberately."""
|
||||
declare("nightly tick", "s_tick", "daily at 06:00")
|
||||
queue = JobQueue(jobs, {}, [], tz="UTC")
|
||||
queue._reconcile_schedules()
|
||||
declared.clear()
|
||||
queue._reconcile_schedules()
|
||||
assert [s.name for s in jobs.schedules()] == ["nightly tick"]
|
||||
|
||||
|
||||
def test_declaring_one_name_twice_differently_is_refused(declared):
|
||||
declare("nightly", "s_tick", "daily at 06:00")
|
||||
declare("nightly", "s_tick", "daily at 06:00") # the same: fine
|
||||
with pytest.raises(ValueError) as refused:
|
||||
declare("nightly", "s_tick", "daily at 07:00")
|
||||
assert "nightly" in str(refused.value)
|
||||
|
||||
|
||||
def test_a_declared_cadence_is_parsed_where_it_is_written(declared):
|
||||
"""At import, with a traceback pointing at the line — rather than at 06:00
|
||||
in a log nobody is reading."""
|
||||
with pytest.raises(ValueError):
|
||||
declare("bad", "s_tick", "every so often")
|
||||
assert "bad" not in declared
|
||||
|
||||
|
||||
# ------------------------------------------------------------- a real queue
|
||||
|
||||
def test_a_started_queue_fires_what_is_overdue(jobs, declared):
|
||||
"""The one test that drives the loop rather than the method.
|
||||
|
||||
Everything above proves the firing decides correctly; this proves it is
|
||||
wired into `start()` at all, which is the wire nobody notices is missing
|
||||
until a schedule silently stops running.
|
||||
"""
|
||||
ran: list[str] = []
|
||||
jobs.add_schedule("regular", "s_tick", "every 15m",
|
||||
payload={"note": "from the loop"}, next_run=ago(30))
|
||||
queue = JobQueue(jobs, {}, ran, tz="UTC", workers={"cpu": 1})
|
||||
assert queue.drain(timeout=10) is True
|
||||
assert ran == ["from the loop"]
|
||||
assert jobs.schedules()[0].last_state == "done"
|
||||
|
||||
|
||||
def test_a_queue_with_schedules_off_fires_nothing(jobs, declared):
|
||||
ran: list[str] = []
|
||||
jobs.add_schedule("regular", "s_tick", "every 15m", next_run=ago(30))
|
||||
queue = JobQueue(jobs, {}, ran, tz="UTC", schedules=False)
|
||||
assert queue.drain(timeout=10) is True
|
||||
assert ran == []
|
||||
assert jobs.schedules()[0].last_job_id == 0
|
||||
|
||||
|
||||
def test_an_unknown_timezone_is_a_warning_and_not_a_refusal(jobs, caplog):
|
||||
"""A mistyped zone must not be the reason a worker will not start."""
|
||||
queue = JobQueue(jobs, {}, [], tz="Mars/Olympus_Mons")
|
||||
assert queue.zone is not None
|
||||
assert Cadence.parse("daily at 06:00").next_after(datetime.now(UTC), queue.zone)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the web
|
||||
|
||||
@pytest.fixture
|
||||
def client(jobs, declared):
|
||||
pytest.importorskip("fastapi")
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from pipeline.web import connect
|
||||
|
||||
app = FastAPI()
|
||||
connect(app, jobs, {}, [], tz="UTC", schedules=False)
|
||||
with TestClient(app) as client:
|
||||
yield client
|
||||
|
||||
|
||||
def test_the_element_is_served(client):
|
||||
answer = client.get("/_jobs/schedules.js")
|
||||
assert answer.status_code == 200
|
||||
assert "javascript" in answer.headers["content-type"]
|
||||
assert "desk-schedules" in answer.text
|
||||
|
||||
|
||||
def test_adding_listing_editing_and_deleting(client):
|
||||
added = client.post("/api/schedules", json={
|
||||
"kind": "s_tick", "cadence": "weekly monday 07:00",
|
||||
"payload": {"note": "over http"}})
|
||||
assert added.status_code == 201
|
||||
row = added.json()
|
||||
# Named after its kind, since none was given, and stored canonically.
|
||||
assert row["name"] == "s_tick" and row["cadence"] == "weekly mon 07:00"
|
||||
assert row["next_run"] and row["unknown"] is False
|
||||
|
||||
assert [s["id"] for s in client.get("/api/schedules").json()] == [row["id"]]
|
||||
|
||||
off = client.patch(f"/api/schedules/{row['id']}", json={"enabled": False})
|
||||
assert off.status_code == 200 and off.json()["enabled"] is False
|
||||
# And only that: a PATCH of one field does not resubmit the rest.
|
||||
assert off.json()["cadence"] == "weekly mon 07:00"
|
||||
assert off.json()["payload"] == {"note": "over http"}
|
||||
|
||||
faster = client.patch(f"/api/schedules/{row['id']}", json={"cadence": "every 15m"})
|
||||
assert faster.json()["cadence"] == "every 15m"
|
||||
# A new cadence moves the time with it, rather than leaving the old one.
|
||||
assert faster.json()["next_run"] != row["next_run"]
|
||||
|
||||
gone = client.delete(f"/api/schedules/{row['id']}")
|
||||
assert gone.status_code == 200 and gone.json() == {"deleted": True}
|
||||
assert client.delete(f"/api/schedules/{row['id']}").json() == {"deleted": False}
|
||||
|
||||
|
||||
def test_a_second_schedule_of_one_kind_is_named_by_counting(client):
|
||||
first = client.post("/api/schedules",
|
||||
json={"kind": "s_tick", "cadence": "every 15m"}).json()
|
||||
second = client.post("/api/schedules",
|
||||
json={"kind": "s_tick", "cadence": "every 6h"}).json()
|
||||
assert (first["name"], second["name"]) == ("s_tick", "s_tick 2")
|
||||
|
||||
|
||||
def test_a_name_somebody_typed_is_refused_rather_than_altered(client):
|
||||
client.post("/api/schedules",
|
||||
json={"name": "nightly", "kind": "s_tick", "cadence": "every 15m"})
|
||||
clash = client.post("/api/schedules",
|
||||
json={"name": "nightly", "kind": "s_tick", "cadence": "every 6h"})
|
||||
assert clash.status_code == 409
|
||||
assert "nightly" in clash.json()["detail"]
|
||||
|
||||
|
||||
def test_an_unregistered_kind_is_a_404_naming_what_there_is(client):
|
||||
refused = client.post("/api/schedules",
|
||||
json={"kind": "s_nonesuch", "cadence": "every 15m"})
|
||||
assert refused.status_code == 404
|
||||
detail = refused.json()["detail"]
|
||||
assert "s_nonesuch" in detail and "s_tick" in detail
|
||||
|
||||
|
||||
def test_a_payload_missing_what_the_handler_needs_is_a_422_naming_it(client):
|
||||
refused = client.post("/api/schedules",
|
||||
json={"kind": "s_needs", "cadence": "every 15m"})
|
||||
assert refused.status_code == 422
|
||||
assert "target" in refused.json()["detail"]
|
||||
# And with it, the same request is fine.
|
||||
assert client.post("/api/schedules", json={
|
||||
"kind": "s_needs", "cadence": "every 15m",
|
||||
"payload": {"target": "somewhere"}}).status_code == 201
|
||||
|
||||
|
||||
def test_a_bad_cadence_is_a_422_listing_the_accepted_wordings(client):
|
||||
refused = client.post("/api/schedules",
|
||||
json={"kind": "s_tick", "cadence": "sometimes"})
|
||||
assert refused.status_code == 422
|
||||
detail = refused.json()["detail"]
|
||||
assert isinstance(detail, str)
|
||||
for form in ("every 15m", "every 6h", "every 2d",
|
||||
"daily at 06:00", "weekly mon 07:00"):
|
||||
assert form in detail
|
||||
|
||||
|
||||
def test_an_edit_is_validated_the_same_way(client):
|
||||
row = client.post("/api/schedules",
|
||||
json={"kind": "s_tick", "cadence": "every 15m"}).json()
|
||||
assert client.patch(f"/api/schedules/{row['id']}",
|
||||
json={"cadence": "now and then"}).status_code == 422
|
||||
assert client.patch(f"/api/schedules/{row['id']}",
|
||||
json={"kind": "s_needs"}).status_code == 422
|
||||
assert client.patch("/api/schedules/9999",
|
||||
json={"enabled": False}).status_code == 404
|
||||
|
||||
|
||||
def test_run_now_queues_the_job(client, jobs):
|
||||
row = client.post("/api/schedules",
|
||||
json={"kind": "s_tick", "cadence": "daily at 06:00"}).json()
|
||||
started = client.post(f"/api/schedules/{row['id']}/run")
|
||||
assert started.status_code == 202
|
||||
assert started.json()["kind"] == "s_tick"
|
||||
after = client.get("/api/schedules").json()[0]
|
||||
assert after["last_job_id"] == started.json()["id"]
|
||||
assert after["next_run"] == row["next_run"]
|
||||
assert client.post("/api/schedules/9999/run").status_code == 404
|
||||
|
||||
|
||||
def test_a_row_whose_handler_went_away_is_flagged_not_hidden(client, jobs):
|
||||
jobs.add_schedule("orphan", "s_vanished", "every 15m")
|
||||
row = next(s for s in client.get("/api/schedules").json()
|
||||
if s["name"] == "orphan")
|
||||
assert row["unknown"] is True
|
||||
# Listed and deletable, which is the whole of what can be done about it.
|
||||
assert client.delete(f"/api/schedules/{row['id']}").json() == {"deleted": True}
|
||||
|
||||
|
||||
def test_the_socket_pushes_the_schedule_list_on_every_change(client):
|
||||
"""So a row edited in one tab redraws in the others without polling.
|
||||
|
||||
The opening message is the *job* list — the socket is shared — which is
|
||||
why `<desk-schedules>` reads `/api/schedules` once on connect and follows
|
||||
this afterwards.
|
||||
"""
|
||||
with client.websocket_connect("/ws/jobs") as socket:
|
||||
assert socket.receive_json()["type"] == "jobs"
|
||||
added = client.post("/api/schedules",
|
||||
json={"kind": "s_tick", "cadence": "every 15m"}).json()
|
||||
pushed = _next_of(socket, "schedules")
|
||||
assert [s["id"] for s in pushed["schedules"]] == [added["id"]]
|
||||
assert pushed["schedules"][0]["cadence"] == "every 15m"
|
||||
|
||||
client.patch(f"/api/schedules/{added['id']}", json={"cadence": "every 6h"})
|
||||
assert _next_of(socket, "schedules")["schedules"][0]["cadence"] == "every 6h"
|
||||
|
||||
client.delete(f"/api/schedules/{added['id']}")
|
||||
assert _next_of(socket, "schedules")["schedules"] == []
|
||||
|
||||
|
||||
def _next_of(socket, type_: str, tries: int = 12) -> dict:
|
||||
"""The next message of one type, stepping over the job pushes between."""
|
||||
for _ in range(tries):
|
||||
message = socket.receive_json()
|
||||
if message["type"] == type_:
|
||||
return message
|
||||
raise AssertionError(f"no {type_} message arrived")
|
||||
|
||||
|
||||
# --------------------------------------------------------------- the worker
|
||||
|
||||
def _worker(db: Path, marks: Path, *extra: str) -> subprocess.CompletedProcess:
|
||||
argv = [sys.executable, "-m", "pipeline.worker", "--db", str(db),
|
||||
"--bootstrap", "pipeline.tests.scheduled_handlers:marking",
|
||||
"--bootstrap-arg", str(marks), "--until-idle", *extra]
|
||||
return subprocess.run(argv, env=dict(os.environ, PYTHONPATH=str(ROOT)),
|
||||
cwd=ROOT, capture_output=True, timeout=120)
|
||||
|
||||
|
||||
def test_the_standalone_worker_fires_schedules(tmp_path):
|
||||
"""A machine running only `python -m pipeline.worker` still gets its 06:00
|
||||
job, which is what makes it a complete replacement for the web app's
|
||||
worker rather than half of one."""
|
||||
db, marks = tmp_path / "w.db", tmp_path / "marks"
|
||||
marks.mkdir()
|
||||
jobs = Jobs(db)
|
||||
jobs.add_schedule("marker", "sw_mark", "every 1h", next_run=ago(60))
|
||||
|
||||
done = _worker(db, marks)
|
||||
assert done.returncode == 0, done.stdout.decode("utf-8", "replace")
|
||||
assert (marks / "fired").exists()
|
||||
assert jobs.schedules()[0].last_job_id
|
||||
|
||||
|
||||
def test_no_schedules_turns_it_off_for_a_second_worker(tmp_path):
|
||||
"""One scheduler per file, as many workers as you like: the pile-up guard
|
||||
reads `last_job_id`, which neither of two simultaneous firings has written
|
||||
yet when the other looks."""
|
||||
db, marks = tmp_path / "w.db", tmp_path / "marks"
|
||||
marks.mkdir()
|
||||
jobs = Jobs(db)
|
||||
jobs.add_schedule("marker", "sw_mark", "every 1h", next_run=ago(60))
|
||||
|
||||
done = _worker(db, marks, "--no-schedules")
|
||||
assert done.returncode == 0, done.stdout.decode("utf-8", "replace")
|
||||
assert not (marks / "fired").exists()
|
||||
assert jobs.schedules()[0].last_job_id == 0
|
||||
# Not even the declared default was reconciled: nothing about schedules
|
||||
# happens in a worker that was told not to do them.
|
||||
assert [s.name for s in jobs.schedules()] == ["marker"]
|
||||
@@ -0,0 +1,59 @@
|
||||
"""The parts of a web app that are the pipeline's, not the application's.
|
||||
|
||||
from pipeline.web import connect
|
||||
queue = connect(app, Jobs("app.db"), config, store)
|
||||
|
||||
An app of this shape has a queue, and therefore has the same endpoints and the
|
||||
same socket over it: list, kinds, enqueue, cancel, and `/ws/jobs`. None of that
|
||||
knows what the work is *for*, so it is here rather than copied into each app —
|
||||
and a socket's reconnect and fan-out behaviour is exactly the sort of thing a
|
||||
second copy gets subtly wrong.
|
||||
|
||||
app.py `connect`: all of the below wired into a FastAPI app, and
|
||||
the worker started and stopped with it, in one call
|
||||
broadcaster.py one message out to every connected socket
|
||||
jobs.py the queue over HTTP, the socket that reports on it, and
|
||||
the one route that serves the element below
|
||||
schedules.py schedules over HTTP: list, add, edit, delete, run now,
|
||||
with the readable refusals a thing that runs while nobody
|
||||
is watching has to have
|
||||
deps.py how those routes reach the queue and the bus
|
||||
schemas.py the request bodies they take
|
||||
static/jobs.js `<desk-jobs>`: the queue on screen. Two lines in a
|
||||
template and a new app has a job list with progress and a
|
||||
Kill button, before it has written any UI of its own
|
||||
static/schedules.js
|
||||
`<desk-schedules>`: the same two lines for the schedule
|
||||
table, themed from the same custom properties
|
||||
|
||||
<script type="module" src="/_jobs/jobs.js"></script>
|
||||
<desk-jobs></desk-jobs>
|
||||
|
||||
The element is the one thing here with an opinion about how something looks,
|
||||
and it is deliberate: this is the part nobody wants to write twice. It keeps
|
||||
to a shadow root and themes from custom properties, so an app overrides it
|
||||
without forking it, and it says nothing about *failures* — it raises an event
|
||||
and leaves the wording to the app, which is the same split `notices.py` makes
|
||||
on the server. newsdesk does not use it: it had a richer strip of its own
|
||||
before this existed, which is what an app is expected to grow into.
|
||||
|
||||
What an app still owns: its own factory, its own store and config
|
||||
dependencies, its own routers, its own page.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pipeline.web.app import connect
|
||||
from pipeline.web.broadcaster import Broadcaster
|
||||
from pipeline.web.deps import BroadcasterDep, QueueDep, get_broadcaster, get_queue
|
||||
from pipeline.web.jobs import STATIC, router as jobs_router
|
||||
from pipeline.web.schedules import router as schedules_router, served
|
||||
from pipeline.web.schemas import (
|
||||
EnqueueBody, JobKind, NewSchedule, ScheduleEdit, ScheduleRow,
|
||||
)
|
||||
|
||||
__all__ = ["STATIC", "Broadcaster", "BroadcasterDep", "EnqueueBody",
|
||||
"JobKind", "NewSchedule", "QueueDep", "ScheduleEdit", "ScheduleRow",
|
||||
"connect", "get_broadcaster", "get_queue", "jobs_router",
|
||||
"schedules_router", "served",
|
||||
]
|
||||
@@ -0,0 +1,86 @@
|
||||
"""The whole web face of the queue, in one call.
|
||||
|
||||
app = FastAPI()
|
||||
queue = connect(app, Jobs("app.db"), config, store)
|
||||
|
||||
That is the HTTP routes, the `/ws/jobs` socket, the `<desk-jobs>` and
|
||||
`<desk-schedules>` elements, the schedule endpoints, the broadcaster that feeds
|
||||
the socket, and the worker starting and stopping with the app. Before this
|
||||
every app wrote the same thirty lines of glue — three callbacks, a lifespan,
|
||||
two `app.state` slots, two `include_router`s — which is the part of "a day of
|
||||
pipeline per project" that was still left.
|
||||
|
||||
Keyword arguments go to `JobQueue`: `workers=`, `bootstrap=`, `retain=`, `tz=`
|
||||
and the rest. `tz` is the one a deployed app has to pass: a server runs on UTC
|
||||
and `daily at 06:00` means 06:00 where the person reading it lives. The queue
|
||||
is returned for the app's own routes and commands to enqueue on; it is also on
|
||||
`app.state.queue`, which is where `QueueDep` finds it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from pipeline import JobQueue, Jobs, JobsSnapshot, Notice, Schedule
|
||||
from pipeline.web.broadcaster import Broadcaster
|
||||
from pipeline.web.jobs import router
|
||||
from pipeline.web.schedules import router as schedules_router, served
|
||||
|
||||
|
||||
def connect(app: FastAPI, jobs: Jobs, config: Any, store: Any,
|
||||
**queue_options: Any) -> JobQueue:
|
||||
"""Give `app` a queue: routes, socket, element, and a worker in its lifespan.
|
||||
|
||||
The lifespan is wrapped rather than replaced, so an app's own lifespan —
|
||||
passed to `FastAPI(lifespan=...)` before or after this — still runs, inside
|
||||
the queue's: the worker is up before the app's startup code and still up
|
||||
through its shutdown code, so either may enqueue.
|
||||
"""
|
||||
bus = Broadcaster()
|
||||
|
||||
async def publish(snapshot: JobsSnapshot) -> None:
|
||||
"""State: the whole queue, replaced wholesale by the next one."""
|
||||
await bus.broadcast({"type": "jobs", **snapshot.model_dump()})
|
||||
|
||||
async def announce(notice: Notice) -> None:
|
||||
"""An event: one failure, out to every open tab. The row it was
|
||||
written to, so a tab can tell news from a repeat — see `NoticeEntry`."""
|
||||
await bus.broadcast({"type": "notice", **notice.model_dump()})
|
||||
|
||||
async def scheduled(rows: list[Schedule]) -> None:
|
||||
"""State, on its own clock: the schedule list, whenever one changes.
|
||||
|
||||
Separate from `jobs` rather than folded into it because the two change
|
||||
at completely different rates — a running job pushes four times a
|
||||
second and a schedule a few times a day — and because a client that
|
||||
only draws the table should not have to ignore everything else.
|
||||
"""
|
||||
await bus.broadcast({"type": "schedules",
|
||||
"schedules": [served(row).model_dump() for row in rows]})
|
||||
|
||||
queue = JobQueue(jobs, config, store, listener=publish, on_notice=announce,
|
||||
on_schedules=scheduled, **queue_options)
|
||||
app.state.queue = queue
|
||||
app.state.bus = bus
|
||||
app.include_router(router)
|
||||
app.include_router(schedules_router)
|
||||
|
||||
inner = app.router.lifespan_context
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(this: FastAPI) -> AsyncIterator[Any]:
|
||||
# The worker needs a running event loop, which exists here and not
|
||||
# where `connect` was called.
|
||||
await queue.start()
|
||||
try:
|
||||
async with inner(this) as state:
|
||||
yield state
|
||||
finally:
|
||||
await queue.stop()
|
||||
|
||||
app.router.lifespan_context = lifespan
|
||||
return queue
|
||||
@@ -0,0 +1,56 @@
|
||||
"""One message out to every connected socket.
|
||||
|
||||
Deliberately knows nothing about jobs. It holds sockets, sends dicts, and drops
|
||||
the ones that have gone away — which is the whole of what a fan-out is, and is
|
||||
the reason the next thing that wants to push (a tag landing, a transcript
|
||||
finishing, another reader in another tab) can use this rather than grow a
|
||||
second copy of it.
|
||||
|
||||
A dead socket is discovered by writing to it, not by asking: a browser tab that
|
||||
was closed, slept or navigated away leaves a connection that looks open until
|
||||
the send fails. So every failure is a removal, and `broadcast` never raises —
|
||||
one stale subscriber must not cost the others their update.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import asyncio
|
||||
from typing import Any
|
||||
|
||||
from fastapi import WebSocket
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Broadcaster:
|
||||
def __init__(self) -> None:
|
||||
self._subscribers: set[WebSocket] = set()
|
||||
|
||||
@property
|
||||
def count(self) -> int:
|
||||
return len(self._subscribers)
|
||||
|
||||
async def subscribe(self, ws: WebSocket, hello: dict[str, Any] | None = None) -> None:
|
||||
"""Register an accepted socket, and send it the current state.
|
||||
|
||||
The immediate send matters: a client that connects mid-job would
|
||||
otherwise see nothing at all until the next change, which for a long
|
||||
job is a blank progress bar next to work that is visibly happening.
|
||||
"""
|
||||
self._subscribers.add(ws)
|
||||
if hello is not None:
|
||||
await self._send(ws, hello)
|
||||
|
||||
def unsubscribe(self, ws: WebSocket) -> None:
|
||||
self._subscribers.discard(ws)
|
||||
|
||||
async def broadcast(self, message: dict[str, Any]) -> None:
|
||||
await asyncio.gather(*(self._send(ws, message)
|
||||
for ws in tuple(self._subscribers)))
|
||||
|
||||
async def _send(self, ws: WebSocket, message: dict[str, Any]) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(ws.send_json(message), timeout=1.0)
|
||||
except Exception:
|
||||
self._subscribers.discard(ws)
|
||||
@@ -0,0 +1,34 @@
|
||||
"""How a route reaches the queue and the socket bus.
|
||||
|
||||
Both are built once by the app's factory and parked on `app.state`. Routes ask
|
||||
for them as dependencies rather than closing over them, so a router module is
|
||||
importable on its own and a test can replace either through
|
||||
`app.dependency_overrides`.
|
||||
|
||||
Only the two things that are the *pipeline's*. An app's own store and config
|
||||
are its own business and stay in its own `deps` — the queue and the bus are
|
||||
here because `pipeline.web.jobs` is here, and it needs them whichever app it is
|
||||
mounted in.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
from pipeline import JobQueue
|
||||
from pipeline.web.broadcaster import Broadcaster
|
||||
|
||||
|
||||
def get_queue(request: HTTPConnection) -> JobQueue:
|
||||
return request.app.state.queue
|
||||
|
||||
|
||||
def get_broadcaster(request: HTTPConnection) -> Broadcaster:
|
||||
return request.app.state.bus
|
||||
|
||||
|
||||
QueueDep = Annotated[JobQueue, Depends(get_queue)]
|
||||
BroadcasterDep = Annotated[Broadcaster, Depends(get_broadcaster)]
|
||||
@@ -0,0 +1,219 @@
|
||||
"""The work queue over HTTP, and the socket that reports on it.
|
||||
|
||||
Four operations and one socket. Anything the UI can start, it starts by
|
||||
enqueueing a job here; anything it needs to know about running work, it learns
|
||||
from `/ws/jobs`. There is no polling endpoint on purpose — a queue that pushes
|
||||
is both less code and less latency than one that is asked every second.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from pipeline import FINAL, JOBS, TASKS
|
||||
from pipeline.models import Job, JobsSnapshot, NoticeEntry, Run
|
||||
from pipeline.web.deps import BroadcasterDep, QueueDep
|
||||
from pipeline.web.schemas import EnqueueBody, JobKind
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(tags=["jobs"])
|
||||
|
||||
#: The element that renders all this, shipped with the package. Served from a
|
||||
#: route rather than mounted as a static directory because a router cannot
|
||||
#: mount one — only an app can — and needing the app to mount something is
|
||||
#: exactly the line of setup this is meant to remove. One file, so one route.
|
||||
STATIC = Path(__file__).resolve().parent / "static"
|
||||
|
||||
|
||||
@router.get("/_jobs/jobs.js", include_in_schema=False)
|
||||
def element() -> FileResponse:
|
||||
"""`<desk-jobs>`: the queue on screen, for an app with no UI for it yet.
|
||||
|
||||
<script type="module" src="/_jobs/jobs.js"></script>
|
||||
<desk-jobs></desk-jobs>
|
||||
|
||||
Cached for an hour rather than forever. It changes when the package is
|
||||
copied into a project and then hardly ever, and an app that wants it
|
||||
fingerprinted can serve the file itself from `STATIC`.
|
||||
"""
|
||||
return FileResponse(STATIC / "jobs.js", media_type="text/javascript",
|
||||
headers={"Cache-Control": "public, max-age=3600"})
|
||||
|
||||
|
||||
@router.get("/api/jobs")
|
||||
def list_jobs(queue: QueueDep) -> JobsSnapshot:
|
||||
"""What is running and what recently finished.
|
||||
|
||||
The same shape the socket pushes, so a client that cannot hold a socket
|
||||
open — a script, a `curl`, another tab — sees exactly what the
|
||||
UI sees.
|
||||
"""
|
||||
return queue.snapshot()
|
||||
|
||||
|
||||
@router.get("/api/jobs/kinds")
|
||||
def kinds() -> list[JobKind]:
|
||||
"""What can be started. Populated by `@job` and by the app's `TASKS`, so a
|
||||
new handler or a new task appears here without being registered twice.
|
||||
|
||||
Tasks and jobs in one list because they are one thing to press: a task is
|
||||
what a person means by "tag everything", and that it becomes eight jobs is
|
||||
the pipeline's business rather than the button's.
|
||||
|
||||
Which is also why a job some task queues is not listed. `tag_batch` without
|
||||
a list of uids is a button that does nothing, and the final job without the
|
||||
batches it reports on is a button that reports nothing. If a task queues
|
||||
it, the task is the thing to offer — worked out from `TASKS` rather than
|
||||
declared, so a new batch kind cannot be forgotten here.
|
||||
|
||||
Nor is a job that needs a payload, such as `mark_reclassify` without a
|
||||
`source`: a press sends none, so its button could only ever fail.
|
||||
"""
|
||||
made_for_you = {build().kind for build in TASKS.values()} | {FINAL}
|
||||
listed = [
|
||||
JobKind(kind=spec.kind, label=spec.label, solo=spec.solo)
|
||||
for spec in JOBS.values()
|
||||
if spec.kind not in made_for_you and not spec.needs
|
||||
]
|
||||
listed += [JobKind(kind=name, label=build().called, solo=True)
|
||||
for name, build in TASKS.items()]
|
||||
return sorted(listed, key=lambda k: k.kind)
|
||||
|
||||
|
||||
@router.post("/api/jobs", status_code=202)
|
||||
def enqueue(body: EnqueueBody, queue: QueueDep) -> Job | Run:
|
||||
"""Start work: one job, or one task.
|
||||
|
||||
A kind that names a task starts it rather than being refused — one task,
|
||||
on its own, which is a cascade of one and comes back as that run. Refusing
|
||||
with "use /api/cascades" was the other reading and is worse: every caller
|
||||
would have to know which names had become tasks, when what they asked for
|
||||
is exactly what this does.
|
||||
|
||||
A name that is both a job and a task is the job. A task of a single batch
|
||||
is that batch, so the two are the same work, and one row says it more
|
||||
plainly than two.
|
||||
|
||||
202 rather than 201, because the useful thing has not happened yet — this
|
||||
only promises that it will. A `solo` job that is already underway comes
|
||||
back as itself rather than as an error: the caller asked for that work to
|
||||
be in progress, and it is.
|
||||
"""
|
||||
if body.kind not in JOBS and body.kind in TASKS:
|
||||
task_id = queue.start_task(TASKS[body.kind]())
|
||||
run = queue.jobs.run(task_id)
|
||||
if run is None:
|
||||
raise HTTPException(409, f"{body.kind!r} would not start")
|
||||
return run
|
||||
spec = JOBS.get(body.kind)
|
||||
missing = [k for k in (spec.needs if spec else ()) if k not in body.payload]
|
||||
if missing:
|
||||
raise HTTPException(422, f"{body.kind!r} needs {', '.join(missing)} "
|
||||
"in its payload")
|
||||
try:
|
||||
job = queue.enqueue(body.kind, body.payload)
|
||||
except KeyError as exc:
|
||||
raise HTTPException(404, str(exc)) from exc
|
||||
if job is None:
|
||||
raise HTTPException(409, "the run it belongs to was killed")
|
||||
return job
|
||||
|
||||
|
||||
@router.post("/api/jobs/{job_id}/cancel")
|
||||
def cancel(job_id: int, queue: QueueDep) -> Job:
|
||||
"""Stop a job.
|
||||
|
||||
A queued one stops immediately. A running one stops at its next checkpoint,
|
||||
so this returns while it is still `running` with `cancel_requested` set —
|
||||
the socket reports the actual end. A job that has already finished is
|
||||
returned unchanged rather than treated as an error; the button was simply
|
||||
pressed a moment too late.
|
||||
"""
|
||||
job = queue.cancel(job_id)
|
||||
if job is None:
|
||||
raise HTTPException(404, "no such job")
|
||||
return job
|
||||
|
||||
|
||||
@router.post("/api/tasks/{task_id}/cancel")
|
||||
def cancel_task(task_id: int, queue: QueueDep) -> dict[str, int]:
|
||||
"""Stop one task: every batch of it.
|
||||
|
||||
Not the cascade it belongs to, and not the task's final job: that still
|
||||
runs, reports the cancels and calls back, so the cascade knows how this
|
||||
ended — "skip this", rather than "stop". Stopping is the cascade's own
|
||||
button, below.
|
||||
"""
|
||||
return {"ended": queue.cancel_run(task_id)}
|
||||
|
||||
|
||||
@router.post("/api/cascades/{cascade_id}/cancel")
|
||||
def cancel_cascade(cascade_id: int, queue: QueueDep) -> dict[str, int]:
|
||||
"""Stop a whole cascade: every task of it, and every job of those.
|
||||
|
||||
The same operation as above — a task and a cascade are one thing to the
|
||||
pipeline — and two endpoints because they are two buttons and mean
|
||||
different things to whoever presses one.
|
||||
|
||||
An id nobody has heard of, or one already over, ends nothing and says so
|
||||
rather than being an error — the same reading as pressing Kill a moment
|
||||
too late.
|
||||
"""
|
||||
return {"ended": queue.cancel_run(cascade_id)}
|
||||
|
||||
|
||||
@router.get("/api/notices")
|
||||
def list_notices(queue: QueueDep) -> list[NoticeEntry]:
|
||||
"""Everything the queue has said that nobody has cleared, newest first.
|
||||
|
||||
What a line on screen was always missing: somewhere to be found again. A
|
||||
notice fades in seconds and is raised, as often as not, while no tab is
|
||||
open to show it. This is the same sentence with its fix and its raw error,
|
||||
how many times it has been said, and when it last was.
|
||||
"""
|
||||
return queue.jobs.notices()
|
||||
|
||||
|
||||
@router.delete("/api/notices/{notice_id}")
|
||||
def clear_notice(notice_id: int, queue: QueueDep) -> dict[str, bool]:
|
||||
"""Done with this one. Not an error when it has already gone: two tabs, or
|
||||
the fortnight running out between the list and the press."""
|
||||
return {"cleared": queue.jobs.clear_notice(notice_id)}
|
||||
|
||||
|
||||
@router.delete("/api/notices")
|
||||
def clear_notices(queue: QueueDep) -> dict[str, int]:
|
||||
return {"cleared": queue.jobs.clear_notices()}
|
||||
|
||||
|
||||
@router.websocket("/ws/jobs")
|
||||
async def jobs_socket(ws: WebSocket, queue: QueueDep, bus: BroadcasterDep) -> None:
|
||||
"""Push queue state on connect and on every change.
|
||||
|
||||
Two message types go out here: `jobs`, the whole queue state, replaced by
|
||||
each new one; and `notice`, a single failure as the log now holds it, its
|
||||
count included. See `pipeline/notices.py` for what a client does with that.
|
||||
|
||||
The receive loop exists only to notice the client leaving — nothing the
|
||||
browser sends is read. Starting and cancelling go over HTTP, where they get
|
||||
status codes and error bodies; making the socket bidirectional would mean
|
||||
inventing a second request protocol inside it for no gain.
|
||||
"""
|
||||
await ws.accept()
|
||||
snapshot = await asyncio.to_thread(queue.snapshot)
|
||||
await bus.subscribe(ws, {"type": "jobs", **snapshot.model_dump()})
|
||||
try:
|
||||
while True:
|
||||
await ws.receive_text()
|
||||
except WebSocketDisconnect:
|
||||
pass
|
||||
except Exception:
|
||||
log.debug("jobs socket closed unexpectedly", exc_info=True)
|
||||
finally:
|
||||
bus.unsubscribe(ws)
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Schedules over HTTP: list, add, edit, delete, run now.
|
||||
|
||||
Five routes and no socket of its own. Changes go out on `/ws/jobs` as a
|
||||
`schedules` message — the same socket the job list already holds open, because
|
||||
a second one would be a second reconnect loop to get subtly wrong, for a list
|
||||
that changes a few times a day.
|
||||
|
||||
Every refusal here is a sentence a person can act on, which is the whole
|
||||
reason these are not four lines of `queue.jobs.*` behind a generic validator:
|
||||
|
||||
an unregistered kind 404, and it names what is registered
|
||||
a payload missing keys 422, and it names the keys
|
||||
a cadence it cannot read 422, and it lists all five accepted wordings
|
||||
|
||||
They arrive as `{"detail": "<sentence>"}`, which is FastAPI's own shape, so an
|
||||
app's `api()` helper turns any of them into one toast carrying the server's
|
||||
words rather than inventing its own — see `app/static/app.js` in the starter.
|
||||
|
||||
Validating the kind and the payload *here*, rather than letting the job fail at
|
||||
06:00, is the point of the endpoint existing at all. A schedule is the one way
|
||||
of starting work where nobody is watching when it runs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from pipeline import JOBS, JobQueue
|
||||
from pipeline.models import Job, Schedule
|
||||
from pipeline.schedule import Cadence
|
||||
from pipeline.web.deps import QueueDep
|
||||
from pipeline.web.jobs import STATIC
|
||||
from pipeline.web.schemas import NewSchedule, ScheduleEdit, ScheduleRow
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(tags=["schedules"])
|
||||
|
||||
|
||||
@router.get("/_jobs/schedules.js", include_in_schema=False)
|
||||
def element() -> FileResponse:
|
||||
"""`<desk-schedules>`: the schedule table on screen.
|
||||
|
||||
<script type="module" src="/_jobs/schedules.js"></script>
|
||||
<desk-schedules></desk-schedules>
|
||||
|
||||
Served and cached exactly as `/_jobs/jobs.js` is, and it reads the same
|
||||
`--desk-jobs-*` custom properties, so an app that themed one has themed
|
||||
both.
|
||||
"""
|
||||
return FileResponse(STATIC / "schedules.js", media_type="text/javascript",
|
||||
headers={"Cache-Control": "public, max-age=3600"})
|
||||
|
||||
|
||||
def listed(queue: JobQueue) -> list[ScheduleRow]:
|
||||
"""The list as it goes out, over HTTP and over the socket alike.
|
||||
|
||||
One function for both, so the two cannot drift: a client that reads the
|
||||
REST list once and then follows the socket must not be handed two shapes of
|
||||
the same thing.
|
||||
"""
|
||||
return [served(row) for row in queue.jobs.schedules()]
|
||||
|
||||
|
||||
def served(row: Schedule) -> ScheduleRow:
|
||||
"""One schedule, plus the one thing the table cannot know: whether anything
|
||||
is still registered to run it. Worked out here rather than stored, because
|
||||
it is a fact about the code that is loaded and not about the row."""
|
||||
return ScheduleRow(**row.model_dump(), unknown=row.kind not in JOBS)
|
||||
|
||||
|
||||
def _check(kind: str, payload: dict) -> None:
|
||||
"""Refuse a kind nothing runs, or a payload its handler cannot take."""
|
||||
spec = JOBS.get(kind)
|
||||
if spec is None:
|
||||
raise HTTPException(404, f"no job is registered as {kind!r}. Registered: "
|
||||
f"{', '.join(sorted(JOBS)) or 'none'}.")
|
||||
missing = [k for k in spec.needs if k not in payload]
|
||||
if missing:
|
||||
raise HTTPException(422, f"{kind!r} cannot run without "
|
||||
f"{', '.join(missing)} in its payload.")
|
||||
|
||||
|
||||
def _cadence(text: str) -> Cadence:
|
||||
"""Parse, or 422 with the accepted wordings.
|
||||
|
||||
`Cadence` already writes that sentence — see `pipeline/schedule.py` — so it
|
||||
is passed through rather than reworded here. One description of the grammar
|
||||
in the codebase, and it is next to the parser that enforces it.
|
||||
"""
|
||||
try:
|
||||
return Cadence.parse(text)
|
||||
except ValueError as bad:
|
||||
raise HTTPException(422, str(bad)) from bad
|
||||
|
||||
|
||||
def _free_name(queue: JobQueue, wanted: str) -> str:
|
||||
"""A name nothing else has: `tick`, then `tick 2`, then `tick 3`.
|
||||
|
||||
Only for a request that gave none. A name somebody typed is refused rather
|
||||
than quietly altered, because the name is the handle, and being handed a
|
||||
different one is how you edit the wrong row a month later.
|
||||
"""
|
||||
taken = {row.name for row in queue.jobs.schedules()}
|
||||
if wanted not in taken:
|
||||
return wanted
|
||||
n = 2
|
||||
while f"{wanted} {n}" in taken:
|
||||
n += 1
|
||||
return f"{wanted} {n}"
|
||||
|
||||
|
||||
@router.get("/api/schedules")
|
||||
def list_schedules(queue: QueueDep) -> list[ScheduleRow]:
|
||||
"""Every schedule, by name, with how its last job ended.
|
||||
|
||||
The same list the socket pushes, so a client that cannot hold one open — a
|
||||
script, a `curl`, a second tab — sees exactly what the element sees.
|
||||
"""
|
||||
return listed(queue)
|
||||
|
||||
|
||||
@router.post("/api/schedules", status_code=201)
|
||||
def add_schedule(body: NewSchedule, queue: QueueDep) -> ScheduleRow:
|
||||
"""Add one. 201, because unlike a job this really does now exist.
|
||||
|
||||
`name` is optional and defaults to the kind, made unique by counting. A
|
||||
name that was given and is taken is a conflict rather than a second row:
|
||||
two rows answering to one name is exactly the ambiguity the handle exists
|
||||
to prevent.
|
||||
|
||||
`next_run` is worked out here, from now, so a schedule added at 09:00
|
||||
saying `every 6h` is next at 15:00 rather than at some moment the table
|
||||
invented.
|
||||
"""
|
||||
_check(body.kind, body.payload)
|
||||
cadence = _cadence(body.cadence)
|
||||
name = body.name.strip() or _free_name(queue, body.kind)
|
||||
row = queue.jobs.add_schedule(
|
||||
name, body.kind, str(cadence), payload=body.payload, label=body.label,
|
||||
enabled=body.enabled,
|
||||
next_run=cadence.next_after(datetime.now(UTC), queue.zone).isoformat(),
|
||||
only_if_new=True)
|
||||
if row is None:
|
||||
raise HTTPException(409, f"there is already a schedule called {name!r}.")
|
||||
queue.notify_schedules()
|
||||
return served(row)
|
||||
|
||||
|
||||
@router.patch("/api/schedules/{schedule_id}")
|
||||
def edit_schedule(schedule_id: int, body: ScheduleEdit, queue: QueueDep) -> ScheduleRow:
|
||||
"""Change part of one. Only what was sent is touched.
|
||||
|
||||
A new cadence moves `next_run` with it, because the alternative is a row
|
||||
that says `every 15m` and does not run for six hours: the old time is an
|
||||
answer to a question nobody is asking any more. Switching a row back on
|
||||
does the same. It would come right on its own — a firing advances to the
|
||||
next slot from now and never walks through the missed ones — but the time
|
||||
on screen would be a lie until it did.
|
||||
"""
|
||||
row = queue.jobs.schedule(schedule_id)
|
||||
if row is None:
|
||||
raise HTTPException(404, f"there is no schedule {schedule_id}.")
|
||||
# Unset and null are one thing here: every field is optional and every
|
||||
# column is NOT NULL, so `{"cadence": null}` can only be a client leaving
|
||||
# a key in rather than a request to blank the column.
|
||||
fields = {k: v for k, v in body.model_dump(exclude_unset=True).items()
|
||||
if v is not None}
|
||||
if "kind" in fields or "payload" in fields:
|
||||
_check(fields.get("kind", row.kind), fields.get("payload", row.payload))
|
||||
now = datetime.now(UTC)
|
||||
if "cadence" in fields:
|
||||
cadence = _cadence(fields["cadence"])
|
||||
fields["cadence"] = str(cadence)
|
||||
fields["next_run"] = cadence.next_after(now, queue.zone).isoformat()
|
||||
elif fields.get("enabled") and not row.enabled:
|
||||
fields["next_run"] = _cadence(row.cadence).next_after(now, queue.zone).isoformat()
|
||||
changed = queue.jobs.update_schedule(schedule_id, **fields)
|
||||
queue.notify_schedules()
|
||||
return served(changed or row)
|
||||
|
||||
|
||||
@router.delete("/api/schedules/{schedule_id}")
|
||||
def delete_schedule(schedule_id: int, queue: QueueDep) -> dict[str, bool]:
|
||||
"""Forget one. Not an error when it has already gone — two tabs.
|
||||
|
||||
A declared default deleted here comes back on the next start, which is
|
||||
what declaring one means. Switching it off is how to be rid of it for
|
||||
good: that edit is the row's, and reconciliation never overwrites one.
|
||||
"""
|
||||
deleted = queue.jobs.delete_schedule(schedule_id)
|
||||
if deleted:
|
||||
queue.notify_schedules()
|
||||
return {"deleted": deleted}
|
||||
|
||||
|
||||
@router.post("/api/schedules/{schedule_id}/run", status_code=202)
|
||||
def run_now(schedule_id: int, queue: QueueDep) -> Job:
|
||||
"""Queue this schedule's job now, without moving when it is next due.
|
||||
|
||||
202 like `/api/jobs`, and for the same reason: what comes back is the
|
||||
promise, not the outcome. A `solo` kind already under way comes back as
|
||||
that job rather than as an error — the caller asked for the work to be in
|
||||
progress, and it is.
|
||||
"""
|
||||
row = queue.jobs.schedule(schedule_id)
|
||||
if row is None:
|
||||
raise HTTPException(404, f"there is no schedule {schedule_id}.")
|
||||
_check(row.kind, row.payload)
|
||||
job = queue.run_schedule(schedule_id)
|
||||
if job is None:
|
||||
raise HTTPException(409, f"{row.kind!r} was not queued: the run it "
|
||||
"would have belonged to was killed.")
|
||||
return job
|
||||
@@ -0,0 +1,75 @@
|
||||
"""What the queue's endpoints accept and promise back.
|
||||
|
||||
Job-shaped, so they live with the router that serves them rather than in an
|
||||
app's schema module. The `Job`, `JobsSnapshot` and `Schedule` they sit
|
||||
alongside are in `pipeline/models.py`; these are only ever request bodies, or a
|
||||
stored shape with one derived field added for the client.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from pipeline.models import Schedule
|
||||
|
||||
|
||||
class EnqueueBody(BaseModel):
|
||||
"""Start a job. `payload` is spread over the handler's keyword arguments,
|
||||
so what a kind accepts is the signature of its function and nothing else
|
||||
has to restate it."""
|
||||
|
||||
kind: str
|
||||
payload: dict[str, Any] = {}
|
||||
|
||||
|
||||
class JobKind(BaseModel):
|
||||
"""One entry from the job registry, for a UI that wants to offer them."""
|
||||
|
||||
kind: str
|
||||
label: str
|
||||
solo: bool
|
||||
|
||||
|
||||
class NewSchedule(BaseModel):
|
||||
"""Add a schedule. `name` is optional because the kind is usually answer
|
||||
enough — one `tick` schedule wants no name of its own, and the second is
|
||||
named by counting rather than by making somebody think of one."""
|
||||
|
||||
kind: str
|
||||
cadence: str
|
||||
name: str = ""
|
||||
label: str = ""
|
||||
payload: dict[str, Any] = {}
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
class ScheduleEdit(BaseModel):
|
||||
"""Change part of a schedule. Every field optional, and read with
|
||||
`exclude_unset`, so sending `{"enabled": false}` switches one off without
|
||||
also resubmitting a cadence somebody edited in another tab a second ago.
|
||||
|
||||
`name` is not here: it is the handle a declared default is matched on, and
|
||||
renaming a row would hand its edits to a fresh one inserted under the old
|
||||
name on the next start.
|
||||
"""
|
||||
|
||||
kind: str | None = None
|
||||
label: str | None = None
|
||||
cadence: str | None = None
|
||||
payload: dict[str, Any] | None = None
|
||||
enabled: bool | None = None
|
||||
|
||||
|
||||
class ScheduleRow(Schedule):
|
||||
"""A schedule as it is served: the row, plus whether anything still runs it.
|
||||
|
||||
`unknown` is not stored, because it is a fact about the code that is loaded
|
||||
rather than about the row — a handler removed in a deploy makes it true
|
||||
without the table changing, and reinstalling the plugin makes it false
|
||||
again. The element greys such a row and says `unknown kind`; the queue
|
||||
never fires one.
|
||||
"""
|
||||
|
||||
unknown: bool = False
|
||||
@@ -0,0 +1,280 @@
|
||||
/* The queue, on screen, for an app that has not written a UI yet.
|
||||
*
|
||||
* <script type="module" src="/_jobs/jobs.js"></script>
|
||||
* <desk-jobs></desk-jobs>
|
||||
*
|
||||
* That is the whole of it. The element finds `/ws/jobs` itself, shows what is
|
||||
* running with a progress bar and a Kill button each, says how many are
|
||||
* waiting, and reconnects when the server restarts under you.
|
||||
*
|
||||
* One file, and the styles are in it rather than beside it, because the
|
||||
* shadow root scopes them anyway and a second file is a second route, a
|
||||
* second cache header and a second thing to forget to copy. Themed from
|
||||
* outside through the custom properties at the top of `CSS` — set them on
|
||||
* `desk-jobs` or on `:root` and nothing here has to be forked.
|
||||
*
|
||||
* What it deliberately does not do is talk about failures. It raises a
|
||||
* `desk-jobs:notice` event and leaves the wording to the app, because a
|
||||
* useful sentence names things only the app knows — its backends, its feeds,
|
||||
* its config keys. `pipeline/notices.py` is the same split on the server.
|
||||
*
|
||||
* Attributes, all optional:
|
||||
* base="/api" prefix for the HTTP calls, if the router is mounted deeper
|
||||
* socket="..." the socket's URL, if it is not `/ws/jobs` on this origin
|
||||
* probe="/api/status"
|
||||
* an authenticated endpoint to try when the socket closes. A
|
||||
* refused handshake and a dead server look identical from
|
||||
* here — the server rejects an unauthenticated upgrade before
|
||||
* completing it, so the browser reports the same abnormal
|
||||
* close for "your session expired" as for "the server is
|
||||
* gone". One cheap request tells them apart. Without this the
|
||||
* element assumes the server went away, which is the more
|
||||
* common reason and the less confusing thing to be wrong about.
|
||||
* idle="hide" with nothing running or queued, take up no space. The
|
||||
* default, since a chip that is always there is furniture.
|
||||
*/
|
||||
|
||||
const CSS = `
|
||||
:host {
|
||||
--desk-jobs-bg: #16181d;
|
||||
--desk-jobs-fg: #e7e9ee;
|
||||
--desk-jobs-dim: #9aa1ad;
|
||||
--desk-jobs-accent: #4c8dff;
|
||||
--desk-jobs-error: #ff6b6b;
|
||||
--desk-jobs-radius: 10px;
|
||||
--desk-jobs-font: system-ui, -apple-system, "Segoe UI", sans-serif;
|
||||
--desk-jobs-gap: 8px;
|
||||
|
||||
display: block;
|
||||
font-family: var(--desk-jobs-font);
|
||||
font-size: 13px;
|
||||
color: var(--desk-jobs-fg);
|
||||
}
|
||||
:host([hidden]) { display: none; }
|
||||
|
||||
.strip {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--desk-jobs-gap);
|
||||
background: var(--desk-jobs-bg);
|
||||
border-radius: var(--desk-jobs-radius);
|
||||
padding: 10px 12px;
|
||||
}
|
||||
.job { display: grid; grid-template-columns: 1fr auto; gap: 2px 10px; align-items: center; }
|
||||
.what { font-weight: 600; }
|
||||
.said { color: var(--desk-jobs-dim); grid-column: 1; }
|
||||
.count { color: var(--desk-jobs-dim); font-variant-numeric: tabular-nums; }
|
||||
|
||||
/* A bar only when there is a fraction to draw. A handler that reports no
|
||||
* total is working on one thing, and an empty trough reads as stalled. */
|
||||
.bar {
|
||||
grid-column: 1 / -1;
|
||||
height: 3px;
|
||||
border-radius: 2px;
|
||||
background: color-mix(in srgb, var(--desk-jobs-fg) 15%, transparent);
|
||||
overflow: hidden;
|
||||
}
|
||||
.bar > i { display: block; height: 100%; background: var(--desk-jobs-accent); transition: width .2s; }
|
||||
|
||||
button {
|
||||
font: inherit;
|
||||
color: var(--desk-jobs-dim);
|
||||
background: none;
|
||||
border: 1px solid currentColor;
|
||||
border-radius: 6px;
|
||||
padding: 1px 8px;
|
||||
cursor: pointer;
|
||||
}
|
||||
button:hover { color: var(--desk-jobs-error); }
|
||||
button[disabled] { opacity: .4; cursor: default; }
|
||||
|
||||
.offline { color: var(--desk-jobs-error); }
|
||||
`;
|
||||
|
||||
/** Backing off rather than hammering. The usual reason a socket closed is the
|
||||
* server restarting under --reload, which comes back inside a second; the
|
||||
* ceiling is for when it is not coming back at all. */
|
||||
const FIRST_WAIT = 500;
|
||||
const LONGEST_WAIT = 30000;
|
||||
|
||||
/** How long down is long enough to say so. A reload blinks the socket every
|
||||
* time a file is saved, and a warning that flashes on every save is one you
|
||||
* stop reading — which would cost us the case that matters, where the server
|
||||
* really is gone and the strip is quietly showing state from a minute ago. */
|
||||
const QUIET_ABOUT_IT_FOR = 4000;
|
||||
|
||||
class DeskJobs extends HTMLElement {
|
||||
#root;
|
||||
#socket = null;
|
||||
#wait = FIRST_WAIT;
|
||||
#timer = null;
|
||||
#state = { running: [], queued: [], recent: [] };
|
||||
#offline = false;
|
||||
#killing = new Set();
|
||||
|
||||
constructor() {
|
||||
super();
|
||||
this.#root = this.attachShadow({ mode: "open" });
|
||||
const style = document.createElement("style");
|
||||
style.textContent = CSS;
|
||||
this.#root.append(style, document.createElement("div"));
|
||||
}
|
||||
|
||||
connectedCallback() { this.#connect(); this.#render(); }
|
||||
|
||||
disconnectedCallback() {
|
||||
clearTimeout(this.#timer);
|
||||
this.#socket?.close();
|
||||
this.#socket = null;
|
||||
}
|
||||
|
||||
get #base() { return this.getAttribute("base") ?? ""; }
|
||||
|
||||
get #socketUrl() {
|
||||
const given = this.getAttribute("socket");
|
||||
if (given) return given;
|
||||
const scheme = location.protocol === "https:" ? "wss" : "ws";
|
||||
return `${scheme}://${location.host}${this.#base}/ws/jobs`;
|
||||
}
|
||||
|
||||
#connect() {
|
||||
let socket;
|
||||
try {
|
||||
socket = new WebSocket(this.#socketUrl);
|
||||
} catch {
|
||||
this.#reconnect(); // a malformed URL, or no network
|
||||
return;
|
||||
}
|
||||
this.#socket = socket;
|
||||
|
||||
socket.onopen = () => {
|
||||
this.#wait = FIRST_WAIT;
|
||||
this.#offline = false;
|
||||
this.#render();
|
||||
};
|
||||
socket.onmessage = (event) => {
|
||||
let message;
|
||||
try { message = JSON.parse(event.data); } catch { return; }
|
||||
if (message.type === "jobs") {
|
||||
const { type, ...state } = message;
|
||||
this.#state = state;
|
||||
this.#render();
|
||||
} else if (message.type === "notice") {
|
||||
// The app's to word. Composed, so a listener on `document` hears it
|
||||
// through the shadow boundary.
|
||||
this.dispatchEvent(new CustomEvent("desk-jobs:notice", {
|
||||
detail: message, bubbles: true, composed: true,
|
||||
}));
|
||||
}
|
||||
};
|
||||
socket.onerror = () => socket.close();
|
||||
socket.onclose = async () => {
|
||||
if (this.#socket !== socket) return; // a later connection owns us now
|
||||
this.#socket = null;
|
||||
await this.#whyClosed();
|
||||
this.#reconnect();
|
||||
};
|
||||
}
|
||||
|
||||
/** Ask before assuming. See the `probe` attribute. */
|
||||
async #whyClosed() {
|
||||
const probe = this.getAttribute("probe");
|
||||
if (!probe) return;
|
||||
try {
|
||||
const answer = await fetch(this.#base + probe, { credentials: "same-origin" });
|
||||
if (answer.status === 401 || answer.status === 403) {
|
||||
this.dispatchEvent(new CustomEvent("desk-jobs:signed-out", {
|
||||
bubbles: true, composed: true,
|
||||
}));
|
||||
}
|
||||
} catch { /* offline; the strip says so below */ }
|
||||
}
|
||||
|
||||
#reconnect() {
|
||||
if (this.#wait >= QUIET_ABOUT_IT_FOR) {
|
||||
this.#offline = true;
|
||||
this.#render();
|
||||
}
|
||||
clearTimeout(this.#timer);
|
||||
this.#timer = setTimeout(() => this.#connect(), this.#wait);
|
||||
this.#wait = Math.min(this.#wait * 2, LONGEST_WAIT);
|
||||
}
|
||||
|
||||
async #kill(id) {
|
||||
this.#killing.add(id);
|
||||
this.#render();
|
||||
try {
|
||||
await fetch(`${this.#base}/api/jobs/${id}/cancel`,
|
||||
{ method: "POST", credentials: "same-origin" });
|
||||
} finally {
|
||||
// The socket reports the actual ending — a running job stops at its next
|
||||
// checkpoint, so the button going quiet is the whole of what happens here.
|
||||
this.#killing.delete(id);
|
||||
}
|
||||
}
|
||||
|
||||
#render() {
|
||||
const { running = [], queued = [] } = this.#state;
|
||||
const strip = this.#root.lastElementChild;
|
||||
this.hidden = this.getAttribute("idle") !== "show"
|
||||
&& !running.length && !queued.length && !this.#offline;
|
||||
|
||||
strip.className = "strip";
|
||||
strip.replaceChildren();
|
||||
|
||||
if (this.#offline) {
|
||||
const said = document.createElement("div");
|
||||
said.className = "offline";
|
||||
said.textContent = "Not connected — what you see here may be out of date.";
|
||||
strip.append(said);
|
||||
}
|
||||
|
||||
for (const job of running) {
|
||||
strip.append(this.#row(job));
|
||||
}
|
||||
if (queued.length) {
|
||||
const waiting = document.createElement("div");
|
||||
waiting.className = "count";
|
||||
waiting.textContent = `${queued.length} waiting`;
|
||||
strip.append(waiting);
|
||||
}
|
||||
}
|
||||
|
||||
#row(job) {
|
||||
const row = document.createElement("div");
|
||||
row.className = "job";
|
||||
|
||||
const what = document.createElement("span");
|
||||
what.className = "what";
|
||||
what.textContent = job.label || job.kind;
|
||||
row.append(what);
|
||||
|
||||
const kill = document.createElement("button");
|
||||
kill.textContent = "Kill";
|
||||
kill.disabled = this.#killing.has(job.id) || job.cancel_requested;
|
||||
kill.addEventListener("click", () => this.#kill(job.id));
|
||||
row.append(kill);
|
||||
|
||||
if (job.message) {
|
||||
const said = document.createElement("span");
|
||||
said.className = "said";
|
||||
said.textContent = job.message;
|
||||
row.append(said);
|
||||
}
|
||||
if (job.total > 0) {
|
||||
const bar = document.createElement("div");
|
||||
bar.className = "bar";
|
||||
const fill = document.createElement("i");
|
||||
fill.style.width = `${Math.min(100, (job.done / job.total) * 100)}%`;
|
||||
bar.append(fill);
|
||||
row.append(bar);
|
||||
}
|
||||
return row;
|
||||
}
|
||||
}
|
||||
|
||||
if (!customElements.get("desk-jobs")) {
|
||||
customElements.define("desk-jobs", DeskJobs);
|
||||
}
|
||||
|
||||
export { DeskJobs };
|
||||
@@ -0,0 +1,543 @@
|
||||
/* The schedule table, on screen, for an app that has not written a UI yet.
|
||||
*
|
||||
* <script type="module" src="/_jobs/schedules.js"></script>
|
||||
* <desk-schedules></desk-schedules>
|
||||
*
|
||||
* A row per schedule — name, kind, cadence, on or off, next run, last run and
|
||||
* how that job ended — plus Run now, Delete, and a row at the bottom for
|
||||
* adding one. The kinds come from `/api/jobs/kinds`, so a new `@job` is a new
|
||||
* option here without this file being touched.
|
||||
*
|
||||
* It reads `/ws/jobs` for `schedules` messages, so a row edited in another tab
|
||||
* redraws in this one, and a firing moves the times without anybody polling.
|
||||
* The socket's opening message is the *job* list, though, so the table is
|
||||
* fetched once on connect: the push says what changed, not what there is. The
|
||||
* job list is read as it goes past as well, for the last column — a schedule
|
||||
* is pushed when it fires, which is before its job has ended, so the state it
|
||||
* ended in can only come from there.
|
||||
*
|
||||
* The same one-file, shadow-root, custom-property shape as `<desk-jobs>`, and
|
||||
* deliberately the same `--desk-jobs-*` names: an app that themed one element
|
||||
* has themed both, and two schemes for two halves of one page is the thing
|
||||
* nobody wants to maintain.
|
||||
*
|
||||
* Failures are not worded here. A refused change raises `desk-schedules:error`
|
||||
* carrying the server's own `detail` — which the endpoints promise is one
|
||||
* readable sentence — and the app toasts it. Notices arriving on the socket
|
||||
* are re-raised as `desk-jobs:notice`, the same event `<desk-jobs>` raises, so
|
||||
* an app that listens for one is listening for both.
|
||||
*
|
||||
* Attributes, all optional:
|
||||
* base="/api" prefix for the HTTP calls, if the router is mounted deeper
|
||||
* socket="..." the socket's URL, if it is not `/ws/jobs` on this origin
|
||||
*/
|
||||
|
||||
const CSS = `
|
||||
:host {
|
||||
--desk-jobs-bg: #16181d;
|
||||
--desk-jobs-fg: #e7e9ee;
|
||||
--desk-jobs-dim: #9aa1ad;
|
||||
--desk-jobs-accent: #4c8dff;
|
||||
--desk-jobs-error: #ff6b6b;
|
||||
--desk-jobs-radius: 10px;
|
||||
--desk-jobs-font: system-ui, -apple-system, "Segoe UI", sans-serif;
|
||||
--desk-jobs-gap: 8px;
|
||||
|
||||
display: block;
|
||||
font-family: var(--desk-jobs-font);
|
||||
font-size: 13px;
|
||||
color: var(--desk-jobs-fg);
|
||||
}
|
||||
:host([hidden]) { display: none; }
|
||||
|
||||
.panel {
|
||||
background: var(--desk-jobs-bg);
|
||||
border-radius: var(--desk-jobs-radius);
|
||||
padding: 10px 12px;
|
||||
}
|
||||
|
||||
table { width: 100%; border-collapse: collapse; }
|
||||
th, td {
|
||||
text-align: left;
|
||||
padding: 6px 8px 6px 0;
|
||||
vertical-align: baseline;
|
||||
border-bottom: 1px solid color-mix(in srgb, var(--desk-jobs-fg) 12%, transparent);
|
||||
}
|
||||
th { color: var(--desk-jobs-dim); font-weight: 600; white-space: nowrap; }
|
||||
tr:last-child > td { border-bottom: 0; }
|
||||
td.actions { text-align: right; white-space: nowrap; }
|
||||
|
||||
.name { font-weight: 600; }
|
||||
.kind, .when { color: var(--desk-jobs-dim); font-variant-numeric: tabular-nums; }
|
||||
.off { opacity: .55; }
|
||||
.bad { color: var(--desk-jobs-error); }
|
||||
|
||||
/* The cadence is edited in place: a box that looks like text until you are in
|
||||
* it, because a table of eight input fields reads as a form to fill in rather
|
||||
* than as a list of what is running. */
|
||||
input.cadence, input.text {
|
||||
font: inherit;
|
||||
color: inherit;
|
||||
background: none;
|
||||
border: 1px solid transparent;
|
||||
border-radius: 6px;
|
||||
padding: 1px 4px;
|
||||
min-width: 9em;
|
||||
}
|
||||
input.cadence:hover { border-color: color-mix(in srgb, var(--desk-jobs-fg) 20%, transparent); }
|
||||
input:focus { outline: none; border-color: var(--desk-jobs-accent); }
|
||||
input::placeholder { color: var(--desk-jobs-dim); }
|
||||
|
||||
select {
|
||||
font: inherit;
|
||||
color: inherit;
|
||||
background: var(--desk-jobs-bg);
|
||||
border: 1px solid color-mix(in srgb, var(--desk-jobs-fg) 20%, transparent);
|
||||
border-radius: 6px;
|
||||
padding: 1px 4px;
|
||||
}
|
||||
|
||||
button {
|
||||
font: inherit;
|
||||
color: var(--desk-jobs-dim);
|
||||
background: none;
|
||||
border: 1px solid currentColor;
|
||||
border-radius: 6px;
|
||||
padding: 1px 8px;
|
||||
margin-left: 6px;
|
||||
cursor: pointer;
|
||||
}
|
||||
button:hover { color: var(--desk-jobs-accent); }
|
||||
button.remove:hover { color: var(--desk-jobs-error); }
|
||||
button[disabled] { opacity: .4; cursor: default; }
|
||||
|
||||
.state { font-size: 11px; padding: 0 5px; border-radius: 999px;
|
||||
border: 1px solid currentColor; margin-left: 6px; }
|
||||
.state.failed, .state.cancelled { color: var(--desk-jobs-error); }
|
||||
.state.running, .state.queued { color: var(--desk-jobs-accent); }
|
||||
|
||||
.empty, .offline { color: var(--desk-jobs-dim); padding: 6px 0; }
|
||||
.offline { color: var(--desk-jobs-error); }
|
||||
`;
|
||||
|
||||
/** The same backoff as `<desk-jobs>`: the usual reason a socket closed is the
|
||||
* server restarting under --reload, which comes back inside a second. */
|
||||
const FIRST_WAIT = 500;
|
||||
const LONGEST_WAIT = 30000;
|
||||
|
||||
/** The grammar, in the one place a person is asked to use it. Two of the five
|
||||
* as the placeholder, because a box wide enough for all five is a box wide
|
||||
* enough to unbalance the row it is in — and all five on hover, which is also
|
||||
* what the server's refusal says if they get it wrong anyway. */
|
||||
const FORMS = "every 15m · every 6h · every 2d · daily at 06:00 · weekly mon 07:00";
|
||||
const HINT = "every 15m · daily at 06:00";
|
||||
|
||||
/** A stamp as a person reads it. Today is a time, anything else is a date and
|
||||
* a time, and "never" is said rather than left blank — an empty cell reads as
|
||||
* a bug, and "never" is a fact. */
|
||||
function when(stamp) {
|
||||
if (!stamp) return "never";
|
||||
const at = new Date(stamp);
|
||||
if (isNaN(at)) return stamp;
|
||||
const today = at.toDateString() === new Date().toDateString();
|
||||
const clock = at.toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" });
|
||||
return today ? clock : `${at.toLocaleDateString()} ${clock}`;
|
||||
}
|
||||
|
||||
class DeskSchedules extends HTMLElement {
|
||||
#root;
|
||||
#socket = null;
|
||||
#wait = FIRST_WAIT;
|
||||
#timer = null;
|
||||
#rows = [];
|
||||
#kinds = [];
|
||||
#offline = false;
|
||||
#busy = new Set();
|
||||
#loaded = false;
|
||||
|
||||
constructor() {
|
||||
super();
|
||||
this.#root = this.attachShadow({ mode: "open" });
|
||||
const style = document.createElement("style");
|
||||
style.textContent = CSS;
|
||||
this.#root.append(style, document.createElement("div"));
|
||||
}
|
||||
|
||||
connectedCallback() {
|
||||
this.#render();
|
||||
this.#load();
|
||||
this.#connect();
|
||||
}
|
||||
|
||||
disconnectedCallback() {
|
||||
clearTimeout(this.#timer);
|
||||
this.#socket?.close();
|
||||
this.#socket = null;
|
||||
}
|
||||
|
||||
get #base() { return this.getAttribute("base") ?? ""; }
|
||||
|
||||
get #socketUrl() {
|
||||
const given = this.getAttribute("socket");
|
||||
if (given) return given;
|
||||
const scheme = location.protocol === "https:" ? "wss" : "ws";
|
||||
return `${scheme}://${location.host}${this.#base}/ws/jobs`;
|
||||
}
|
||||
|
||||
/* -- the server ------------------------------------------------------- */
|
||||
|
||||
/** Every call goes through here, so one place raises the event and one
|
||||
* place knows that a refusal carries `{"detail": "<sentence>"}`. */
|
||||
async #ask(path, options) {
|
||||
const o = Object.assign({ credentials: "same-origin" }, options);
|
||||
if (o.body !== undefined && typeof o.body !== "string") {
|
||||
o.headers = Object.assign({ "Content-Type": "application/json" }, o.headers);
|
||||
o.body = JSON.stringify(o.body);
|
||||
}
|
||||
let answer;
|
||||
try {
|
||||
answer = await fetch(this.#base + path, o);
|
||||
} catch (e) {
|
||||
this.#failed("Could not reach the server.");
|
||||
throw e;
|
||||
}
|
||||
const text = await answer.text();
|
||||
let body = null;
|
||||
if (text) { try { body = JSON.parse(text); } catch (e) { body = null; } }
|
||||
if (!answer.ok) {
|
||||
this.#failed((body && body.detail)
|
||||
|| `The server answered ${answer.status} and said nothing about why.`);
|
||||
throw new Error(String((body && body.detail) || answer.status));
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
#failed(detail) {
|
||||
// Composed, so a listener on `document` hears it through the shadow
|
||||
// boundary. The wording is the server's; this element has no opinion.
|
||||
this.dispatchEvent(new CustomEvent("desk-schedules:error", {
|
||||
detail: { detail: String(detail) }, bubbles: true, composed: true,
|
||||
}));
|
||||
}
|
||||
|
||||
async #load() {
|
||||
try {
|
||||
const [rows, kinds] = await Promise.all([
|
||||
this.#ask("/api/schedules"),
|
||||
this.#ask("/api/jobs/kinds"),
|
||||
]);
|
||||
this.#rows = rows || [];
|
||||
this.#kinds = kinds || [];
|
||||
} catch (e) {
|
||||
/* already raised */
|
||||
} finally {
|
||||
this.#loaded = true;
|
||||
this.#render();
|
||||
}
|
||||
}
|
||||
|
||||
#connect() {
|
||||
let socket;
|
||||
try {
|
||||
socket = new WebSocket(this.#socketUrl);
|
||||
} catch {
|
||||
this.#reconnect();
|
||||
return;
|
||||
}
|
||||
this.#socket = socket;
|
||||
|
||||
socket.onopen = () => {
|
||||
this.#wait = FIRST_WAIT;
|
||||
if (this.#offline) {
|
||||
// Down and back: what changed while we were away came as nothing, so
|
||||
// the list is read again rather than waited for.
|
||||
this.#offline = false;
|
||||
this.#load();
|
||||
}
|
||||
this.#render();
|
||||
};
|
||||
socket.onmessage = (event) => {
|
||||
let message;
|
||||
try { message = JSON.parse(event.data); } catch { return; }
|
||||
if (message.type === "schedules") {
|
||||
this.#rows = message.schedules || [];
|
||||
this.#render();
|
||||
} else if (message.type === "jobs") {
|
||||
this.#catchUp(message);
|
||||
} else if (message.type === "notice") {
|
||||
this.dispatchEvent(new CustomEvent("desk-jobs:notice", {
|
||||
detail: message, bubbles: true, composed: true,
|
||||
}));
|
||||
}
|
||||
};
|
||||
socket.onerror = () => socket.close();
|
||||
socket.onclose = () => {
|
||||
if (this.#socket !== socket) return; // a later connection owns us
|
||||
this.#socket = null;
|
||||
this.#reconnect();
|
||||
};
|
||||
}
|
||||
|
||||
/* The last column says how the last job ended, and the `schedules` message
|
||||
* is pushed when a schedule *fires* — at which point that job has not ended
|
||||
* and the cell reads `running`. Nothing pushes a schedule again when the
|
||||
* job finishes, so without this the table goes on reporting a job as
|
||||
* running long after it is done, which is the one thing this column is for.
|
||||
*
|
||||
* The job list arrives on this same socket four times a second while
|
||||
* anything is going, and it carries the state of every job worth knowing
|
||||
* about, so the answer is already here: take it off the snapshot rather
|
||||
* than ask the server a second time. A job old enough to have left `recent`
|
||||
* keeps whatever it was last seen as; its schedule has fired since, or it
|
||||
* has been on screen for an hour. */
|
||||
#catchUp(snapshot) {
|
||||
const states = new Map();
|
||||
for (const list of ["running", "queued", "recent"]) {
|
||||
for (const job of snapshot[list] || []) states.set(job.id, job.state);
|
||||
}
|
||||
let moved = false;
|
||||
for (const row of this.#rows) {
|
||||
const state = states.get(row.last_job_id);
|
||||
if (state && state !== row.last_state) {
|
||||
row.last_state = state;
|
||||
moved = true;
|
||||
}
|
||||
}
|
||||
if (moved) this.#render();
|
||||
}
|
||||
|
||||
#reconnect() {
|
||||
if (this.#wait >= 4000) {
|
||||
this.#offline = true;
|
||||
this.#render();
|
||||
}
|
||||
clearTimeout(this.#timer);
|
||||
this.#timer = setTimeout(() => this.#connect(), this.#wait);
|
||||
this.#wait = Math.min(this.#wait * 2, LONGEST_WAIT);
|
||||
}
|
||||
|
||||
/* One wrapper for every change: the row is greyed while the request is in
|
||||
* the air, and the new list arrives on the socket rather than being patched
|
||||
* in here — so what is on screen is always what the server holds. */
|
||||
async #change(id, run) {
|
||||
this.#busy.add(id);
|
||||
this.#render();
|
||||
try {
|
||||
await run();
|
||||
} catch (e) {
|
||||
/* already raised */
|
||||
} finally {
|
||||
this.#busy.delete(id);
|
||||
this.#render();
|
||||
}
|
||||
}
|
||||
|
||||
/* -- drawing ---------------------------------------------------------- */
|
||||
|
||||
#render() {
|
||||
const panel = this.#root.lastElementChild;
|
||||
panel.className = "panel";
|
||||
panel.replaceChildren();
|
||||
|
||||
if (this.#offline) {
|
||||
const said = document.createElement("div");
|
||||
said.className = "offline";
|
||||
said.textContent = "Not connected — what you see here may be out of date.";
|
||||
panel.append(said);
|
||||
}
|
||||
|
||||
const table = document.createElement("table");
|
||||
table.append(this.#head());
|
||||
const body = document.createElement("tbody");
|
||||
for (const row of this.#rows) body.append(this.#row(row));
|
||||
body.append(this.#adder());
|
||||
table.append(body);
|
||||
panel.append(table);
|
||||
|
||||
if (this.#loaded && !this.#rows.length) {
|
||||
const empty = document.createElement("div");
|
||||
empty.className = "empty";
|
||||
empty.textContent = "Nothing is scheduled. Add one below.";
|
||||
panel.insertBefore(empty, table);
|
||||
}
|
||||
}
|
||||
|
||||
#head() {
|
||||
const head = document.createElement("thead");
|
||||
const row = document.createElement("tr");
|
||||
for (const label of ["Name", "Kind", "Cadence", "On", "Next", "Last", ""]) {
|
||||
const cell = document.createElement("th");
|
||||
cell.textContent = label;
|
||||
row.append(cell);
|
||||
}
|
||||
head.append(row);
|
||||
return head;
|
||||
}
|
||||
|
||||
#row(schedule) {
|
||||
const row = document.createElement("tr");
|
||||
const busy = this.#busy.has(schedule.id);
|
||||
if (!schedule.enabled) row.classList.add("off");
|
||||
|
||||
row.append(this.#cell("name", schedule.name));
|
||||
|
||||
const kind = this.#cell("kind", schedule.kind);
|
||||
if (schedule.unknown) {
|
||||
// Never fires, and says so rather than sitting there with a next run it
|
||||
// is not going to keep. Deleting it is the only thing left to do.
|
||||
const bad = document.createElement("span");
|
||||
bad.className = "state bad";
|
||||
bad.textContent = "unknown kind";
|
||||
kind.append(bad);
|
||||
}
|
||||
row.append(kind);
|
||||
|
||||
const cadence = document.createElement("td");
|
||||
const box = document.createElement("input");
|
||||
box.className = "cadence";
|
||||
box.value = schedule.cadence;
|
||||
box.title = FORMS;
|
||||
box.disabled = busy;
|
||||
const save = () => {
|
||||
const wanted = box.value.trim();
|
||||
if (!wanted || wanted === schedule.cadence) {
|
||||
box.value = schedule.cadence;
|
||||
return;
|
||||
}
|
||||
this.#change(schedule.id, () => this.#ask(`/api/schedules/${schedule.id}`,
|
||||
{ method: "PATCH", body: { cadence: wanted } }));
|
||||
};
|
||||
box.addEventListener("change", save);
|
||||
box.addEventListener("keydown", (e) => { if (e.key === "Enter") box.blur(); });
|
||||
cadence.append(box);
|
||||
row.append(cadence);
|
||||
|
||||
const onOff = document.createElement("td");
|
||||
const toggle = document.createElement("input");
|
||||
toggle.type = "checkbox";
|
||||
toggle.checked = !!schedule.enabled;
|
||||
toggle.disabled = busy;
|
||||
toggle.title = schedule.enabled ? "switch this schedule off" : "switch it on";
|
||||
toggle.addEventListener("change", () => this.#change(schedule.id,
|
||||
() => this.#ask(`/api/schedules/${schedule.id}`,
|
||||
{ method: "PATCH", body: { enabled: toggle.checked } })));
|
||||
onOff.append(toggle);
|
||||
row.append(onOff);
|
||||
|
||||
row.append(this.#cell("when", schedule.enabled ? when(schedule.next_run) : "—"));
|
||||
|
||||
const last = this.#cell("when", when(schedule.last_run));
|
||||
if (schedule.last_state) {
|
||||
const state = document.createElement("span");
|
||||
state.className = `state ${schedule.last_state}`;
|
||||
state.textContent = schedule.last_state;
|
||||
last.append(state);
|
||||
}
|
||||
row.append(last);
|
||||
|
||||
const actions = document.createElement("td");
|
||||
actions.className = "actions";
|
||||
const now = document.createElement("button");
|
||||
now.textContent = "Run now";
|
||||
now.disabled = busy || schedule.unknown;
|
||||
now.addEventListener("click", () => this.#change(schedule.id,
|
||||
() => this.#ask(`/api/schedules/${schedule.id}/run`, { method: "POST" })));
|
||||
const remove = document.createElement("button");
|
||||
remove.className = "remove";
|
||||
remove.textContent = "Delete";
|
||||
remove.disabled = busy;
|
||||
remove.addEventListener("click", () => this.#change(schedule.id,
|
||||
() => this.#ask(`/api/schedules/${schedule.id}`, { method: "DELETE" })));
|
||||
actions.append(now, remove);
|
||||
row.append(actions);
|
||||
|
||||
return row;
|
||||
}
|
||||
|
||||
#cell(className, text) {
|
||||
const cell = document.createElement("td");
|
||||
cell.className = className;
|
||||
const span = document.createElement("span");
|
||||
span.textContent = text;
|
||||
cell.append(span);
|
||||
return cell;
|
||||
}
|
||||
|
||||
/** The add row. In the table rather than in a form above it, because what
|
||||
* is being added is another one of these and the columns already say what
|
||||
* each box is for. */
|
||||
#adder() {
|
||||
const row = document.createElement("tr");
|
||||
const busy = this.#busy.has(0);
|
||||
|
||||
const name = document.createElement("input");
|
||||
name.className = "text";
|
||||
name.placeholder = "optional";
|
||||
name.disabled = busy;
|
||||
row.append(this.#wrap(name));
|
||||
|
||||
const kind = document.createElement("select");
|
||||
kind.disabled = busy || !this.#kinds.length;
|
||||
for (const k of this.#kinds) {
|
||||
const option = document.createElement("option");
|
||||
option.value = k.kind;
|
||||
option.textContent = k.label || k.kind;
|
||||
kind.append(option);
|
||||
}
|
||||
if (!this.#kinds.length) {
|
||||
const none = document.createElement("option");
|
||||
none.textContent = this.#loaded ? "nothing registered" : "loading…";
|
||||
kind.append(none);
|
||||
}
|
||||
row.append(this.#wrap(kind));
|
||||
|
||||
const cadence = document.createElement("input");
|
||||
cadence.className = "text";
|
||||
cadence.placeholder = HINT;
|
||||
cadence.title = FORMS;
|
||||
cadence.disabled = busy;
|
||||
row.append(this.#wrap(cadence));
|
||||
|
||||
row.append(document.createElement("td"), document.createElement("td"),
|
||||
document.createElement("td"));
|
||||
|
||||
const actions = document.createElement("td");
|
||||
actions.className = "actions";
|
||||
const add = document.createElement("button");
|
||||
add.textContent = "Add";
|
||||
add.disabled = busy;
|
||||
const submit = () => {
|
||||
if (!kind.value) return;
|
||||
this.#change(0, async () => {
|
||||
await this.#ask("/api/schedules", {
|
||||
method: "POST",
|
||||
body: { name: name.value.trim(), kind: kind.value,
|
||||
cadence: cadence.value.trim() },
|
||||
});
|
||||
// Cleared only on success, so a refused cadence is still there to be
|
||||
// corrected rather than retyped from memory.
|
||||
name.value = "";
|
||||
cadence.value = "";
|
||||
});
|
||||
};
|
||||
add.addEventListener("click", submit);
|
||||
for (const box of [name, cadence]) {
|
||||
box.addEventListener("keydown", (e) => { if (e.key === "Enter") submit(); });
|
||||
}
|
||||
actions.append(add);
|
||||
row.append(actions);
|
||||
return row;
|
||||
}
|
||||
|
||||
#wrap(control) {
|
||||
const cell = document.createElement("td");
|
||||
cell.append(control);
|
||||
return cell;
|
||||
}
|
||||
}
|
||||
|
||||
if (!customElements.get("desk-schedules")) {
|
||||
customElements.define("desk-schedules", DeskSchedules);
|
||||
}
|
||||
|
||||
export { DeskSchedules };
|
||||
@@ -0,0 +1,113 @@
|
||||
"""A worker with no web app around it.
|
||||
|
||||
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap
|
||||
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap \\
|
||||
--bootstrap-arg config.yaml --workers cpu=2,http=8 --until-idle
|
||||
|
||||
What a second machine, a cron entry or a batch script runs. It is the same
|
||||
`JobQueue` the web app starts, on the same table, so it races correctly against
|
||||
any other worker on that file: `Jobs.claim` is one atomic statement and every
|
||||
claimed job is held on a lease. Start as many as the file will bear.
|
||||
|
||||
The bootstrap is the one `isolate=True` already asks for — `module:function`
|
||||
returning `(config, store)` — and it is used for the same two reasons here:
|
||||
importing it fills `JOBS`, and calling it builds the pair the handlers take.
|
||||
An app that has written one for isolation has written this worker's too.
|
||||
|
||||
`--until-idle` drains what is queued and exits, which is the batch shape. Without
|
||||
it the worker runs until it is interrupted, and a SIGINT or SIGTERM stops it the
|
||||
way the web app stops: handlers in flight get `JobQueue.STOP_GRACE` to land,
|
||||
and what is still running after that goes back to the queue on its lease.
|
||||
|
||||
**This worker fires schedules too**, which is what makes it a complete
|
||||
replacement for the web app's own worker — a machine running only this still
|
||||
gets its 06:00 job. `--no-schedules` turns that off, and is what every worker
|
||||
after the first passes: they all share the jobs table safely, but the pile-up
|
||||
guard reads `last_job_id`, which neither of two simultaneous firings has
|
||||
written yet when the other looks. One scheduler per file, as many workers as
|
||||
you like. `--timezone` is what the clock cadences are read in; without it they
|
||||
are read in the machine's own zone, which on a server is UTC and is usually not
|
||||
what the person who wrote `daily at 06:00` meant.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
import signal
|
||||
|
||||
from pipeline.child import resolve
|
||||
from pipeline.queue import JobQueue
|
||||
from pipeline.table import Jobs
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def workers(text: str) -> dict[str, int]:
|
||||
"""`cpu=2,http=8` to `{"cpu": 2, "http": 8}`."""
|
||||
counts: dict[str, int] = {}
|
||||
for part in filter(None, (p.strip() for p in text.split(","))):
|
||||
name, _, n = part.partition("=")
|
||||
if not name or not n.isdigit():
|
||||
raise argparse.ArgumentTypeError(f"not resource=count: {part!r}")
|
||||
counts[name] = int(n)
|
||||
return counts
|
||||
|
||||
|
||||
async def serve(queue: JobQueue) -> None:
|
||||
"""Run until SIGINT or SIGTERM."""
|
||||
stop = asyncio.Event()
|
||||
loop = asyncio.get_running_loop()
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
# Windows has no loop signal handlers; there Ctrl+C arrives as
|
||||
# KeyboardInterrupt instead, and `main` turns it into the same stop.
|
||||
with contextlib.suppress(NotImplementedError, RuntimeError):
|
||||
loop.add_signal_handler(sig, stop.set)
|
||||
await queue.start()
|
||||
try:
|
||||
await stop.wait()
|
||||
finally:
|
||||
await queue.stop()
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(prog="pipeline.worker", description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--db", required=True, help="the SQLite file the jobs table is on")
|
||||
parser.add_argument("--bootstrap", required=True,
|
||||
help="module:function returning (config, store)")
|
||||
parser.add_argument("--bootstrap-arg", default="",
|
||||
help="one string handed to the bootstrap, e.g. a config path")
|
||||
parser.add_argument("--workers", type=workers, default=None,
|
||||
help="resource=count,... (default: JobQueue.WORKERS)")
|
||||
parser.add_argument("--until-idle", action="store_true",
|
||||
help="drain what is queued, then exit")
|
||||
parser.add_argument("--no-schedules", action="store_true",
|
||||
help="do not fire schedules here; for a second worker "
|
||||
"on a file that already has a scheduler")
|
||||
parser.add_argument("--timezone", default="",
|
||||
help="what 'daily at 06:00' is 06:00 in "
|
||||
"(default: this machine's own zone)")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
build = resolve(args.bootstrap)
|
||||
config, store = build(args.bootstrap_arg) if args.bootstrap_arg else build()
|
||||
queue = JobQueue(Jobs(args.db), config, store, workers=args.workers,
|
||||
bootstrap=args.bootstrap, bootstrap_arg=args.bootstrap_arg,
|
||||
tz=args.timezone, schedules=not args.no_schedules)
|
||||
try:
|
||||
if args.until_idle:
|
||||
queue.drain()
|
||||
else:
|
||||
asyncio.run(serve(queue))
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s")
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user