Files
jobboard/pipeline/README.md
T
WolfandClaude Opus 5.5 9d666d0937 Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and
scheduler), auth (one password, sessions, passkeys), the deploy kit and its
docs, plus the coupling and packaging tests. Downstream copies, unedited.

Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:16 +02:00

173 lines
9.2 KiB
Markdown

# pipeline
A job queue that lives in a SQLite table. Workers per resource, batching with a report job, dependency chains, dedupe, retries, cancellation that can kill, a WebSocket job list, and a standalone worker. No broker.
Needs Python 3.12+ and `pydantic`. `pipeline/web/` also needs `fastapi`; an app without a web face never imports it.
The source of truth is `boilerplate/pipeline/`. Copies elsewhere — news, health — are copies: take the whole folder, and run its tests where it lands.
```bash
cp -r boilerplate/pipeline yourproject/pipeline
python -m pytest yourproject/pipeline/tests
```
## A job
A job is a plain function. `@job` registers it and returns it unchanged, so it stays callable from a terminal or a test with no queue.
```python
from pipeline import NULL, JobQueue, Jobs, job
@job("greet", "Saying hello", resource="cpu")
def greet(config, store, *, context=NULL, name: str = "") -> dict:
context.progress(0, 1, f"greeting {name}")
context.check() # stops here if Kill was pressed
return {"said": f"hello {name}"}
greet(config, store, name="you") # direct call, no queue
queue = JobQueue(Jobs("app.db"), config, store)
queue.enqueue("greet", {"name": "you"})
queue.drain() # run everything queued, then stop
```
`config` and `store` are whatever your app uses. The queue passes them through untouched. The payload is the keyword arguments, stored as JSON on the row.
| `@job(...)` | |
|---|---|
| `resource="cpu"` | which worker pool runs it. Pool sizes: `JobQueue(workers={"cpu": 1, "http": 8, "llm": 4})` |
| `solo=True` | at most one queued or running at a time; a second enqueue returns the first |
| `max_attempts=3, retry_delay=60` | retried by the queue, not by a sleeping worker |
| `isolate=True, timeout=600` | runs in its own process, so Kill and the deadline actually end it |
## A web app
```python
from fastapi import FastAPI
from pipeline import Jobs
from pipeline.web import connect
app = FastAPI()
queue = connect(app, Jobs("app.db"), config, store)
```
That mounts `GET /api/jobs`, `POST /api/jobs`, `POST /api/jobs/{id}/cancel`, the `/ws/jobs` socket, and starts the worker with the app. For a job list with progress and Kill buttons, put this in a page:
```html
<script type="module" src="/_jobs/jobs.js"></script>
<desk-jobs></desk-jobs>
```
It themes from CSS custom properties (`--desk-jobs-bg`, `--desk-jobs-accent`, …) and raises a `desk-jobs:notice` DOM event for failures instead of wording them. The wording is the app's.
## Many jobs and one report
```python
@job("fetch_all")
def fetch_all(config, store, *, context=NULL):
urls = store.pending_urls()
context.fan_out("fetch", [{"urls": urls[i:i + 10]} for i in range(0, len(urls), 10)],
then="myapp.jobs:after_fetch")
def after_fetch(config, store, tally, carries, *, context):
# tally.done / .failed / .cancelled, and tally.results: what each child returned
context.fan_out("parse", ...) # the next step; this is a cascade
```
`then` is `module:function` because it is stored on a row. The report job runs whether the children succeeded or not. `Task(kind, finder=...)` is the same thing with the finding done for you; register it in `TASKS` to make it a button.
## Chains
```python
a = queue.enqueue("fetch", {"url": url})
b = queue.enqueue("extract", {}, depends_on=[a.id], after="ok") # "done" = however A ended
@job("extract")
def extract(config, store, *, context=NULL, html: str = ""):
html = html or context.upstream().get("html", "") # A's result
```
B reads A's result off A's row. Do not queue a row you mean to fill in later. The fallback keyword keeps `extract` callable by hand, since `upstream()` is `{}` outside the queue.
Results are reports: counts, ids, errors. A result over 64 KB fails the job with `ResultTooLarge` rather than being cut down. Put data in your store and return its ids, or raise `pipeline.table.MAX_RESULT` if a project really needs more.
## Ending work that does not look up
A thread cannot be interrupted, so `check()` is cooperative. Two cases can be killed:
- **A command** (ffmpeg, stockfish): `context.spawn([...], timeout=600)` kills the process tree on Kill or deadline.
- **Python that blocks** (`sklearn.fit`): `@job(..., isolate=True, timeout=600)`. The job runs as `python -m pipeline.child`, which rebuilds `(config, store)` from a bootstrap you provide:
```python
def bootstrap(config_path: str = ""): # myapp/jobs.py; importing it registers the jobs
config = load_config(config_path)
return config, Store(config.database)
JobQueue(jobs, config, store, bootstrap="myapp.jobs:bootstrap", bootstrap_arg=config_path)
```
`timeout` without `isolate` is refused at registration: a thread's deadline could only mark the row failed while the work carried on.
## Schedules
A schedule is a row that says "enqueue this kind with this payload on this cadence". Same SQLite file as the jobs, so there is no second process to keep alive and a restart between two runs loses nothing.
```python
from pipeline.schedule import declare
declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
declare("hourly count", "count", "every 1h", {"n": 5})
```
`declare` goes next to `@job` in the app's jobs module and fills `SCHEDULES` the way `@job` fills `JOBS`. On `JobQueue.start()` every declared name with no row gets one, and a row that already exists is left exactly as it was edited — cadence, payload and enabled alike. So a declaration is a starting point, not a setting: switching one off on screen survives the next deploy.
Five cadences and nothing else:
| | |
|---|---|
| `every 15m` `every 6h` `every 2d` | elapsed time. Unaffected by a daylight-saving change, because an interval is an interval |
| `daily at 06:00` | 06:00 on the wall clock of `JobQueue(tz=...)`, on both sides of a change |
| `weekly mon 07:00` | the same, on that weekday. Long names (`monday`) are accepted and stored short |
Anything else raises `ValueError` listing all five, which is the sentence the API hands a person who typed something else. There is no seconds unit: the loop looks every `SCHEDULE_TICK` (30 s), and a cadence it cannot keep is one to refuse rather than round.
The tick runs inside the drain loop — no second thread, no second clock — and once on `start()`, so an app that was down over 06:00 runs that job when it comes back. Once: a firing sets `next_run` from *now* rather than stepping through the slots it missed, so there is nowhere for a backlog to accumulate. A schedule whose last job is still queued or running skips its turn and advances instead of piling copies of it up, and so does one whose kind is `solo` while another is under way. A row whose kind is no longer registered never fires, keeps its place, and is listed as `unknown kind` for somebody to delete.
```python
queue = JobQueue(jobs, config, store, tz="Europe/Copenhagen") # or schedules=False
```
Over HTTP, from `pipeline.web.connect`: `GET/POST /api/schedules`, `PATCH`/`DELETE /api/schedules/{id}`, `POST /api/schedules/{id}/run` (fires now without moving the next run). Every refusal is `{"detail": "<sentence>"}` — an unregistered kind 404s naming what there is, a payload missing keys 422s naming them, a bad cadence 422s with all five wordings. Any change pushes a `schedules` message on `/ws/jobs`.
```html
<script type="module" src="/_jobs/schedules.js"></script>
<desk-schedules></desk-schedules>
```
`<desk-schedules>` is the table: name, kind, cadence edited in place, an on/off toggle, next and last run with the last job's state, Run now, Delete, and a row for adding one. Same `--desk-jobs-*` custom properties as `<desk-jobs>`, and just as silent about failures — it raises `desk-schedules:error` carrying the server's `detail` and leaves the wording to the app.
## A worker without the web app
```bash
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
python -m pipeline.worker ... --until-idle # drain and exit, for cron
python -m pipeline.worker ... --timezone Europe/Copenhagen
python -m pipeline.worker ... --no-schedules # every worker after the first
```
Any number of workers may share one file. A claim is one atomic statement and a claimed job is held on a lease; a worker that dies has its jobs re-queued by whichever is still alive.
This worker fires schedules too, which is what makes it a complete replacement for the web app's own rather than half of one. Exactly one process per file should: `--no-schedules` is what the second and third pass, because the pile-up guard reads `last_job_id` and neither of two simultaneous firings has written it when the other looks.
## Housekeeping
Finished jobs older than `JobQueue(retain=30 * 86400)` are purged hourly while idle. Unfinished jobs and the newest ending of each kind are kept regardless. `retain=0` keeps everything.
The table is WAL-mode SQLite. Back it up with `VACUUM INTO`, not a file copy, and keep it off network filesystems.
## More
- `pipeline.md` — the reference card: every entity, the timings, the limits.
- `__init__.py` — what each module is for.
- `tests/` — every claim above, including two worker processes racing on one file and a schedule either side of a daylight-saving change.