Files
jobboard/pipeline
WolfandClaude Sonnet 5 36860f0cfe Pull pipeline/ from boilerplate: the phone schedule table no longer overflows
Refreshes pipeline/ with boilerplate's fix for the schedule table on a
phone (grid-template-columns squeezing name/cadence to a sliver at
375px, plus a content-box input overlapping the next column). Verified
on /settings/pulls at 375px: no overflow anywhere in the element or
the page. Removes the desk-schedules { overflow-x: auto } workaround
and its comment from app/static/app.css now that nothing overflows it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-25 22:23:09 +02:00
..

pipeline

A job queue that lives in a SQLite table. Workers per resource, batching with a report job, dependency chains, dedupe, retries, cancellation that can kill, a WebSocket job list, and a standalone worker. No broker.

Needs Python 3.12+ and pydantic. pipeline/web/ also needs fastapi; an app without a web face never imports it.

The source of truth is boilerplate/pipeline/. Copies elsewhere — news, health — are copies: take the whole folder, and run its tests where it lands.

cp -r boilerplate/pipeline yourproject/pipeline
python -m pytest yourproject/pipeline/tests

A job

A job is a plain function. @job registers it and returns it unchanged, so it stays callable from a terminal or a test with no queue.

from pipeline import NULL, JobQueue, Jobs, job

@job("greet", "Saying hello", resource="cpu")
def greet(config, store, *, context=NULL, name: str = "") -> dict:
    context.progress(0, 1, f"greeting {name}")
    context.check()                        # stops here if Kill was pressed
    return {"said": f"hello {name}"}

greet(config, store, name="you")          # direct call, no queue

queue = JobQueue(Jobs("app.db"), config, store)
queue.enqueue("greet", {"name": "you"})
queue.drain()                             # run everything queued, then stop

config and store are whatever your app uses. The queue passes them through untouched. The payload is the keyword arguments, stored as JSON on the row.

@job(...)
resource="cpu" which worker pool runs it. Pool sizes: JobQueue(workers={"cpu": 1, "http": 8, "llm": 4})
solo=True at most one queued or running at a time; a second enqueue returns the first
max_attempts=3, retry_delay=60 retried by the queue, not by a sleeping worker
isolate=True, timeout=600 runs in its own process, so Kill and the deadline actually end it

A web app

from fastapi import FastAPI
from pipeline import Jobs
from pipeline.web import connect

app = FastAPI()
queue = connect(app, Jobs("app.db"), config, store)

That mounts GET /api/jobs, POST /api/jobs, POST /api/jobs/{id}/cancel, the /ws/jobs socket, and starts the worker with the app. For a job list with progress and Kill buttons, put this in a page:

<script type="module" src="/_jobs/jobs.js"></script>
<desk-jobs></desk-jobs>

It themes from CSS custom properties (--desk-jobs-bg, --desk-jobs-accent, …) and raises a desk-jobs:notice DOM event for failures instead of wording them. The wording is the app's. On a phone it drops its own padding (the app's card around it is the gutter) and under a coarse pointer Kill is a thumb's height; the width it calls a phone is 40rem, the same number the app's stylesheet uses.

Many jobs and one report

@job("fetch_all")
def fetch_all(config, store, *, context=NULL):
    urls = store.pending_urls()
    context.fan_out("fetch", [{"urls": urls[i:i + 10]} for i in range(0, len(urls), 10)],
                    then="myapp.jobs:after_fetch")

def after_fetch(config, store, tally, carries, *, context):
    # tally.done / .failed / .cancelled, and tally.results: what each child returned
    context.fan_out("parse", ...)          # the next step; this is a cascade

then is module:function because it is stored on a row. The report job runs whether the children succeeded or not. Task(kind, finder=...) is the same thing with the finding done for you; register it in TASKS to make it a button.

Chains

a = queue.enqueue("fetch", {"url": url})
b = queue.enqueue("extract", {}, depends_on=[a.id], after="ok")    # "done" = however A ended

@job("extract")
def extract(config, store, *, context=NULL, html: str = ""):
    html = html or context.upstream().get("html", "")               # A's result

B reads A's result off A's row. Do not queue a row you mean to fill in later. The fallback keyword keeps extract callable by hand, since upstream() is {} outside the queue.

Results are reports: counts, ids, errors. A result over 64 KB fails the job with ResultTooLarge rather than being cut down. Put data in your store and return its ids, or raise pipeline.table.MAX_RESULT if a project really needs more.

Ending work that does not look up

A thread cannot be interrupted, so check() is cooperative. Two cases can be killed:

  • A command (ffmpeg, stockfish): context.spawn([...], timeout=600) kills the process tree on Kill or deadline.
  • Python that blocks (sklearn.fit): @job(..., isolate=True, timeout=600). The job runs as python -m pipeline.child, which rebuilds (config, store) from a bootstrap you provide:
def bootstrap(config_path: str = ""):          # myapp/jobs.py; importing it registers the jobs
    config = load_config(config_path)
    return config, Store(config.database)

JobQueue(jobs, config, store, bootstrap="myapp.jobs:bootstrap", bootstrap_arg=config_path)

timeout without isolate is refused at registration: a thread's deadline could only mark the row failed while the work carried on.

Schedules

A schedule is a row that says "enqueue this kind with this payload on this cadence". Same SQLite file as the jobs, so there is no second process to keep alive and a restart between two runs loses nothing.

from pipeline.schedule import declare

declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
declare("hourly count", "count", "every 1h", {"n": 5})

declare goes next to @job in the app's jobs module and fills SCHEDULES the way @job fills JOBS. On JobQueue.start() every declared name with no row gets one, and a row that already exists is left exactly as it was edited — cadence, payload and enabled alike. So a declaration is a starting point, not a setting: switching one off on screen survives the next deploy.

Five cadences and nothing else:

every 15m every 6h every 2d elapsed time. Unaffected by a daylight-saving change, because an interval is an interval
daily at 06:00 06:00 on the wall clock of JobQueue(tz=...), on both sides of a change
weekly mon 07:00 the same, on that weekday. Long names (monday) are accepted and stored short

Anything else raises ValueError listing all five, which is the sentence the API hands a person who typed something else. There is no seconds unit: the loop looks every SCHEDULE_TICK (30 s), and a cadence it cannot keep is one to refuse rather than round.

The tick runs inside the drain loop — no second thread, no second clock — and once on start(), so an app that was down over 06:00 runs that job when it comes back. Once: a firing sets next_run from now rather than stepping through the slots it missed, so there is nowhere for a backlog to accumulate. A schedule whose last job is still queued or running skips its turn and advances instead of piling copies of it up, and so does one whose kind is solo while another is under way. A row whose kind is no longer registered never fires, keeps its place, and is listed as unknown kind for somebody to delete.

queue = JobQueue(jobs, config, store, tz="Europe/Copenhagen")   # or schedules=False

Over HTTP, from pipeline.web.connect: GET/POST /api/schedules, PATCH/DELETE /api/schedules/{id}, POST /api/schedules/{id}/run (fires now without moving the next run). Every refusal is {"detail": "<sentence>"} — an unregistered kind 404s naming what there is, a payload missing keys 422s naming them, a bad cadence 422s with all five wordings. Any change pushes a schedules message on /ws/jobs.

<script type="module" src="/_jobs/schedules.js"></script>
<desk-schedules></desk-schedules>

<desk-schedules> is the table: name, kind, cadence edited in place, an on/off toggle, next and last run with the last job's state, Run now, Delete, and a row for adding one. Same --desk-jobs-* custom properties as <desk-jobs>, and just as silent about failures — it raises desk-schedules:error carrying the server's detail and leaves the wording to the app. Under 40rem the table is a card per schedule — the name and the switch on the first line, the other columns labelled under it, the two buttons across the bottom — from the same markup, because each cell carries its column's name as data-label.

A worker without the web app

python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
python -m pipeline.worker ... --until-idle          # drain and exit, for cron
python -m pipeline.worker ... --timezone Europe/Copenhagen
python -m pipeline.worker ... --no-schedules        # every worker after the first

Any number of workers may share one file. A claim is one atomic statement and a claimed job is held on a lease; a worker that dies has its jobs re-queued by whichever is still alive.

This worker fires schedules too, which is what makes it a complete replacement for the web app's own rather than half of one. Exactly one process per file should: --no-schedules is what the second and third pass, because the pile-up guard reads last_job_id and neither of two simultaneous firings has written it when the other looks.

Housekeeping

Finished jobs older than JobQueue(retain=30 * 86400) are purged hourly while idle. Unfinished jobs and the newest ending of each kind are kept regardless. retain=0 keeps everything.

The table is WAL-mode SQLite. Back it up with VACUUM INTO, not a file copy, and keep it off network filesystems.

More

  • pipeline.md — the reference card: every entity, the timings, the limits.
  • __init__.py — what each module is for.
  • tests/ — every claim above, including two worker processes racing on one file and a schedule either side of a daylight-saving change.