Files
jobboard/pipeline
..

pipeline

A job queue that lives in a SQLite table. Workers per resource, batching with a report job, dependency chains, dedupe, retries, cancellation that can kill, a WebSocket job list, and a standalone worker. No broker.

Needs Python 3.12+ and pydantic. pipeline/web/ also needs fastapi; an app without a web face never imports it.

The source of truth is boilerplate/pipeline/. Copies elsewhere — news, health — are copies: take the whole folder, and run its tests where it lands.

cp -r boilerplate/pipeline yourproject/pipeline
python -m pytest yourproject/pipeline/tests

A job

A job is a plain function. @job registers it and returns it unchanged, so it stays callable from a terminal or a test with no queue.

from pipeline import NULL, JobQueue, Jobs, job

@job("greet", "Saying hello", resource="cpu")
def greet(config, store, *, context=NULL, name: str = "") -> dict:
    context.progress(0, 1, f"greeting {name}")
    context.check()                        # stops here if Kill was pressed
    return {"said": f"hello {name}"}

greet(config, store, name="you")          # direct call, no queue

queue = JobQueue(Jobs("app.db"), config, store)
queue.enqueue("greet", {"name": "you"})
queue.drain()                             # run everything queued, then stop

config and store are whatever your app uses. The queue passes them through untouched. The payload is the keyword arguments, stored as JSON on the row.

@job(...)
resource="cpu" which worker pool runs it. Pool sizes: JobQueue(workers={"cpu": 1, "http": 8, "llm": 4})
solo=True at most one queued or running at a time; a second enqueue returns the first
max_attempts=3, retry_delay=60 retried by the queue, not by a sleeping worker
isolate=True, timeout=600 runs in its own process, so Kill and the deadline actually end it

A web app

from fastapi import FastAPI
from pipeline import Jobs
from pipeline.web import connect

app = FastAPI()
queue = connect(app, Jobs("app.db"), config, store)

That mounts GET /api/jobs, POST /api/jobs, POST /api/jobs/{id}/cancel, the /ws/jobs socket, and starts the worker with the app. For a job list with progress and Kill buttons, put this in a page:

<script type="module" src="/_jobs/jobs.js"></script>
<desk-jobs></desk-jobs>

It themes from CSS custom properties (--desk-jobs-bg, --desk-jobs-accent, …) and raises a desk-jobs:notice DOM event for failures instead of wording them. The wording is the app's. On a phone it drops its own padding (the app's card around it is the gutter) and under a coarse pointer Kill is a thumb's height; the width it calls a phone is 40rem, the same number the app's stylesheet uses.

Attributes, all optional: idle="hide" (the default) takes up no space while nothing is running or queued; idle="show" stays and says "Nothing running."; base prefixes the HTTP calls; socket overrides the socket URL; probe="/api/status" names an authenticated endpoint to try when the socket closes, and raises desk-jobs:signed-out on a 401 or 403.

The same file defines <desk-queue>, a chip for a top bar:

<desk-queue href="/jobs">All runs</desk-queue>

The chip is a dot, the running count and the first running job's label and progress ("Embedding roles, 6 of 6, +2 waiting"), with the job's message as its tooltip. It is grey and says "Idle" when nothing runs, and shows a red dot and "Offline" while the socket is down; under 40rem it shrinks to the dot and the count. Pressing it opens a popover listing every running and queued job with its message, done/total, a ticking age, a countdown for one waiting out a retry, and Kill, one Stop run per live cascade of more than one job (POST /api/cascades/{id}/cancel), and, with href, a link to the app's own page whose text is the element's content ("All runs" without any). It reads the --desk-jobs-* properties with fallbacks instead of setting them, so a theme on :root or an ancestor reaches it and without one it takes the bar's colour and font; --desk-jobs-warn colours the countdown. It does not re-raise notices, because it sits next to a <desk-jobs> on some pages. idle="hide" hides it while idle; base, socket and probe are as above.

Many jobs and one report

@job("fetch_all")
def fetch_all(config, store, *, context=NULL):
    urls = store.pending_urls()
    context.fan_out("fetch", [{"urls": urls[i:i + 10]} for i in range(0, len(urls), 10)],
                    then="myapp.jobs:after_fetch")

def after_fetch(config, store, tally, carries, *, context):
    # tally.done / .failed / .cancelled, and tally.results: what each child returned
    context.fan_out("parse", ...)          # the next step; this is a cascade

then is module:function because it is stored on a row. The report job runs whether the children succeeded or not. Task(kind, finder=...) is the same thing with the finding done for you; register it in TASKS to make it a button.

Chains

a = queue.enqueue("fetch", {"url": url})
b = queue.enqueue("extract", {}, depends_on=[a.id], after="ok")    # "done" = however A ended

@job("extract")
def extract(config, store, *, context=NULL, html: str = ""):
    html = html or context.upstream().get("html", "")               # A's result

B reads A's result off A's row. Do not queue a row you mean to fill in later. The fallback keyword keeps extract callable by hand, since upstream() is {} outside the queue.

Results are reports: counts, ids, errors. A result over 64 KB fails the job with ResultTooLarge rather than being cut down. Put data in your store and return its ids, or raise pipeline.table.MAX_RESULT if a project really needs more.

Ending work that does not look up

A thread cannot be interrupted, so check() is cooperative. Two cases can be killed:

  • A command (ffmpeg, stockfish): context.spawn([...], timeout=600) kills the process tree on Kill or deadline.
  • Python that blocks (sklearn.fit): @job(..., isolate=True, timeout=600). The job runs as python -m pipeline.child, which rebuilds (config, store) from a bootstrap you provide:
def bootstrap(config_path: str = ""):          # myapp/jobs.py; importing it registers the jobs
    config = load_config(config_path)
    return config, Store(config.database)

JobQueue(jobs, config, store, bootstrap="myapp.jobs:bootstrap", bootstrap_arg=config_path)

timeout without isolate is refused at registration: a thread's deadline could only mark the row failed while the work carried on.

Schedules

A schedule is a row that says "enqueue this kind with this payload on this cadence". Same SQLite file as the jobs, so there is no second process to keep alive and a restart between two runs loses nothing.

from pipeline.schedule import declare

declare("nightly tick", "tick", "daily at 06:00", label="A tick, nightly")
declare("hourly count", "count", "every 1h", {"n": 5})

declare goes next to @job in the app's jobs module and fills SCHEDULES the way @job fills JOBS. On JobQueue.start() every declared name with no row gets one, and a row that already exists keeps its cadence, payload and enabled as they were edited. So a declaration is a starting point, not a setting: switching one off on screen survives the next deploy. The label is the exception: no screen edits it, so it is refreshed from the declaration on every start while the row still has the declared kind.

Five cadences and nothing else:

every 15m every 6h every 2d elapsed time. Unaffected by a daylight-saving change, because an interval is an interval
daily at 06:00 06:00 on the wall clock of JobQueue(tz=...), on both sides of a change
weekly mon 07:00 the same, on that weekday. Long names (monday) are accepted and stored short

Anything else raises ValueError listing all five, which is the sentence the API hands a person who typed something else. There is no seconds unit: the loop looks every SCHEDULE_TICK (30 s), and a cadence it cannot keep is one to refuse rather than round.

The tick runs inside the drain loop — no second thread, no second clock — and once on start(), so an app that was down over 06:00 runs that job when it comes back. Once: a firing sets next_run from now rather than stepping through the slots it missed, so there is nowhere for a backlog to accumulate. A schedule whose last job is still queued or running skips its turn and advances instead of piling copies of it up, and so does one whose kind is solo while another is under way. A row whose kind is no longer registered never fires, keeps its place, and is listed as unknown kind for somebody to delete.

queue = JobQueue(jobs, config, store, tz="Europe/Copenhagen")   # or schedules=False

Over HTTP, from pipeline.web.connect: GET/POST /api/schedules, PATCH/DELETE /api/schedules/{id}, POST /api/schedules/{id}/run (fires now without moving the next run). Every refusal is {"detail": "<sentence>"} — an unregistered kind 404s naming what there is, a payload missing keys 422s naming them, a bad cadence 422s with all five wordings. Any change pushes a schedules message on /ws/jobs.

<script type="module" src="/_jobs/schedules.js"></script>
<desk-schedules></desk-schedules>

<desk-schedules> is the table: name, kind, cadence, an on/off toggle, next and last run with the last job's state, Run now, Delete, and an "Add a schedule" button that opens a row for adding one, its kind starting on "Choose a job…". The cadence is a picker rather than a text box: Every (15m to 7d), Daily at (a time, in 15-minute steps) or Weekly on (a day and a time). It builds the canonical string the API takes, so the grammar is unchanged and a 422 is still the backstop; a stored cadence outside the presets, such as every 50m, is added to the options. add="off" removes adding altogether, for an app whose schedules are all declared; base and socket are as for <desk-jobs>. Same --desk-jobs-* custom properties as <desk-jobs>, and just as silent about failures — it raises desk-schedules:error carrying the server's detail and leaves the wording to the app. Under 40rem the table is a card per schedule — the name and the switch on the first line, the other columns labelled under it, the two buttons across the bottom — from the same markup, because each cell carries its column's name as data-label.

A worker without the web app

python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
python -m pipeline.worker ... --until-idle          # drain and exit, for cron
python -m pipeline.worker ... --timezone Europe/Copenhagen
python -m pipeline.worker ... --no-schedules        # every worker after the first

Any number of workers may share one file. A claim is one atomic statement and a claimed job is held on a lease; a worker that dies has its jobs re-queued by whichever is still alive.

This worker fires schedules too, which is what makes it a complete replacement for the web app's own rather than half of one. Exactly one process per file should: --no-schedules is what the second and third pass, because the pile-up guard reads last_job_id and neither of two simultaneous firings has written it when the other looks.

Housekeeping

Finished jobs older than JobQueue(retain=30 * 86400) are purged hourly while idle. Unfinished jobs and the newest ending of each kind are kept regardless. retain=0 keeps everything.

The table is WAL-mode SQLite. Back it up with VACUUM INTO, not a file copy, and keep it off network filesystems.

More

  • pipeline.md — the reference card: every entity, the timings, the limits.
  • __init__.py — what each module is for.
  • tests/ — every claim above, including two worker processes racing on one file and a schedule either side of a daylight-saving change.