Files
jobboard/pipeline/__init__.py
T
WolfandClaude Opus 5.5 9d666d0937 Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and
scheduler), auth (one password, sessions, passkeys), the deploy kit and its
docs, plus the coupling and packaging tests. Downstream copies, unedited.

Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:16 +02:00

128 lines
5.1 KiB
Python

"""The pipeline: how work is queued, run, watched and stopped.
from pipeline import JobQueue, Jobs, job, RunContext, Task, NULL
**This folder is meant to be copied.** It imports nothing from the app around
it and nothing outside the standard library but pydantic — and FastAPI, in
`pipeline/web/`, which an app that has no web face never imports. Drop the
directory into another project, point `Jobs` at a SQLite file, and it works.
`pipeline.md` beside this file is the write-up, and it travels with the folder.
**The source of truth is the boilerplate repo's copy.** news and health hold
copies; a fix lands here first and is copied out with `git archive`.
Nothing here knows what a job *does*. The jobs themselves are the app's —
`app/jobs.py` for the starter — and import `@job` from here. That is the one
rule: the pipeline is imported by the work, never the other way round, so
there is no cycle to break and no lazy import to remember.
models.py `Job`, `Run`, `JobsSnapshot`, `Notice` and the `Model` base
they share — the shapes the queue stores and serves
notices.py a failure turned into one line worth putting on screen.
Only the part the queue needs; classifying a failure is the
app's, because a useful sentence names things only it knows
registry.py `@job`, `Spec`, and `JOBS` — how a function becomes a job,
how many attempts it is worth, and whether it runs in a
thread or in a child process of its own
schedule.py `Cadence`, `declare` and `SCHEDULES` — the five wordings a
standing instruction may be written in, when each next comes
round, and how an app ships a default schedule next to `@job`
context.py `RunContext` — what a running job is handed: progress,
cancellation, notices, its parents' results, the fan-out,
the subprocess helper, and the two things it can raise
table.py `Jobs` — the jobs table, on whatever SQLite file it is given
child.py the other side of `isolate=True`: one job, one process,
reconstructed from its row
queue.py `JobQueue` — the worker loop that drains that table
task.py `Task` — plain code that finds work, cuts it into batches
and queues them, plus the one final job every task ends
with and the tally it hands to the callback. `Bound` is
what a batch carries when how often to try it is the
batch's own business rather than the job kind's
worker.py `python -m pipeline.worker`: the same queue with no web app
around it, for a second machine, a cron entry or a script
web/ the queue over HTTP, the socket that reports on it, and the
`<desk-jobs>` element that renders it. Optional: an app
with no web face never imports it, and so never needs
FastAPI
What the pipeline asks of an app is small and is typed where it is asked:
handlers take `(config, store)` and the queue passes those through untouched,
`Jobs` wants a file path, and `isolate=True` wants a `bootstrap` that can
rebuild that pair in a child process. That is the whole of what a second app
has to provide. `README.md` beside this file is the ten-minute version.
Within the package the order is `models, notices > registry > schedule >
context > table > child > task > queue > worker > web`, and `tests/test_architecture.py` holds
every file to it, and to one door: what is listed below is the whole of what
anything outside may use.
"""
from __future__ import annotations
from pipeline import notices
from pipeline.context import NULL, Cancelled, Retry, RunContext, spawn
from pipeline.models import (
TERMINAL, CascadeProgress, Job, JobState, JobsSnapshot, Notice,
NoticeEntry, NoticeKind, NoticeLevel, Run, Schedule, TaskProgress,
)
from pipeline.queue import (
ChildFailed, JobQueue, Listener, NoticeListener, ScheduleListener,
local_zone,
)
from pipeline.registry import JOBS, Handler, Spec, job
from pipeline.schedule import (
DAYS, FORMS, SCHEDULES, BadCadence, Cadence, Declared, declare,
)
from pipeline.table import MAX_RESULT, Jobs, ResultTooLarge, RunCancelled
from pipeline.task import FINAL, TASKS, Batch, Bound, Tally, Task, fan_out
__all__ = [
"DAYS",
"FINAL",
"FORMS",
"JOBS",
"MAX_RESULT",
"NULL",
"SCHEDULES",
"TASKS",
"TERMINAL",
"BadCadence",
"Batch",
"Bound",
"Cadence",
"Cancelled",
"CascadeProgress",
"ChildFailed",
"Declared",
"Handler",
"Job",
"JobQueue",
"JobState",
"Jobs",
"JobsSnapshot",
"Listener",
"Notice",
"NoticeEntry",
"NoticeKind",
"NoticeLevel",
"NoticeListener",
"ResultTooLarge",
"Retry",
"Run",
"RunCancelled",
"RunContext",
"Schedule",
"ScheduleListener",
"Spec",
"Tally",
"Task",
"TaskProgress",
"declare",
"fan_out",
"job",
"local_zone",
"notices",
"spawn",
]