"""The pipeline: how work is queued, run, watched and stopped. from pipeline import JobQueue, Jobs, job, RunContext, Task, NULL **This folder is meant to be copied.** It imports nothing from the app around it and nothing outside the standard library but pydantic — and FastAPI, in `pipeline/web/`, which an app that has no web face never imports. Drop the directory into another project, point `Jobs` at a SQLite file, and it works. `pipeline.md` beside this file is the write-up, and it travels with the folder. **The source of truth is the boilerplate repo's copy.** news and health hold copies; a fix lands here first and is copied out with `git archive`. Nothing here knows what a job *does*. The jobs themselves are the app's — `app/jobs.py` for the starter — and import `@job` from here. That is the one rule: the pipeline is imported by the work, never the other way round, so there is no cycle to break and no lazy import to remember. models.py `Job`, `Run`, `JobsSnapshot`, `Notice` and the `Model` base they share — the shapes the queue stores and serves notices.py a failure turned into one line worth putting on screen. Only the part the queue needs; classifying a failure is the app's, because a useful sentence names things only it knows registry.py `@job`, `Spec`, and `JOBS` — how a function becomes a job, how many attempts it is worth, and whether it runs in a thread or in a child process of its own schedule.py `Cadence`, `declare` and `SCHEDULES` — the five wordings a standing instruction may be written in, when each next comes round, and how an app ships a default schedule next to `@job` context.py `RunContext` — what a running job is handed: progress, cancellation, notices, its parents' results, the fan-out, the subprocess helper, and the two things it can raise table.py `Jobs` — the jobs table, on whatever SQLite file it is given child.py the other side of `isolate=True`: one job, one process, reconstructed from its row queue.py `JobQueue` — the worker loop that drains that table task.py `Task` — plain code that finds work, cuts it into batches and queues them, plus the one final job every task ends with and the tally it hands to the callback. `Bound` is what a batch carries when how often to try it is the batch's own business rather than the job kind's worker.py `python -m pipeline.worker`: the same queue with no web app around it, for a second machine, a cron entry or a script web/ the queue over HTTP, the socket that reports on it, and the `` element that renders it. Optional: an app with no web face never imports it, and so never needs FastAPI What the pipeline asks of an app is small and is typed where it is asked: handlers take `(config, store)` and the queue passes those through untouched, `Jobs` wants a file path, and `isolate=True` wants a `bootstrap` that can rebuild that pair in a child process. That is the whole of what a second app has to provide. `README.md` beside this file is the ten-minute version. Within the package the order is `models, notices > registry > schedule > context > table > child > task > queue > worker > web`, and `tests/test_architecture.py` holds every file to it, and to one door: what is listed below is the whole of what anything outside may use. """ from __future__ import annotations from pipeline import notices from pipeline.context import NULL, Cancelled, Retry, RunContext, spawn from pipeline.models import ( TERMINAL, CascadeProgress, Job, JobState, JobsSnapshot, Notice, NoticeEntry, NoticeKind, NoticeLevel, Run, Schedule, TaskProgress, ) from pipeline.queue import ( ChildFailed, JobQueue, Listener, NoticeListener, ScheduleListener, local_zone, ) from pipeline.registry import JOBS, Handler, Spec, job from pipeline.schedule import ( DAYS, FORMS, SCHEDULES, BadCadence, Cadence, Declared, declare, ) from pipeline.table import MAX_RESULT, Jobs, ResultTooLarge, RunCancelled from pipeline.task import FINAL, TASKS, Batch, Bound, Tally, Task, fan_out __all__ = [ "DAYS", "FINAL", "FORMS", "JOBS", "MAX_RESULT", "NULL", "SCHEDULES", "TASKS", "TERMINAL", "BadCadence", "Batch", "Bound", "Cadence", "Cancelled", "CascadeProgress", "ChildFailed", "Declared", "Handler", "Job", "JobQueue", "JobState", "Jobs", "JobsSnapshot", "Listener", "Notice", "NoticeEntry", "NoticeKind", "NoticeLevel", "NoticeListener", "ResultTooLarge", "Retry", "Run", "RunCancelled", "RunContext", "Schedule", "ScheduleListener", "Spec", "Tally", "Task", "TaskProgress", "declare", "fan_out", "job", "local_zone", "notices", "spawn", ]