roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
128 lines
5.1 KiB
Python
128 lines
5.1 KiB
Python
"""The pipeline: how work is queued, run, watched and stopped.
|
|
|
|
from pipeline import JobQueue, Jobs, job, RunContext, Task, NULL
|
|
|
|
**This folder is meant to be copied.** It imports nothing from the app around
|
|
it and nothing outside the standard library but pydantic — and FastAPI, in
|
|
`pipeline/web/`, which an app that has no web face never imports. Drop the
|
|
directory into another project, point `Jobs` at a SQLite file, and it works.
|
|
`pipeline.md` beside this file is the write-up, and it travels with the folder.
|
|
|
|
**The source of truth is the boilerplate repo's copy.** news and health hold
|
|
copies; a fix lands here first and is copied out with `git archive`.
|
|
|
|
Nothing here knows what a job *does*. The jobs themselves are the app's —
|
|
`app/jobs.py` for the starter — and import `@job` from here. That is the one
|
|
rule: the pipeline is imported by the work, never the other way round, so
|
|
there is no cycle to break and no lazy import to remember.
|
|
|
|
models.py `Job`, `Run`, `JobsSnapshot`, `Notice` and the `Model` base
|
|
they share — the shapes the queue stores and serves
|
|
notices.py a failure turned into one line worth putting on screen.
|
|
Only the part the queue needs; classifying a failure is the
|
|
app's, because a useful sentence names things only it knows
|
|
registry.py `@job`, `Spec`, and `JOBS` — how a function becomes a job,
|
|
how many attempts it is worth, and whether it runs in a
|
|
thread or in a child process of its own
|
|
schedule.py `Cadence`, `declare` and `SCHEDULES` — the five wordings a
|
|
standing instruction may be written in, when each next comes
|
|
round, and how an app ships a default schedule next to `@job`
|
|
context.py `RunContext` — what a running job is handed: progress,
|
|
cancellation, notices, its parents' results, the fan-out,
|
|
the subprocess helper, and the two things it can raise
|
|
table.py `Jobs` — the jobs table, on whatever SQLite file it is given
|
|
child.py the other side of `isolate=True`: one job, one process,
|
|
reconstructed from its row
|
|
queue.py `JobQueue` — the worker loop that drains that table
|
|
task.py `Task` — plain code that finds work, cuts it into batches
|
|
and queues them, plus the one final job every task ends
|
|
with and the tally it hands to the callback. `Bound` is
|
|
what a batch carries when how often to try it is the
|
|
batch's own business rather than the job kind's
|
|
worker.py `python -m pipeline.worker`: the same queue with no web app
|
|
around it, for a second machine, a cron entry or a script
|
|
web/ the queue over HTTP, the socket that reports on it, and the
|
|
`<desk-jobs>` element that renders it. Optional: an app
|
|
with no web face never imports it, and so never needs
|
|
FastAPI
|
|
|
|
What the pipeline asks of an app is small and is typed where it is asked:
|
|
handlers take `(config, store)` and the queue passes those through untouched,
|
|
`Jobs` wants a file path, and `isolate=True` wants a `bootstrap` that can
|
|
rebuild that pair in a child process. That is the whole of what a second app
|
|
has to provide. `README.md` beside this file is the ten-minute version.
|
|
|
|
Within the package the order is `models, notices > registry > schedule >
|
|
context > table > child > task > queue > worker > web`, and `tests/test_architecture.py` holds
|
|
every file to it, and to one door: what is listed below is the whole of what
|
|
anything outside may use.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pipeline import notices
|
|
from pipeline.context import NULL, Cancelled, Retry, RunContext, spawn
|
|
from pipeline.models import (
|
|
TERMINAL, CascadeProgress, Job, JobState, JobsSnapshot, Notice,
|
|
NoticeEntry, NoticeKind, NoticeLevel, Run, Schedule, TaskProgress,
|
|
)
|
|
from pipeline.queue import (
|
|
ChildFailed, JobQueue, Listener, NoticeListener, ScheduleListener,
|
|
local_zone,
|
|
)
|
|
from pipeline.registry import JOBS, Handler, Spec, job
|
|
from pipeline.schedule import (
|
|
DAYS, FORMS, SCHEDULES, BadCadence, Cadence, Declared, declare,
|
|
)
|
|
from pipeline.table import MAX_RESULT, Jobs, ResultTooLarge, RunCancelled
|
|
from pipeline.task import FINAL, TASKS, Batch, Bound, Tally, Task, fan_out
|
|
|
|
__all__ = [
|
|
"DAYS",
|
|
"FINAL",
|
|
"FORMS",
|
|
"JOBS",
|
|
"MAX_RESULT",
|
|
"NULL",
|
|
"SCHEDULES",
|
|
"TASKS",
|
|
"TERMINAL",
|
|
"BadCadence",
|
|
"Batch",
|
|
"Bound",
|
|
"Cadence",
|
|
"Cancelled",
|
|
"CascadeProgress",
|
|
"ChildFailed",
|
|
"Declared",
|
|
"Handler",
|
|
"Job",
|
|
"JobQueue",
|
|
"JobState",
|
|
"Jobs",
|
|
"JobsSnapshot",
|
|
"Listener",
|
|
"Notice",
|
|
"NoticeEntry",
|
|
"NoticeKind",
|
|
"NoticeLevel",
|
|
"NoticeListener",
|
|
"ResultTooLarge",
|
|
"Retry",
|
|
"Run",
|
|
"RunCancelled",
|
|
"RunContext",
|
|
"Schedule",
|
|
"ScheduleListener",
|
|
"Spec",
|
|
"Tally",
|
|
"Task",
|
|
"TaskProgress",
|
|
"declare",
|
|
"fan_out",
|
|
"job",
|
|
"local_zone",
|
|
"notices",
|
|
"spawn",
|
|
]
|