Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f436104f3
commit
9d666d0937
76 files changed
+15235
No files matched your search
@@ -0,0 +1,127 @@
|
||||
"""The pipeline: how work is queued, run, watched and stopped.
|
||||
|
||||
from pipeline import JobQueue, Jobs, job, RunContext, Task, NULL
|
||||
|
||||
**This folder is meant to be copied.** It imports nothing from the app around
|
||||
it and nothing outside the standard library but pydantic — and FastAPI, in
|
||||
`pipeline/web/`, which an app that has no web face never imports. Drop the
|
||||
directory into another project, point `Jobs` at a SQLite file, and it works.
|
||||
`pipeline.md` beside this file is the write-up, and it travels with the folder.
|
||||
|
||||
**The source of truth is the boilerplate repo's copy.** news and health hold
|
||||
copies; a fix lands here first and is copied out with `git archive`.
|
||||
|
||||
Nothing here knows what a job *does*. The jobs themselves are the app's —
|
||||
`app/jobs.py` for the starter — and import `@job` from here. That is the one
|
||||
rule: the pipeline is imported by the work, never the other way round, so
|
||||
there is no cycle to break and no lazy import to remember.
|
||||
|
||||
models.py `Job`, `Run`, `JobsSnapshot`, `Notice` and the `Model` base
|
||||
they share — the shapes the queue stores and serves
|
||||
notices.py a failure turned into one line worth putting on screen.
|
||||
Only the part the queue needs; classifying a failure is the
|
||||
app's, because a useful sentence names things only it knows
|
||||
registry.py `@job`, `Spec`, and `JOBS` — how a function becomes a job,
|
||||
how many attempts it is worth, and whether it runs in a
|
||||
thread or in a child process of its own
|
||||
schedule.py `Cadence`, `declare` and `SCHEDULES` — the five wordings a
|
||||
standing instruction may be written in, when each next comes
|
||||
round, and how an app ships a default schedule next to `@job`
|
||||
context.py `RunContext` — what a running job is handed: progress,
|
||||
cancellation, notices, its parents' results, the fan-out,
|
||||
the subprocess helper, and the two things it can raise
|
||||
table.py `Jobs` — the jobs table, on whatever SQLite file it is given
|
||||
child.py the other side of `isolate=True`: one job, one process,
|
||||
reconstructed from its row
|
||||
queue.py `JobQueue` — the worker loop that drains that table
|
||||
task.py `Task` — plain code that finds work, cuts it into batches
|
||||
and queues them, plus the one final job every task ends
|
||||
with and the tally it hands to the callback. `Bound` is
|
||||
what a batch carries when how often to try it is the
|
||||
batch's own business rather than the job kind's
|
||||
worker.py `python -m pipeline.worker`: the same queue with no web app
|
||||
around it, for a second machine, a cron entry or a script
|
||||
web/ the queue over HTTP, the socket that reports on it, and the
|
||||
`<desk-jobs>` element that renders it. Optional: an app
|
||||
with no web face never imports it, and so never needs
|
||||
FastAPI
|
||||
|
||||
What the pipeline asks of an app is small and is typed where it is asked:
|
||||
handlers take `(config, store)` and the queue passes those through untouched,
|
||||
`Jobs` wants a file path, and `isolate=True` wants a `bootstrap` that can
|
||||
rebuild that pair in a child process. That is the whole of what a second app
|
||||
has to provide. `README.md` beside this file is the ten-minute version.
|
||||
|
||||
Within the package the order is `models, notices > registry > schedule >
|
||||
context > table > child > task > queue > worker > web`, and `tests/test_architecture.py` holds
|
||||
every file to it, and to one door: what is listed below is the whole of what
|
||||
anything outside may use.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pipeline import notices
|
||||
from pipeline.context import NULL, Cancelled, Retry, RunContext, spawn
|
||||
from pipeline.models import (
|
||||
TERMINAL, CascadeProgress, Job, JobState, JobsSnapshot, Notice,
|
||||
NoticeEntry, NoticeKind, NoticeLevel, Run, Schedule, TaskProgress,
|
||||
)
|
||||
from pipeline.queue import (
|
||||
ChildFailed, JobQueue, Listener, NoticeListener, ScheduleListener,
|
||||
local_zone,
|
||||
)
|
||||
from pipeline.registry import JOBS, Handler, Spec, job
|
||||
from pipeline.schedule import (
|
||||
DAYS, FORMS, SCHEDULES, BadCadence, Cadence, Declared, declare,
|
||||
)
|
||||
from pipeline.table import MAX_RESULT, Jobs, ResultTooLarge, RunCancelled
|
||||
from pipeline.task import FINAL, TASKS, Batch, Bound, Tally, Task, fan_out
|
||||
|
||||
__all__ = [
|
||||
"DAYS",
|
||||
"FINAL",
|
||||
"FORMS",
|
||||
"JOBS",
|
||||
"MAX_RESULT",
|
||||
"NULL",
|
||||
"SCHEDULES",
|
||||
"TASKS",
|
||||
"TERMINAL",
|
||||
"BadCadence",
|
||||
"Batch",
|
||||
"Bound",
|
||||
"Cadence",
|
||||
"Cancelled",
|
||||
"CascadeProgress",
|
||||
"ChildFailed",
|
||||
"Declared",
|
||||
"Handler",
|
||||
"Job",
|
||||
"JobQueue",
|
||||
"JobState",
|
||||
"Jobs",
|
||||
"JobsSnapshot",
|
||||
"Listener",
|
||||
"Notice",
|
||||
"NoticeEntry",
|
||||
"NoticeKind",
|
||||
"NoticeLevel",
|
||||
"NoticeListener",
|
||||
"ResultTooLarge",
|
||||
"Retry",
|
||||
"Run",
|
||||
"RunCancelled",
|
||||
"RunContext",
|
||||
"Schedule",
|
||||
"ScheduleListener",
|
||||
"Spec",
|
||||
"Tally",
|
||||
"Task",
|
||||
"TaskProgress",
|
||||
"declare",
|
||||
"fan_out",
|
||||
"job",
|
||||
"local_zone",
|
||||
"notices",
|
||||
"spawn",
|
||||
]
|
||||
Reference in new issue
Block a user