roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
396 lines
54 KiB
Markdown
396 lines
54 KiB
Markdown
# `pipeline` — the job queue
|
||
|
||
## What it is
|
||
|
||
A jobs **table in SQLite** with an async worker draining it. Handlers are plain synchronous functions run in threads.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Packages | stdlib here — `sqlite3`, `asyncio`, `dataclasses`, `inspect` — plus pydantic, which `pipeline/models.py` wants |
|
||
| Not used | no Celery / RQ / Redis / Huey, no `multiprocessing`, no in-memory queue |
|
||
| Workers | threads, several at once, one per running job, from a pool of the queue's own |
|
||
| Storage | one SQLite file (WAL). Point it at the app's DB or one of its own |
|
||
| Concurrency | threads within a worker; any number of workers on one file — `claim` is one atomic `UPDATE`, and a claimed job is held on a lease its worker renews. `python -m pipeline.worker` starts one with no web app; `pipeline/tests/test_housekeeping.py` races two |
|
||
| Web | `pipeline.web.connect(app, jobs, config, store)` — routes, socket, the `<desk-jobs>` and `<desk-schedules>` elements, and the worker in the app's lifespan |
|
||
| States | `queued → running → done \| failed \| cancelled`, and `running → queued` again for a job with an attempt left |
|
||
| Scheduling | five cadences on a `schedules` row in the same file — `every 15m`, `daily at 06:00`, `weekly mon 07:00` — fired from the drain loop, defaults declared next to `@job` |
|
||
| Door | `from pipeline import DAYS, FINAL, FORMS, JOBS, MAX_RESULT, NULL, SCHEDULES, TASKS, TERMINAL, BadCadence, Batch, Bound, Cadence, Cancelled, CascadeProgress, ChildFailed, Declared, Handler, Job, JobQueue, JobState, Jobs, JobsSnapshot, Listener, Notice, NoticeEntry, NoticeKind, NoticeLevel, NoticeListener, ResultTooLarge, Retry, Run, RunCancelled, RunContext, Schedule, ScheduleListener, Spec, Tally, Task, TaskProgress, declare, fan_out, job, local_zone, notices, spawn` |
|
||
| Copyable | yes, and that is the point — `cp -r pipeline yourproject/`. Nothing here imports `desk` or `newsdesk`, and `tests/test_architecture.py` fails the build if it ever does. See `docs/pipeline-package.md` |
|
||
|
||
The table *is* the queue, which is why a job survives a restart, why the UI's job list is a `SELECT`, and why a worker can live in another process.
|
||
|
||
## The Entities
|
||
|
||
**Job** — one physical unit of work, admitted to the queue and run on one worker. It has a resource class, so the queue knows which worker to run it on. A job is the only thing the queue runs, and the only thing that has a row in `jobs`.
|
||
|
||
**Handler** — the function that defines the job, aka the code. `@job` registers the handler in the queue. The handler is the piece of code which is called when the job is about to run.
|
||
|
||
**Task** — **plain code that creates jobs, and is never a job.** A business step comes in — fetch every feed, give every article a body — and is translated into jobs the queue can run, typically by cutting the work into batch payloads. A task mints a run id for itself, queues one job per payload tagged with that id, and queues one **final job** behind them with `after="done"` on all of them, which is how it comes to be reported on. Then it is over: nothing of the task is alive afterwards, only its jobs and the callback the final job carries.
|
||
|
||
Everything a task queues, it queues in one go. That is what makes it plain synchronous code and not a job: finding the work is a query, and a query is cheap enough to run inside the request that pressed the button, or in the worker thread that has just finished the job before. If the planning itself needs real work done first, that work is a job, and what follows it is the next task of a cascade.
|
||
|
||
**Cascade** — a series of tasks, for when each can only be planned once the one before it is over, or where there is a natural breaking point between them. Structurally it is the same shape as a task, and the pipeline does not distinguish the two. Both are rows in `runs`, both group jobs by an id, both are killed by the same call, and neither has an object of its own. The difference is a **convention**, and it is worth stating because nothing enforces it:
|
||
|
||
> A **task** queues everything it wants at once.
|
||
> A **cascade** is where the breakpoints are: it queues the next task only once the last one is over.
|
||
|
||
The final job of a task is what calls back to the cascade to start the next task.
|
||
|
||
Jobs are tied to a task and a cascade by `task_id` and `cascade_id`.
|
||
|
||
**Schedule** — **a row that queues a job, and is never a job itself.** "Enqueue this kind with this payload on this cadence": a name, a kind, a payload, a cadence, whether it is on, when it is next due and what it queued last time. It is a row rather than a timer for the same reason a queued job is: nothing is alive between runs, so a process restarted at 05:59 still fires the 06:00 one, and two workers on the file read one schedule instead of each holding a copy of it in memory. It queues *jobs*, not tasks — a schedule that wants a whole cascade queues the job that starts one.
|
||
|
||
**Cadence** — one of five wordings, and the only five: `every 15m`, `every 6h`, `every 2d`, `daily at 06:00`, `weekly mon 07:00`. Not cron, which is five fields of positional punctuation that nobody reads back correctly. The interval forms are elapsed time and ignore the timezone; the clock forms are wall-clock in `JobQueue(tz=...)` and are 06:00 on both sides of a daylight-saving change. The two mean different things and a person picking one has already chosen which.
|
||
|
||
**Declaration** — a default schedule an app ships, written next to `@job` with `declare(name, kind, cadence, payload, label)`. `SCHEDULES` is to it what `JOBS` is to `@job`. On start, a declared name with no row gets one and a name that has one is not touched, so the declarations are where a schedule *comes from* and the table is what it *is*.
|
||
|
||
## How a job has to look
|
||
|
||
```python
|
||
def handler(config, store, *, context=NULL, limit=None) -> dict: ...
|
||
```
|
||
|
||
`config` and `store` are the app's, passed through untouched. Payload keys arrive as keyword arguments — **declare the ones you take**; an undeclared key is a `TypeError` recorded against the job, and `**kwargs` would swallow a typo. `context` is optional: a handler that takes neither progress nor cancellation says so by not declaring the parameter, and `registry.py` works that out once at registration rather than inspecting a signature per run. One payload key is reserved — a job whose payload names `context` fails, because the worker's own is what goes there.
|
||
|
||
`limit` is a **convention, not a contract**. Nothing in the pipeline reads it. It is an ordinary payload key that most of newsdesk's sweeps happen to accept, so that a person can ask for a short run of one, and a handler is free not to have it.
|
||
|
||
The return value should be a JSON-serialisable dict, and is stored as the job's result. A handler that returns something else — `None`, a list, a count — is not an error: the worker keeps the ending and stores an empty result, so the job is `done` with `{}`. Which means a result nobody can read is silent, and worth knowing when a handler's tally goes missing rather than its job failing.
|
||
|
||
**An exception ends the job in `failed`.** The traceback goes to the log, the exception is stored on the row as `Type: message`, and a `job_failed` notice goes to the screen saying it looks like a bug — *unless* the handler already announced a notice about that same failure, in which case the handler's sentence stands alone rather than being contradicted underneath. The `except` is deliberately broad: a handler is a source plugin, a model call, a subprocess, and anything it raises is that one job's failure and nothing else's. Letting it escape would take the worker down and strand every job behind it.
|
||
|
||
## Minimum example
|
||
|
||
```python
|
||
from pipeline import Jobs, JobQueue, job, RunContext, NULL
|
||
|
||
@job("greet", "Saying hello", solo=True, resource="cpu")
|
||
def run_greet(config, store, *, context: RunContext = NULL, name="world") -> dict:
|
||
context.step(0, 1, f"greeting {name}") # progress + cancellation check
|
||
return {"greeting": f"hello {name}"}
|
||
|
||
run_greet(cfg, store, name="wolf") # 1. straight call — no queue at all
|
||
|
||
queue = JobQueue(Jobs(cfg.database), cfg, store) # 2. queued
|
||
await queue.start()
|
||
queue.enqueue("greet", {"name": "wolf"})
|
||
```
|
||
|
||
`@job` returns the function unchanged, so every job stays importable and callable on its own. Registration is by import — importing `newsdesk.jobs` fills `JOBS`.
|
||
|
||
## Workers
|
||
|
||
`JobQueue.WORKERS = {"llm": 4, "http": 8, "cpu": 1}` — how many workers of each kind there are, and so how many jobs of that resource run at once. An app passes its own; newsdesk has a `workers` section. A resource named in neither gets one. Counted from the table, not from this process, so a second worker on the same file shares the four model workers rather than helping itself to four more.
|
||
|
||
The count belongs to the worker, not the job kind: tagging and summarising both ask a model, so they share the same four.
|
||
|
||
Our queue is fanning out threads not processes. Therefore, a worker is a thread, not a program.
|
||
|
||
## `depends_on`
|
||
|
||
A job may name jobs it waits for, and say what it wants of them.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `after="ok"` | admit only if all of them succeeded; fail with them otherwise. The default, and what one job needing another's output means. |
|
||
| `after="done"` | admit once all of them are over, however they ended. |
|
||
|
||
Waiting costs nothing: `claim` passes the job over and it stays queued. No timer, no listener, nothing holding a place, nothing lost to a restart.
|
||
|
||
`done` exists for the job that reports on the others — success-only semantics would mean the report never exists when there is something to report. An id naming nothing is satisfied.
|
||
|
||
A **cancel** and a **failure** are the same thing here: both travel to the `after="ok"` jobs waiting on them, and neither holds back an `after="done"` one. Killing a job is dropping that batch, not stopping the run it belongs to, so the job that reports still runs and counts it among the ones that did not land.
|
||
|
||
**What a waiting job is handed.** `context.waited_for` is a tuple of `pipeline.models.Job` — the whole row for each job this one named, oldest id first, and empty for a job that waited for nothing. Each carries its `state` and its `result`, which is everything a report needs. Read at claim time off the jobs themselves, never counted up as they landed, so there is no tally to keep in step and nothing to lose to a restart half way through. An id that named a job which no longer exists is simply absent from the tuple, so `len(waited_for)` is what was found, not what was asked for.
|
||
|
||
## `solo`
|
||
|
||
**One job of this kind queued or running at a time.** Two tabs pressing Fetch is one pull, and the second caller is handed the job already going rather than an error, because what they asked for is that the work be under way and it is. Enforced by a partial unique index over the unfinished states, so it holds across every tab, every client and every process at once — not by a flag in whichever browser happened to ask.
|
||
|
||
**Off unless a job says otherwise.** A sweep that forgets to say `solo=True` runs twice and wastes a minute. A batch kind that forgets to say `solo=False` has seven of its eight batches quietly handed the first one, and the work simply never happens. The default belongs on the side that fails loudly.
|
||
|
||
A job that names `depends_on` never dedups, whatever it declared, since the whole point of a batch is that there are several.
|
||
|
||
**The job you are handed back may not be yours.** Dedup matches on the kind alone, so a duplicate is answered with whatever live job holds that key — which can belong to another cascade entirely. Press Fetch while an overnight pull is part way through its own `pull` job and you are handed *that* job: the right answer to "is a pull under way", and the wrong one to "is *my* pull under way". Nothing rewrites its `cascade_id`, so a caller that means to follow the run it started has to notice the ids it got back are not the ones it minted.
|
||
|
||
## Retrying
|
||
|
||
```python
|
||
@job("extract", "Reading the page", resource="http",
|
||
max_attempts=3, retry_delay=60, retry_backoff=2.0)
|
||
def run_extract(config, store, *, context=NULL, url=""): ...
|
||
```
|
||
|
||
**`max_attempts` counts the first attempt**, so `1` is the default and means what every job here meant before the column existed: run it once, and an exception is the end of it. An attempt that fails with attempts left goes back to `queued` with a `run_after` on it, `retry_delay × retry_backoff ⁿ` seconds hence.
|
||
|
||
**Waiting costs exactly what `depends_on` costs, which is nothing.** The wait is a column, not a sleeping worker: `claim` passes the row over until its time comes round, so no worker, no thread and no lease is spent on an hour's backoff, and the poll that exists to notice another process's enqueue is what notices the hour is up. Sleeping in a handler would hold a worker for the duration, and is the thing this exists instead of.
|
||
|
||
**And because it is a column it survives the process that wrote it.** A retry scheduled for 09:40 is taken at 09:40 by whichever worker is alive then, including one started at 09:38 in a different process. There is nothing to hand over, because nothing was alive to hand.
|
||
|
||
**Three endings, and `Jobs.retry` picks between them.** `cancelled`, if a Kill landed while the attempt ran — a retry that put the job straight back would be a button that stopped nothing and said it had. Back to `queued`, if there are attempts left. `failed`, if there are not, and `Job.exhausted` says so from then on: the policy ran its course and the work still did not land, which is the only one of the three where trying again is a person's decision. A job that failed at `max_attempts=1` is not exhausted — nothing was ever used up.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `job.waiting` | queued, with its next attempt still to come |
|
||
| `job.exhausted` | failed on the last attempt it was allowed, rather than the only one |
|
||
| `job.attempt` | how many times it has been *started*, stamped by `claim` in the statement that takes the row |
|
||
| `jobs.waiting()` | every row in the first of those states, oldest first |
|
||
| `jobs.blocked()` | everything `claim` would pass over — waiting on a time, or on another job |
|
||
|
||
**A handler can name the wait but not the bound.** `raise Retry("rate limited", after=120)` puts the other end's own answer in place of the backoff, which is worth having because the alternative is a guess: a minute is either a minute of doing nothing when the service was ready, or another refusal charged against the bound. It buys no extra attempt — a job queued with one attempt that raises `Retry` fails exactly as it would on any other exception. The bound is ours to declare and the wait is the world's to name.
|
||
|
||
**A retry says nothing to the screen.** A notice is for what a person has to act on, and while attempts remain there is nothing to act on: the queue is already doing what they would have asked for. The log has every attempt either way; only the one that runs out announces.
|
||
|
||
**Retrying is off by default**, for the reason `solo` is off by default. A fetch that should have been retried and was not costs one item until the next sweep finds it again. Work retried three times that never should have been is three times the writes, or three times the bill.
|
||
|
||
`queue.enqueue(kind, delay=600)` schedules one job for later with no retry involved at all. `queue.enqueue(kind, max_attempts=n, retry_delay=s, retry_backoff=b)` overrides what the kind declared for one piece of work, and `dedup_key=` overrides the kind's own key — which is how one *item's* acquisition is made unrepeatable while the kind stays freely repeatable, so two presses of a reader's retry button collide in the unique index rather than in one browser tab.
|
||
|
||
**A batch can carry its own bound.** A task's finder may return `Batch(payload, Bound(max_attempts=3, delay=300, backoff=2.0))` instead of a bare payload, and those numbers go onto that job's row. That is where a plugin's declared retry policy lands: newsdesk cuts its body batches per plugin profile precisely because the row holds one bound, so a batch mixing two outlets could only carry one of their declarations. A finder returning plain dicts — every task that existed before this — gets `Bound()`, which is one attempt and no wait. The direct path (`task.run`) ignores bounds entirely: a bound is a number the queue enforces by putting a row back with a time on it, and there is no row.
|
||
|
||
## Typical way of dealing with batches
|
||
|
||
A task queues its batches as independent jobs, and one **final job** behind them. It depends on all of them with `after="done"`, so it runs however they ended, and it is handed each one's state and result in `context.waited_for`. It is the only job of the task that calls back.
|
||
|
||
A task that finds nothing to do is the ordinary case, not the empty one: it queues no batches and its final job runs immediately, with an empty `waited_for`, and moves the cascade on. Which is why a switch turned off, a backlog already cleared and a stage this pull does not want are all the same thing here — a finder that returns no payloads — and none of them stops the chain.
|
||
|
||
**The callback is a name in the final job's payload, not a function.** Nothing in a table can hold a function, and a name is what makes the callback survive a restart and be callable from another process; the worker running the final job resolves `module:function` to imported code. `newsdesk.jobs.cascade:step` reads as documentation in the payload, and carries its own import, where a registry would have to be filled before the worker looked in it.
|
||
|
||
It is called with the task's tally and whatever the task fixed at the time it was made:
|
||
|
||
```python
|
||
def step(config, store, tally: Tally, payload: dict, *, context: RunContext) -> dict | None: ...
|
||
```
|
||
|
||
`payload` is `Task.carries`, stored in the final job's row, so it survives a restart and a worker in another process. Whatever the callback returns is merged into the final job's own result; `None` is fine. A task with no callback stops there, which is what being asked for on its own means.
|
||
|
||
**`settle` is a second name in the same payload, resolved the same way, and it belongs to the task rather than to what follows.** The final job runs it before the callback, over the same tally. It is for the decision a batch is in no position to make: a batch is one of forty and cannot see what the other thirty-nine are doing, so anything that must hold across the whole run — a cap on what the run may add, a merge of what several batches each asked for — is worked out once, here. Newsdesk's tagger is the one that has one: a batch proposes new tags and adds none, and the settle merges the proposals, takes at most `max_new_tags_per_run` of them and goes back to tag the items that asked. A task started on its own has no callback and still settles, because settling is its own work and not a message to anybody.
|
||
|
||
**`Tally`** is how the batches ended, read off their rows at the moment the final job runs:
|
||
|
||
| | |
|
||
|---|---|
|
||
| `done` `failed` `cancelled` | how many batches ended each way, and `jobs` for the sum |
|
||
| `results` | what each batch returned, in the order they were queued |
|
||
| `killed` | whether the **task itself** was killed |
|
||
|
||
`killed` is the one that cannot be inferred from the counts, and is the reason it is there. A task killed while its batches ran leaves every one of them `cancelled`, and so does a person killing every batch by hand — identical numbers meaning opposite things: *skip this step*, or *drop this work*. Only the `runs` row knows, so the final job asks it. What to do about it is the cascade's decision and nothing here presumes it; newsdesk's pull skips the stage and carries on to the next.
|
||
|
||
## Passing data
|
||
|
||
Three ways to pass data. Can be mixed and merged.
|
||
|
||
1. **Store.** A job reads its input or persists its output directly into a storage.
|
||
2. **Parent.** A job receives data as a parameter, put in its payload by whatever queued it — a batch's uids, a cascade's options. Fixed at queue time, and the only kind of data a job that runs first can hand to a job that runs later.
|
||
3. **Cross job via `context.waited_for`.** The `after="done"` job gets the jobs it waited on: their `state` and their `result`. Read, never pushed — nothing is counted while the work runs, so there is no tally to lose.
|
||
|
||
The usual arrangement is the third feeding the callback: the final job reads how its batches went and hands the tally to the cascade, which turns it into the next `Task`.
|
||
|
||
`context.upstream()` is the third one merged into a dict, later parents winning, which is what a chain of one parent wants to read like. It is `{}` under `NULL`, which is what keeps a chained handler callable by hand:
|
||
|
||
```python
|
||
@job("extract")
|
||
def extract(config, store, *, context=NULL, html: str = ""):
|
||
html = html or context.upstream().get("html", "")
|
||
```
|
||
|
||
From the queue nobody passes `html`; from a terminal you pass it yourself. A handler reading `waited_for[0].result` directly would be the one kind in the pipeline you cannot simply call.
|
||
|
||
**The rule is that whoever has the data writes the row.** Never insert a row meaning an earlier job to fill its payload in later: a worker claims it the moment the parent is `done`, so it can be claimed between that write and its commit; `dedup_key` has to be known at queue time; and `max_attempts` re-runs it from its own row. Reading a parent has none of that, because the row was complete when written.
|
||
|
||
Bulk data belongs in the store. `Tally.results` is a query across the children's `result` TEXT columns and those columns never shrink — counts, ids and errors are free, a few hundred large dicts are not.
|
||
|
||
## `fan_out` — N jobs and the one that reports on them
|
||
|
||
```python
|
||
@job("body")
|
||
def body(config, store, *, context=NULL):
|
||
context.fan_out("body_batch", [{"uids": c} for c in chunks],
|
||
then="newsdesk.jobs.cascade:step")
|
||
```
|
||
|
||
One call, one transaction. The reporting job is `FINAL`, the pipeline's own kind, and an app never writes one. The children go into a run of their own inside the caller's cascade, which is what lets the callback fan out again — step N queues step N+1, and that is the whole of what a cascade is.
|
||
|
||
Because the thing fanning out is itself a job it has a context, so this works from inside a handler as well as from a request. `Task` is the same three statements with a finder in front, which is why a task and a cascade are one shape and one primitive covers both.
|
||
|
||
`()` under `NULL`: a handler called directly is being run by somebody standing there, and the useful answer is to do its own work and report that nothing was queued behind it.
|
||
|
||
## Running it
|
||
|
||
```python
|
||
queue = JobQueue(Jobs("app.db"), config, store) # then, in an app:
|
||
connect(app, Jobs("app.db"), config, store) # pipeline.web: routes, socket, element, lifespan
|
||
queue.drain(timeout=60) # in a script: run what is queued, stop
|
||
```
|
||
|
||
```bash
|
||
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
|
||
```
|
||
|
||
The worker is the same `JobQueue` with no web app, on the same table, and may run beside the web app's own. `--until-idle` drains and exits. The bootstrap is the one `isolate=True` uses.
|
||
|
||
## Ending work that will not end itself
|
||
|
||
Cancellation is cooperative because the work is synchronous — but two kinds of work never look up, and for those there is a real kill.
|
||
|
||
**A command.** `context.spawn(argv, timeout=...)` runs it, polls for Kill and for the deadline, and kills the **process tree** on either — `taskkill /T` on Windows, `killpg` elsewhere, which is why the child is started detached. `proc.kill()` alone would leave the ffmpeg a shell started still holding the core. Blocking a worker thread on this costs nothing: the thread is only sleeping. Output goes to temporary files, not `PIPE`, because a child that fills the pipe buffer while the loop polls is a deadlock. A non-zero exit is handed back rather than raised.
|
||
|
||
**Python that cannot be interrupted.** `@job(..., isolate=True, timeout=600)` runs the job in a child process, which is the only thing that can be killed mid-`sklearn.fit`. Nothing is pickled: a job row is already a complete description of the work, so the child is handed `--db` and `--job` and reads the same row. It writes *progress* to that table — which is how its progress reaches the browser with nothing forwarding it, the snapshot being a `SELECT` — and hands its result back through a file. The worker still writes the ending, so there is exactly one writer of the terminal state, as before.
|
||
|
||
`isolate=True` needs `bootstrap="module:function"` on the queue, returning `(config, store)`: neither survives a process boundary, so both are rebuilt in the child, and the same import is what fills `JOBS` there. `bootstrap_arg` is one string passed to it — an app's config path, which the child cannot learn any other way.
|
||
|
||
**`timeout` without `isolate` raises at registration.** A deadline over a thread could only mark the row failed and walk away — the work would carry on holding the core, and two things would intend to write one row. The refusal names both ways out.
|
||
|
||
## `context.task` — a job asking for what comes next
|
||
|
||
Newsdesk's body task fetches each eligible article and passes its HTML to `desk.tools.article.extract_article`. Its storage writer saves structured content, format, provenance and a derived plain-text projection atomically; existing enrichment jobs continue to read the generated `body` view. Empty or late fetch results preserve previously acquired content. The body job still uses the existing wall detector and candidate rechecks, and does not yet execute plugin-declared acquisition *graphs* — one declared step, the one whose tool is `extract_article`, is what it runs. Its declared **retry policy** is executed: a failure is classified with `desk.tools.failures`, the plugin decides whether that category is worth another attempt, and the batch raises so the queue puts it back under the bound its row carries. Batches are therefore cut per plugin profile. A second attempt asks the table again and so re-fetches only what is still missing, and never overwrites text that has since arrived. Structured bodies are retained without the former 60,000-character clipping; downstream model tasks remain responsible for their own input limits.
|
||
|
||
Hacker News discovery now runs as the `hacker_news` outlet through shared RSS fetching. Its item destination stays the publisher URL, so the subsequent body task extracts that article and records that URL as provenance. Its supplied discussion link is a separate typed input, and feed points/comment statistics stay in source evidence rather than the summary or preview. Its declared tagging policy runs: the destination article is what the tagger reads, and a link whose page would not extract is tagged from its title. Its summary policy stays off — the summariser selects only profiles that ask for an abstract, and nobody has decided that a front-page link earns one.
|
||
|
||
A handler that discovers work only by doing its own can name the task that should follow it:
|
||
|
||
```python
|
||
context.task(cascade.task("body", cascade="pull"))
|
||
```
|
||
|
||
**Recorded, not queued.** The worker starts it after this job lands in `done`, on the same thread that ran the handler, and hands it this job's `cascade_id` so its batches land in the run this one belongs to. Three consequences worth knowing: a task never outlives the failure of the job that described it, a handler still cannot reach the queue, and a job's row lands a moment *before* the jobs that follow from it exist.
|
||
|
||
It returns whether anything took it — `False` under `NULL`, where there is no queue. A callback invoked from a notebook or a Claude Code session gets that answer and has to do the work itself.
|
||
|
||
## `Cancelled` and `RunCancelled`
|
||
|
||
Two exceptions with similar names, and only one of them is a handler's business.
|
||
|
||
**`Cancelled`** is raised by `context.check()` inside a handler when this job has been asked to stop. The worker catches it and records `cancelled` rather than `failed`. A handler should let it travel; catching it is how you build the thing the Kill button cannot kill. A handler that never checks is not lost either — the worker asks once more when the handler returns, and a job asked to stop while it ran ends `cancelled`, keeping whatever it made and queueing nothing.
|
||
|
||
**`RunCancelled`** is raised by `Jobs.enqueue` for a job whose `cascade_id` or `task_id` names a killed run, except for an `after='done'` job under a killed *task* — the same carve-out `cancel_run` and `stop_task` make, so a kill landing while a task is still fanning out its batches cannot take the final job with it. A killed cascade refuses it too, which is what "nothing reports and nothing follows" means. It is not a handler's business and a handler will not see it: `JobQueue.enqueue` catches it, logs it, and returns **`None`** instead of a `Job`. So a caller that queues a job has to expect no job back — `Task.start` skips the id it did not get, and a `depends_on` list is that much shorter. Nothing is told that a run is over; a callback tries to queue the next task and the table says no, which is the same answer across processes, after a restart, and when the kill landed mid-run.
|
||
|
||
## Ids
|
||
|
||
Every job has an `id` of its own, and carries a `task_id` and a `cascade_id`. Both are grouping ids: nothing depends on a task, but everything belonging to one can be found by it and killed together. Two of them rather than one because they are two groupings and both have a button.
|
||
|
||
**Killing a run** is one operation for a task and a cascade alike — `cancel_run`. Three things: flag the row, end its queued jobs, ask its running ones. And a fourth that is not an operation at all: **`enqueue` refuses a job whose `cascade_id` or `task_id` names a killed run**, which is what stops a callback from reviving a killed task or moving a killed cascade on.
|
||
|
||
Three buttons, three sizes of stop: killing a **job** drops one batch, killing a **task** skips one step, killing a **cascade** ends the run.
|
||
|
||
**Stopping a run is not killing it.** `Jobs.stop_task` ends a task's queued batches and nothing else: the row is not flagged, so `Tally.killed` stays false and nobody is told a person pressed anything; running batches keep their requests and their own endings; the final job still reports. It is what a handler reaches through `RunContext.stop(why)`, for the failure that is the backend's answer to the whole run rather than to one batch — a spend limit, an expired login, no model on the machine. The reason is written to `error` on each cancelled row, so a batch that never ran says on screen why it didn't. `desk/modeltask.py` is the one caller, and `notices.stops_the_run` is what decides.
|
||
|
||
The one asymmetry between killing a `cascade` and killing a `task` is what the last two of those mean. Kill a **task** and its final job is spared: it still runs, and what it reports says the *task* was killed rather than only how many of its batches ended cancelled — see `Tally.killed`. Kill the **cascade** and the final job dies with it, so nothing reports and nothing follows — the one stop no cascade gets a say in, because there is nothing left of it to ask.
|
||
|
||
cascade_id = jobs.new_run("cascade", "pull") # before a single job exists
|
||
jobs.of_cascade(n) · jobs.of_task(n) · jobs.run(n) · jobs.cancel_run(n)
|
||
jobs.cascades()
|
||
POST /api/cascades/{id}/cancel · /api/tasks/{id}/cancel · /api/jobs/{id}/cancel
|
||
|
||
A job queued with neither begins both: something asked for it on its own, so it is a cascade of one task of one job. A task started with no cascade does the same, for the same reason.
|
||
|
||
The cascade that comes of it says so in its name — `pull on its own` rather than `pull` — because a chain named `pull` is a different run, and `live_run("cascade", "pull")` is how a caller asks whether that chain is already going. Answering it with a lone job would be a Fetch button that starts nothing and greys itself over a run that is one job long. The task keeps its plain name (`pull`), since the task really is that task however it was started, and the strip draws its box from that.
|
||
|
||
## Schedules
|
||
|
||
The tick is in the drain loop, at the top of `_turn`, before the claim — not in a thread of its own and not on the idle path with the reclaim and the purge. Not a second thread, because then two things would race to enqueue one row. Not on the idle path, because reclaim and purge are repairs that can wait for a quiet moment and a schedule is *work*: on a queue that is never idle, 06:00 would never come round. The check itself is a clock comparison, so paying it every pass costs nothing. It also runs once in `start()`, after reconciliation, so a default declared today fires on the start that introduced it.
|
||
|
||
**Overdue fires once, not once per slot missed.** A firing sets `next_run` from *now*, never by stepping through the times it was down for. There is one `next_run` column and nowhere for a backlog to accumulate even in principle, so an app off for three days runs its hourly job once when it comes back.
|
||
|
||
**No pile-up.** Before firing, two questions: is the job from the last firing still queued or running (`last_job_id`), and is this kind `solo` with one already under way? Either way `next_run` advances and nothing is queued. A job slower than its own cadence must not grow a queue of itself, and a `solo` kind somebody has already pressed must not be recorded as this schedule's firing.
|
||
|
||
**A row the code no longer supports keeps its place.** An unregistered kind, or a cadence nothing can parse (only reachable by editing the file by hand), is logged once per process and left exactly as it is — `next_run` included — so registering the handler again runs it rather than starting another cadence. `<desk-schedules>` marks the row `unknown kind`; deleting it is deliberate.
|
||
|
||
**Reconciliation never overwrites and never deletes.** `Jobs.add_schedule(..., only_if_new=True)` is the whole of it: one statement, so two workers starting at once insert a declared default once between them. A declaration removed from the code leaves its row behind, because a deploy that silently forgot a schedule somebody relied on is worse than an orphan that says so.
|
||
|
||
**One scheduler per file.** `JobQueue(schedules=False)` and `pipeline.worker --no-schedules` are what every worker after the first passes. The workers themselves race correctly — that is `claim` — but the pile-up guard reads `last_job_id`, which neither of two simultaneous firings has written when the other looks.
|
||
|
||
declare(name, kind, cadence, payload=None, label="") # next to @job
|
||
Cadence.parse(text) · str(cadence) · cadence.next_after(moment, zone)
|
||
jobs.schedules() · jobs.schedule(id) · jobs.add_schedule(...)
|
||
jobs.update_schedule(id, **fields) · jobs.delete_schedule(id)
|
||
jobs.due_schedules(now) · jobs.mark_fired(id, job_id, next_run)
|
||
queue.run_schedule(id) · queue.notify_schedules()
|
||
GET/POST /api/schedules · PATCH/DELETE /api/schedules/{id}
|
||
POST /api/schedules/{id}/run
|
||
|
||
## The three tables
|
||
|
||
**`jobs`** is the queue, and holds rather more than the states: the `payload` it was queued with and the `result` it ended with, both JSON; `done`, `total` and `message`, which are the progress bar and are written by `context.progress`; `error`, the exception string of a failure; `dedup_key`, `depends_on` and `after`; `resource`; `cascade_id` and `task_id`; the retry policy and where it has got to, as `attempt`, `max_attempts`, `retry_delay` and `retry_backoff`; and five times — `created_at`, `started_at`, `seen_at` (the lease), `run_after` (the next attempt) and `finished_at`. `cancel_requested` is a flag, because a kill is a request in the table rather than a message to whichever process happens to be running the job.
|
||
|
||
The policy is on the row rather than read off the job kind when a failure happens, for the same reason the payload is: the attempt that has to decide may be in another process, after a restart, running a version of the code where the kind's numbers are different or the kind is gone.
|
||
|
||
**`schedules`** is the standing instructions: `id`, `name` (unique, and the handle a declaration is matched on), `kind`, `label`, `payload`, `cadence`, `enabled`, `next_run`, `last_run`, `last_job_id`, `created_at`. `next_run` and `last_run` are UTC ISO stamps like every other time here, and `next_run` is compared as text in `due_schedules` for the reason `run_after` is in `claim`: `''` sorts before every stamp, so "never" and "not due" are one comparison with no coalesce in it. A cadence worded in a local timezone is turned into one of these before it is stored, so `pipeline/schedule.py` is the only file that reads a zone at all.
|
||
|
||
**`runs`** is where a cascade's id and a task's come from, and it is deliberately almost empty: `id`, `kind` (`cascade` or `task`), `name`, `label`, `created_at`, `cancelled_at`.
|
||
|
||
What it does **not** have is the point of it. No state, no progress, no counts, no `finished_at`: the jobs carrying the id are the record of what happened, and a second copy of that could only go out of step with them. `cascades()` tallies a run by reading its jobs. So the table exists for exactly two facts its jobs cannot supply — that a number was handed out *before* there was a job to take it from, and that somebody killed this run. `name` carries a third job: it is how `live_run` answers "is one already going", which is why a cascade minted only because a lone job needed an id is named apart from a chain somebody asked for (see `on_its_own`).
|
||
|
||
## `ModelTask` — next door, not here
|
||
|
||
`ModelTask` left this package: it imports `desk.config` and `desk.tools.agent`, so it was never the queue's. It is `desk/modeltask.py` now, and this section stays because it is still the thing most jobs in newsdesk are.
|
||
|
||
`desk/modeltask.py` is one layer above `desk/tools/agent.py`. `agent` answers *how do I reach a model*; this answers *what do I send it and what do I do with what comes back*, once, for every task that needs one. Tagging and summarising are the two, and what differs between them is all content: which rows are waiting, what goes in the batch, what the model is told, what shape it must answer in, what to do with the answer, what the run has to settle once every batch is in, and how a row that never gets one is given up on. That set of callables **is** a `ModelTask`; the module is everything that is not one.
|
||
|
||
`modeltask.of(TASK)` turns it into an ordinary `Task`, so it is queued, watched and killed like any other stage. `modeltask.one` is what a single batch job does: check for a kill, build the payload from the uids it was given, ask, apply. Two things about it are worth reading before changing:
|
||
|
||
- **The prompt is built when the batch runs, not when it was handed out.** Tagging's carries the vocabulary as it stands at that moment, so a tag somebody adds by hand mid-run reaches the batches still to come. It does not carry what a sibling batch proposed, because a batch proposes and adds nothing: the run's own additions are worked out once, in `settle`.
|
||
- **`pending`, `waiting` and `build_batch` are given the config as well as the store.** Which rows are waiting is no longer a question a table can answer on its own: in newsdesk both backlogs are whatever the installed plugins declare about their own items, and the plugins are found through the config. Both tasks now use it; the argument is still there for one that does not.
|
||
- **A kill during the call does not throw the answer away.** `check()` is called before the model is asked, so a Kill pressed while the job sat queued makes no paid call at all. Once the request is in the air the money is spent, so the answer is applied and the worker records the cancellation at return. The asymmetry is deliberate and looks like an oversight otherwise.
|
||
|
||
**A row is given up on eventually.** "The next run finds the work again" is unbounded, and for a row no model will ever answer usefully — a segment whose only honest abstract is one sentence, a uid dropped from every reply — unbounded means paying for the same call on every pull for as long as the row exists. So a row handed over and left unanswered is charged a turn through `ModelTask.missed`, and one charged three of them stops being offered until a person asks for it deliberately. A batch that *failed* charges nothing: it never reached a model, so it proved nothing about its rows. Every `ModelTask` must supply `missed`, and there is no default, because the default is the bug.
|
||
|
||
**A refusal is the one failure that is about a row.** The platform runs a classifier over the request and rejects the whole thing at once, naming nothing — and a news archive meets it regularly, because reporting on a worm or a supply-chain compromise is written in the vocabulary it watches for. `one` answers it by asking again in halves (`_narrow`) until the batch is the row, which is then charged and leaves the backlog after three. Halves rather than one at a time: two flagged rows in a batch is ordinary, since they arrive from the same day's news, and stepping through twenty singly would pay eighteen calls for what four find. The notice is `agent.refused` and is deliberately not `standing` — the account is fine and the next batch is about different rows.
|
||
|
||
**What was given up on is listed, and said once.** A refused row is charged with `why="refused"`, kept in `raw.why.<task>`, so the list can tell a refusal from a row the model left out. The strip interrupts only for given-up rows without `raw.seen.<task>`; its × or opening Jobs & history sets that mark on the server, so a reload does not say them again and a row given up on later still does. Try again there reopens a row with one attempt (`reopen_given_up(..., attempts=1)`) and queues its `_batch` job at once, so a row refused again is back on the list after one call rather than after three pulls.
|
||
|
||
## `notices`
|
||
|
||
**Two halves since the extraction.** `pipeline/notices.py` is what the queue itself needs — the notice a failed job gets, the check that stops it being said twice, the rendering a terminal wants. `desk/notices.py` is newsdesk's vocabulary: `Rule`, `RULES`, `classify`, `stops_the_run`, `source_failed`. An app writes its own, because a useful sentence names things only the app knows.
|
||
|
||
`pipeline/notices.py` turns a failure into one line worth putting on screen. A handler calls `context.notice(...)`, and the queue writes it to the `notices` table and pushes it to whoever is watching — the counterpart to `log.exception`, not a replacement for it: log the traceback for afterwards, raise a notice for now.
|
||
|
||
**A notice is stored, one row per `(code, key)`.** The first design stored nothing, on the argument that a table would duplicate the log. That assumed somebody was watching when the notice was raised; a scheduled pull and a reader on a phone mean nobody usually is. `Jobs.record_notice` upserts the row and returns it as a `NoticeEntry` with `count`, `first_at` and `last_at`, and that row is what goes over the socket. `count == 1` is news and anything above is a repeat, so a client interrupts for the first and only counts the rest — a refused article is announced on the pull that met it, not on the two retries after. Every notice still reaches `on_notice`; what interrupts a person is the interface's decision.
|
||
|
||
A row goes when it is cleared (`DELETE /api/notices/{id}`, or `/api/notices` for all), after which the same failure is news again, or when `JobQueue.NOTICE_QUIET` (14 days) has passed since it was *last* said. `GET /api/notices` lists the log, newest first. A write that fails is logged and the notice is pushed anyway: it never costs the job that raised it. The table holds no traceback and no per-occurrence history; those stay in the log.
|
||
|
||
A notice carries three things beyond its text: `kind`, which says whether this is the world misbehaving or a bug of ours — the fork in what you do next, and exactly the guess you should not have to make from an error string; `fix`, the command when there is one; and `key`, which with `code` names the row in the log — so a feed that has been failing all week is one line with a count rather than one per pull, and two refused articles are two. `classify` is the interface, and `RULES` is the catalogue it reads, which doubles as the list of what this will ever interrupt you about. Everything in the catalogue is marked `expected`; the fallback is marked `bug`, which is the whole point of enumerating them — a failure nobody wrote a rule for is one to go and read about.
|
||
|
||
For a job author, two rules. Use a notice for what a person has to act on, not for what the program can get on with. And classify your own failure if you can: the worker announces "looks like a bug" for every job that ends `failed`, and `already_said` is what stops it contradicting a handler that had already explained itself.
|
||
|
||
A rule also says whether the failure it names is `standing` — caused by the account or the machine rather than by this batch, so every sibling of the run would be told the same thing. `stops_the_run(notice)` is that answer, and a task for the model cancels what is still queued rather than paying for twenty-two more refusals. A limit, an expired login and an unreachable model are standing; a timeout, an oversized answer and a reply that was not JSON are not.
|
||
|
||
## Timing
|
||
|
||
Seven numbers, all on `JobQueue`, plus one on `Jobs`. None of them is tunable and none is meant to be; they are here so that "why did that take a minute" has an answer.
|
||
|
||
| | | |
|
||
|---|---|---|
|
||
| `IDLE_POLL` | 1 s | How long to sleep before looking again when the table was empty. Only reached when nothing woke us — an enqueue in this process is instant, so this is what notices one made by another, and what notices a scheduled attempt whose time has come. |
|
||
| `PUSH_EVERY` | 0.25 s | Fastest a running job's progress may reach the browser. The write is not throttled; the push is. |
|
||
| `CANCEL_CACHE` | 0.25 s | How long `check()` caches its answer. It reads the table so Kill works across processes, and a handler calling it in a loop would otherwise turn one button press into a query storm. |
|
||
| `HEARTBEAT_EVERY` | 5 s | How often a worker restamps `seen_at` on the jobs it holds. |
|
||
| `STALE_AFTER` | 30 s | How long a `running` row may go unsaid-about before another worker reads it as abandoned. |
|
||
| `STOP_GRACE` | 5 s | How long a shutdown waits for handlers already in a thread. |
|
||
| `SCHEDULE_TICK` | 30 s | How often the loop looks for a schedule whose time has come, and the resolution the cadences promise — which is why `pipeline/schedule.py` has no seconds unit. |
|
||
| `Jobs.LINGER` | 15 s | How long a finished cascade stays in `cascades()`, so a run you watched is still on screen the moment after it lands. |
|
||
|
||
`HEARTBEAT_EVERY` and `STALE_AFTER` are far apart on purpose: the gap is how many beats may be missed to a stalled loop or a slow disk before live work is re-queued underneath a worker that is still there, which is the one mistake this must not make. And because reclamation runs at most once per `STALE_AFTER` on the idle path, a crashed worker's job comes back after up to about a minute rather than after thirty seconds.
|
||
|
||
## Limits
|
||
|
||
- **Cancellation in a thread is cooperative.** No `context.check()` in the loop, no Kill. A running handler in a thread is never interrupted, not even on `stop()`, because Python cannot kill a thread from outside. Work that might need killing runs in a process: `context.spawn` for a command, `isolate=True` for Python. See "Ending work that will not end itself".
|
||
- **A shutdown waits `STOP_GRACE` for its handlers, then walks away.** A thread cannot be cut off, only waited for or abandoned, and both are wrong as an absolute: waiting is a reload held up by a model call, abandoning is a handler still writing to a file the process considers finished with. Most handlers use the moment and end properly; the rest are left running, and land their own ending in the table whenever they reach it.
|
||
- **A shutdown hands its leases back, a crash cannot.** `stop()` expires the lease on whatever was still going when the grace ran out, because a process on its way out knows outright what silence can only imply. So a restart re-queues that work at once instead of a window later, and only a real crash pays the wait. Expired rather than re-queued, so a handler thread outliving the shutdown still writes its own ending rather than being overwritten by a second run of itself.
|
||
- **A job's ending is written by the thread that ran it.** The handler call and the terminal `done` / `failed` / `cancelled` are one function on one thread; the loop's part is claiming the job and pushing the new state out afterwards. Written back on the loop, the ending would be the first thing a `stop()` costs: cancelling an abandoned job's coroutine cancels the write, while the handler itself carries on to the end regardless — so a reload landing mid-run would leave finished work looking unstarted, and the next start would run it a second time. For a batch that asks a model that is a second bill for something already paid for. The task a job named with `context.task` is queued by that same thread, for the same reason and still only once the job has landed `done`. Which is the one thing to know when reading the table from outside: a job's row lands a moment before the jobs that follow from it exist, so an empty `running` and `queued` is not by itself the end of a run.
|
||
- **A kill outlives the crash that interrupted it.** A job asked to stop, whose worker then died, is reclaimed to `cancelled` rather than back to `queued`. The request is in the table exactly so that it survives the process that was meant to act on it.
|
||
- **A dead worker is noticed late, on purpose.** A `running` row carries `seen_at`, stamped by `claim` and renewed every `HEARTBEAT_EVERY`. SQLite reads and writes run off the event loop so they do not delay its heartbeat. A lease expires after `STALE_AFTER`; reclamation runs at startup and at most once per `STALE_AFTER` on the idle path, so recovery can take about a minute. A worker excludes jobs it still holds from reclamation, renews their expired leases and warns that its loop stalled. Another process can still reclaim an expired lease.
|
||
- **Every handler must be safe to run twice.** Not advice — the rule the rest of this depends on. A reclaimed lease, a shutdown that abandoned a thread, a second process on the same file, an attempt that failed half way through: each of them can put a handler on the same rows again, and the queue does not promise otherwise. So write for it. Read what is still waiting rather than trusting the uids you were handed, upsert rather than insert, and write each unit as it lands so a re-run has less to redo. `max_attempts` bounds how often that happens on a failure; it is not something to lean on, because the paths that re-run a handler without asking it are the ones that were always there.
|
||
- **Handlers must be sync.** An async handler is not awaited.
|
||
- **A crashed worker holds its slots until it is reclaimed.** `claim` counts `running` rows, so the survivor cannot use a crashed worker's slots until the lease expires and its next reclamation pass runs, about a minute in the worst case with a responsive loop and database. A stalled worker never re-queues its own held jobs. Socket sends run concurrently with a one-second timeout per subscriber, so an unresponsive reader cannot delay drain wake-ups indefinitely.
|
||
- **Handlers run in a pool of the queue's own**, sized to the worker counts, because the interpreter's shared pool is sized from the cpu count instead. On a machine smaller than `workers` the shared one would run fewer jobs than the table admits and, worse, queue the loop's own claim and heartbeat behind the handlers, which is a worker starving itself of the thread it needs to hold its leases.
|
||
- **Progress is a write per call.** `context.step` is one `UPDATE` in its own transaction, on its own connection. The push to the browser is throttled; the write is not, because the row is the record and a second process reads it. Report per batch rather than per row in anything with thousands of items.
|
||
- **One bad turn does not stop the loop.** A locked database is logged and retried on the next pass rather than ending the drain task, because a worker that died silently looks exactly like a healthy one with nothing to do.
|
||
- **`depends_on` names jobs, not tasks.** A job waits on ids it was given; a task id is for finding and cancelling, not for waiting.
|
||
- **No priorities.** Jobs are claimed oldest first within a resource; there is no way to jump the queue but to be queued earlier.
|
||
- **Five cadences, not cron.** `every 15m`, `every 6h`, `every 2d`, `daily at 06:00`, `weekly mon 07:00`, and nothing else — no seconds, no day-of-month, no "every weekday", no ranges. Anything the five do not cover is a job that decides for itself whether today is its day, or cron running `pipeline.worker --until-idle` as before.
|
||
- **A schedule is not an alarm.** It is checked every `SCHEDULE_TICK`, so "06:00" means within half a minute of it, and a machine asleep at 06:00 runs the job when it wakes rather than at 06:00. What it will not do is run it once per slot missed.
|
||
- **History is purged.** Finished jobs older than `JobQueue(retain=...)`, 30 days by default, are deleted hourly while idle. Kept regardless: unfinished jobs, the newest ending of each kind in each state (what `last(kind)` reads), and runs still pointed at. `retain=0` keeps everything.
|
||
- **A result is at most 64 KB of JSON, and a bigger one fails the job.** Past `MAX_RESULT` the job ends `failed` with `ResultTooLarge`, the size and the keys it returned, a notice goes out, and nothing of the result is written. It is not retried, since the same handler returns the same result. Every snapshot and tally reads results back, so data belongs in the app's store; raise `MAX_RESULT` if a project really needs bigger results.
|
||
- **A task for the model gives up on a row eventually.** See `ModelTask` above: three unanswered turns and a row stops being offered until a person asks for it deliberately (`reset_tagging` with no slug, `reset_summaries`). `ModelTask.missed` is the seam and `status.py` says how many.
|
||
- **Migrations are `_reconcile`.** Columns are added, and dropped, by reading the catalogue on open, because `CREATE TABLE IF NOT EXISTS` leaves an existing table alone and `PRAGMA user_version` on that file belongs to the app.
|
||
- **SQLite means one machine.** Not a network queue.
|