The schedule cadence is a picker, the add row is behind a button and off here, a running-jobs chip with Kill and Stop run sits in every header, and a declared schedule's label is refreshed at startup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
54 KiB
pipeline — the job queue
What it is
A jobs table in SQLite with an async worker draining it. Handlers are plain synchronous functions run in threads.
| Packages | stdlib here — sqlite3, asyncio, dataclasses, inspect — plus pydantic, which pipeline/models.py wants |
| Not used | no Celery / RQ / Redis / Huey, no multiprocessing, no in-memory queue |
| Workers | threads, several at once, one per running job, from a pool of the queue's own |
| Storage | one SQLite file (WAL). Point it at the app's DB or one of its own |
| Concurrency | threads within a worker; any number of workers on one file — claim is one atomic UPDATE, and a claimed job is held on a lease its worker renews. python -m pipeline.worker starts one with no web app; pipeline/tests/test_housekeeping.py races two |
| Web | pipeline.web.connect(app, jobs, config, store) — routes, socket, the <desk-jobs>, <desk-queue> and <desk-schedules> elements, and the worker in the app's lifespan |
| States | queued → running → done | failed | cancelled, and running → queued again for a job with an attempt left |
| Scheduling | five cadences on a schedules row in the same file — every 15m, daily at 06:00, weekly mon 07:00 — fired from the drain loop, defaults declared next to @job |
| Door | from pipeline import DAYS, FINAL, FORMS, JOBS, MAX_RESULT, NULL, SCHEDULES, TASKS, TERMINAL, BadCadence, Batch, Bound, Cadence, Cancelled, CascadeProgress, ChildFailed, Declared, Handler, Job, JobQueue, JobState, Jobs, JobsSnapshot, Listener, Notice, NoticeEntry, NoticeKind, NoticeLevel, NoticeListener, ResultTooLarge, Retry, Run, RunCancelled, RunContext, Schedule, ScheduleListener, Spec, Tally, Task, TaskProgress, declare, fan_out, job, local_zone, notices, spawn |
| Copyable | yes, and that is the point — cp -r pipeline yourproject/. Nothing here imports desk or newsdesk, and tests/test_architecture.py fails the build if it ever does. See docs/pipeline-package.md |
The table is the queue, which is why a job survives a restart, why the UI's job list is a SELECT, and why a worker can live in another process.
The Entities
Job — one physical unit of work, admitted to the queue and run on one worker. It has a resource class, so the queue knows which worker to run it on. A job is the only thing the queue runs, and the only thing that has a row in jobs.
Handler — the function that defines the job, aka the code. @job registers the handler in the queue. The handler is the piece of code which is called when the job is about to run.
Task — plain code that creates jobs, and is never a job. A business step comes in — fetch every feed, give every article a body — and is translated into jobs the queue can run, typically by cutting the work into batch payloads. A task mints a run id for itself, queues one job per payload tagged with that id, and queues one final job behind them with after="done" on all of them, which is how it comes to be reported on. Then it is over: nothing of the task is alive afterwards, only its jobs and the callback the final job carries.
Everything a task queues, it queues in one go. That is what makes it plain synchronous code and not a job: finding the work is a query, and a query is cheap enough to run inside the request that pressed the button, or in the worker thread that has just finished the job before. If the planning itself needs real work done first, that work is a job, and what follows it is the next task of a cascade.
Cascade — a series of tasks, for when each can only be planned once the one before it is over, or where there is a natural breaking point between them. Structurally it is the same shape as a task, and the pipeline does not distinguish the two. Both are rows in runs, both group jobs by an id, both are killed by the same call, and neither has an object of its own. The difference is a convention, and it is worth stating because nothing enforces it:
A task queues everything it wants at once. A cascade is where the breakpoints are: it queues the next task only once the last one is over.
The final job of a task is what calls back to the cascade to start the next task.
Jobs are tied to a task and a cascade by task_id and cascade_id.
Schedule — a row that queues a job, and is never a job itself. "Enqueue this kind with this payload on this cadence": a name, a kind, a payload, a cadence, whether it is on, when it is next due and what it queued last time. It is a row rather than a timer for the same reason a queued job is: nothing is alive between runs, so a process restarted at 05:59 still fires the 06:00 one, and two workers on the file read one schedule instead of each holding a copy of it in memory. It queues jobs, not tasks — a schedule that wants a whole cascade queues the job that starts one.
Cadence — one of five wordings, and the only five: every 15m, every 6h, every 2d, daily at 06:00, weekly mon 07:00. Not cron, which is five fields of positional punctuation that nobody reads back correctly. The interval forms are elapsed time and ignore the timezone; the clock forms are wall-clock in JobQueue(tz=...) and are 06:00 on both sides of a daylight-saving change. The two mean different things and a person picking one has already chosen which.
Declaration — a default schedule an app ships, written next to @job with declare(name, kind, cadence, payload, label). SCHEDULES is to it what JOBS is to @job. On start, a declared name with no row gets one and a name that has one keeps everything but its label, so the declarations are where a schedule comes from and the table is what it is.
How a job has to look
def handler(config, store, *, context=NULL, limit=None) -> dict: ...
config and store are the app's, passed through untouched. Payload keys arrive as keyword arguments — declare the ones you take; an undeclared key is a TypeError recorded against the job, and **kwargs would swallow a typo. context is optional: a handler that takes neither progress nor cancellation says so by not declaring the parameter, and registry.py works that out once at registration rather than inspecting a signature per run. One payload key is reserved — a job whose payload names context fails, because the worker's own is what goes there.
limit is a convention, not a contract. Nothing in the pipeline reads it. It is an ordinary payload key that most of newsdesk's sweeps happen to accept, so that a person can ask for a short run of one, and a handler is free not to have it.
The return value should be a JSON-serialisable dict, and is stored as the job's result. A handler that returns something else — None, a list, a count — is not an error: the worker keeps the ending and stores an empty result, so the job is done with {}. Which means a result nobody can read is silent, and worth knowing when a handler's tally goes missing rather than its job failing.
An exception ends the job in failed. The traceback goes to the log, the exception is stored on the row as Type: message, and a job_failed notice goes to the screen saying it looks like a bug — unless the handler already announced a notice about that same failure, in which case the handler's sentence stands alone rather than being contradicted underneath. The except is deliberately broad: a handler is a source plugin, a model call, a subprocess, and anything it raises is that one job's failure and nothing else's. Letting it escape would take the worker down and strand every job behind it.
Minimum example
from pipeline import Jobs, JobQueue, job, RunContext, NULL
@job("greet", "Saying hello", solo=True, resource="cpu")
def run_greet(config, store, *, context: RunContext = NULL, name="world") -> dict:
context.step(0, 1, f"greeting {name}") # progress + cancellation check
return {"greeting": f"hello {name}"}
run_greet(cfg, store, name="wolf") # 1. straight call — no queue at all
queue = JobQueue(Jobs(cfg.database), cfg, store) # 2. queued
await queue.start()
queue.enqueue("greet", {"name": "wolf"})
@job returns the function unchanged, so every job stays importable and callable on its own. Registration is by import — importing newsdesk.jobs fills JOBS.
Workers
JobQueue.WORKERS = {"llm": 4, "http": 8, "cpu": 1} — how many workers of each kind there are, and so how many jobs of that resource run at once. An app passes its own; newsdesk has a workers section. A resource named in neither gets one. Counted from the table, not from this process, so a second worker on the same file shares the four model workers rather than helping itself to four more.
The count belongs to the worker, not the job kind: tagging and summarising both ask a model, so they share the same four.
Our queue is fanning out threads not processes. Therefore, a worker is a thread, not a program.
depends_on
A job may name jobs it waits for, and say what it wants of them.
after="ok" |
admit only if all of them succeeded; fail with them otherwise. The default, and what one job needing another's output means. |
after="done" |
admit once all of them are over, however they ended. |
Waiting costs nothing: claim passes the job over and it stays queued. No timer, no listener, nothing holding a place, nothing lost to a restart.
done exists for the job that reports on the others — success-only semantics would mean the report never exists when there is something to report. An id naming nothing is satisfied.
A cancel and a failure are the same thing here: both travel to the after="ok" jobs waiting on them, and neither holds back an after="done" one. Killing a job is dropping that batch, not stopping the run it belongs to, so the job that reports still runs and counts it among the ones that did not land.
What a waiting job is handed. context.waited_for is a tuple of pipeline.models.Job — the whole row for each job this one named, oldest id first, and empty for a job that waited for nothing. Each carries its state and its result, which is everything a report needs. Read at claim time off the jobs themselves, never counted up as they landed, so there is no tally to keep in step and nothing to lose to a restart half way through. An id that named a job which no longer exists is simply absent from the tuple, so len(waited_for) is what was found, not what was asked for.
solo
One job of this kind queued or running at a time. Two tabs pressing Fetch is one pull, and the second caller is handed the job already going rather than an error, because what they asked for is that the work be under way and it is. Enforced by a partial unique index over the unfinished states, so it holds across every tab, every client and every process at once — not by a flag in whichever browser happened to ask.
Off unless a job says otherwise. A sweep that forgets to say solo=True runs twice and wastes a minute. A batch kind that forgets to say solo=False has seven of its eight batches quietly handed the first one, and the work simply never happens. The default belongs on the side that fails loudly.
A job that names depends_on never dedups, whatever it declared, since the whole point of a batch is that there are several.
The job you are handed back may not be yours. Dedup matches on the kind alone, so a duplicate is answered with whatever live job holds that key — which can belong to another cascade entirely. Press Fetch while an overnight pull is part way through its own pull job and you are handed that job: the right answer to "is a pull under way", and the wrong one to "is my pull under way". Nothing rewrites its cascade_id, so a caller that means to follow the run it started has to notice the ids it got back are not the ones it minted.
Retrying
@job("extract", "Reading the page", resource="http",
max_attempts=3, retry_delay=60, retry_backoff=2.0)
def run_extract(config, store, *, context=NULL, url=""): ...
max_attempts counts the first attempt, so 1 is the default and means what every job here meant before the column existed: run it once, and an exception is the end of it. An attempt that fails with attempts left goes back to queued with a run_after on it, retry_delay × retry_backoff ⁿ seconds hence.
Waiting costs exactly what depends_on costs, which is nothing. The wait is a column, not a sleeping worker: claim passes the row over until its time comes round, so no worker, no thread and no lease is spent on an hour's backoff, and the poll that exists to notice another process's enqueue is what notices the hour is up. Sleeping in a handler would hold a worker for the duration, and is the thing this exists instead of.
And because it is a column it survives the process that wrote it. A retry scheduled for 09:40 is taken at 09:40 by whichever worker is alive then, including one started at 09:38 in a different process. There is nothing to hand over, because nothing was alive to hand.
Three endings, and Jobs.retry picks between them. cancelled, if a Kill landed while the attempt ran — a retry that put the job straight back would be a button that stopped nothing and said it had. Back to queued, if there are attempts left. failed, if there are not, and Job.exhausted says so from then on: the policy ran its course and the work still did not land, which is the only one of the three where trying again is a person's decision. A job that failed at max_attempts=1 is not exhausted — nothing was ever used up.
job.waiting |
queued, with its next attempt still to come |
job.exhausted |
failed on the last attempt it was allowed, rather than the only one |
job.attempt |
how many times it has been started, stamped by claim in the statement that takes the row |
jobs.waiting() |
every row in the first of those states, oldest first |
jobs.blocked() |
everything claim would pass over — waiting on a time, or on another job |
A handler can name the wait but not the bound. raise Retry("rate limited", after=120) puts the other end's own answer in place of the backoff, which is worth having because the alternative is a guess: a minute is either a minute of doing nothing when the service was ready, or another refusal charged against the bound. It buys no extra attempt — a job queued with one attempt that raises Retry fails exactly as it would on any other exception. The bound is ours to declare and the wait is the world's to name.
A retry says nothing to the screen. A notice is for what a person has to act on, and while attempts remain there is nothing to act on: the queue is already doing what they would have asked for. The log has every attempt either way; only the one that runs out announces.
Retrying is off by default, for the reason solo is off by default. A fetch that should have been retried and was not costs one item until the next sweep finds it again. Work retried three times that never should have been is three times the writes, or three times the bill.
queue.enqueue(kind, delay=600) schedules one job for later with no retry involved at all. queue.enqueue(kind, max_attempts=n, retry_delay=s, retry_backoff=b) overrides what the kind declared for one piece of work, and dedup_key= overrides the kind's own key — which is how one item's acquisition is made unrepeatable while the kind stays freely repeatable, so two presses of a reader's retry button collide in the unique index rather than in one browser tab.
A batch can carry its own bound. A task's finder may return Batch(payload, Bound(max_attempts=3, delay=300, backoff=2.0)) instead of a bare payload, and those numbers go onto that job's row. That is where a plugin's declared retry policy lands: newsdesk cuts its body batches per plugin profile precisely because the row holds one bound, so a batch mixing two outlets could only carry one of their declarations. A finder returning plain dicts — every task that existed before this — gets Bound(), which is one attempt and no wait. The direct path (task.run) ignores bounds entirely: a bound is a number the queue enforces by putting a row back with a time on it, and there is no row.
Typical way of dealing with batches
A task queues its batches as independent jobs, and one final job behind them. It depends on all of them with after="done", so it runs however they ended, and it is handed each one's state and result in context.waited_for. It is the only job of the task that calls back.
A task that finds nothing to do is the ordinary case, not the empty one: it queues no batches and its final job runs immediately, with an empty waited_for, and moves the cascade on. Which is why a switch turned off, a backlog already cleared and a stage this pull does not want are all the same thing here — a finder that returns no payloads — and none of them stops the chain.
The callback is a name in the final job's payload, not a function. Nothing in a table can hold a function, and a name is what makes the callback survive a restart and be callable from another process; the worker running the final job resolves module:function to imported code. newsdesk.jobs.cascade:step reads as documentation in the payload, and carries its own import, where a registry would have to be filled before the worker looked in it.
It is called with the task's tally and whatever the task fixed at the time it was made:
def step(config, store, tally: Tally, payload: dict, *, context: RunContext) -> dict | None: ...
payload is Task.carries, stored in the final job's row, so it survives a restart and a worker in another process. Whatever the callback returns is merged into the final job's own result; None is fine. A task with no callback stops there, which is what being asked for on its own means.
settle is a second name in the same payload, resolved the same way, and it belongs to the task rather than to what follows. The final job runs it before the callback, over the same tally. It is for the decision a batch is in no position to make: a batch is one of forty and cannot see what the other thirty-nine are doing, so anything that must hold across the whole run — a cap on what the run may add, a merge of what several batches each asked for — is worked out once, here. Newsdesk's tagger is the one that has one: a batch proposes new tags and adds none, and the settle merges the proposals, takes at most max_new_tags_per_run of them and goes back to tag the items that asked. A task started on its own has no callback and still settles, because settling is its own work and not a message to anybody.
Tally is how the batches ended, read off their rows at the moment the final job runs:
done failed cancelled |
how many batches ended each way, and jobs for the sum |
results |
what each batch returned, in the order they were queued |
killed |
whether the task itself was killed |
killed is the one that cannot be inferred from the counts, and is the reason it is there. A task killed while its batches ran leaves every one of them cancelled, and so does a person killing every batch by hand — identical numbers meaning opposite things: skip this step, or drop this work. Only the runs row knows, so the final job asks it. What to do about it is the cascade's decision and nothing here presumes it; newsdesk's pull skips the stage and carries on to the next.
Passing data
Three ways to pass data. Can be mixed and merged.
- Store. A job reads its input or persists its output directly into a storage.
- Parent. A job receives data as a parameter, put in its payload by whatever queued it — a batch's uids, a cascade's options. Fixed at queue time, and the only kind of data a job that runs first can hand to a job that runs later.
- Cross job via
context.waited_for. Theafter="done"job gets the jobs it waited on: theirstateand theirresult. Read, never pushed — nothing is counted while the work runs, so there is no tally to lose.
The usual arrangement is the third feeding the callback: the final job reads how its batches went and hands the tally to the cascade, which turns it into the next Task.
context.upstream() is the third one merged into a dict, later parents winning, which is what a chain of one parent wants to read like. It is {} under NULL, which is what keeps a chained handler callable by hand:
@job("extract")
def extract(config, store, *, context=NULL, html: str = ""):
html = html or context.upstream().get("html", "")
From the queue nobody passes html; from a terminal you pass it yourself. A handler reading waited_for[0].result directly would be the one kind in the pipeline you cannot simply call.
The rule is that whoever has the data writes the row. Never insert a row meaning an earlier job to fill its payload in later: a worker claims it the moment the parent is done, so it can be claimed between that write and its commit; dedup_key has to be known at queue time; and max_attempts re-runs it from its own row. Reading a parent has none of that, because the row was complete when written.
Bulk data belongs in the store. Tally.results is a query across the children's result TEXT columns and those columns never shrink — counts, ids and errors are free, a few hundred large dicts are not.
fan_out — N jobs and the one that reports on them
@job("body")
def body(config, store, *, context=NULL):
context.fan_out("body_batch", [{"uids": c} for c in chunks],
then="newsdesk.jobs.cascade:step")
One call, one transaction. The reporting job is FINAL, the pipeline's own kind, and an app never writes one. The children go into a run of their own inside the caller's cascade, which is what lets the callback fan out again — step N queues step N+1, and that is the whole of what a cascade is.
Because the thing fanning out is itself a job it has a context, so this works from inside a handler as well as from a request. Task is the same three statements with a finder in front, which is why a task and a cascade are one shape and one primitive covers both.
() under NULL: a handler called directly is being run by somebody standing there, and the useful answer is to do its own work and report that nothing was queued behind it.
Running it
queue = JobQueue(Jobs("app.db"), config, store) # then, in an app:
connect(app, Jobs("app.db"), config, store) # pipeline.web: routes, socket, element, lifespan
queue.drain(timeout=60) # in a script: run what is queued, stop
python -m pipeline.worker --db app.db --bootstrap myapp.jobs:bootstrap --bootstrap-arg config.yaml
The worker is the same JobQueue with no web app, on the same table, and may run beside the web app's own. --until-idle drains and exits. The bootstrap is the one isolate=True uses.
Ending work that will not end itself
Cancellation is cooperative because the work is synchronous — but two kinds of work never look up, and for those there is a real kill.
A command. context.spawn(argv, timeout=...) runs it, polls for Kill and for the deadline, and kills the process tree on either — taskkill /T on Windows, killpg elsewhere, which is why the child is started detached. proc.kill() alone would leave the ffmpeg a shell started still holding the core. Blocking a worker thread on this costs nothing: the thread is only sleeping. Output goes to temporary files, not PIPE, because a child that fills the pipe buffer while the loop polls is a deadlock. A non-zero exit is handed back rather than raised.
Python that cannot be interrupted. @job(..., isolate=True, timeout=600) runs the job in a child process, which is the only thing that can be killed mid-sklearn.fit. Nothing is pickled: a job row is already a complete description of the work, so the child is handed --db and --job and reads the same row. It writes progress to that table — which is how its progress reaches the browser with nothing forwarding it, the snapshot being a SELECT — and hands its result back through a file. The worker still writes the ending, so there is exactly one writer of the terminal state, as before.
isolate=True needs bootstrap="module:function" on the queue, returning (config, store): neither survives a process boundary, so both are rebuilt in the child, and the same import is what fills JOBS there. bootstrap_arg is one string passed to it — an app's config path, which the child cannot learn any other way.
timeout without isolate raises at registration. A deadline over a thread could only mark the row failed and walk away — the work would carry on holding the core, and two things would intend to write one row. The refusal names both ways out.
context.task — a job asking for what comes next
Newsdesk's body task fetches each eligible article and passes its HTML to desk.tools.article.extract_article. Its storage writer saves structured content, format, provenance and a derived plain-text projection atomically; existing enrichment jobs continue to read the generated body view. Empty or late fetch results preserve previously acquired content. The body job still uses the existing wall detector and candidate rechecks, and does not yet execute plugin-declared acquisition graphs — one declared step, the one whose tool is extract_article, is what it runs. Its declared retry policy is executed: a failure is classified with desk.tools.failures, the plugin decides whether that category is worth another attempt, and the batch raises so the queue puts it back under the bound its row carries. Batches are therefore cut per plugin profile. A second attempt asks the table again and so re-fetches only what is still missing, and never overwrites text that has since arrived. Structured bodies are retained without the former 60,000-character clipping; downstream model tasks remain responsible for their own input limits.
Hacker News discovery now runs as the hacker_news outlet through shared RSS fetching. Its item destination stays the publisher URL, so the subsequent body task extracts that article and records that URL as provenance. Its supplied discussion link is a separate typed input, and feed points/comment statistics stay in source evidence rather than the summary or preview. Its declared tagging policy runs: the destination article is what the tagger reads, and a link whose page would not extract is tagged from its title. Its summary policy stays off — the summariser selects only profiles that ask for an abstract, and nobody has decided that a front-page link earns one.
A handler that discovers work only by doing its own can name the task that should follow it:
context.task(cascade.task("body", cascade="pull"))
Recorded, not queued. The worker starts it after this job lands in done, on the same thread that ran the handler, and hands it this job's cascade_id so its batches land in the run this one belongs to. Three consequences worth knowing: a task never outlives the failure of the job that described it, a handler still cannot reach the queue, and a job's row lands a moment before the jobs that follow from it exist.
It returns whether anything took it — False under NULL, where there is no queue. A callback invoked from a notebook or a Claude Code session gets that answer and has to do the work itself.
Cancelled and RunCancelled
Two exceptions with similar names, and only one of them is a handler's business.
Cancelled is raised by context.check() inside a handler when this job has been asked to stop. The worker catches it and records cancelled rather than failed. A handler should let it travel; catching it is how you build the thing the Kill button cannot kill. A handler that never checks is not lost either — the worker asks once more when the handler returns, and a job asked to stop while it ran ends cancelled, keeping whatever it made and queueing nothing.
RunCancelled is raised by Jobs.enqueue for a job whose cascade_id or task_id names a killed run, except for an after='done' job under a killed task — the same carve-out cancel_run and stop_task make, so a kill landing while a task is still fanning out its batches cannot take the final job with it. A killed cascade refuses it too, which is what "nothing reports and nothing follows" means. It is not a handler's business and a handler will not see it: JobQueue.enqueue catches it, logs it, and returns None instead of a Job. So a caller that queues a job has to expect no job back — Task.start skips the id it did not get, and a depends_on list is that much shorter. Nothing is told that a run is over; a callback tries to queue the next task and the table says no, which is the same answer across processes, after a restart, and when the kill landed mid-run.
Ids
Every job has an id of its own, and carries a task_id and a cascade_id. Both are grouping ids: nothing depends on a task, but everything belonging to one can be found by it and killed together. Two of them rather than one because they are two groupings and both have a button.
Killing a run is one operation for a task and a cascade alike — cancel_run. Three things: flag the row, end its queued jobs, ask its running ones. And a fourth that is not an operation at all: enqueue refuses a job whose cascade_id or task_id names a killed run, which is what stops a callback from reviving a killed task or moving a killed cascade on.
Three buttons, three sizes of stop: killing a job drops one batch, killing a task skips one step, killing a cascade ends the run.
Stopping a run is not killing it. Jobs.stop_task ends a task's queued batches and nothing else: the row is not flagged, so Tally.killed stays false and nobody is told a person pressed anything; running batches keep their requests and their own endings; the final job still reports. It is what a handler reaches through RunContext.stop(why), for the failure that is the backend's answer to the whole run rather than to one batch — a spend limit, an expired login, no model on the machine. The reason is written to error on each cancelled row, so a batch that never ran says on screen why it didn't. desk/modeltask.py is the one caller, and notices.stops_the_run is what decides.
The one asymmetry between killing a cascade and killing a task is what the last two of those mean. Kill a task and its final job is spared: it still runs, and what it reports says the task was killed rather than only how many of its batches ended cancelled — see Tally.killed. Kill the cascade and the final job dies with it, so nothing reports and nothing follows — the one stop no cascade gets a say in, because there is nothing left of it to ask.
cascade_id = jobs.new_run("cascade", "pull") # before a single job exists
jobs.of_cascade(n) · jobs.of_task(n) · jobs.run(n) · jobs.cancel_run(n)
jobs.cascades()
POST /api/cascades/{id}/cancel · /api/tasks/{id}/cancel · /api/jobs/{id}/cancel
A job queued with neither begins both: something asked for it on its own, so it is a cascade of one task of one job. A task started with no cascade does the same, for the same reason.
The cascade that comes of it says so in its name — pull on its own rather than pull — because a chain named pull is a different run, and live_run("cascade", "pull") is how a caller asks whether that chain is already going. Answering it with a lone job would be a Fetch button that starts nothing and greys itself over a run that is one job long. The task keeps its plain name (pull), since the task really is that task however it was started, and the strip draws its box from that.
Schedules
The tick is in the drain loop, at the top of _turn, before the claim — not in a thread of its own and not on the idle path with the reclaim and the purge. Not a second thread, because then two things would race to enqueue one row. Not on the idle path, because reclaim and purge are repairs that can wait for a quiet moment and a schedule is work: on a queue that is never idle, 06:00 would never come round. The check itself is a clock comparison, so paying it every pass costs nothing. It also runs once in start(), after reconciliation, so a default declared today fires on the start that introduced it.
Overdue fires once, not once per slot missed. A firing sets next_run from now, never by stepping through the times it was down for. There is one next_run column and nowhere for a backlog to accumulate even in principle, so an app off for three days runs its hourly job once when it comes back.
No pile-up. Before firing, two questions: is the job from the last firing still queued or running (last_job_id), and is this kind solo with one already under way? Either way next_run advances and nothing is queued. A job slower than its own cadence must not grow a queue of itself, and a solo kind somebody has already pressed must not be recorded as this schedule's firing.
A row the code no longer supports keeps its place. An unregistered kind, or a cadence nothing can parse (only reachable by editing the file by hand), is logged once per process and left exactly as it is — next_run included — so registering the handler again runs it rather than starting another cadence. <desk-schedules> marks the row unknown kind; deleting it is deliberate.
Reconciliation overwrites only the label and never deletes. Jobs.add_schedule(..., only_if_new=True) is one statement, so two workers starting at once insert a declared default once between them. Jobs.relabel_schedule then refreshes the label of a row that still has the declared kind: no screen edits a label, so the row's copy is only ever an older declaration's. A declaration removed from the code leaves its row behind, because a deploy that silently forgot a schedule somebody relied on is worse than an orphan that says so.
One scheduler per file. JobQueue(schedules=False) and pipeline.worker --no-schedules are what every worker after the first passes. The workers themselves race correctly — that is claim — but the pile-up guard reads last_job_id, which neither of two simultaneous firings has written when the other looks.
declare(name, kind, cadence, payload=None, label="") # next to @job
Cadence.parse(text) · str(cadence) · cadence.next_after(moment, zone)
jobs.schedules() · jobs.schedule(id) · jobs.add_schedule(...)
jobs.update_schedule(id, **fields) · jobs.delete_schedule(id)
jobs.due_schedules(now) · jobs.mark_fired(id, job_id, next_run)
queue.run_schedule(id) · queue.notify_schedules()
GET/POST /api/schedules · PATCH/DELETE /api/schedules/{id}
POST /api/schedules/{id}/run
The three tables
jobs is the queue, and holds rather more than the states: the payload it was queued with and the result it ended with, both JSON; done, total and message, which are the progress bar and are written by context.progress; error, the exception string of a failure; dedup_key, depends_on and after; resource; cascade_id and task_id; the retry policy and where it has got to, as attempt, max_attempts, retry_delay and retry_backoff; and five times — created_at, started_at, seen_at (the lease), run_after (the next attempt) and finished_at. cancel_requested is a flag, because a kill is a request in the table rather than a message to whichever process happens to be running the job.
The policy is on the row rather than read off the job kind when a failure happens, for the same reason the payload is: the attempt that has to decide may be in another process, after a restart, running a version of the code where the kind's numbers are different or the kind is gone.
schedules is the standing instructions: id, name (unique, and the handle a declaration is matched on), kind, label, payload, cadence, enabled, next_run, last_run, last_job_id, created_at. next_run and last_run are UTC ISO stamps like every other time here, and next_run is compared as text in due_schedules for the reason run_after is in claim: '' sorts before every stamp, so "never" and "not due" are one comparison with no coalesce in it. A cadence worded in a local timezone is turned into one of these before it is stored, so pipeline/schedule.py is the only file that reads a zone at all.
runs is where a cascade's id and a task's come from, and it is deliberately almost empty: id, kind (cascade or task), name, label, created_at, cancelled_at.
What it does not have is the point of it. No state, no progress, no counts, no finished_at: the jobs carrying the id are the record of what happened, and a second copy of that could only go out of step with them. cascades() tallies a run by reading its jobs. So the table exists for exactly two facts its jobs cannot supply — that a number was handed out before there was a job to take it from, and that somebody killed this run. name carries a third job: it is how live_run answers "is one already going", which is why a cascade minted only because a lone job needed an id is named apart from a chain somebody asked for (see on_its_own).
ModelTask — next door, not here
ModelTask left this package: it imports desk.config and desk.tools.agent, so it was never the queue's. It is desk/modeltask.py now, and this section stays because it is still the thing most jobs in newsdesk are.
desk/modeltask.py is one layer above desk/tools/agent.py. agent answers how do I reach a model; this answers what do I send it and what do I do with what comes back, once, for every task that needs one. Tagging and summarising are the two, and what differs between them is all content: which rows are waiting, what goes in the batch, what the model is told, what shape it must answer in, what to do with the answer, what the run has to settle once every batch is in, and how a row that never gets one is given up on. That set of callables is a ModelTask; the module is everything that is not one.
modeltask.of(TASK) turns it into an ordinary Task, so it is queued, watched and killed like any other stage. modeltask.one is what a single batch job does: check for a kill, build the payload from the uids it was given, ask, apply. Two things about it are worth reading before changing:
- The prompt is built when the batch runs, not when it was handed out. Tagging's carries the vocabulary as it stands at that moment, so a tag somebody adds by hand mid-run reaches the batches still to come. It does not carry what a sibling batch proposed, because a batch proposes and adds nothing: the run's own additions are worked out once, in
settle. pending,waitingandbuild_batchare given the config as well as the store. Which rows are waiting is no longer a question a table can answer on its own: in newsdesk both backlogs are whatever the installed plugins declare about their own items, and the plugins are found through the config. Both tasks now use it; the argument is still there for one that does not.- A kill during the call does not throw the answer away.
check()is called before the model is asked, so a Kill pressed while the job sat queued makes no paid call at all. Once the request is in the air the money is spent, so the answer is applied and the worker records the cancellation at return. The asymmetry is deliberate and looks like an oversight otherwise.
A row is given up on eventually. "The next run finds the work again" is unbounded, and for a row no model will ever answer usefully — a segment whose only honest abstract is one sentence, a uid dropped from every reply — unbounded means paying for the same call on every pull for as long as the row exists. So a row handed over and left unanswered is charged a turn through ModelTask.missed, and one charged three of them stops being offered until a person asks for it deliberately. A batch that failed charges nothing: it never reached a model, so it proved nothing about its rows. Every ModelTask must supply missed, and there is no default, because the default is the bug.
A refusal is the one failure that is about a row. The platform runs a classifier over the request and rejects the whole thing at once, naming nothing — and a news archive meets it regularly, because reporting on a worm or a supply-chain compromise is written in the vocabulary it watches for. one answers it by asking again in halves (_narrow) until the batch is the row, which is then charged and leaves the backlog after three. Halves rather than one at a time: two flagged rows in a batch is ordinary, since they arrive from the same day's news, and stepping through twenty singly would pay eighteen calls for what four find. The notice is agent.refused and is deliberately not standing — the account is fine and the next batch is about different rows.
What was given up on is listed, and said once. A refused row is charged with why="refused", kept in raw.why.<task>, so the list can tell a refusal from a row the model left out. The strip interrupts only for given-up rows without raw.seen.<task>; its × or opening Jobs & history sets that mark on the server, so a reload does not say them again and a row given up on later still does. Try again there reopens a row with one attempt (reopen_given_up(..., attempts=1)) and queues its _batch job at once, so a row refused again is back on the list after one call rather than after three pulls.
notices
Two halves since the extraction. pipeline/notices.py is what the queue itself needs — the notice a failed job gets, the check that stops it being said twice, the rendering a terminal wants. desk/notices.py is newsdesk's vocabulary: Rule, RULES, classify, stops_the_run, source_failed. An app writes its own, because a useful sentence names things only the app knows.
pipeline/notices.py turns a failure into one line worth putting on screen. A handler calls context.notice(...), and the queue writes it to the notices table and pushes it to whoever is watching — the counterpart to log.exception, not a replacement for it: log the traceback for afterwards, raise a notice for now.
A notice is stored, one row per (code, key). The first design stored nothing, on the argument that a table would duplicate the log. That assumed somebody was watching when the notice was raised; a scheduled pull and a reader on a phone mean nobody usually is. Jobs.record_notice upserts the row and returns it as a NoticeEntry with count, first_at and last_at, and that row is what goes over the socket. count == 1 is news and anything above is a repeat, so a client interrupts for the first and only counts the rest — a refused article is announced on the pull that met it, not on the two retries after. Every notice still reaches on_notice; what interrupts a person is the interface's decision.
A row goes when it is cleared (DELETE /api/notices/{id}, or /api/notices for all), after which the same failure is news again, or when JobQueue.NOTICE_QUIET (14 days) has passed since it was last said. GET /api/notices lists the log, newest first. A write that fails is logged and the notice is pushed anyway: it never costs the job that raised it. The table holds no traceback and no per-occurrence history; those stay in the log.
A notice carries three things beyond its text: kind, which says whether this is the world misbehaving or a bug of ours — the fork in what you do next, and exactly the guess you should not have to make from an error string; fix, the command when there is one; and key, which with code names the row in the log — so a feed that has been failing all week is one line with a count rather than one per pull, and two refused articles are two. classify is the interface, and RULES is the catalogue it reads, which doubles as the list of what this will ever interrupt you about. Everything in the catalogue is marked expected; the fallback is marked bug, which is the whole point of enumerating them — a failure nobody wrote a rule for is one to go and read about.
For a job author, two rules. Use a notice for what a person has to act on, not for what the program can get on with. And classify your own failure if you can: the worker announces "looks like a bug" for every job that ends failed, and already_said is what stops it contradicting a handler that had already explained itself.
A rule also says whether the failure it names is standing — caused by the account or the machine rather than by this batch, so every sibling of the run would be told the same thing. stops_the_run(notice) is that answer, and a task for the model cancels what is still queued rather than paying for twenty-two more refusals. A limit, an expired login and an unreachable model are standing; a timeout, an oversized answer and a reply that was not JSON are not.
Timing
Seven numbers, all on JobQueue, plus one on Jobs. None of them is tunable and none is meant to be; they are here so that "why did that take a minute" has an answer.
IDLE_POLL |
1 s | How long to sleep before looking again when the table was empty. Only reached when nothing woke us — an enqueue in this process is instant, so this is what notices one made by another, and what notices a scheduled attempt whose time has come. |
PUSH_EVERY |
0.25 s | Fastest a running job's progress may reach the browser. The write is not throttled; the push is. |
CANCEL_CACHE |
0.25 s | How long check() caches its answer. It reads the table so Kill works across processes, and a handler calling it in a loop would otherwise turn one button press into a query storm. |
HEARTBEAT_EVERY |
5 s | How often a worker restamps seen_at on the jobs it holds. |
STALE_AFTER |
30 s | How long a running row may go unsaid-about before another worker reads it as abandoned. |
STOP_GRACE |
5 s | How long a shutdown waits for handlers already in a thread. |
SCHEDULE_TICK |
30 s | How often the loop looks for a schedule whose time has come, and the resolution the cadences promise — which is why pipeline/schedule.py has no seconds unit. |
Jobs.LINGER |
15 s | How long a finished cascade stays in cascades(), so a run you watched is still on screen the moment after it lands. |
HEARTBEAT_EVERY and STALE_AFTER are far apart on purpose: the gap is how many beats may be missed to a stalled loop or a slow disk before live work is re-queued underneath a worker that is still there, which is the one mistake this must not make. And because reclamation runs at most once per STALE_AFTER on the idle path, a crashed worker's job comes back after up to about a minute rather than after thirty seconds.
Limits
- Cancellation in a thread is cooperative. No
context.check()in the loop, no Kill. A running handler in a thread is never interrupted, not even onstop(), because Python cannot kill a thread from outside. Work that might need killing runs in a process:context.spawnfor a command,isolate=Truefor Python. See "Ending work that will not end itself". - A shutdown waits
STOP_GRACEfor its handlers, then walks away. A thread cannot be cut off, only waited for or abandoned, and both are wrong as an absolute: waiting is a reload held up by a model call, abandoning is a handler still writing to a file the process considers finished with. Most handlers use the moment and end properly; the rest are left running, and land their own ending in the table whenever they reach it. - A shutdown hands its leases back, a crash cannot.
stop()expires the lease on whatever was still going when the grace ran out, because a process on its way out knows outright what silence can only imply. So a restart re-queues that work at once instead of a window later, and only a real crash pays the wait. Expired rather than re-queued, so a handler thread outliving the shutdown still writes its own ending rather than being overwritten by a second run of itself. - A job's ending is written by the thread that ran it. The handler call and the terminal
done/failed/cancelledare one function on one thread; the loop's part is claiming the job and pushing the new state out afterwards. Written back on the loop, the ending would be the first thing astop()costs: cancelling an abandoned job's coroutine cancels the write, while the handler itself carries on to the end regardless — so a reload landing mid-run would leave finished work looking unstarted, and the next start would run it a second time. For a batch that asks a model that is a second bill for something already paid for. The task a job named withcontext.taskis queued by that same thread, for the same reason and still only once the job has landeddone. Which is the one thing to know when reading the table from outside: a job's row lands a moment before the jobs that follow from it exist, so an emptyrunningandqueuedis not by itself the end of a run. - A kill outlives the crash that interrupted it. A job asked to stop, whose worker then died, is reclaimed to
cancelledrather than back toqueued. The request is in the table exactly so that it survives the process that was meant to act on it. - A dead worker is noticed late, on purpose. A
runningrow carriesseen_at, stamped byclaimand renewed everyHEARTBEAT_EVERY. SQLite reads and writes run off the event loop so they do not delay its heartbeat. A lease expires afterSTALE_AFTER; reclamation runs at startup and at most once perSTALE_AFTERon the idle path, so recovery can take about a minute. A worker excludes jobs it still holds from reclamation, renews their expired leases and warns that its loop stalled. Another process can still reclaim an expired lease. - Every handler must be safe to run twice. Not advice — the rule the rest of this depends on. A reclaimed lease, a shutdown that abandoned a thread, a second process on the same file, an attempt that failed half way through: each of them can put a handler on the same rows again, and the queue does not promise otherwise. So write for it. Read what is still waiting rather than trusting the uids you were handed, upsert rather than insert, and write each unit as it lands so a re-run has less to redo.
max_attemptsbounds how often that happens on a failure; it is not something to lean on, because the paths that re-run a handler without asking it are the ones that were always there. - Handlers must be sync. An async handler is not awaited.
- A crashed worker holds its slots until it is reclaimed.
claimcountsrunningrows, so the survivor cannot use a crashed worker's slots until the lease expires and its next reclamation pass runs, about a minute in the worst case with a responsive loop and database. A stalled worker never re-queues its own held jobs. Socket sends run concurrently with a one-second timeout per subscriber, so an unresponsive reader cannot delay drain wake-ups indefinitely. - Handlers run in a pool of the queue's own, sized to the worker counts, because the interpreter's shared pool is sized from the cpu count instead. On a machine smaller than
workersthe shared one would run fewer jobs than the table admits and, worse, queue the loop's own claim and heartbeat behind the handlers, which is a worker starving itself of the thread it needs to hold its leases. - Progress is a write per call.
context.stepis oneUPDATEin its own transaction, on its own connection. The push to the browser is throttled; the write is not, because the row is the record and a second process reads it. Report per batch rather than per row in anything with thousands of items. - One bad turn does not stop the loop. A locked database is logged and retried on the next pass rather than ending the drain task, because a worker that died silently looks exactly like a healthy one with nothing to do.
depends_onnames jobs, not tasks. A job waits on ids it was given; a task id is for finding and cancelling, not for waiting.- No priorities. Jobs are claimed oldest first within a resource; there is no way to jump the queue but to be queued earlier.
- Five cadences, not cron.
every 15m,every 6h,every 2d,daily at 06:00,weekly mon 07:00, and nothing else — no seconds, no day-of-month, no "every weekday", no ranges. Anything the five do not cover is a job that decides for itself whether today is its day, or cron runningpipeline.worker --until-idleas before. - A schedule is not an alarm. It is checked every
SCHEDULE_TICK, so "06:00" means within half a minute of it, and a machine asleep at 06:00 runs the job when it wakes rather than at 06:00. What it will not do is run it once per slot missed. - History is purged. Finished jobs older than
JobQueue(retain=...), 30 days by default, are deleted hourly while idle. Kept regardless: unfinished jobs, the newest ending of each kind in each state (whatlast(kind)reads), and runs still pointed at.retain=0keeps everything. - A result is at most 64 KB of JSON, and a bigger one fails the job. Past
MAX_RESULTthe job endsfailedwithResultTooLarge, the size and the keys it returned, a notice goes out, and nothing of the result is written. It is not retried, since the same handler returns the same result. Every snapshot and tally reads results back, so data belongs in the app's store; raiseMAX_RESULTif a project really needs bigger results. - A task for the model gives up on a row eventually. See
ModelTaskabove: three unanswered turns and a row stops being offered until a person asks for it deliberately (reset_taggingwith no slug,reset_summaries).ModelTask.missedis the seam andstatus.pysays how many. - Migrations are
_reconcile. Columns are added, and dropped, by reading the catalogue on open, becauseCREATE TABLE IF NOT EXISTSleaves an existing table alone andPRAGMA user_versionon that file belongs to the app. - SQLite means one machine. Not a network queue.