One pull at a time: the run is claimed in the database and its marker rides in each batch
A second pull (schedule, Run button, `app job pull` in another process) could start while one was fetching and overwrite the stored run marker, so the first run's boards failed their stamp check or a settle closed on the other run's id lists. `claim_run` sets `pull:run` in one conditional statement, only while it equals `pull:settled` or is older than an hour; a refused `pull` job queues no stage and says so in its result and a notice, and the CLI exits 1. Batches carry the marker in their payload, the plugins stamp with it, and the settle reads it off its own batches' payloads before releasing the run. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
c7da12b93d
commit
b43fc54565
11 files changed
+328
-70
No files matched your search
+33
-6
@@ -34,7 +34,7 @@ from typing import Any
|
||||
|
||||
from app.jobs import embed, extract, pull, score, summarise # noqa: F401 -- registers the jobs
|
||||
from desk.tools import agent
|
||||
from pipeline import NULL, TASKS, RunContext, Tally, Task, declare, job
|
||||
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
|
||||
|
||||
STEP = "app.jobs.cascade:step"
|
||||
|
||||
@@ -129,17 +129,44 @@ def chain(name: str) -> Task:
|
||||
return task(CASCADES[name][0], cascade=name, at=0)
|
||||
|
||||
|
||||
def pull_chain(run: str) -> Task:
|
||||
"""The pull cascade for a run `pull.claim_run` has claimed: its finder
|
||||
reads under that marker rather than claiming one of its own."""
|
||||
return replace(chain("pull"), finder=pull.pull_task(run).finder)
|
||||
|
||||
|
||||
def already_running(since: str) -> Notice:
|
||||
"""The notice for a pull refused because another is in flight."""
|
||||
return Notice(
|
||||
code="pull.running", level="info", kind="expected", key="pull",
|
||||
text=f"Not started: {pull.ALREADY_RUNNING}"
|
||||
+ (f" (since {since[:19].replace('T', ' ')} UTC)." if since else "."),
|
||||
where="app.jobs.pull")
|
||||
|
||||
|
||||
@job("pull", "Pulling every board", resource="http")
|
||||
def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Ask for the pull cascade once this job is done. Its own work is nil.
|
||||
"""Claim a pull run and ask for the pull cascade once this job is done.
|
||||
Its own work is nil.
|
||||
|
||||
A pull whose claim is refused, because another run is still unsettled,
|
||||
queues nothing, embed to summarise included, and says so in its result
|
||||
and a notice: a second cascade would fetch every board again and could
|
||||
send the same roles to the model twice.
|
||||
|
||||
Under `NULL` there is no queue to take the chain, so this raises rather
|
||||
than report a pull that never happened; `app job pull` starts the chain
|
||||
on a queue of its own instead of calling this.
|
||||
than report a pull that never happened; `app job pull` claims the run and
|
||||
starts the chain on a queue of its own instead of calling this.
|
||||
"""
|
||||
if not context.task(chain("pull")):
|
||||
run = pull.claim_run(store)
|
||||
if not run:
|
||||
since = pull.running(store)
|
||||
context.notice(already_running(since))
|
||||
return {"skipped": pull.ALREADY_RUNNING, "running": since}
|
||||
if not context.task(pull_chain(run)):
|
||||
pull.release_run(store, run)
|
||||
raise RuntimeError("no queue to start the pull cascade on")
|
||||
return {"started": " > ".join(CASCADES["pull"])}
|
||||
return {"started": " > ".join(CASCADES["pull"]), "run": run}
|
||||
|
||||
|
||||
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
|
||||
|
||||
Reference in new issue
Block a user