Pull pipeline/ from boilerplate a1c6f0d: a dedup key is looked up and inserted in one transaction
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
471ace703c
commit
2e4624cbc4
3 files changed
+195
-24
No files matched your search
@@ -106,6 +106,8 @@ A **cancel** and a **failure** are the same thing here: both travel to the `afte
|
||||
|
||||
**One job of this kind queued or running at a time.** Two tabs pressing Fetch is one pull, and the second caller is handed the job already going rather than an error, because what they asked for is that the work be under way and it is. Enforced by a partial unique index over the unfinished states, so it holds across every tab, every client and every process at once — not by a flag in whichever browser happened to ask.
|
||||
|
||||
**A live job that finishes during an enqueue is not an error.** `Jobs.enqueue` looks the key up and inserts in one `BEGIN IMMEDIATE` transaction, so a worker finishing the live job lands either before it, and a new job is queued, or after it, and the caller is handed the live one. The earlier version inserted first and looked the job up only after the index refused, and a finish between those two statements surfaced as `IntegrityError: UNIQUE constraint failed: jobs.dedup_key`. The write lock is waited for under the connection's 30-second busy timeout, as every other write on the file is.
|
||||
|
||||
**Off unless a job says otherwise.** A sweep that forgets to say `solo=True` runs twice and wastes a minute. A batch kind that forgets to say `solo=False` has seven of its eight batches quietly handed the first one, and the work simply never happens. The default belongs on the side that fails loudly.
|
||||
|
||||
A job that names `depends_on` never dedups, whatever it declared, since the whole point of a batch is that there are several.
|
||||
|
||||
+39
-24
@@ -447,15 +447,24 @@ class Jobs:
|
||||
Returning the existing job rather than raising is deliberate: the
|
||||
caller asked for that work to be underway, and it is.
|
||||
|
||||
The lookup and the insert share one `BEGIN IMMEDIATE` transaction, so
|
||||
a live job that finishes during an enqueue finishes either before it,
|
||||
and a new job is queued, or after it, and the caller is handed the
|
||||
live one. An `IntegrityError` on the key cannot reach the caller. It
|
||||
used to, when the insert failed on the index and the job had finished
|
||||
by the time a second statement looked for it. Waiting for the write
|
||||
lock goes through the connection's busy timeout like any other write.
|
||||
|
||||
`depends_on` holds this job back until those jobs are over — see the
|
||||
column comments in `SCHEMA` for what `after` chooses between.
|
||||
|
||||
A job under a killed run is refused with `RunCancelled`. That is where
|
||||
a kill reaches everything that had not been queued when the button was
|
||||
pressed: the next task of a killed cascade, the batches a callback was
|
||||
about to fan out. Checked in the job's own transaction, so a kill
|
||||
landing between the check and the insert loses the race by a statement
|
||||
rather than by a process.
|
||||
about to fan out. Checked in the job's own transaction, under its write
|
||||
lock, so a kill cannot land between the check and the insert: it lands
|
||||
before, and the job is refused, or after, and `cancel_run` finds the
|
||||
job queued and ends it.
|
||||
|
||||
The refusal spares an `after='done'` job under a killed *task*, which
|
||||
is the same carve-out `cancel_run` and `stop_task` make and for the
|
||||
@@ -481,8 +490,14 @@ class Jobs:
|
||||
if max_attempts < 1:
|
||||
raise ValueError(f"max_attempts counts the first attempt, so it is "
|
||||
f"at least 1, not {max_attempts!r}")
|
||||
try:
|
||||
with closing(self._connect()) as conn, conn:
|
||||
# Taken before the first read, so no other writer, in this process
|
||||
# or another, can finish, cancel or queue a job between the lookup
|
||||
# below and the insert. Waits out a busy file like any write, under
|
||||
# the connection's busy timeout: IMMEDIATE takes the write lock at
|
||||
# BEGIN, where SQLite calls the busy handler, rather than upgrading
|
||||
# a read snapshot later, where it fails at once.
|
||||
conn.execute("BEGIN IMMEDIATE")
|
||||
killed = conn.execute(
|
||||
"""SELECT id FROM runs WHERE cancelled_at <> ''
|
||||
AND (id = ?1 OR (id = ?2 AND ?3 <> 'done'))""",
|
||||
@@ -491,11 +506,23 @@ class Jobs:
|
||||
if killed is not None:
|
||||
raise RunCancelled(
|
||||
f"run {killed['id']} was killed; {kind!r} is not queued")
|
||||
# The same predicate as `idx_jobs_dedup`, read under the write
|
||||
# lock: what it finds is still live when this commits, and when it
|
||||
# finds nothing the insert cannot collide.
|
||||
if dedup_key:
|
||||
existing = conn.execute(
|
||||
"""SELECT * FROM jobs
|
||||
WHERE dedup_key=? AND state IN ('queued','running')
|
||||
ORDER BY id DESC LIMIT 1""",
|
||||
(dedup_key,),
|
||||
).fetchone()
|
||||
if existing is not None:
|
||||
return Job.from_row(existing)
|
||||
# A job queued with neither begins both: something asked for
|
||||
# this on its own, so it is a cascade of one task of one job.
|
||||
# Minted in the job's own transaction, so a duplicate that
|
||||
# rolls back takes its ids with it — a `runs` row with no job
|
||||
# under it would be an identity for nothing.
|
||||
# Minted in the job's own transaction, so a refusal that rolls
|
||||
# back takes its ids with it — a `runs` row with no job under it
|
||||
# would be an identity for nothing.
|
||||
#
|
||||
# The cascade says it stands alone and the task does not: the
|
||||
# task is the `pull` task however it was started, but a chain
|
||||
@@ -504,32 +531,20 @@ class Jobs:
|
||||
cascade_id = int(cascade_id) or self._mint(
|
||||
conn, "cascade", on_its_own(kind), label)
|
||||
task_id = int(task_id) or self._mint(conn, "task", kind, label)
|
||||
job_id = conn.execute(
|
||||
row = conn.execute(
|
||||
"""INSERT INTO jobs (kind, label, payload, state, dedup_key,
|
||||
depends_on, after, resource, cascade_id, task_id,
|
||||
max_attempts, retry_delay, retry_backoff,
|
||||
run_after, created_at)
|
||||
VALUES (?,?,?,'queued',?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
VALUES (?,?,?,'queued',?,?,?,?,?,?,?,?,?,?,?)
|
||||
RETURNING *""",
|
||||
(kind, label, json.dumps(payload or {}), dedup_key,
|
||||
json.dumps([int(i) for i in depends_on]), after,
|
||||
resource or "cpu", int(cascade_id), int(task_id),
|
||||
int(max_attempts), float(retry_delay), float(retry_backoff),
|
||||
_in(delay), _now()),
|
||||
).lastrowid
|
||||
except sqlite3.IntegrityError:
|
||||
existing = self._row(
|
||||
"""SELECT * FROM jobs
|
||||
WHERE dedup_key=? AND state IN ('queued','running')
|
||||
ORDER BY id DESC LIMIT 1""",
|
||||
(dedup_key,),
|
||||
)
|
||||
if existing is None:
|
||||
raise
|
||||
return Job.from_row(existing)
|
||||
job = self.get(job_id) if job_id is not None else None
|
||||
if job is None:
|
||||
raise RuntimeError(f"job {job_id} disappeared between write and read")
|
||||
return job
|
||||
).fetchone()
|
||||
return Job.from_row(row)
|
||||
|
||||
def claim(self, workers: Mapping[str, int] | None = None) -> Job | None:
|
||||
"""Take the oldest ready job and mark it running, atomically.
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""A `dedup_key` enqueue racing the live job it collides with.
|
||||
|
||||
`Jobs.enqueue` either hands back the live job holding the key or queues a new
|
||||
one. It must never raise `IntegrityError` because that job finished while the
|
||||
enqueue was between looking for it and inserting.
|
||||
|
||||
The first test makes the finish land at that exact moment, by hooking the
|
||||
enqueue's own connection: when it starts the statement that looks the key up,
|
||||
a second thread tries to finish the live job on a connection of its own. The
|
||||
second test is the same race left to the scheduler, many times over.
|
||||
|
||||
python3 -m pytest pipeline/tests/test_dedup.py -q
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
from contextlib import closing
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from pipeline import Jobs
|
||||
from pipeline.models import Job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def jobs():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
yield Jobs(Path(tmp) / "t.db")
|
||||
|
||||
|
||||
def made(queued: Job | None) -> Job:
|
||||
assert queued is not None
|
||||
return queued
|
||||
|
||||
|
||||
def live(jobs: Jobs, key: str) -> list[int]:
|
||||
with closing(sqlite3.connect(jobs.path)) as conn:
|
||||
return [r[0] for r in conn.execute(
|
||||
"SELECT id FROM jobs WHERE dedup_key=? AND state IN ('queued','running')",
|
||||
(key,))]
|
||||
|
||||
|
||||
def finish_when_looked_up(jobs: Jobs, monkeypatch: pytest.MonkeyPatch,
|
||||
job_id: int) -> list[tuple[threading.Thread, bool]]:
|
||||
"""Finish `job_id` when an enqueue starts the SELECT that looks up its key.
|
||||
|
||||
A trace callback runs as a statement starts, before it takes its read
|
||||
snapshot, so a write committed from inside it is visible to that SELECT.
|
||||
The finish runs on its own thread and connection, and the callback waits
|
||||
for it up to half a second: without a write lock held by the enqueue it
|
||||
lands at once, and with one it waits until the enqueue commits. Either way
|
||||
the enqueue then goes on, so the outcome does not depend on timing.
|
||||
|
||||
Returns the finishing thread, and whether it was still blocked when the
|
||||
callback stopped waiting for it.
|
||||
"""
|
||||
fired: list[tuple[threading.Thread, bool]] = []
|
||||
enqueuer = threading.get_ident()
|
||||
connect = jobs._connect
|
||||
|
||||
def trace(sql: str) -> None:
|
||||
if fired or threading.get_ident() != enqueuer:
|
||||
return
|
||||
text = " ".join(sql.split()).upper()
|
||||
if not (text.startswith("SELECT") and "DEDUP_KEY" in text):
|
||||
return
|
||||
worker = threading.Thread(target=jobs.finish, args=(job_id, "done"))
|
||||
worker.start()
|
||||
worker.join(timeout=0.5)
|
||||
fired.append((worker, worker.is_alive()))
|
||||
|
||||
def hooked() -> sqlite3.Connection:
|
||||
conn = connect()
|
||||
conn.set_trace_callback(trace)
|
||||
return conn
|
||||
|
||||
monkeypatch.setattr(jobs, "_connect", hooked)
|
||||
return fired
|
||||
|
||||
|
||||
def test_live_job_finishing_mid_enqueue_is_not_an_error(jobs, monkeypatch):
|
||||
first = made(jobs.enqueue("pull", dedup_key="pull"))
|
||||
assert made(jobs.claim()).id == first.id
|
||||
fired = finish_when_looked_up(jobs, monkeypatch, first.id)
|
||||
|
||||
again = jobs.enqueue("pull", dedup_key="pull")
|
||||
|
||||
assert fired, "the hook never saw the key being looked up"
|
||||
finisher, blocked = fired[0]
|
||||
finisher.join(timeout=35)
|
||||
assert not finisher.is_alive()
|
||||
# The finish could not come between the lookup and the insert: it waited
|
||||
# for the enqueue's write lock, so the enqueue saw the job live and was
|
||||
# handed it, and the finish landed after.
|
||||
assert blocked
|
||||
assert again is not None and again.id == first.id
|
||||
assert made(jobs.get(first.id)).state == "done"
|
||||
assert live(jobs, "pull") == []
|
||||
# And with the key free, the next enqueue queues a job of its own.
|
||||
fresh = made(jobs.enqueue("pull", dedup_key="pull"))
|
||||
assert fresh.id != first.id and fresh.state == "queued"
|
||||
|
||||
|
||||
def test_enqueues_racing_a_worker_never_raise_or_double_up(jobs):
|
||||
"""Several threads enqueue one key while another claims and finishes it."""
|
||||
key, rounds, threads = "solo", 60, 6
|
||||
errors: list[BaseException] = []
|
||||
doubled: list[list[int]] = []
|
||||
stop = threading.Event()
|
||||
|
||||
def enqueue() -> None:
|
||||
try:
|
||||
for _ in range(rounds):
|
||||
job = made(jobs.enqueue("solo", dedup_key=key))
|
||||
assert job.dedup_key == key
|
||||
except BaseException as exc: # noqa: BLE001 - reported below
|
||||
errors.append(exc)
|
||||
|
||||
def work() -> None:
|
||||
try:
|
||||
while not stop.is_set():
|
||||
job = jobs.claim()
|
||||
if job is not None:
|
||||
jobs.finish(job.id, "done")
|
||||
except BaseException as exc: # noqa: BLE001
|
||||
errors.append(exc)
|
||||
|
||||
def watch() -> None:
|
||||
while not stop.is_set():
|
||||
ids = live(jobs, key)
|
||||
if len(ids) > 1:
|
||||
doubled.append(ids)
|
||||
|
||||
helpers = [threading.Thread(target=work), threading.Thread(target=watch)]
|
||||
senders = [threading.Thread(target=enqueue) for _ in range(threads)]
|
||||
for t in helpers + senders:
|
||||
t.start()
|
||||
for t in senders:
|
||||
t.join(timeout=120)
|
||||
stop.set()
|
||||
for t in helpers:
|
||||
t.join(timeout=35)
|
||||
|
||||
assert not errors, errors
|
||||
assert not doubled, doubled
|
||||
done = [j for j in jobs.snapshot(recent=10_000).recent if j.dedup_key == key]
|
||||
assert done, "the worker never finished a job, so nothing raced"
|
||||
Reference in new issue
Block a user