Runs is the queue's elements and a Problems log; a Kill lands between boards

Run one stage goes: the cascade is the unit of work and app job keeps the
per-stage entry points. The pull button is disabled while a pull runs and
the lede no longer hardcodes the cadence. Messages folds into Runs as
Problems. pull_batch checks for a cancel before every board, one plugin per
target, and both plugins now close their HTTP client.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-28 16:10:38 +02:00
1 parent cebe62ddc9
commit 8532e4e739
15 files changed
+284 -242

No files matched your search

+23 -24
View File
@@ -78,41 +78,40 @@ def run_batch(config: Config, store: Store, *, targets: list[str], source: str =
they list.
`targets` is keyword-only without a default, which is what makes it a
required payload key to the queue: the Pulls page offers no button for a
batch that could only fail without one.
required payload key to the queue: `/api/jobs/kinds` lists no batch that
could only fail without one.
Builds the plugin scoped to just these targets — `Source.targets()` reads
`options[subscription_field]`, so handing it only this batch's entries is
what keeps one job from re-fetching every other job's boards.
One plugin per target, scoped to just that entry — `Source.targets()`
reads `options[subscription_field]` — so one job never re-fetches another
job's boards, and `context.check()` runs before each fetch: a Kill lands
between two boards rather than after the tenth.
"""
from plugins.ats import ATSSource
from plugins.search import SearchSource
# Fetchable now, not when the pull was queued: a board switched off on the
# Searches or Companies page since then is skipped.
# Job boards or Company boards since then is skipped.
subs = {s.target: s for s in store.subscriptions(source=source, fetchable=True)
if s.target in targets}
options = dict(config.sources.get(source) or {})
plugin: ATSSource | SearchSource
if source == "search":
options["searches"] = [{"url": t, "label": subs[t].label, "params": subs[t].params}
for t in targets if t in subs]
plugin = SearchSource(options)
else:
options["boards"] = [{"url": t, "label": subs[t].company or subs[t].label}
for t in targets if t in subs]
plugin = ATSSource(options)
plugin.state = store.state
stored = 0
for item in plugin.fetch():
item.source = source
for target in targets:
sub = subs.get(target)
if sub is None:
continue
context.check()
options = dict(config.sources.get(source) or {})
plugin: ATSSource | SearchSource
if source == "search":
target = str(item.raw.get("search", ""))
options["searches"] = [{"url": target, "label": sub.label, "params": sub.params}]
plugin = SearchSource(options)
else:
target = f"{item.raw.get('board', '')}:{item.raw.get('slug', '')}"
store.upsert_role(item, subs.get(target))
stored += 1
options["boards"] = [{"url": target, "label": sub.company or sub.label}]
plugin = ATSSource(options)
plugin.state = store.state
for item in plugin.fetch():
item.source = source
store.upsert_role(item, sub)
stored += 1
return {"roles": stored}