Runs is the queue's elements and a Problems log; a Kill lands between boards
Run one stage goes: the cascade is the unit of work and app job keeps the per-stage entry points. The pull button is disabled while a pull runs and the lede no longer hardcodes the cadence. Messages folds into Runs as Problems. pull_batch checks for a cancel before every board, one plugin per target, and both plugins now close their HTTP client. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
cebe62ddc9
commit
8532e4e739
15 files changed
+284
-242
No files matched your search
+23
-24
@@ -78,41 +78,40 @@ def run_batch(config: Config, store: Store, *, targets: list[str], source: str =
|
||||
they list.
|
||||
|
||||
`targets` is keyword-only without a default, which is what makes it a
|
||||
required payload key to the queue: the Pulls page offers no button for a
|
||||
batch that could only fail without one.
|
||||
required payload key to the queue: `/api/jobs/kinds` lists no batch that
|
||||
could only fail without one.
|
||||
|
||||
Builds the plugin scoped to just these targets — `Source.targets()` reads
|
||||
`options[subscription_field]`, so handing it only this batch's entries is
|
||||
what keeps one job from re-fetching every other job's boards.
|
||||
One plugin per target, scoped to just that entry — `Source.targets()`
|
||||
reads `options[subscription_field]` — so one job never re-fetches another
|
||||
job's boards, and `context.check()` runs before each fetch: a Kill lands
|
||||
between two boards rather than after the tenth.
|
||||
"""
|
||||
from plugins.ats import ATSSource
|
||||
from plugins.search import SearchSource
|
||||
|
||||
# Fetchable now, not when the pull was queued: a board switched off on the
|
||||
# Searches or Companies page since then is skipped.
|
||||
# Job boards or Company boards since then is skipped.
|
||||
subs = {s.target: s for s in store.subscriptions(source=source, fetchable=True)
|
||||
if s.target in targets}
|
||||
options = dict(config.sources.get(source) or {})
|
||||
plugin: ATSSource | SearchSource
|
||||
if source == "search":
|
||||
options["searches"] = [{"url": t, "label": subs[t].label, "params": subs[t].params}
|
||||
for t in targets if t in subs]
|
||||
plugin = SearchSource(options)
|
||||
else:
|
||||
options["boards"] = [{"url": t, "label": subs[t].company or subs[t].label}
|
||||
for t in targets if t in subs]
|
||||
plugin = ATSSource(options)
|
||||
plugin.state = store.state
|
||||
|
||||
stored = 0
|
||||
for item in plugin.fetch():
|
||||
item.source = source
|
||||
for target in targets:
|
||||
sub = subs.get(target)
|
||||
if sub is None:
|
||||
continue
|
||||
context.check()
|
||||
options = dict(config.sources.get(source) or {})
|
||||
plugin: ATSSource | SearchSource
|
||||
if source == "search":
|
||||
target = str(item.raw.get("search", ""))
|
||||
options["searches"] = [{"url": target, "label": sub.label, "params": sub.params}]
|
||||
plugin = SearchSource(options)
|
||||
else:
|
||||
target = f"{item.raw.get('board', '')}:{item.raw.get('slug', '')}"
|
||||
store.upsert_role(item, subs.get(target))
|
||||
stored += 1
|
||||
options["boards"] = [{"url": target, "label": sub.company or sub.label}]
|
||||
plugin = ATSSource(options)
|
||||
plugin.state = store.state
|
||||
for item in plugin.fetch():
|
||||
item.source = source
|
||||
store.upsert_role(item, sub)
|
||||
stored += 1
|
||||
return {"roles": stored}
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user