Extract is bounded to the ranking: pull → embed → score → extract → score → summarise
The free stages run first and their score bounds the two model stages. Extract reads only the best extract.per_agent roles of each agent at extract.min_score or more, then score runs again so the gates read what it filled; summarise gets a min_score of its own. The cascade steps by position, since score is in it twice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
c4f2bcc0c9
commit
7db77e83a3
15 files changed
+215
-89
No files matched your search
+12
-1
@@ -42,11 +42,19 @@ class ServerConfig(Section):
|
||||
class ExtractConfig(ModelTaskConfig):
|
||||
"""Turning a posting's title and text into the gate columns. See
|
||||
app/jobs/extract.py. Off by default, so nobody is billed by surprise:
|
||||
on the API backend every batch is a paid call."""
|
||||
on the API backend every batch is a paid call. On, it reads only the
|
||||
best `per_agent` roles of each enabled agent that score at least
|
||||
`min_score`, so the cost follows the ranking rather than the boards."""
|
||||
|
||||
enabled: bool = False
|
||||
model: str = "claude-haiku-4-5-20251001"
|
||||
batch_size: int = Field(20, ge=1)
|
||||
#: How deep into each enabled agent's ranking extract reads: the backstop.
|
||||
per_agent: int = Field(500, ge=0)
|
||||
#: Below this score a role is not read, however few are above it. Loose,
|
||||
#: since a fresh role with the worst fit scores about 0.27 on the default
|
||||
#: weights and a good one 0.6 or more.
|
||||
min_score: float = Field(0.3, ge=0, le=1)
|
||||
|
||||
|
||||
class SummariseConfig(ModelTaskConfig):
|
||||
@@ -63,6 +71,9 @@ class SummariseConfig(ModelTaskConfig):
|
||||
batch_size: int = Field(10, ge=1)
|
||||
#: How deep into each enabled agent's ranking a pull summarises.
|
||||
per_agent: int = Field(30, ge=0)
|
||||
#: Below this score a role is not summarised. Stricter than extract's,
|
||||
#: since a summary is the expensive answer.
|
||||
min_score: float = Field(0.5, ge=0, le=1)
|
||||
|
||||
|
||||
class EmbedConfig(Section):
|
||||
|
||||
@@ -6,7 +6,7 @@ task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
|
||||
embed.py one vector per open role, for fit
|
||||
score.py every open role scored for every enabled search agent
|
||||
summarise.py about 60 words on each of the best roles, on Haiku
|
||||
cascade.py pull → extract → embed → score → summarise, what `TASKS`
|
||||
cascade.py pull → embed → score → extract → score → summarise, what `TASKS`
|
||||
offers, the `pull` job a schedule fires, and that schedule
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
added, switched on or off, pruned by region
|
||||
|
||||
+22
-20
@@ -1,18 +1,21 @@
|
||||
"""The tasks roledesk offers, and the order a pull runs them in.
|
||||
|
||||
pull → extract → embed → score → summarise
|
||||
pull → embed → score → extract → score → summarise
|
||||
|
||||
newsdesk's cascade convention: a cascade is a tuple of task names, and the
|
||||
final job of one task starts the next (`step` is the callback it carries).
|
||||
Nothing is alive between two tasks.
|
||||
Nothing is alive between two tasks. A stage is addressed by its position in
|
||||
the tuple, not its name, since `score` runs twice.
|
||||
|
||||
Every stage is always in the chain, and a stage that is switched off finds
|
||||
nothing: `extract.enabled: false`, or an `on_ingest` this pull does not
|
||||
satisfy, is an extract stage with no batches whose final job moves straight
|
||||
on. Extract runs before score so the gates read what it filled; before embed
|
||||
too, since embedding does not read its columns and the order costs nothing.
|
||||
Summarise runs last because it reads the scores: it takes only the best
|
||||
`summarise.per_agent` roles of each enabled agent.
|
||||
on. The free stages go first: embed and score cost no tokens, and the score
|
||||
they produce is what bounds the two that do. Extract reads only the best
|
||||
`extract.per_agent` roles of each enabled agent, not below
|
||||
`extract.min_score`; the second score then applies the gates to the columns
|
||||
it filled. Summarise runs last, on the best `summarise.per_agent` roles not
|
||||
below `summarise.min_score`.
|
||||
|
||||
A batch that failed, or a stage somebody killed, does not stop the chain: the
|
||||
work comes round on the next pull, and a rescore of what did arrive is worth
|
||||
@@ -37,7 +40,7 @@ from pipeline import NULL, TASKS, RunContext, Tally, Task, declare, job
|
||||
STEP = "app.jobs.cascade:step"
|
||||
|
||||
CASCADES: dict[str, tuple[str, ...]] = {
|
||||
"pull": ("pull", "extract", "embed", "score", "summarise"),
|
||||
"pull": ("pull", "embed", "score", "extract", "score", "summarise"),
|
||||
}
|
||||
|
||||
|
||||
@@ -92,21 +95,19 @@ STAGES: dict[str, Callable[[str], Task]] = {
|
||||
}
|
||||
|
||||
|
||||
def task(name: str, *, cascade: str = "") -> Task:
|
||||
def task(name: str, *, cascade: str = "", at: int = 0) -> Task:
|
||||
"""One stage as a `Task`. With `cascade` its final job carries `step`,
|
||||
and the next stage follows; without, it stops when its batches do."""
|
||||
and the stage after position `at` follows; without, it stops when its
|
||||
batches do."""
|
||||
return replace(STAGES[name](cascade), name=name,
|
||||
callback=STEP if cascade else "",
|
||||
carries={"cascade": cascade, "task": name})
|
||||
carries={"cascade": cascade, "task": name, "at": at})
|
||||
|
||||
|
||||
def follows(cascade: str, name: str) -> str:
|
||||
"""What runs after `name`, or "" at the end or for an unknown name."""
|
||||
def follows(cascade: str, at: int) -> str:
|
||||
"""The stage after position `at`, or "" at the end or off the cascade."""
|
||||
tasks = CASCADES.get(cascade, ())
|
||||
if name not in tasks:
|
||||
return ""
|
||||
at = tasks.index(name)
|
||||
return tasks[at + 1] if at + 1 < len(tasks) else ""
|
||||
return tasks[at + 1] if 0 <= at < len(tasks) - 1 else ""
|
||||
|
||||
|
||||
def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
|
||||
@@ -114,9 +115,10 @@ def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
|
||||
"""The callback: make the next task. A killed stage is skipped and the
|
||||
chain goes on, as in newsdesk's cascade."""
|
||||
name, done = payload.get("cascade", ""), payload.get("task", "")
|
||||
following = follows(name, done)
|
||||
at = int(payload.get("at", -1))
|
||||
following = follows(name, at)
|
||||
if following:
|
||||
context.task(task(following, cascade=name))
|
||||
context.task(task(following, cascade=name, at=at + 1))
|
||||
out: dict[str, Any] = {"task": done, "next": following}
|
||||
if tally.killed:
|
||||
out["skipped"] = "cancelled"
|
||||
@@ -125,7 +127,7 @@ def step(config: Any, store: Any, tally: Tally, payload: dict[str, Any], *,
|
||||
|
||||
def chain(name: str) -> Task:
|
||||
"""The first stage of cascade `name`, carrying `step`, so the rest follow."""
|
||||
return task(CASCADES[name][0], cascade=name)
|
||||
return task(CASCADES[name][0], cascade=name, at=0)
|
||||
|
||||
|
||||
@job("pull", "Pulling every board", resource="http")
|
||||
@@ -154,4 +156,4 @@ for _name in ("extract", "embed", "score", "summarise"):
|
||||
# setting: a cadence edited or switched off on the Pulls page survives
|
||||
# every restart (see pipeline/schedule.py).
|
||||
declare("pull", "pull", "every 6h",
|
||||
label="Pull every board, then extract, embed, score and summarise")
|
||||
label="Pull every board, then embed, score, extract, rescore and summarise")
|
||||
+15
-7
@@ -8,12 +8,17 @@ refused batch, and failing with a notice when no model is reachable are
|
||||
`desk.modeltask`'s.
|
||||
|
||||
A role is waiting while it is open, has never been answered (`extracted_at IS
|
||||
NULL`) and has not been given up on. The answer only fills columns that are
|
||||
still empty: a board's structured field wins over what a model read out of the
|
||||
prose. See `Store.set_extracted`.
|
||||
NULL`), has not been given up on, and is in the best `extract.per_agent` of an
|
||||
enabled agent's ungated scores at `extract.min_score` or more. That bound is
|
||||
what keeps the cost proportional to the rankings rather than to the boards:
|
||||
the pull scores every role for free first, then reads the few hundred that
|
||||
rank, and scores again so the gates see what was filled. A batch built for
|
||||
named uids finishes what it was dealt whatever their rank. The answer only
|
||||
fills columns that are still empty: a board's structured field wins over what
|
||||
a model read out of the prose. See `Store.set_extracted`.
|
||||
|
||||
Off by default (`extract.enabled: false`), because on the API backend every
|
||||
batch is a paid call and ~30k open roles is ~1,500 of them.
|
||||
batch is a paid call.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -128,7 +133,8 @@ def _item(role: Role) -> dict[str, Any]:
|
||||
def build_batch(config: Config, store: Store, limit: int = 20,
|
||||
uids: Sequence[str] | None = None) -> dict[str, Any]:
|
||||
return {"instructions": SYSTEM,
|
||||
"roles": [_item(r) for r in store.extract_waiting(limit, uids)]}
|
||||
"roles": [_item(r) for r in store.extract_waiting(
|
||||
limit, config.extract.per_agent, config.extract.min_score, uids)]}
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- answer
|
||||
@@ -199,9 +205,11 @@ TASK: ModelTask[ExtractResult] = ModelTask(
|
||||
kind="extract", label="Reading postings for gate fields",
|
||||
verb="extracting", noun="fields", idle="nothing to extract", items="roles",
|
||||
settings=lambda config: config.extract,
|
||||
pending=lambda config, store: store.extract_pending(),
|
||||
pending=lambda config, store: store.extract_pending(
|
||||
config.extract.per_agent, config.extract.min_score),
|
||||
waiting=lambda config, store, limit: [
|
||||
r.uid for r in store.extract_waiting(limit)],
|
||||
r.uid for r in store.extract_waiting(
|
||||
limit, config.extract.per_agent, config.extract.min_score)],
|
||||
build_batch=build_batch,
|
||||
system=lambda store: SYSTEM,
|
||||
schema=SCHEMA,
|
||||
|
||||
@@ -6,9 +6,9 @@ what the model is told, the schema it answers in, what to do with the
|
||||
answer), and running it is `desk.modeltask`'s.
|
||||
|
||||
A role is waiting while it has no summary, has some text, and is in the top
|
||||
`summarise.per_agent` of an enabled agent's ungated scores. That bound is what
|
||||
keeps the cost flat: a pull summarises the roles somebody will look at, not the
|
||||
30k that are open. A batch built for named uids (the read page's Summarise
|
||||
`summarise.per_agent` of an enabled agent's ungated scores at
|
||||
`summarise.min_score` or more. That bound is what keeps the cost flat: a pull
|
||||
summarises the roles somebody will look at, not the 30k that are open. A batch built for named uids (the read page's Summarise
|
||||
button) takes those roles whatever their rank. See `Store.summarise_waiting`.
|
||||
|
||||
The text is `article_text` when the posting page was fetched, else
|
||||
@@ -121,7 +121,8 @@ def _item(role: Role) -> dict[str, Any]:
|
||||
|
||||
def _waiting(config: Config, store: Store, limit: int,
|
||||
uids: Sequence[str] | None = None) -> list[Role]:
|
||||
return store.summarise_waiting(limit, config.summarise.per_agent, uids)
|
||||
return store.summarise_waiting(limit, config.summarise.per_agent, uids,
|
||||
min_score=config.summarise.min_score)
|
||||
|
||||
|
||||
def build_batch(config: Config, store: Store, limit: int = 10,
|
||||
@@ -165,7 +166,8 @@ TASK: ModelTask[SummariseResult] = ModelTask(
|
||||
verb="summarising", noun="summaries", idle="nothing to summarise",
|
||||
items="summaries",
|
||||
settings=lambda config: config.summarise,
|
||||
pending=lambda config, store: store.summarise_pending(config.summarise.per_agent),
|
||||
pending=lambda config, store: store.summarise_pending(
|
||||
config.summarise.per_agent, config.summarise.min_score),
|
||||
waiting=lambda config, store, limit: [r.uid for r in _waiting(config, store, limit)],
|
||||
build_batch=build_batch,
|
||||
system=lambda store: SYSTEM,
|
||||
|
||||
+56
-33
@@ -619,21 +619,51 @@ class Store:
|
||||
|
||||
# -------------------------------------------------------------- extract
|
||||
|
||||
def extract_pending(self) -> int:
|
||||
row = self._row(f"SELECT COUNT(*) AS n FROM roles WHERE {_EXTRACT_WAITING}")
|
||||
def _top_scope(self, per_agent: int, min_score: float) -> tuple[str, list[Any]]:
|
||||
"""The `WHERE` fragment for a role among the `per_agent` best ungated
|
||||
scores of an enabled agent, not below `min_score`: what the two model
|
||||
stages, extract and summarise, are bounded to. The role is `r`. A
|
||||
company an agent hid needs no clause: `score` writes no rows for it
|
||||
under that agent."""
|
||||
return ("""r.uid IN (SELECT uid FROM (
|
||||
SELECT s.uid, s.score, ROW_NUMBER() OVER (
|
||||
PARTITION BY s.agent_id
|
||||
ORDER BY s.score DESC, top.first_seen DESC, s.uid) AS n
|
||||
FROM scores s
|
||||
JOIN agents a ON a.id = s.agent_id AND a.enabled = 1
|
||||
JOIN roles top ON top.uid = s.uid AND top.closed_at IS NULL
|
||||
WHERE s.gated_reason = '')
|
||||
WHERE n <= ? AND score >= ?)""", [per_agent, min_score])
|
||||
|
||||
def _extract_scope(self, per_agent: int, min_score: float,
|
||||
uids: Sequence[str] | None) -> tuple[str, list[Any]]:
|
||||
"""The `WHERE` for a role `extract` may read: open, never answered,
|
||||
not given up. Given `uids`, those of them and nothing else, since a
|
||||
batch asks again when it runs and must finish what it was dealt;
|
||||
otherwise only the roles in `_top_scope`."""
|
||||
where = [_EXTRACT_WAITING.replace("extracted_at", "r.extracted_at")
|
||||
.replace("closed_at", "r.closed_at").replace("extract_tries", "r.extract_tries")]
|
||||
if uids is not None:
|
||||
where.append("r.uid IN (SELECT value FROM json_each(?))")
|
||||
return " AND ".join(where), [json.dumps(list(uids))]
|
||||
top, params = self._top_scope(per_agent, min_score)
|
||||
return " AND ".join([*where, top]), params
|
||||
|
||||
def extract_pending(self, per_agent: int, min_score: float = 0.0) -> int:
|
||||
where, params = self._extract_scope(per_agent, min_score, None)
|
||||
row = self._row(f"SELECT COUNT(*) AS n FROM roles r WHERE {where}", params)
|
||||
return int(row["n"]) if row else 0
|
||||
|
||||
def extract_waiting(self, limit: int,
|
||||
def extract_waiting(self, limit: int, per_agent: int, min_score: float = 0.0,
|
||||
uids: Sequence[str] | None = None) -> list[Role]:
|
||||
"""Roles `extract` has yet to read, newest first. Given `uids`, only
|
||||
those of them still waiting: a batch asks again when it runs, since a
|
||||
role may have been answered since the task shared the work out."""
|
||||
sql, params = f"SELECT * FROM roles WHERE {_EXTRACT_WAITING}", []
|
||||
if uids is not None:
|
||||
sql += " AND uid IN (SELECT value FROM json_each(?))"
|
||||
params.append(json.dumps(list(uids)))
|
||||
sql += " ORDER BY first_seen DESC, uid LIMIT ?"
|
||||
return [Role.from_row(r) for r in self._rows(sql, [*params, limit])]
|
||||
"""Roles `extract` has yet to read, newest first: the best
|
||||
`per_agent` of every enabled agent's scores not below `min_score`, so
|
||||
the backlog follows the rankings rather than the boards. Given
|
||||
`uids`, only those of them still waiting, whatever their rank."""
|
||||
where, params = self._extract_scope(per_agent, min_score, uids)
|
||||
return [Role.from_row(r) for r in self._rows(
|
||||
f"SELECT r.* FROM roles r WHERE {where} "
|
||||
"ORDER BY r.first_seen DESC, r.uid LIMIT ?", [*params, limit])]
|
||||
|
||||
def set_extracted(self, uid: str, fields: dict[str, Any]) -> int | None:
|
||||
"""Write what `extract` read for one role and stamp `extracted_at`.
|
||||
@@ -692,48 +722,41 @@ class Store:
|
||||
#: roles in some agent's top list.
|
||||
_SUMMARY_MISSED = "summarise.missed:"
|
||||
|
||||
def _summary_scope(self, per_agent: int,
|
||||
def _summary_scope(self, per_agent: int, min_score: float,
|
||||
uids: Sequence[str] | None) -> tuple[str, list[Any]]:
|
||||
"""The `WHERE` for a role `summarise` may be sent: no summary yet and
|
||||
some text to summarise. Given `uids`, those of them and nothing else,
|
||||
open or not and given up or not, because somebody asked for them.
|
||||
Otherwise open, not given up, and among the `per_agent` best ungated
|
||||
scores of an enabled agent. A company an agent hid needs no clause:
|
||||
`score` writes no rows for it under that agent."""
|
||||
Otherwise open, not given up, and in `_top_scope`."""
|
||||
where = ["r.summary = ''",
|
||||
"(TRIM(r.article_text) <> '' OR TRIM(r.description) <> '')"]
|
||||
if uids is not None:
|
||||
where.append("r.uid IN (SELECT value FROM json_each(?))")
|
||||
return " AND ".join(where), [json.dumps(list(uids))]
|
||||
top, params = self._top_scope(per_agent, min_score)
|
||||
where += [
|
||||
"r.closed_at IS NULL",
|
||||
f"""NOT EXISTS (SELECT 1 FROM state
|
||||
WHERE key = '{self._SUMMARY_MISSED}' || r.uid
|
||||
AND CAST(value AS INTEGER) >= {GIVE_UP_AFTER})""",
|
||||
"""r.uid IN (SELECT uid FROM (
|
||||
SELECT s.uid, ROW_NUMBER() OVER (
|
||||
PARTITION BY s.agent_id
|
||||
ORDER BY s.score DESC, top.first_seen DESC, s.uid) AS n
|
||||
FROM scores s
|
||||
JOIN agents a ON a.id = s.agent_id AND a.enabled = 1
|
||||
JOIN roles top ON top.uid = s.uid AND top.closed_at IS NULL
|
||||
WHERE s.gated_reason = '')
|
||||
WHERE n <= ?)""",
|
||||
top,
|
||||
]
|
||||
return " AND ".join(where), [per_agent]
|
||||
return " AND ".join(where), params
|
||||
|
||||
def summarise_pending(self, per_agent: int) -> int:
|
||||
where, params = self._summary_scope(per_agent, None)
|
||||
def summarise_pending(self, per_agent: int, min_score: float = 0.0) -> int:
|
||||
where, params = self._summary_scope(per_agent, min_score, None)
|
||||
row = self._row(f"SELECT COUNT(*) AS n FROM roles r WHERE {where}", params)
|
||||
return int(row["n"]) if row else 0
|
||||
|
||||
def summarise_waiting(self, limit: int, per_agent: int,
|
||||
uids: Sequence[str] | None = None) -> list[Role]:
|
||||
uids: Sequence[str] | None = None,
|
||||
min_score: float = 0.0) -> list[Role]:
|
||||
"""Roles `summarise` has yet to write, newest first. The top
|
||||
`per_agent` of every enabled agent's scores, so the backlog is bounded
|
||||
by the number of agents rather than by the number of open roles; given
|
||||
`uids`, those still without a summary. A batch asks again when it runs."""
|
||||
where, params = self._summary_scope(per_agent, uids)
|
||||
`per_agent` of every enabled agent's scores not below `min_score`, so
|
||||
the backlog is bounded by the number of agents rather than by the
|
||||
number of open roles; given `uids`, those still without a summary,
|
||||
whatever their rank. A batch asks again when it runs."""
|
||||
where, params = self._summary_scope(per_agent, min_score, uids)
|
||||
return [Role.from_row(r) for r in self._rows(
|
||||
f"SELECT r.* FROM roles r WHERE {where} "
|
||||
"ORDER BY r.first_seen DESC, r.uid LIMIT ?", [*params, limit])]
|
||||
|
||||
@@ -126,7 +126,7 @@
|
||||
<h2>Runs</h2>
|
||||
<button type="button" id="pull-now" class="secondary">Pull now</button>
|
||||
</div>
|
||||
<p class="lede">Every 6 hours the pull reads every board, then extract fills in what each posting says, embed and score rank them, and summarise writes the summaries. Change the cadence below.</p>
|
||||
<p class="lede">Every 6 hours the pull reads every board, embed and score rank every role, extract fills in what the best postings say, score runs again with what it filled, and summarise writes the summaries. Change the cadence below.</p>
|
||||
|
||||
<h3>What is running</h3>
|
||||
{# The heading and the idle line are the app's: the element takes up no
|
||||
|
||||
Reference in new issue
Block a user