Prep stage: host words stored raw, mapped centrally in app/prep after every pull

upsert_role stores the host's own words (roles.*_host), the place texts
unresolved when they change, and the company key unresolved on insert, and
marks the role prep_due. app.prep maps seniority, employment, remote and
country through the host tables in app/prep/hosts, resolves places and
company aliases, lists unknown words in `unmapped`, remaps every open role
when the tables' fingerprint changes, and runs rebuild_dupes, which moves
out of the pull's settle. Migration 22 adds the columns and the table and
fills the host words from raw. Cascade: pull, prep, embed, score, enrich,
prep, embed, score, extract, score, summarise. `app prep [unmapped]
[--remap]`, /api/prep/unmapped and a fold on the Sources page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 11:19:22 +02:00
1 parent a66aac95cf
commit d04ceae7e0
31 files changed
+1468 -330

No files matched your search

+2 -1
View File
@@ -2,11 +2,12 @@
task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
pull.py read every board and search and store the roles they yield
prep.py the `prep` stage: `app.prep` over what a pull stored, then the fold
extract.py title and description into the gate columns, on Haiku
embed.py one vector per open role, for fit
score.py every open role scored for every enabled search agent
summarise.py about 60 words on each of the best roles, on Haiku
cascade.py pull → embed → score → extract → score → summarise, what `TASKS`
cascade.py pull → prep → embed → score → ... → summarise, what `TASKS`
offers, the `pull` job a schedule fires, and that schedule
stages.py whether extract and summarise run: the switch, else config.yaml
sources.py what a pull reads, changed by hand: boards and searches
+19 -13
View File
@@ -1,10 +1,13 @@
"""The tasks roledesk offers, and the order a pull runs them in.
pull → embed → score → enrich → embed → score → extract → score → summarise
pull → prep → embed → score → enrich → prep → embed → score → extract → score
→ summarise
newsdesk's cascade convention: a cascade is a tuple of task names, and the
final job of one task starts the next (`step` is the callback it carries).
Nothing is alive between two tasks. A stage is addressed by its position in
Nothing is alive between two tasks. `prep` (`app/jobs/prep.py`) maps what
the pull stored and folds the copies before anything reads the roles; after
`enrich` it maps the roles their own pages filled. A stage is addressed by its position in
the tuple, not its name, since `score` runs twice.
Every stage is always in the chain, and a stage that is switched off finds
@@ -12,8 +15,9 @@ nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this pull d
satisfy, is an extract stage with no batches whose final job moves straight
on. The free stages go first: embed and score cost no tokens, and the score
they produce is what bounds the rest. Enrich reads the best roles without
text on their own pages (`app/jobs/enrich.py`), and embed and score follow for
the roles it filled; when it filled none, `step` passes over both. Extract
text on their own pages (`app/jobs/enrich.py`), and prep, embed and score
follow for the roles it filled; when it filled none, `step` passes over all
three. Extract
reads only the best
roles of each enabled agent, as deep and as low as that agent's own `limits`
say; the second score then applies the gates to the columns it filled.
@@ -27,7 +31,7 @@ having either way.
and a cascade is started by one: the `pull` job's only work is to ask for the
chain. Its default schedule, `every 6h`, is declared here beside it.
`enrich` is a job and a cascade the same way, `enrich → embed → score`, queued
`enrich` is a job and a cascade the same way, `enrich → prep → embed → score`, queued
behind the `score` that follows a change to the ranking other than a pull's
(`enrich_after`): an agent saved, a company's affinity changed. A vote moves
one agent's taste a little and queues neither.
@@ -40,20 +44,20 @@ from dataclasses import replace
from functools import partial
from typing import Any
from app.jobs import embed, enrich, extract, pull, score, summarise # noqa: F401 -- registers the jobs
from app.jobs import embed, enrich, extract, prep, pull, score, summarise # noqa: F401 -- registers the jobs
from desk.tools import agent
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
STEP = "app.jobs.cascade:step"
CASCADES: dict[str, tuple[str, ...]] = {
"pull": ("pull", "embed", "score", "enrich", "embed", "score", "extract", "score",
"summarise"),
"enrich": ("enrich", "embed", "score"),
"pull": ("pull", "prep", "embed", "score", "enrich", "prep", "embed", "score", "extract",
"score", "summarise"),
"enrich": ("enrich", "prep", "embed", "score"),
}
#: What follows `enrich` for the roles it filled, passed over when it filled none.
REFILL = ("embed", "score")
REFILL = ("prep", "embed", "score")
def _once(config: Any, store: Any) -> list[dict[str, Any]]:
@@ -103,6 +107,8 @@ STAGES: dict[str, Callable[[str], Task]] = {
"embed": lambda cascade: Task(kind="embed", finder=_once, label="Embedding roles"),
"score": lambda cascade: Task(kind="score", finder=_once,
label="Scoring roles for every agent"),
"prep": lambda cascade: Task(kind="prep", finder=_once,
label="Mapping the roles' host words"),
"summarise": _summarise,
"enrich": lambda cascade: enrich.enrich_task(),
}
@@ -189,7 +195,7 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
@job("enrich", "Reading the best roles on their own pages", solo=True, resource="http")
def run_enrich(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
"""Ask for the enrich cascade, `enrich → embed → score`. Its own work is
"""Ask for the enrich cascade, `enrich → prep → embed → score`. Its own work is
nil, as the `pull` job's. Under `NULL` there is no queue to take it."""
if not context.task(chain("enrich")):
raise RuntimeError("no queue to start the enrich cascade on")
@@ -210,7 +216,7 @@ def enrich_after(queue: Any, score_job: Any) -> Any:
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
#: each, on its own. `pull` is not a task here, since it is a job: a name that
#: is both would be listed twice by `/api/jobs/kinds`.
for _name in ("extract", "embed", "score", "summarise"):
for _name in ("extract", "embed", "score", "summarise", "prep"):
TASKS[_name] = partial(task, _name)
@@ -220,5 +226,5 @@ for _name in ("extract", "embed", "score", "summarise"):
# setting: a cadence edited or switched off under Runs survives
# every restart (see pipeline/schedule.py).
declare("pull", "pull", "every 6h",
label="Pull every board, then embed, score, read the best roles' pages, "
label="Pull every board, then map, embed, score, read the best roles' pages, "
"extract, rescore and summarise")
+31
View File
@@ -0,0 +1,31 @@
"""`prep`: the stage that maps what a pull stored, `app.prep.run`.
In the pull cascade it follows the pull, whose settle has closed what is
gone, and maps every role the pull changed, then folds the copies across
the whole database (`Store.rebuild_dupes`, which the pull's settle used to
run). After `enrich` it maps the roles their own pages filled, since only
those are due. `solo`, so two cascades never write the same rows at once.
"""
from __future__ import annotations
import logging
from typing import Any
from app import prep
from app.config import Config
from app.storage import Store
from pipeline import NULL, RunContext, job
log = logging.getLogger(__name__)
@job("prep", "Mapping the roles' host words", solo=True, resource="cpu")
def run_prep(config: Config, store: Store, *, remap: bool = False,
context: RunContext = NULL) -> dict[str, Any]:
"""Map the due roles, every open one when the tables changed, and fold
the copies. `remap` maps every role, closed ones included."""
done = prep.run(store, remap=remap)
log.info("prep: %d read, %d written, %d words unmapped, %s copies%s", done.read,
done.written, done.unmapped, done.copies, ", tables changed" if done.remapped else "")
return done._asdict()
+4 -6
View File
@@ -244,10 +244,9 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
roles reopened by that batch, after the switch closed them, so every
switched-off one is closed again here.
After the closes, `Store.rebuild_dupes` recomputes which open roles are
copies of one posting, so a closed kept role hands its group on and the
roles this run added join theirs; `copies` is how many open roles it
leaves hidden as copies.
The `prep` stage that follows maps what this run stored and folds the
copies (`Store.rebuild_dupes`), so a closed kept role hands its group on
and the roles this run added join theirs.
"""
answered = searched = failed = incomplete = closed = 0
stale = (datetime.now(UTC) - timedelta(days=STALE_DAYS)).isoformat(timespec="seconds")
@@ -284,10 +283,9 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
closed += store.close_unseen("search", stale)
if run_marker:
release_run(store, run_marker)
copies = store.rebuild_dupes().copies
roles = sum(int(result.get("roles", 0)) for result in tally.results)
return {"boards": answered, "searches": searched, "failed": failed,
"incomplete": incomplete, "roles": roles, "closed": closed, "copies": copies}
"incomplete": incomplete, "roles": roles, "closed": closed}
def _close_unlisted(store: Store, source: str, seen_ids: list[str], subscription_id: int) -> int: