Prep stage: host words stored raw, mapped centrally in app/prep after every pull
upsert_role stores the host's own words (roles.*_host), the place texts unresolved when they change, and the company key unresolved on insert, and marks the role prep_due. app.prep maps seniority, employment, remote and country through the host tables in app/prep/hosts, resolves places and company aliases, lists unknown words in `unmapped`, remaps every open role when the tables' fingerprint changes, and runs rebuild_dupes, which moves out of the pull's settle. Migration 22 adds the columns and the table and fills the host words from raw. Cascade: pull, prep, embed, score, enrich, prep, embed, score, extract, score, summarise. `app prep [unmapped] [--remap]`, /api/prep/unmapped and a fold on the Sources page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a66aac95cf
commit
d04ceae7e0
31 files changed
+1468
-330
No files matched your search
@@ -2,11 +2,12 @@
|
||||
task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
|
||||
|
||||
pull.py read every board and search and store the roles they yield
|
||||
prep.py the `prep` stage: `app.prep` over what a pull stored, then the fold
|
||||
extract.py title and description into the gate columns, on Haiku
|
||||
embed.py one vector per open role, for fit
|
||||
score.py every open role scored for every enabled search agent
|
||||
summarise.py about 60 words on each of the best roles, on Haiku
|
||||
cascade.py pull → embed → score → extract → score → summarise, what `TASKS`
|
||||
cascade.py pull → prep → embed → score → ... → summarise, what `TASKS`
|
||||
offers, the `pull` job a schedule fires, and that schedule
|
||||
stages.py whether extract and summarise run: the switch, else config.yaml
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
|
||||
+19
-13
@@ -1,10 +1,13 @@
|
||||
"""The tasks roledesk offers, and the order a pull runs them in.
|
||||
|
||||
pull → embed → score → enrich → embed → score → extract → score → summarise
|
||||
pull → prep → embed → score → enrich → prep → embed → score → extract → score
|
||||
→ summarise
|
||||
|
||||
newsdesk's cascade convention: a cascade is a tuple of task names, and the
|
||||
final job of one task starts the next (`step` is the callback it carries).
|
||||
Nothing is alive between two tasks. A stage is addressed by its position in
|
||||
Nothing is alive between two tasks. `prep` (`app/jobs/prep.py`) maps what
|
||||
the pull stored and folds the copies before anything reads the roles; after
|
||||
`enrich` it maps the roles their own pages filled. A stage is addressed by its position in
|
||||
the tuple, not its name, since `score` runs twice.
|
||||
|
||||
Every stage is always in the chain, and a stage that is switched off finds
|
||||
@@ -12,8 +15,9 @@ nothing: a stage switched off (`stages.stage_on`), or an `on_ingest` this pull d
|
||||
satisfy, is an extract stage with no batches whose final job moves straight
|
||||
on. The free stages go first: embed and score cost no tokens, and the score
|
||||
they produce is what bounds the rest. Enrich reads the best roles without
|
||||
text on their own pages (`app/jobs/enrich.py`), and embed and score follow for
|
||||
the roles it filled; when it filled none, `step` passes over both. Extract
|
||||
text on their own pages (`app/jobs/enrich.py`), and prep, embed and score
|
||||
follow for the roles it filled; when it filled none, `step` passes over all
|
||||
three. Extract
|
||||
reads only the best
|
||||
roles of each enabled agent, as deep and as low as that agent's own `limits`
|
||||
say; the second score then applies the gates to the columns it filled.
|
||||
@@ -27,7 +31,7 @@ having either way.
|
||||
and a cascade is started by one: the `pull` job's only work is to ask for the
|
||||
chain. Its default schedule, `every 6h`, is declared here beside it.
|
||||
|
||||
`enrich` is a job and a cascade the same way, `enrich → embed → score`, queued
|
||||
`enrich` is a job and a cascade the same way, `enrich → prep → embed → score`, queued
|
||||
behind the `score` that follows a change to the ranking other than a pull's
|
||||
(`enrich_after`): an agent saved, a company's affinity changed. A vote moves
|
||||
one agent's taste a little and queues neither.
|
||||
@@ -40,20 +44,20 @@ from dataclasses import replace
|
||||
from functools import partial
|
||||
from typing import Any
|
||||
|
||||
from app.jobs import embed, enrich, extract, pull, score, summarise # noqa: F401 -- registers the jobs
|
||||
from app.jobs import embed, enrich, extract, prep, pull, score, summarise # noqa: F401 -- registers the jobs
|
||||
from desk.tools import agent
|
||||
from pipeline import NULL, TASKS, Notice, RunContext, Tally, Task, declare, job
|
||||
|
||||
STEP = "app.jobs.cascade:step"
|
||||
|
||||
CASCADES: dict[str, tuple[str, ...]] = {
|
||||
"pull": ("pull", "embed", "score", "enrich", "embed", "score", "extract", "score",
|
||||
"summarise"),
|
||||
"enrich": ("enrich", "embed", "score"),
|
||||
"pull": ("pull", "prep", "embed", "score", "enrich", "prep", "embed", "score", "extract",
|
||||
"score", "summarise"),
|
||||
"enrich": ("enrich", "prep", "embed", "score"),
|
||||
}
|
||||
|
||||
#: What follows `enrich` for the roles it filled, passed over when it filled none.
|
||||
REFILL = ("embed", "score")
|
||||
REFILL = ("prep", "embed", "score")
|
||||
|
||||
|
||||
def _once(config: Any, store: Any) -> list[dict[str, Any]]:
|
||||
@@ -103,6 +107,8 @@ STAGES: dict[str, Callable[[str], Task]] = {
|
||||
"embed": lambda cascade: Task(kind="embed", finder=_once, label="Embedding roles"),
|
||||
"score": lambda cascade: Task(kind="score", finder=_once,
|
||||
label="Scoring roles for every agent"),
|
||||
"prep": lambda cascade: Task(kind="prep", finder=_once,
|
||||
label="Mapping the roles' host words"),
|
||||
"summarise": _summarise,
|
||||
"enrich": lambda cascade: enrich.enrich_task(),
|
||||
}
|
||||
@@ -189,7 +195,7 @@ def run_pull(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str
|
||||
|
||||
@job("enrich", "Reading the best roles on their own pages", solo=True, resource="http")
|
||||
def run_enrich(config: Any, store: Any, *, context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Ask for the enrich cascade, `enrich → embed → score`. Its own work is
|
||||
"""Ask for the enrich cascade, `enrich → prep → embed → score`. Its own work is
|
||||
nil, as the `pull` job's. Under `NULL` there is no queue to take it."""
|
||||
if not context.task(chain("enrich")):
|
||||
raise RuntimeError("no queue to start the enrich cascade on")
|
||||
@@ -210,7 +216,7 @@ def enrich_after(queue: Any, score_job: Any) -> Any:
|
||||
#: What `app job <name>` and `/api/jobs` start besides `pull`: one stage
|
||||
#: each, on its own. `pull` is not a task here, since it is a job: a name that
|
||||
#: is both would be listed twice by `/api/jobs/kinds`.
|
||||
for _name in ("extract", "embed", "score", "summarise"):
|
||||
for _name in ("extract", "embed", "score", "summarise", "prep"):
|
||||
TASKS[_name] = partial(task, _name)
|
||||
|
||||
|
||||
@@ -220,5 +226,5 @@ for _name in ("extract", "embed", "score", "summarise"):
|
||||
# setting: a cadence edited or switched off under Runs survives
|
||||
# every restart (see pipeline/schedule.py).
|
||||
declare("pull", "pull", "every 6h",
|
||||
label="Pull every board, then embed, score, read the best roles' pages, "
|
||||
label="Pull every board, then map, embed, score, read the best roles' pages, "
|
||||
"extract, rescore and summarise")
|
||||
@@ -0,0 +1,31 @@
|
||||
"""`prep`: the stage that maps what a pull stored, `app.prep.run`.
|
||||
|
||||
In the pull cascade it follows the pull, whose settle has closed what is
|
||||
gone, and maps every role the pull changed, then folds the copies across
|
||||
the whole database (`Store.rebuild_dupes`, which the pull's settle used to
|
||||
run). After `enrich` it maps the roles their own pages filled, since only
|
||||
those are due. `solo`, so two cascades never write the same rows at once.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from app import prep
|
||||
from app.config import Config
|
||||
from app.storage import Store
|
||||
from pipeline import NULL, RunContext, job
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@job("prep", "Mapping the roles' host words", solo=True, resource="cpu")
|
||||
def run_prep(config: Config, store: Store, *, remap: bool = False,
|
||||
context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Map the due roles, every open one when the tables changed, and fold
|
||||
the copies. `remap` maps every role, closed ones included."""
|
||||
done = prep.run(store, remap=remap)
|
||||
log.info("prep: %d read, %d written, %d words unmapped, %s copies%s", done.read,
|
||||
done.written, done.unmapped, done.copies, ", tables changed" if done.remapped else "")
|
||||
return done._asdict()
|
||||
+4
-6
@@ -244,10 +244,9 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
|
||||
roles reopened by that batch, after the switch closed them, so every
|
||||
switched-off one is closed again here.
|
||||
|
||||
After the closes, `Store.rebuild_dupes` recomputes which open roles are
|
||||
copies of one posting, so a closed kept role hands its group on and the
|
||||
roles this run added join theirs; `copies` is how many open roles it
|
||||
leaves hidden as copies.
|
||||
The `prep` stage that follows maps what this run stored and folds the
|
||||
copies (`Store.rebuild_dupes`), so a closed kept role hands its group on
|
||||
and the roles this run added join theirs.
|
||||
"""
|
||||
answered = searched = failed = incomplete = closed = 0
|
||||
stale = (datetime.now(UTC) - timedelta(days=STALE_DAYS)).isoformat(timespec="seconds")
|
||||
@@ -284,10 +283,9 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
|
||||
closed += store.close_unseen("search", stale)
|
||||
if run_marker:
|
||||
release_run(store, run_marker)
|
||||
copies = store.rebuild_dupes().copies
|
||||
roles = sum(int(result.get("roles", 0)) for result in tally.results)
|
||||
return {"boards": answered, "searches": searched, "failed": failed,
|
||||
"incomplete": incomplete, "roles": roles, "closed": closed, "copies": copies}
|
||||
"incomplete": incomplete, "roles": roles, "closed": closed}
|
||||
|
||||
|
||||
def _close_unlisted(store: Store, source: str, seen_ids: list[str], subscription_id: int) -> int:
|
||||
|
||||
Reference in new issue
Block a user