Prep stage: host words stored raw, mapped centrally in app/prep after every pull
upsert_role stores the host's own words (roles.*_host), the place texts unresolved when they change, and the company key unresolved on insert, and marks the role prep_due. app.prep maps seniority, employment, remote and country through the host tables in app/prep/hosts, resolves places and company aliases, lists unknown words in `unmapped`, remaps every open role when the tables' fingerprint changes, and runs rebuild_dupes, which moves out of the pull's settle. Migration 22 adds the columns and the table and fills the host words from raw. Cascade: pull, prep, embed, score, enrich, prep, embed, score, extract, score, summarise. `app prep [unmapped] [--remap]`, /api/prep/unmapped and a fold on the Sources page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a66aac95cf
commit
d04ceae7e0
31 files changed
+1468
-330
No files matched your search
@@ -0,0 +1,31 @@
|
||||
"""`prep`: the stage that maps what a pull stored, `app.prep.run`.
|
||||
|
||||
In the pull cascade it follows the pull, whose settle has closed what is
|
||||
gone, and maps every role the pull changed, then folds the copies across
|
||||
the whole database (`Store.rebuild_dupes`, which the pull's settle used to
|
||||
run). After `enrich` it maps the roles their own pages filled, since only
|
||||
those are due. `solo`, so two cascades never write the same rows at once.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from app import prep
|
||||
from app.config import Config
|
||||
from app.storage import Store
|
||||
from pipeline import NULL, RunContext, job
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@job("prep", "Mapping the roles' host words", solo=True, resource="cpu")
|
||||
def run_prep(config: Config, store: Store, *, remap: bool = False,
|
||||
context: RunContext = NULL) -> dict[str, Any]:
|
||||
"""Map the due roles, every open one when the tables changed, and fold
|
||||
the copies. `remap` maps every role, closed ones included."""
|
||||
done = prep.run(store, remap=remap)
|
||||
log.info("prep: %d read, %d written, %d words unmapped, %s copies%s", done.read,
|
||||
done.written, done.unmapped, done.copies, ", tables changed" if done.remapped else "")
|
||||
return done._asdict()
|
||||
Reference in new issue
Block a user