Files
WolfandClaude Opus 5.5 d04ceae7e0 Prep stage: host words stored raw, mapped centrally in app/prep after every pull
upsert_role stores the host's own words (roles.*_host), the place texts
unresolved when they change, and the company key unresolved on insert, and
marks the role prep_due. app.prep maps seniority, employment, remote and
country through the host tables in app/prep/hosts, resolves places and
company aliases, lists unknown words in `unmapped`, remaps every open role
when the tables' fingerprint changes, and runs rebuild_dupes, which moves
out of the pull's settle. Migration 22 adds the columns and the table and
fills the host words from raw. Cascade: pull, prep, embed, score, enrich,
prep, embed, score, extract, score, summarise. `app prep [unmapped]
[--remap]`, /api/prep/unmapped and a fold on the Sources page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:19:22 +02:00

62 lines
2.6 KiB
Python

"""Each host's own words, one module per reader, named as the reader is.
A module defines any of three dicts, keyed by the host's word lowercased:
- `SENIORITY`, values in `app.models.SENIORITIES`
- `EMPLOYMENT`, values in `app.models.EMPLOYMENTS`
- `REMOTE`, values in `app.models.REMOTE` (remote, hybrid, onsite)
A value of "" is a word the host uses that names no value ("Experienced",
"Permanent"): known, so it is not listed as unmapped.
A host whose employment word is a code of several parts (`permanent_full_time`,
`SALARIED_FT`) may add `MATCH`, `[(fragment, value), ...]`, tried in order
after `EMPLOYMENT` with the first hit winning, and must then say how a
fragment matches in `MATCH_MODE`: "substring" (the fragment is in the
lowercased word) or "token" (it equals one of the word's parts split at `_`
and `-`, so `ft` does not match inside `software`). `TABLES` loads every
module of this package by name, so a table is added by adding its file.
"""
from __future__ import annotations
import importlib
import pkgutil
from typing import Any
#: The dict a module names, by the field it maps.
FIELDS = {"seniority": "SENIORITY", "employment": "EMPLOYMENT", "remote": "REMOTE"}
#: How a `MATCH` fragment may match.
MATCH_MODES = ("substring", "token")
def load() -> dict[str, dict[str, Any]]:
"""`host -> field -> {word: value}` for every module of the package, the
words lowercased and stripped. A host without a table for a field has
no entry for it. A `MATCH` is `host -> "match" -> {"mode": ..., "rules":
[[fragment, value], ...]}`; a module with `MATCH` and no valid
`MATCH_MODE` fails the import by name."""
tables: dict[str, dict[str, Any]] = {}
for info in sorted(pkgutil.iter_modules(__path__), key=lambda i: i.name):
if info.name.startswith("_"):
continue
module = importlib.import_module(f"{__name__}.{info.name}")
for field, name in FIELDS.items():
table = getattr(module, name, None)
if isinstance(table, dict):
tables.setdefault(info.name, {})[field] = {
str(k).strip().lower(): str(v) for k, v in table.items()}
rules = getattr(module, "MATCH", None)
if rules is not None:
mode = getattr(module, "MATCH_MODE", None)
if mode not in MATCH_MODES:
raise ValueError(f"app/prep/hosts/{info.name}.py has MATCH and its MATCH_MODE "
f"is {mode!r}, not one of {', '.join(MATCH_MODES)}")
tables.setdefault(info.name, {})["match"] = {
"mode": mode, "rules": [[str(f).lower(), str(v)] for f, v in rules]}
return tables
TABLES = load()