upsert_role stores the host's own words (roles.*_host), the place texts unresolved when they change, and the company key unresolved on insert, and marks the role prep_due. app.prep maps seniority, employment, remote and country through the host tables in app/prep/hosts, resolves places and company aliases, lists unknown words in `unmapped`, remaps every open role when the tables' fingerprint changes, and runs rebuild_dupes, which moves out of the pull's settle. Migration 22 adds the columns and the table and fills the host words from raw. Cascade: pull, prep, embed, score, enrich, prep, embed, score, extract, score, summarise. `app prep [unmapped] [--remap]`, /api/prep/unmapped and a fold on the Sources page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
62 lines
2.6 KiB
Python
62 lines
2.6 KiB
Python
"""Each host's own words, one module per reader, named as the reader is.
|
|
|
|
A module defines any of three dicts, keyed by the host's word lowercased:
|
|
|
|
- `SENIORITY`, values in `app.models.SENIORITIES`
|
|
- `EMPLOYMENT`, values in `app.models.EMPLOYMENTS`
|
|
- `REMOTE`, values in `app.models.REMOTE` (remote, hybrid, onsite)
|
|
|
|
A value of "" is a word the host uses that names no value ("Experienced",
|
|
"Permanent"): known, so it is not listed as unmapped.
|
|
|
|
A host whose employment word is a code of several parts (`permanent_full_time`,
|
|
`SALARIED_FT`) may add `MATCH`, `[(fragment, value), ...]`, tried in order
|
|
after `EMPLOYMENT` with the first hit winning, and must then say how a
|
|
fragment matches in `MATCH_MODE`: "substring" (the fragment is in the
|
|
lowercased word) or "token" (it equals one of the word's parts split at `_`
|
|
and `-`, so `ft` does not match inside `software`). `TABLES` loads every
|
|
module of this package by name, so a table is added by adding its file.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib
|
|
import pkgutil
|
|
from typing import Any
|
|
|
|
#: The dict a module names, by the field it maps.
|
|
FIELDS = {"seniority": "SENIORITY", "employment": "EMPLOYMENT", "remote": "REMOTE"}
|
|
|
|
#: How a `MATCH` fragment may match.
|
|
MATCH_MODES = ("substring", "token")
|
|
|
|
|
|
def load() -> dict[str, dict[str, Any]]:
|
|
"""`host -> field -> {word: value}` for every module of the package, the
|
|
words lowercased and stripped. A host without a table for a field has
|
|
no entry for it. A `MATCH` is `host -> "match" -> {"mode": ..., "rules":
|
|
[[fragment, value], ...]}`; a module with `MATCH` and no valid
|
|
`MATCH_MODE` fails the import by name."""
|
|
tables: dict[str, dict[str, Any]] = {}
|
|
for info in sorted(pkgutil.iter_modules(__path__), key=lambda i: i.name):
|
|
if info.name.startswith("_"):
|
|
continue
|
|
module = importlib.import_module(f"{__name__}.{info.name}")
|
|
for field, name in FIELDS.items():
|
|
table = getattr(module, name, None)
|
|
if isinstance(table, dict):
|
|
tables.setdefault(info.name, {})[field] = {
|
|
str(k).strip().lower(): str(v) for k, v in table.items()}
|
|
rules = getattr(module, "MATCH", None)
|
|
if rules is not None:
|
|
mode = getattr(module, "MATCH_MODE", None)
|
|
if mode not in MATCH_MODES:
|
|
raise ValueError(f"app/prep/hosts/{info.name}.py has MATCH and its MATCH_MODE "
|
|
f"is {mode!r}, not one of {', '.join(MATCH_MODES)}")
|
|
tables.setdefault(info.name, {})["match"] = {
|
|
"mode": mode, "rules": [[str(f).lower(), str(v)] for f, v in rules]}
|
|
return tables
|
|
|
|
|
|
TABLES = load()
|