Prep stage: host words stored raw, mapped centrally in app/prep after every pull
upsert_role stores the host's own words (roles.*_host), the place texts unresolved when they change, and the company key unresolved on insert, and marks the role prep_due. app.prep maps seniority, employment, remote and country through the host tables in app/prep/hosts, resolves places and company aliases, lists unknown words in `unmapped`, remaps every open role when the tables' fingerprint changes, and runs rebuild_dupes, which moves out of the pull's settle. Migration 22 adds the columns and the table and fills the host words from raw. Cascade: pull, prep, embed, score, enrich, prep, embed, score, extract, score, summarise. `app prep [unmapped] [--remap]`, /api/prep/unmapped and a fold on the Sources page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a66aac95cf
commit
d04ceae7e0
31 files changed
+1468
-330
No files matched your search
@@ -0,0 +1,61 @@
|
||||
"""Each host's own words, one module per reader, named as the reader is.
|
||||
|
||||
A module defines any of three dicts, keyed by the host's word lowercased:
|
||||
|
||||
- `SENIORITY`, values in `app.models.SENIORITIES`
|
||||
- `EMPLOYMENT`, values in `app.models.EMPLOYMENTS`
|
||||
- `REMOTE`, values in `app.models.REMOTE` (remote, hybrid, onsite)
|
||||
|
||||
A value of "" is a word the host uses that names no value ("Experienced",
|
||||
"Permanent"): known, so it is not listed as unmapped.
|
||||
|
||||
A host whose employment word is a code of several parts (`permanent_full_time`,
|
||||
`SALARIED_FT`) may add `MATCH`, `[(fragment, value), ...]`, tried in order
|
||||
after `EMPLOYMENT` with the first hit winning, and must then say how a
|
||||
fragment matches in `MATCH_MODE`: "substring" (the fragment is in the
|
||||
lowercased word) or "token" (it equals one of the word's parts split at `_`
|
||||
and `-`, so `ft` does not match inside `software`). `TABLES` loads every
|
||||
module of this package by name, so a table is added by adding its file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import pkgutil
|
||||
from typing import Any
|
||||
|
||||
#: The dict a module names, by the field it maps.
|
||||
FIELDS = {"seniority": "SENIORITY", "employment": "EMPLOYMENT", "remote": "REMOTE"}
|
||||
|
||||
#: How a `MATCH` fragment may match.
|
||||
MATCH_MODES = ("substring", "token")
|
||||
|
||||
|
||||
def load() -> dict[str, dict[str, Any]]:
|
||||
"""`host -> field -> {word: value}` for every module of the package, the
|
||||
words lowercased and stripped. A host without a table for a field has
|
||||
no entry for it. A `MATCH` is `host -> "match" -> {"mode": ..., "rules":
|
||||
[[fragment, value], ...]}`; a module with `MATCH` and no valid
|
||||
`MATCH_MODE` fails the import by name."""
|
||||
tables: dict[str, dict[str, Any]] = {}
|
||||
for info in sorted(pkgutil.iter_modules(__path__), key=lambda i: i.name):
|
||||
if info.name.startswith("_"):
|
||||
continue
|
||||
module = importlib.import_module(f"{__name__}.{info.name}")
|
||||
for field, name in FIELDS.items():
|
||||
table = getattr(module, name, None)
|
||||
if isinstance(table, dict):
|
||||
tables.setdefault(info.name, {})[field] = {
|
||||
str(k).strip().lower(): str(v) for k, v in table.items()}
|
||||
rules = getattr(module, "MATCH", None)
|
||||
if rules is not None:
|
||||
mode = getattr(module, "MATCH_MODE", None)
|
||||
if mode not in MATCH_MODES:
|
||||
raise ValueError(f"app/prep/hosts/{info.name}.py has MATCH and its MATCH_MODE "
|
||||
f"is {mode!r}, not one of {', '.join(MATCH_MODES)}")
|
||||
tables.setdefault(info.name, {})["match"] = {
|
||||
"mode": mode, "rules": [[str(f).lower(), str(v)] for f, v in rules]}
|
||||
return tables
|
||||
|
||||
|
||||
TABLES = load()
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Platsbanken's words, for prep."""
|
||||
|
||||
#: `workplace_model.label`, in Swedish; mostly empty. Only "Arbete på plats"
|
||||
#: is in the fixtures; the others are JobTech's other labels as the site
|
||||
#: writes them, and a label not here is listed as unmapped.
|
||||
REMOTE: dict[str, str] = {
|
||||
"arbete på plats": "onsite", "på plats": "onsite",
|
||||
"hybridarbete": "hybrid", "hybrid": "hybrid", "delvis på distans": "hybrid",
|
||||
"delvis distans": "hybrid",
|
||||
"distansarbete": "remote", "arbete på distans": "remote", "helt på distans": "remote",
|
||||
"distans": "remote",
|
||||
}
|
||||
Reference in new issue
Block a user