Unpack structured pay from five ATS hosts; fix the function classifier

Part A: Ashby (?includeCompensation=true), Greenhouse (pay_transparency=true,
already free on the list request), Lever (salaryRange), Recruitee (salary)
and Personio (salaryInformation) all give structured pay; Workable and
SmartRecruiters do not (checked live). Every host converts to a yearly
salary_min/salary_max/currency, multiplying a monthly figure by 12 and
skipping an hourly or daily one rather than annualising a number nobody
stated. A range with no currency is ignored, and Greenhouse's own multi-zone
ranges are skipped outright since nothing says which zone the role is at.

Part B: classify() matched substrings, so " it" matched inside "Italy",
filing European sales roles as engineering; it now matches on word
boundaries (full boundary for the handful of short tokens that are
themselves the start of an unrelated word, prefix matching kept for the
stems). Title is checked before department (a hand-checked sample found
title-first right about twice as often), nothing matching returns '' rather
than "other", and a modest set of German/Dutch/Danish/Norwegian/Swedish/
French keywords covers the seed's other languages.

function is not a board field, so it gets the opposite rule from every
other raw column: extract's answer overwrites the classifier's guess
outright, and once extracted_at is set no later pull may put the guess
back. Migration 9 recomputes function for every not-yet-extracted role from
its stored title and department.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Sonnet 5 committed 2026-09-29 14:29:12 +02:00
1 parent 0558ef19ed
commit 44ff7c0131
10 files changed
+458 -35

No files matched your search

+47
View File
@@ -12,6 +12,7 @@ shipped.
from __future__ import annotations
import json
import logging
import sqlite3
from collections.abc import Callable
@@ -338,12 +339,58 @@ ALTER TABLE companies ADD COLUMN city TEXT NOT NULL DEFAULT '';
(8, "agents.limits: how deep into its ranking the model stages read is each agent's own", """
ALTER TABLE agents ADD COLUMN limits TEXT NOT NULL DEFAULT '{}';
"""),
# A Python step, not SQL: classify() lives in plugins/ats.py, and running
# its current rules is the whole point (see _recompute_function below).
(9, "roles.function recomputed for every not-yet-extracted role, word-boundary "
"classifier fix", lambda conn: _recompute_function(conn)),
)
#: What a file created from `SCHEMA` today is already at.
LATEST = MIGRATIONS[-1][0]
def _recompute_function(conn: sqlite3.Connection) -> None:
"""Migration 9's step. `roles.function` used to be filled by a keyword
classifier that matched substrings (" it" inside "Italy"), never
returned '' (only "other", outside the declared vocabulary) and checked
department before title; `plugins.ats.classify` no longer does any of
that (see its own docstring). A role `extract` has already read keeps
whatever it decided — `function` is extract's from then on, see
`Store.set_extracted` — so only roles with no `extracted_at` are
recomputed here, from their stored title and department (department
lives in `raw`, not its own column: docs/decisions.md, "raw is
transport, columns are storage").
A Python step because the classifier is Python, not SQL; `plugins` is a
top-level package outside `app`'s own tree, so it is reached the same
way `app/jobs/extract.py` reaches `FUNCTIONS`: inserted onto `sys.path`
rather than imported as an installed dependency, since `plugins` is not
in `pyproject.toml`'s packages."""
import sys
from pathlib import Path
root = Path(__file__).resolve().parents[2]
if str(root) not in sys.path:
sys.path.insert(0, str(root))
from plugins.ats import classify
rows = conn.execute(
"SELECT uid, title, raw, function FROM roles WHERE extracted_at IS NULL").fetchall()
updates = []
for uid, title, raw, old in rows:
try:
department = json.loads(raw or "{}").get("department") or ""
except (TypeError, ValueError):
department = ""
new = classify(department, title)
if new != old:
updates.append((new, uid))
if updates:
conn.executemany("UPDATE roles SET function = ? WHERE uid = ?", updates)
log.info("migration 9: recomputed function for %d of %d not-yet-extracted roles",
len(updates), len(rows))
def user_version(conn: sqlite3.Connection) -> int:
return int(conn.execute("PRAGMA user_version").fetchone()[0])