Unpack structured pay from five ATS hosts; fix the function classifier
Part A: Ashby (?includeCompensation=true), Greenhouse (pay_transparency=true, already free on the list request), Lever (salaryRange), Recruitee (salary) and Personio (salaryInformation) all give structured pay; Workable and SmartRecruiters do not (checked live). Every host converts to a yearly salary_min/salary_max/currency, multiplying a monthly figure by 12 and skipping an hourly or daily one rather than annualising a number nobody stated. A range with no currency is ignored, and Greenhouse's own multi-zone ranges are skipped outright since nothing says which zone the role is at. Part B: classify() matched substrings, so " it" matched inside "Italy", filing European sales roles as engineering; it now matches on word boundaries (full boundary for the handful of short tokens that are themselves the start of an unrelated word, prefix matching kept for the stems). Title is checked before department (a hand-checked sample found title-first right about twice as often), nothing matching returns '' rather than "other", and a modest set of German/Dutch/Danish/Norwegian/Swedish/ French keywords covers the seed's other languages. function is not a board field, so it gets the opposite rule from every other raw column: extract's answer overwrites the classifier's guess outright, and once extracted_at is set no later pull may put the guess back. Migration 9 recomputes function for every not-yet-extracted role from its stored title and department. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0558ef19ed
commit
44ff7c0131
10 files changed
+458
-35
No files matched your search
+42
-1
@@ -62,7 +62,7 @@ def test_schema_creates_on_an_empty_file(tmp_path: Path) -> None:
|
||||
"application_events", "agents", "scores", "state"} <= tables
|
||||
assert "idx_roles_cluster" in indexes
|
||||
assert store.migrations_applied == []
|
||||
assert LATEST == 8
|
||||
assert LATEST == 9
|
||||
# A second open finds it current and applies nothing.
|
||||
assert Store(path).migrations_applied == []
|
||||
|
||||
@@ -259,6 +259,47 @@ def test_upsert_unpacks_raw_and_keeps_filled_columns(store: Store) -> None:
|
||||
assert (role.place, role.remote, role.salary_min) == ("Berlin", "remote", 80000)
|
||||
|
||||
|
||||
def test_function_is_the_classifiers_guess_not_a_board_field(store: Store) -> None:
|
||||
"""`function` behaves the opposite of every other raw field: a pull may
|
||||
only set it before extract has run, and once extract has answered — even
|
||||
with a different function than the pull's guess — no later pull may put
|
||||
the guess back. The no-flip property across pull, extract, pull."""
|
||||
store.upsert_role(posting(function="engineering"))
|
||||
uid = posting().uid
|
||||
assert store.role(uid).function == "engineering" # type: ignore[union-attr]
|
||||
|
||||
store.set_extracted(uid, {"function": "sales"})
|
||||
assert store.role(uid).function == "sales" # type: ignore[union-attr]
|
||||
|
||||
# A later pull with the classifier's old (now wrong) guess must not flip
|
||||
# it back — the "board wins" COALESCE that every other raw field gets
|
||||
# would otherwise put "engineering" straight back on every pull.
|
||||
store.upsert_role(posting(function="engineering"))
|
||||
assert store.role(uid).function == "sales" # type: ignore[union-attr]
|
||||
|
||||
|
||||
def test_function_is_frozen_once_extracted_even_with_no_answer(store: Store) -> None:
|
||||
store.upsert_role(posting(function="engineering"))
|
||||
uid = posting().uid
|
||||
store.set_extracted(uid, {}) # extract answered this role, but not function
|
||||
assert store.role(uid).extracted_at is not None # type: ignore[union-attr]
|
||||
|
||||
# extracted_at alone freezes it, whatever a later pull's guess says.
|
||||
store.upsert_role(posting(function="sales"))
|
||||
assert store.role(uid).function == "engineering" # type: ignore[union-attr]
|
||||
|
||||
|
||||
def test_function_is_set_from_raw_before_extract_like_any_other_field(store: Store) -> None:
|
||||
store.upsert_role(posting(function="engineering"))
|
||||
role = store.role(posting().uid)
|
||||
assert role is not None and role.function == "engineering"
|
||||
# Before extract, a fresh guess still overwrites the old one, the same
|
||||
# "board wins" behaviour as place, country and the rest.
|
||||
store.upsert_role(posting(function="sales"))
|
||||
role = store.role(posting().uid)
|
||||
assert role is not None and role.function == "sales"
|
||||
|
||||
|
||||
def test_company_falls_back_to_the_subscription(store: Store) -> None:
|
||||
sub = store.add_subscription("ats", "greenhouse:acme", company="Acme", board="Greenhouse")
|
||||
store.upsert_role(posting(), sub)
|
||||
|
||||
Reference in new issue
Block a user