Country lookup: every ISO country, the widest unit wins, several European countries are EU

country_of folds with _place_fold (ł, đ, ø, æ map to ASCII; _fold keeps building
company_key unchanged), knows every ISO country by English name and European
ones by their own, and the cities among the 150 most frequent unresolved
places on the live data. Within one location a country beats a US state or
Canadian province, which beats a city. A state code that is also an ISO code
counts as the state only when the line names a US place. Several countries,
all European, are EU; otherwise ''. A code in parentheses is read when no
name matched.

Migration 12 recomputes country from place for roles extract has not read
and whose board sent no country in raw. On the live data: open roles with no
country 2,827 -> 1,183 of 16,841; 1,705 open roles change, 48 from a code.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 15:06:15 +02:00
1 parent 706b127bc0
commit 2c4b992c50
6 files changed
+611 -109

No files matched your search

+32 -1
View File
@@ -17,7 +17,7 @@ import logging
import sqlite3
from collections.abc import Callable
from app.models import posted_at, posted_host
from app.models import country_of, posted_at, posted_host
log = logging.getLogger(__name__)
@@ -381,6 +381,8 @@ ALTER TABLE companies ADD COLUMN ats_slug TEXT NOT NULL DEFAULT '';
# its current rules is the whole point (see _recompute_function below).
(11, "roles.function recomputed for every not-yet-extracted role, word-boundary "
"classifier fix", lambda conn: _recompute_function(conn)),
(12, "roles.country recomputed from place for every not-yet-extracted role whose "
"board gave none, country lookup fix", lambda conn: _recompute_country(conn)),
)
#: What a file created from `SCHEMA` today is already at.
@@ -429,6 +431,35 @@ def _recompute_function(conn: sqlite3.Connection) -> None:
len(updates), len(rows))
def _recompute_country(conn: sqlite3.Connection) -> None:
"""Migration 12's step. `app.models.country_of` used to read a US state
code over a country's ("Bogota, CO" was US), return '' for a line naming
several European countries, and miss most countries outside Europe;
`Store.backfill_from_place` fills only an empty column, so a wrong code
would stay. Recomputed from `place` for every role `extract` has not
read (its answer only ever filled an empty column, but a role it read is
left as it is) and whose board sent no `country` of its own in `raw`
(the `search` plugin's Jobindex and Adzuna rows do), which is what
`Store.upsert_role` would write for that role today."""
rows = conn.execute(
"SELECT uid, place, raw, country FROM roles WHERE extracted_at IS NULL").fetchall()
updates = []
for uid, place, raw, old in rows:
try:
given = json.loads(raw or "{}").get("country") or ""
except (TypeError, ValueError, AttributeError):
given = ""
if given:
continue
new = country_of(place or "")
if new != old:
updates.append((new, uid))
if updates:
conn.executemany("UPDATE roles SET country = ? WHERE uid = ?", updates)
log.info("migration 12: recomputed country for %d of %d not-yet-extracted roles",
len(updates), len(rows))
def user_version(conn: sqlite3.Connection) -> int:
return int(conn.execute("PRAGMA user_version").fetchone()[0])