Country lookup: every ISO country, the widest unit wins, several European countries are EU
country_of folds with _place_fold (ł, đ, ø, æ map to ASCII; _fold keeps building company_key unchanged), knows every ISO country by English name and European ones by their own, and the cities among the 150 most frequent unresolved places on the live data. Within one location a country beats a US state or Canadian province, which beats a city. A state code that is also an ISO code counts as the state only when the line names a US place. Several countries, all European, are EU; otherwise ''. A code in parentheses is read when no name matched. Migration 12 recomputes country from place for roles extract has not read and whose board sent no country in raw. On the live data: open roles with no country 2,827 -> 1,183 of 16,841; 1,705 open roles change, 48 from a code. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
706b127bc0
commit
2c4b992c50
6 files changed
+611
-109
No files matched your search
+32
-1
@@ -17,7 +17,7 @@ import logging
|
||||
import sqlite3
|
||||
from collections.abc import Callable
|
||||
|
||||
from app.models import posted_at, posted_host
|
||||
from app.models import country_of, posted_at, posted_host
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -381,6 +381,8 @@ ALTER TABLE companies ADD COLUMN ats_slug TEXT NOT NULL DEFAULT '';
|
||||
# its current rules is the whole point (see _recompute_function below).
|
||||
(11, "roles.function recomputed for every not-yet-extracted role, word-boundary "
|
||||
"classifier fix", lambda conn: _recompute_function(conn)),
|
||||
(12, "roles.country recomputed from place for every not-yet-extracted role whose "
|
||||
"board gave none, country lookup fix", lambda conn: _recompute_country(conn)),
|
||||
)
|
||||
|
||||
#: What a file created from `SCHEMA` today is already at.
|
||||
@@ -429,6 +431,35 @@ def _recompute_function(conn: sqlite3.Connection) -> None:
|
||||
len(updates), len(rows))
|
||||
|
||||
|
||||
def _recompute_country(conn: sqlite3.Connection) -> None:
|
||||
"""Migration 12's step. `app.models.country_of` used to read a US state
|
||||
code over a country's ("Bogota, CO" was US), return '' for a line naming
|
||||
several European countries, and miss most countries outside Europe;
|
||||
`Store.backfill_from_place` fills only an empty column, so a wrong code
|
||||
would stay. Recomputed from `place` for every role `extract` has not
|
||||
read (its answer only ever filled an empty column, but a role it read is
|
||||
left as it is) and whose board sent no `country` of its own in `raw`
|
||||
(the `search` plugin's Jobindex and Adzuna rows do), which is what
|
||||
`Store.upsert_role` would write for that role today."""
|
||||
rows = conn.execute(
|
||||
"SELECT uid, place, raw, country FROM roles WHERE extracted_at IS NULL").fetchall()
|
||||
updates = []
|
||||
for uid, place, raw, old in rows:
|
||||
try:
|
||||
given = json.loads(raw or "{}").get("country") or ""
|
||||
except (TypeError, ValueError, AttributeError):
|
||||
given = ""
|
||||
if given:
|
||||
continue
|
||||
new = country_of(place or "")
|
||||
if new != old:
|
||||
updates.append((new, uid))
|
||||
if updates:
|
||||
conn.executemany("UPDATE roles SET country = ? WHERE uid = ?", updates)
|
||||
log.info("migration 12: recomputed country for %d of %d not-yet-extracted roles",
|
||||
len(updates), len(rows))
|
||||
|
||||
|
||||
def user_version(conn: sqlite3.Connection) -> int:
|
||||
return int(conn.execute("PRAGMA user_version").fetchone()[0])
|
||||
|
||||
|
||||
Reference in new issue
Block a user