Date a role by posted_at, not just first_seen

recency and the newest feed both counted from first_seen, when the app
itself first stored a role, so a newly added board's whole backlog scored
as freshly posted for two weeks: 10,223 of 15,415 board roles were posted
more than 30 days before they were first seen (average gap in days: Ashby
78, Greenhouse 143, Lever 226, Personio 333).

Add roles.posted_at (migration 9): the earlier of first_seen and a trusted
host's published_at, where TRUSTED_POSTED_HOSTS (app/models.py) records
which of the ats and search hosts report a creation date rather than one
that moves on an edit, checked against live boards and the live database.
A future or pre-2015 date is not trusted either. upsert_role sets it on
insert and keeps the earliest on update, so an edit on the board can only
move it earlier, never later. recency, the feed's newest order, the "new
since" days filter and _top_scope's tie-break now read posted_at;
first_seen is unchanged and still what the UI shows as when the app found
a role. The row's age in roles.js now shows the posted date instead.

A partial index (posted_at DESC, uid) WHERE closed_at IS NULL serves the
newest feed; sqlite3's bundled build has no STAT4, so its planner
otherwise sorts every open role instead of using it, measured 30x slower
on the live database, so Store.roles() forces it for that query shape.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Sonnet committed 2026-09-29 14:13:06 +02:00
1 parent 0558ef19ed
commit 75eca1c914
8 files changed
+335 -33

No files matched your search

+81 -4
View File
@@ -228,6 +228,79 @@ STATUSES: tuple[str, ...] = get_args(Status)
AFFINITY = range(-1, 3)
# -------------------------------------------------------------- posting dates
#: Hosts whose date is a creation or first-publication timestamp, safe to
#: read as when a role was posted, checked against live boards on
#: 2026-09-29:
#:
#: - `greenhouse`: `plugins.ats._parse_greenhouse` already prefers
#: `first_published` over `updated_at`. A live board (adyen) showed a job
#: with `first_published` five months before `updated_at`, proving the
#: first stays fixed while the second moves on an edit — the resettable
#: field decisions.md used to warn against is `updated_at`, which this
#: reads only when `first_published` is missing (rare: 0 of 208 on that
#: board). The large real gaps this migration measured for Greenhouse
#: roles (average 143 days between this date and `first_seen`) are further
#: evidence it holds still rather than tracking every edit.
#: - `lever` (`createdAt`), `ashby` (`publishedAt`), `personio` (`createdAt`):
#: named and shaped as creation dates, with no "last edited" field in the
#: feed at all to reset from, and the same large real gaps as Greenhouse's.
#: - `smartrecruiters` (`releasedDate`), `recruitee` (`published_at`, falling
#: back to `created_at`): `releasedDate` reads the same way; a live
#: Recruitee board (bunq) showed `published_at` and `created_at` minutes
#: apart while `updated_at` had moved days later on an edited posting,
#: confirming the field this migration reads is not the one that moves.
#: - `workable` (`published`): named as a publish date, not an edit one, but
#: only one live board had postings to check (huggingface, 8 roles) and
#: none showed a second, "last touched" field to compare against, so this
#: is trusted on the name and shape alone.
#: - `jobindex`: the RSS `pubDate` a saved search returns. The live
#: database's 158 Jobindex-found roles have a `published_at`/`first_seen`
#: gap of 0.4 to 31 days (median 12), which reads as real, spread-out
#: posting ages rather than a date that jumps to "now" on every re-crawl.
#: - `adzuna`: `created`, an aggregator's own record-creation stamp — named
#: as creation, not modification, like the rest. No live check: this
#: deployment carries no Adzuna key (`plugins.search`'s own docstring:
#: "not yet run live") and no Adzuna subscription exists to fetch.
#:
#: A host not listed here, or a role whose stored `published_at` fails the
#: sanity check in `posted_at`, keeps `first_seen` as its posted date — the
#: safe default this migration is replacing everywhere else.
TRUSTED_POSTED_HOSTS: frozenset[str] = frozenset({
"greenhouse", "lever", "ashby", "workable", "smartrecruiters", "recruitee",
"personio", "jobindex", "adzuna",
})
#: A `published_at` before this is a parsing slip or a placeholder, not a
#: real posting date: no board here existed before 2015.
_POSTED_FLOOR = "2015-01-01T00:00:00+00:00"
def posted_host(raw: dict[str, Any]) -> str:
"""The host a role's dates came from, read the way `ats` and `search`
both put it in `Item.raw`: `board` for an ATS board (`greenhouse`,
`lever`, ...), `provider` for a cross-company search (`jobindex`,
`adzuna`). '' when neither plugin wrote one."""
return str(raw.get("board") or raw.get("provider") or "")
def posted_at(published_at: str, first_seen: str, host: str) -> str:
"""The date a role counts as posted: the earlier of `first_seen` (when
the app itself first stored it) and `published_at` (what the board
claims), but only when `host` is in `TRUSTED_POSTED_HOSTS` and
`published_at` is neither absurdly old nor later than `first_seen` — a
role cannot be posted after the app already saw it, trusted host or not,
which also catches a `published_at` the board reports in the future
(`first_seen` is never later than the moment this runs). Both dates are
`Store`'s own UTC, zero-padded `_iso` text, so comparing them as strings
is exact and needs no parsing. A role with no trusted date, or with one
that fails the check, keeps `first_seen`."""
if host in TRUSTED_POSTED_HOSTS and _POSTED_FLOOR <= published_at <= first_seen:
return published_at
return first_seen
def _json(value: Any, empty: str) -> Any:
"""A JSON text column as the Python value, tolerant like `JsonDict`: a
value that is already parsed passes, and one that does not parse is the
@@ -273,6 +346,10 @@ class Role(Model):
feed_text: str = ""
original_url: str = ""
published_at: str
#: The date this role counts as posted: see `posted_at`. Used for
#: `recency` and the feed's newest order; `first_seen` stays what it
#: always was, when the app itself first stored the role.
posted_at: str
first_seen: str
last_seen: str
closed_at: str | None = None
@@ -445,12 +522,12 @@ class Agent(Model):
class Candidate(Model):
"""What scoring reads about one open role for one agent: when it was
first seen, its company's affinity for that agent, and per gate `1`
match, `0` no match, `-1` the column is empty."""
"""What scoring reads about one open role for one agent: its posted
date (for `recency`), its company's affinity for that agent, and per
gate `1` match, `0` no match, `-1` the column is empty."""
uid: str
first_seen: str
posted_at: str
affinity: int = 0
gates: list[int] = []