Embed a role again when its text arrives, and keep the place its own page gave

A role listed before its page was read kept a title-only vector, and the listing's place replaced the page's on every later pull.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-29 22:23:43 +02:00
1 parent fd89381caa
commit 31abad4c8e
5 files changed
+69 -6

No files matched your search

+16 -3
View File
@@ -618,7 +618,9 @@ DETAIL_CAP = 200
#: `location` (`place`, `place_key`, `country`), `remote` and `currency` are
#: `upsert_role`'s keep-filled columns, `salary_min` and `salary_max` are only
#: overwritten by a figure, and `posted` feeds `posted_at`, which keeps the
#: earliest date a role ever had. `title` and `url` are not here, since the
#: earliest date a role ever had. A filled column is still overwritten by
#: another filled value, so `_read_details` withholds the listing's place and
#: remote for a role already read. `title` and `url` are not here, since the
#: listing always states them and would overwrite them; nor are `department`
#: (no column of its own, only `raw` and the classifier's guess) and
#: `salary_text`, both rewritten from the listing on every pull.
@@ -641,7 +643,10 @@ def _read_details(client: httpx.Client, target: str, rows: list[Row], complete:
is left out with a debug line. Every row not read in detail in this call
goes out with an empty description: a role read on an earlier pull keeps
the one it has, since `upsert_role` never overwrites a filled column with
an empty value, and a teaser from the listing must not replace it.
an empty value, and a teaser from the listing must not replace it. A role
read on an earlier pull goes out without the listing's place and remote
as well, for the same reason: the listing states them every pull
("2 Locations"), and would replace what the role's own page said.
A detail that answers 404 or 410 counts as read: the role is gone or has
no page of its own, and asking again would not change that. One that
@@ -691,7 +696,15 @@ def _read_details(client: httpx.Client, target: str, rows: list[Row], complete:
filled[key] = {k: v for k, v in answer.items() if k in DETAIL_FIELDS}
done.add(key)
failing.discard(key)
out = [row | {"description": ""} | filled.get(compound(row), {}) for row in rows]
out = []
for row in rows:
key = compound(row)
if key in filled:
out.append(row | {"description": ""} | filled[key])
elif key in detailed:
out.append(row | {"description": "", "location": "", "remote": None})
else:
out.append(row | {"description": ""})
if complete:
listed = {compound(row) for row in rows}
done &= listed