Embed a role again when its text arrives, and keep the place its own page gave

A role listed before its page was read kept a title-only vector, and the listing's place replaced the page's on every later pull.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-29 22:23:43 +02:00
1 parent fd89381caa
commit 31abad4c8e
5 files changed
+69 -6

No files matched your search

+16 -3
View File
@@ -618,7 +618,9 @@ DETAIL_CAP = 200
#: `location` (`place`, `place_key`, `country`), `remote` and `currency` are
#: `upsert_role`'s keep-filled columns, `salary_min` and `salary_max` are only
#: overwritten by a figure, and `posted` feeds `posted_at`, which keeps the
#: earliest date a role ever had. `title` and `url` are not here, since the
#: earliest date a role ever had. A filled column is still overwritten by
#: another filled value, so `_read_details` withholds the listing's place and
#: remote for a role already read. `title` and `url` are not here, since the
#: listing always states them and would overwrite them; nor are `department`
#: (no column of its own, only `raw` and the classifier's guess) and
#: `salary_text`, both rewritten from the listing on every pull.
@@ -641,7 +643,10 @@ def _read_details(client: httpx.Client, target: str, rows: list[Row], complete:
is left out with a debug line. Every row not read in detail in this call
goes out with an empty description: a role read on an earlier pull keeps
the one it has, since `upsert_role` never overwrites a filled column with
an empty value, and a teaser from the listing must not replace it.
an empty value, and a teaser from the listing must not replace it. A role
read on an earlier pull goes out without the listing's place and remote
as well, for the same reason: the listing states them every pull
("2 Locations"), and would replace what the role's own page said.
A detail that answers 404 or 410 counts as read: the role is gone or has
no page of its own, and asking again would not change that. One that
@@ -691,7 +696,15 @@ def _read_details(client: httpx.Client, target: str, rows: list[Row], complete:
filled[key] = {k: v for k, v in answer.items() if k in DETAIL_FIELDS}
done.add(key)
failing.discard(key)
out = [row | {"description": ""} | filled.get(compound(row), {}) for row in rows]
out = []
for row in rows:
key = compound(row)
if key in filled:
out.append(row | {"description": ""} | filled[key])
elif key in detailed:
out.append(row | {"description": "", "location": "", "remote": None})
else:
out.append(row | {"description": ""})
if complete:
listed = {compound(row) for row in rows}
done &= listed
+2
View File
@@ -113,6 +113,8 @@ listing's empty value; any other key is left out with a debug log line.
on a pull is sent with an empty description, so the one read earlier stays.
A 404 or a 410 counts as read; any other failure is logged and retried on a
later pull; a 429 ends the detail reads of that board for the pull.
A role read in detail on an earlier pull is sent without the
listing's place and remote, so what its own page said about them stays.
Completeness is decided by the listing alone.
**robots.txt.** With `robots_page(slug)`, the shared code reads