Embed a role again when its text arrives, and keep the place its own page gave
A role listed before its page was read kept a title-only vector, and the listing's place replaced the page's on every later pull. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
fd89381caa
commit
31abad4c8e
5 files changed
+69
-6
No files matched your search
+16
-3
@@ -618,7 +618,9 @@ DETAIL_CAP = 200
|
||||
#: `location` (`place`, `place_key`, `country`), `remote` and `currency` are
|
||||
#: `upsert_role`'s keep-filled columns, `salary_min` and `salary_max` are only
|
||||
#: overwritten by a figure, and `posted` feeds `posted_at`, which keeps the
|
||||
#: earliest date a role ever had. `title` and `url` are not here, since the
|
||||
#: earliest date a role ever had. A filled column is still overwritten by
|
||||
#: another filled value, so `_read_details` withholds the listing's place and
|
||||
#: remote for a role already read. `title` and `url` are not here, since the
|
||||
#: listing always states them and would overwrite them; nor are `department`
|
||||
#: (no column of its own, only `raw` and the classifier's guess) and
|
||||
#: `salary_text`, both rewritten from the listing on every pull.
|
||||
@@ -641,7 +643,10 @@ def _read_details(client: httpx.Client, target: str, rows: list[Row], complete:
|
||||
is left out with a debug line. Every row not read in detail in this call
|
||||
goes out with an empty description: a role read on an earlier pull keeps
|
||||
the one it has, since `upsert_role` never overwrites a filled column with
|
||||
an empty value, and a teaser from the listing must not replace it.
|
||||
an empty value, and a teaser from the listing must not replace it. A role
|
||||
read on an earlier pull goes out without the listing's place and remote
|
||||
as well, for the same reason: the listing states them every pull
|
||||
("2 Locations"), and would replace what the role's own page said.
|
||||
|
||||
A detail that answers 404 or 410 counts as read: the role is gone or has
|
||||
no page of its own, and asking again would not change that. One that
|
||||
@@ -691,7 +696,15 @@ def _read_details(client: httpx.Client, target: str, rows: list[Row], complete:
|
||||
filled[key] = {k: v for k, v in answer.items() if k in DETAIL_FIELDS}
|
||||
done.add(key)
|
||||
failing.discard(key)
|
||||
out = [row | {"description": ""} | filled.get(compound(row), {}) for row in rows]
|
||||
out = []
|
||||
for row in rows:
|
||||
key = compound(row)
|
||||
if key in filled:
|
||||
out.append(row | {"description": ""} | filled[key])
|
||||
elif key in detailed:
|
||||
out.append(row | {"description": "", "location": "", "remote": None})
|
||||
else:
|
||||
out.append(row | {"description": ""})
|
||||
if complete:
|
||||
listed = {compound(row) for row in rows}
|
||||
done &= listed
|
||||
|
||||
@@ -113,6 +113,8 @@ listing's empty value; any other key is left out with a debug log line.
|
||||
on a pull is sent with an empty description, so the one read earlier stays.
|
||||
A 404 or a 410 counts as read; any other failure is logged and retried on a
|
||||
later pull; a 429 ends the detail reads of that board for the pull.
|
||||
A role read in detail on an earlier pull is sent without the
|
||||
listing's place and remote, so what its own page said about them stays.
|
||||
Completeness is decided by the listing alone.
|
||||
|
||||
**robots.txt.** With `robots_page(slug)`, the shared code reads
|
||||
|
||||
Reference in new issue
Block a user