Count a page of ids already read as a stall only where the host states no total or the reader hands the page over empty, so a resumed sweep reads on when the list grew at the top

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-10-05 09:56:06 +02:00
1 parent 7f37270eb3
commit c4c17fb019
3 files changed
+39 -23

No files matched your search

+12 -8
View File
@@ -556,10 +556,11 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
A whole listing is complete whenever it parses. A paged one reads each
series' pages, `limits.page_delay` apart, until the host has no next
page, the stated total is reached, a page brings no new id while the
reader still hands a cursor (a stall: a host that ignores the cursor or
sends a page again, with or without a total) or `limits.max_pages`
pages in all. A series is complete when it read as many distinct result
page, the stated total is reached, the read stalls or `limits.max_pages`
pages in all. A stall is a page that brings no new id while the reader
still hands a cursor: any such page on a host that states no total (it
ignores the cursor), and with a total one the reader hands over empty
(it saw the page come again). A series is complete when it read as many distinct result
ids as its stated total, or, without a total, when it ended on the
host's last page; never after a stall, and never for
`complete="never"`. The read is complete when every series is. A 429
@@ -636,10 +637,13 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
if cursor is None or (isinstance(total, int) and len(seen) >= total):
ended = True
break
# Nothing new while the reader says more follow: the host ignores
# the cursor or sent a page again. A stall, whatever the total: the
# series ends incomplete rather than loop to the page cap.
if len(seen) == before:
# Nothing new while the reader says more follow. Without a total
# the host ignores the cursor, and the read would never end. With
# one, only a page the reader hands over empty is a stall (it saw
# the page come again, as SuccessFactors does): a page of ids
# already read is what a resumed sweep starts on when the list
# grew at the top since, and the read goes on from there.
if len(seen) == before and (total is None or not results):
stalled = True
break
if stop: