Count a page of ids already read as a stall only where the host states no total or the reader hands the page over empty, so a resumed sweep reads on when the list grew at the top
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
7f37270eb3
commit
c4c17fb019
3 files changed
+39
-23
No files matched your search
@@ -556,10 +556,11 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
|
||||
A whole listing is complete whenever it parses. A paged one reads each
|
||||
series' pages, `limits.page_delay` apart, until the host has no next
|
||||
page, the stated total is reached, a page brings no new id while the
|
||||
reader still hands a cursor (a stall: a host that ignores the cursor or
|
||||
sends a page again, with or without a total) or `limits.max_pages`
|
||||
pages in all. A series is complete when it read as many distinct result
|
||||
page, the stated total is reached, the read stalls or `limits.max_pages`
|
||||
pages in all. A stall is a page that brings no new id while the reader
|
||||
still hands a cursor: any such page on a host that states no total (it
|
||||
ignores the cursor), and with a total one the reader hands over empty
|
||||
(it saw the page come again). A series is complete when it read as many distinct result
|
||||
ids as its stated total, or, without a total, when it ended on the
|
||||
host's last page; never after a stall, and never for
|
||||
`complete="never"`. The read is complete when every series is. A 429
|
||||
@@ -636,10 +637,13 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
if cursor is None or (isinstance(total, int) and len(seen) >= total):
|
||||
ended = True
|
||||
break
|
||||
# Nothing new while the reader says more follow: the host ignores
|
||||
# the cursor or sent a page again. A stall, whatever the total: the
|
||||
# series ends incomplete rather than loop to the page cap.
|
||||
if len(seen) == before:
|
||||
# Nothing new while the reader says more follow. Without a total
|
||||
# the host ignores the cursor, and the read would never end. With
|
||||
# one, only a page the reader hands over empty is a stall (it saw
|
||||
# the page come again, as SuccessFactors does): a page of ids
|
||||
# already read is what a resumed sweep starts on when the list
|
||||
# grew at the top since, and the read goes on from there.
|
||||
if len(seen) == before and (total is None or not results):
|
||||
stalled = True
|
||||
break
|
||||
if stop:
|
||||
|
||||
Reference in new issue
Block a user