Keep the sweep of a reader whose cursor is a tuple, so Workday and SuccessFactors boards over 100 pages finish across pulls, and end a read incomplete on a page with nothing new while a cursor follows, whatever the total, which SuccessFactors now hands back on a repeated page
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
85aed0807b
commit
7f37270eb3
6 files changed
+219
-18
No files matched your search
+29
-10
@@ -556,17 +556,20 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
|
||||
A whole listing is complete whenever it parses. A paged one reads each
|
||||
series' pages, `limits.page_delay` apart, until the host has no next
|
||||
page, the stated total is reached, a page brings nothing new (a host
|
||||
that ignores the cursor, incomplete) or `limits.max_pages` pages in all.
|
||||
A series is complete when it read as many distinct result ids as its
|
||||
stated total, or, without a total, when it ended on the host's last
|
||||
page; never for `complete="never"`. The read is complete when every
|
||||
series is. A 429 after the first page ends the read, incomplete.
|
||||
page, the stated total is reached, a page brings no new id while the
|
||||
reader still hands a cursor (a stall: a host that ignores the cursor or
|
||||
sends a page again, with or without a total) or `limits.max_pages`
|
||||
pages in all. A series is complete when it read as many distinct result
|
||||
ids as its stated total, or, without a total, when it ended on the
|
||||
host's last page; never after a stall, and never for
|
||||
`complete="never"`. The read is complete when every series is. A 429
|
||||
after the first page ends the read, incomplete.
|
||||
|
||||
With `can_sweep` (boards), a read cut at `max_pages` or by a 429 returns
|
||||
the `sweep` to keep, and the next pull, handed it back, continues from
|
||||
its series and cursor; the sweep is complete when it reaches the end,
|
||||
and `ids` then hold every id it listed, and the listing is `swept`. A
|
||||
and `ids` then hold every id it listed, and the listing is `swept`. The
|
||||
cursor comes back as JSON gives it, a tuple as a list (`_storable`). A
|
||||
read at the reader's `ceiling` is not swept: the host answers nothing
|
||||
past it.
|
||||
|
||||
@@ -633,7 +636,10 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
if cursor is None or (isinstance(total, int) and len(seen) >= total):
|
||||
ended = True
|
||||
break
|
||||
if total is None and len(seen) == before:
|
||||
# Nothing new while the reader says more follow: the host ignores
|
||||
# the cursor or sent a page again. A stall, whatever the total: the
|
||||
# series ends incomplete rather than loop to the page cap.
|
||||
if len(seen) == before:
|
||||
stalled = True
|
||||
break
|
||||
if stop:
|
||||
@@ -663,13 +669,26 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
|
||||
|
||||
def _storable(cursor: Any) -> bool:
|
||||
"""Whether a cursor survives a trip through JSON in state."""
|
||||
"""Whether a cursor survives a trip through JSON in state. A tuple comes
|
||||
back as a list (Workday's `(offset, total)`, SuccessFactors' `(start
|
||||
row, first id)`), which counts as surviving: the next pull hands the
|
||||
reader its cursor as JSON gives it back, and every paged reader takes
|
||||
the list as it took the tuple."""
|
||||
try:
|
||||
return json.loads(json.dumps(cursor)) == cursor
|
||||
return json.loads(json.dumps(cursor)) == _as_json(cursor)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _as_json(value: Any) -> Any:
|
||||
"""`value` with every tuple in it a list, as JSON hands it back."""
|
||||
if isinstance(value, tuple | list):
|
||||
return [_as_json(item) for item in value]
|
||||
if isinstance(value, dict):
|
||||
return {key: _as_json(item) for key, item in value.items()}
|
||||
return value
|
||||
|
||||
|
||||
# ------------------------------------------------------------ with state
|
||||
|
||||
def _load(state: Mapping[str, str], key: str) -> dict[str, Any]:
|
||||
|
||||
Reference in new issue
Block a user