Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s

_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 15:12:14 +02:00
1 parent 2c4b992c50
commit 3abb44e10a
4 files changed
+126 -22

No files matched your search

+7 -3
View File
@@ -47,9 +47,13 @@ from app.storage import Store
log = logging.getLogger(__name__)
#: As many boards in the air per job as there are `http` workers by default
#: (see `config.yaml`), so one pull keeps the pool busy without one job
#: holding all 190-odd boards' worth of retries against a single attempt.
#: Targets per `pull_batch` job. A batch reads its boards one after another,
#: so the parallelism is the number of batches the `http` pool runs at once
#: (8 workers by default, see `config.yaml`), not this number. On the live
#: database of 2026-09-29, 175 boards and 4 searches make 19 batches, enough
#: to keep eight workers busy, while a batch that fails or is killed costs at
#: most ten targets' progress. A board's request is bounded by
#: `plugins.ats.DEADLINE` plus one read timeout.
BATCH_SIZE = 10
SETTLE = "app.jobs.pull:close_boards"