Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is refused at MAX_BYTES instead of after it is held whole, and bounds the whole request, redirects included, by DEADLINE, checked between chunks. The per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests against mongodb, feedzai and the five Greenhouse boards with most open roles, with and without pay_transparency=true: the body arrives in under 1 s, the first byte waited up to 16.6 s, and the flag made no measurable difference. _paged works unchanged through it (tested over a mock transport, 429 included). Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence, and the parallelism is the http pool's. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
2c4b992c50
commit
3abb44e10a
4 files changed
+126
-22
No files matched your search
+7
-3
@@ -47,9 +47,13 @@ from app.storage import Store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: As many boards in the air per job as there are `http` workers by default
|
||||
#: (see `config.yaml`), so one pull keeps the pool busy without one job
|
||||
#: holding all 190-odd boards' worth of retries against a single attempt.
|
||||
#: Targets per `pull_batch` job. A batch reads its boards one after another,
|
||||
#: so the parallelism is the number of batches the `http` pool runs at once
|
||||
#: (8 workers by default, see `config.yaml`), not this number. On the live
|
||||
#: database of 2026-09-29, 175 boards and 4 searches make 19 batches, enough
|
||||
#: to keep eight workers busy, while a batch that fails or is killed costs at
|
||||
#: most ten targets' progress. A board's request is bounded by
|
||||
#: `plugins.ats.DEADLINE` plus one read timeout.
|
||||
BATCH_SIZE = 10
|
||||
|
||||
SETTLE = "app.jobs.pull:close_boards"
|
||||
|
||||
Reference in new issue
Block a user