Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s

_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 15:12:14 +02:00
1 parent 2c4b992c50
commit 3abb44e10a
4 files changed
+126 -22

No files matched your search

+50 -19
View File
@@ -5,7 +5,7 @@ unauthenticated JSON (Personio: XML) — first-party, no scraping, and a role
that drops off the board is *closed*, which is the cheapest staleness check
there is.
The field map, the redirect allow-list and the 10 s timeout are ported from
The field map and the redirect allow-list are ported from
`openings.go` and `cmd/seedgen/main.go` in
[jobhunt-hq](https://github.com/rembrandtreyes/jobhunt-hq) (MIT), including
`classify`'s nine functions and their keywords. Workable, SmartRecruiters,
@@ -47,8 +47,7 @@ from desk.sources import (
log = logging.getLogger(__name__)
#: Matches openings.go: a board answers in 10s or it is skipped. openings.go's
#: 8 MB body cap does not: it was written against arbitrary URLs, and every
#: openings.go's 8 MB body cap is not kept: it was written against arbitrary URLs, and every
#: URL fetched here is `?content=true` against a fixed allow-list of seven ATS
#: API hosts. The real run of 2026-09-23 hit it on the biggest employers —
#: Anduril at 41 MB, Anthropic, OpenAI, Databricks, Carvana, Axon and
@@ -56,9 +55,26 @@ log = logging.getLogger(__name__)
#: to 96 MB so those boards fit; a board that still exceeds it gets one retry
#: (Greenhouse only, see `_fetch_rows`) without `?content=true`, so at least
#: the titles land.
TIMEOUT = 10.0
MAX_BYTES = 96 << 20
#: openings.go's 10 s did not hold: `greenhouse:mongodb` and `greenhouse:feedzai`
#: failed a pull of 2026-09-28 with a read timeout. Measured live on
#: 2026-09-29, three runs each of mongodb, feedzai and the five Greenhouse
#: boards with the most open roles (databricks, stripe, anthropic, feverup,
#: datadog), with and without `pay_transparency=true`, cached and with a
#: cache-busting parameter (84 requests; CloudFront answered every one
#: "Miss", so all reached the origin): the body arrives in under 1 s once it
#: starts (at most 0.09 s between two chunks, 10.4 MB the largest), but the
#: first byte sometimes waits: 16.6 s (feverup, no flag), 14.4 s (anthropic,
#: flag), 10.9 s (feverup, flag), 5.2 s and 3.6 s, and under 2.6 s otherwise.
#: The flag made no difference that three runs could tell apart from that
#: noise. So a read may wait 30 s, about twice the worst first byte seen,
#: and a whole request, redirects and body included, 60 s (see `_get`),
#: about four times the slowest seen, which still leaves a 96 MB body room
#: on a link far slower than the one measured.
TIMEOUTS = httpx.Timeout(10.0, read=30.0)
DEADLINE = 60.0
#: host -> (feed URL template, careers URL template). `{slug}` is filled in;
#: recruitee and personio put the slug in the hostname instead of the path.
_BOARDS: dict[str, tuple[str, str]] = {
@@ -141,29 +157,44 @@ def _get(client: httpx.Client, url: str, host: str, slug: str, *,
"""Fetch one board feed: https only, the host's own allow-list, at most
three redirect hops each checked the same way, and the body cut off at
`MAX_BYTES`. Port of openings.go's `fetchOne`/`feedCache.allow`. A POST
sends `payload` as JSON, `{}` when there is none."""
sends `payload` as JSON, `{}` when there is none.
The body is streamed and counted as it arrives, so an oversized one is
refused at `MAX_BYTES` rather than after it has been read whole, and
the whole request, redirects included, must end within `DEADLINE`: the
per-read `TIMEOUTS` alone let a board that trickles bytes run on for
good. The deadline is checked between chunks, so it can be overrun by
at most one read timeout."""
allow = _allowed_host(host, slug)
deadline = time.monotonic() + DEADLINE
current = url
for _hop in range(4):
parsed = httpx.URL(current)
if parsed.scheme != "https" or not allow(parsed.host):
raise _Refused(f"{current} is not an allowed host for {host}")
response = client.request(
method, current, timeout=TIMEOUT, follow_redirects=False,
with client.stream(
method, current, timeout=TIMEOUTS, follow_redirects=False,
headers={"User-Agent": USER_AGENT, "Accept": "application/json, text/xml"},
json=(payload or {}) if method == "POST" else None,
)
if response.is_redirect:
location = response.headers.get("location") or ""
if not location:
raise ValueError("redirect with no location")
current = str(httpx.URL(current).join(location))
continue
response.raise_for_status()
body = response.content
if len(body) > MAX_BYTES:
raise _TooLarge(f"response too large ({len(body)} bytes)")
return body
) as response:
if response.is_redirect:
location = response.headers.get("location") or ""
if not location:
raise ValueError("redirect with no location")
current = str(httpx.URL(current).join(location))
continue
response.raise_for_status()
chunks: list[bytes] = []
size = 0
for chunk in response.iter_bytes():
size += len(chunk)
if size > MAX_BYTES:
raise _TooLarge(f"response too large (over {MAX_BYTES} bytes)")
if time.monotonic() > deadline:
raise httpx.ReadTimeout(f"no whole answer within {DEADLINE:.0f} s",
request=response.request)
chunks.append(chunk)
return b"".join(chunks)
raise ValueError("too many redirects")