Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is refused at MAX_BYTES instead of after it is held whole, and bounds the whole request, redirects included, by DEADLINE, checked between chunks. The per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests against mongodb, feedzai and the five Greenhouse boards with most open roles, with and without pay_transparency=true: the body arrives in under 1 s, the first byte waited up to 16.6 s, and the flag made no measurable difference. _paged works unchanged through it (tested over a mock transport, 429 included). Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence, and the parallelism is the http pool's. Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
1 parent
2c4b992c50
commit
3abb44e10a
4 files changed
+126
-22
No files matched your search
+50
-19
@@ -5,7 +5,7 @@ unauthenticated JSON (Personio: XML) — first-party, no scraping, and a role
|
||||
that drops off the board is *closed*, which is the cheapest staleness check
|
||||
there is.
|
||||
|
||||
The field map, the redirect allow-list and the 10 s timeout are ported from
|
||||
The field map and the redirect allow-list are ported from
|
||||
`openings.go` and `cmd/seedgen/main.go` in
|
||||
[jobhunt-hq](https://github.com/rembrandtreyes/jobhunt-hq) (MIT), including
|
||||
`classify`'s nine functions and their keywords. Workable, SmartRecruiters,
|
||||
@@ -47,8 +47,7 @@ from desk.sources import (
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: Matches openings.go: a board answers in 10s or it is skipped. openings.go's
|
||||
#: 8 MB body cap does not: it was written against arbitrary URLs, and every
|
||||
#: openings.go's 8 MB body cap is not kept: it was written against arbitrary URLs, and every
|
||||
#: URL fetched here is `?content=true` against a fixed allow-list of seven ATS
|
||||
#: API hosts. The real run of 2026-09-23 hit it on the biggest employers —
|
||||
#: Anduril at 41 MB, Anthropic, OpenAI, Databricks, Carvana, Axon and
|
||||
@@ -56,9 +55,26 @@ log = logging.getLogger(__name__)
|
||||
#: to 96 MB so those boards fit; a board that still exceeds it gets one retry
|
||||
#: (Greenhouse only, see `_fetch_rows`) without `?content=true`, so at least
|
||||
#: the titles land.
|
||||
TIMEOUT = 10.0
|
||||
MAX_BYTES = 96 << 20
|
||||
|
||||
#: openings.go's 10 s did not hold: `greenhouse:mongodb` and `greenhouse:feedzai`
|
||||
#: failed a pull of 2026-09-28 with a read timeout. Measured live on
|
||||
#: 2026-09-29, three runs each of mongodb, feedzai and the five Greenhouse
|
||||
#: boards with the most open roles (databricks, stripe, anthropic, feverup,
|
||||
#: datadog), with and without `pay_transparency=true`, cached and with a
|
||||
#: cache-busting parameter (84 requests; CloudFront answered every one
|
||||
#: "Miss", so all reached the origin): the body arrives in under 1 s once it
|
||||
#: starts (at most 0.09 s between two chunks, 10.4 MB the largest), but the
|
||||
#: first byte sometimes waits: 16.6 s (feverup, no flag), 14.4 s (anthropic,
|
||||
#: flag), 10.9 s (feverup, flag), 5.2 s and 3.6 s, and under 2.6 s otherwise.
|
||||
#: The flag made no difference that three runs could tell apart from that
|
||||
#: noise. So a read may wait 30 s, about twice the worst first byte seen,
|
||||
#: and a whole request, redirects and body included, 60 s (see `_get`),
|
||||
#: about four times the slowest seen, which still leaves a 96 MB body room
|
||||
#: on a link far slower than the one measured.
|
||||
TIMEOUTS = httpx.Timeout(10.0, read=30.0)
|
||||
DEADLINE = 60.0
|
||||
|
||||
#: host -> (feed URL template, careers URL template). `{slug}` is filled in;
|
||||
#: recruitee and personio put the slug in the hostname instead of the path.
|
||||
_BOARDS: dict[str, tuple[str, str]] = {
|
||||
@@ -141,29 +157,44 @@ def _get(client: httpx.Client, url: str, host: str, slug: str, *,
|
||||
"""Fetch one board feed: https only, the host's own allow-list, at most
|
||||
three redirect hops each checked the same way, and the body cut off at
|
||||
`MAX_BYTES`. Port of openings.go's `fetchOne`/`feedCache.allow`. A POST
|
||||
sends `payload` as JSON, `{}` when there is none."""
|
||||
sends `payload` as JSON, `{}` when there is none.
|
||||
|
||||
The body is streamed and counted as it arrives, so an oversized one is
|
||||
refused at `MAX_BYTES` rather than after it has been read whole, and
|
||||
the whole request, redirects included, must end within `DEADLINE`: the
|
||||
per-read `TIMEOUTS` alone let a board that trickles bytes run on for
|
||||
good. The deadline is checked between chunks, so it can be overrun by
|
||||
at most one read timeout."""
|
||||
allow = _allowed_host(host, slug)
|
||||
deadline = time.monotonic() + DEADLINE
|
||||
current = url
|
||||
for _hop in range(4):
|
||||
parsed = httpx.URL(current)
|
||||
if parsed.scheme != "https" or not allow(parsed.host):
|
||||
raise _Refused(f"{current} is not an allowed host for {host}")
|
||||
response = client.request(
|
||||
method, current, timeout=TIMEOUT, follow_redirects=False,
|
||||
with client.stream(
|
||||
method, current, timeout=TIMEOUTS, follow_redirects=False,
|
||||
headers={"User-Agent": USER_AGENT, "Accept": "application/json, text/xml"},
|
||||
json=(payload or {}) if method == "POST" else None,
|
||||
)
|
||||
if response.is_redirect:
|
||||
location = response.headers.get("location") or ""
|
||||
if not location:
|
||||
raise ValueError("redirect with no location")
|
||||
current = str(httpx.URL(current).join(location))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
body = response.content
|
||||
if len(body) > MAX_BYTES:
|
||||
raise _TooLarge(f"response too large ({len(body)} bytes)")
|
||||
return body
|
||||
) as response:
|
||||
if response.is_redirect:
|
||||
location = response.headers.get("location") or ""
|
||||
if not location:
|
||||
raise ValueError("redirect with no location")
|
||||
current = str(httpx.URL(current).join(location))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
chunks: list[bytes] = []
|
||||
size = 0
|
||||
for chunk in response.iter_bytes():
|
||||
size += len(chunk)
|
||||
if size > MAX_BYTES:
|
||||
raise _TooLarge(f"response too large (over {MAX_BYTES} bytes)")
|
||||
if time.monotonic() > deadline:
|
||||
raise httpx.ReadTimeout(f"no whole answer within {DEADLINE:.0f} s",
|
||||
request=response.request)
|
||||
chunks.append(chunk)
|
||||
return b"".join(chunks)
|
||||
raise ValueError("too many redirects")
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user