Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s

_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus committed 2026-09-29 15:12:14 +02:00
1 parent 2c4b992c50
commit 3abb44e10a
4 files changed
+119 -15

No files matched your search

+7 -3
View File
@@ -47,9 +47,13 @@ from app.storage import Store
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
#: As many boards in the air per job as there are `http` workers by default #: Targets per `pull_batch` job. A batch reads its boards one after another,
#: (see `config.yaml`), so one pull keeps the pool busy without one job #: so the parallelism is the number of batches the `http` pool runs at once
#: holding all 190-odd boards' worth of retries against a single attempt. #: (8 workers by default, see `config.yaml`), not this number. On the live
#: database of 2026-09-29, 175 boards and 4 searches make 19 batches, enough
#: to keep eight workers busy, while a batch that fails or is killed costs at
#: most ten targets' progress. A board's request is bounded by
#: `plugins.ats.DEADLINE` plus one read timeout.
BATCH_SIZE = 10 BATCH_SIZE = 10
SETTLE = "app.jobs.pull:close_boards" SETTLE = "app.jobs.pull:close_boards"
+2
View File
@@ -78,6 +78,8 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing. **The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
**A board request is streamed, with a 30 s read timeout and a 60 s deadline.** `_get` counts the body as it arrives and refuses it at `MAX_BYTES`, where reading `response.content` first held the whole body in memory before the check. httpx's timeouts are per phase, so a board that keeps sending a byte now and then never timed out; the deadline covers the whole request, redirects and body included, and is checked between chunks. openings.go's 10 s read timeout failed `greenhouse:mongodb` and `greenhouse:feedzai` in a pull of 2026-09-28. Measured on 2026-09-29, three runs each of those two and the five Greenhouse boards with the most open roles, with and without `pay_transparency=true` (84 requests, every one a CloudFront miss): a body of up to 10.4 MB arrives in under a second once it starts, and the first byte waited up to 16.6 s, with the three slowest at 16.6, 14.4 and 10.9 s. The flag made no measurable difference; the slow first byte is the origin's, with or without it. A read may wait 30 s, about twice the worst seen, and the deadline is 60 s, which leaves a large board room on a slower link.
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`, and `description` stays empty as for SmartRecruiters rather than costing one request per role. **Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`, and `description` stays empty as for SmartRecruiters rather than costing one request per role.
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse. **Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse.
+43 -12
View File
@@ -5,7 +5,7 @@ unauthenticated JSON (Personio: XML) — first-party, no scraping, and a role
that drops off the board is *closed*, which is the cheapest staleness check that drops off the board is *closed*, which is the cheapest staleness check
there is. there is.
The field map, the redirect allow-list and the 10 s timeout are ported from The field map and the redirect allow-list are ported from
`openings.go` and `cmd/seedgen/main.go` in `openings.go` and `cmd/seedgen/main.go` in
[jobhunt-hq](https://github.com/rembrandtreyes/jobhunt-hq) (MIT), including [jobhunt-hq](https://github.com/rembrandtreyes/jobhunt-hq) (MIT), including
`classify`'s nine functions and their keywords. Workable, SmartRecruiters, `classify`'s nine functions and their keywords. Workable, SmartRecruiters,
@@ -47,8 +47,7 @@ from desk.sources import (
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
#: Matches openings.go: a board answers in 10s or it is skipped. openings.go's #: openings.go's 8 MB body cap is not kept: it was written against arbitrary URLs, and every
#: 8 MB body cap does not: it was written against arbitrary URLs, and every
#: URL fetched here is `?content=true` against a fixed allow-list of seven ATS #: URL fetched here is `?content=true` against a fixed allow-list of seven ATS
#: API hosts. The real run of 2026-09-23 hit it on the biggest employers — #: API hosts. The real run of 2026-09-23 hit it on the biggest employers —
#: Anduril at 41 MB, Anthropic, OpenAI, Databricks, Carvana, Axon and #: Anduril at 41 MB, Anthropic, OpenAI, Databricks, Carvana, Axon and
@@ -56,9 +55,26 @@ log = logging.getLogger(__name__)
#: to 96 MB so those boards fit; a board that still exceeds it gets one retry #: to 96 MB so those boards fit; a board that still exceeds it gets one retry
#: (Greenhouse only, see `_fetch_rows`) without `?content=true`, so at least #: (Greenhouse only, see `_fetch_rows`) without `?content=true`, so at least
#: the titles land. #: the titles land.
TIMEOUT = 10.0
MAX_BYTES = 96 << 20 MAX_BYTES = 96 << 20
#: openings.go's 10 s did not hold: `greenhouse:mongodb` and `greenhouse:feedzai`
#: failed a pull of 2026-09-28 with a read timeout. Measured live on
#: 2026-09-29, three runs each of mongodb, feedzai and the five Greenhouse
#: boards with the most open roles (databricks, stripe, anthropic, feverup,
#: datadog), with and without `pay_transparency=true`, cached and with a
#: cache-busting parameter (84 requests; CloudFront answered every one
#: "Miss", so all reached the origin): the body arrives in under 1 s once it
#: starts (at most 0.09 s between two chunks, 10.4 MB the largest), but the
#: first byte sometimes waits: 16.6 s (feverup, no flag), 14.4 s (anthropic,
#: flag), 10.9 s (feverup, flag), 5.2 s and 3.6 s, and under 2.6 s otherwise.
#: The flag made no difference that three runs could tell apart from that
#: noise. So a read may wait 30 s, about twice the worst first byte seen,
#: and a whole request, redirects and body included, 60 s (see `_get`),
#: about four times the slowest seen, which still leaves a 96 MB body room
#: on a link far slower than the one measured.
TIMEOUTS = httpx.Timeout(10.0, read=30.0)
DEADLINE = 60.0
#: host -> (feed URL template, careers URL template). `{slug}` is filled in; #: host -> (feed URL template, careers URL template). `{slug}` is filled in;
#: recruitee and personio put the slug in the hostname instead of the path. #: recruitee and personio put the slug in the hostname instead of the path.
_BOARDS: dict[str, tuple[str, str]] = { _BOARDS: dict[str, tuple[str, str]] = {
@@ -141,18 +157,26 @@ def _get(client: httpx.Client, url: str, host: str, slug: str, *,
"""Fetch one board feed: https only, the host's own allow-list, at most """Fetch one board feed: https only, the host's own allow-list, at most
three redirect hops each checked the same way, and the body cut off at three redirect hops each checked the same way, and the body cut off at
`MAX_BYTES`. Port of openings.go's `fetchOne`/`feedCache.allow`. A POST `MAX_BYTES`. Port of openings.go's `fetchOne`/`feedCache.allow`. A POST
sends `payload` as JSON, `{}` when there is none.""" sends `payload` as JSON, `{}` when there is none.
The body is streamed and counted as it arrives, so an oversized one is
refused at `MAX_BYTES` rather than after it has been read whole, and
the whole request, redirects included, must end within `DEADLINE`: the
per-read `TIMEOUTS` alone let a board that trickles bytes run on for
good. The deadline is checked between chunks, so it can be overrun by
at most one read timeout."""
allow = _allowed_host(host, slug) allow = _allowed_host(host, slug)
deadline = time.monotonic() + DEADLINE
current = url current = url
for _hop in range(4): for _hop in range(4):
parsed = httpx.URL(current) parsed = httpx.URL(current)
if parsed.scheme != "https" or not allow(parsed.host): if parsed.scheme != "https" or not allow(parsed.host):
raise _Refused(f"{current} is not an allowed host for {host}") raise _Refused(f"{current} is not an allowed host for {host}")
response = client.request( with client.stream(
method, current, timeout=TIMEOUT, follow_redirects=False, method, current, timeout=TIMEOUTS, follow_redirects=False,
headers={"User-Agent": USER_AGENT, "Accept": "application/json, text/xml"}, headers={"User-Agent": USER_AGENT, "Accept": "application/json, text/xml"},
json=(payload or {}) if method == "POST" else None, json=(payload or {}) if method == "POST" else None,
) ) as response:
if response.is_redirect: if response.is_redirect:
location = response.headers.get("location") or "" location = response.headers.get("location") or ""
if not location: if not location:
@@ -160,10 +184,17 @@ def _get(client: httpx.Client, url: str, host: str, slug: str, *,
current = str(httpx.URL(current).join(location)) current = str(httpx.URL(current).join(location))
continue continue
response.raise_for_status() response.raise_for_status()
body = response.content chunks: list[bytes] = []
if len(body) > MAX_BYTES: size = 0
raise _TooLarge(f"response too large ({len(body)} bytes)") for chunk in response.iter_bytes():
return body size += len(chunk)
if size > MAX_BYTES:
raise _TooLarge(f"response too large (over {MAX_BYTES} bytes)")
if time.monotonic() > deadline:
raise httpx.ReadTimeout(f"no whole answer within {DEADLINE:.0f} s",
request=response.request)
chunks.append(chunk)
return b"".join(chunks)
raise ValueError("too many redirects") raise ValueError("too many redirects")
+67
View File
@@ -7,6 +7,7 @@ from __future__ import annotations
import json import json
import sys import sys
from pathlib import Path from pathlib import Path
from collections.abc import Callable, Iterator
from typing import Any from typing import Any
import httpx import httpx
@@ -279,6 +280,72 @@ def test_fetch_rows_does_not_retry_a_too_large_body_on_other_hosts(
ats._fetch_rows(object(), "lever:acme") # type: ignore[arg-type] ats._fetch_rows(object(), "lever:acme") # type: ignore[arg-type]
def _client(handler: Callable[[httpx.Request], httpx.Response]) -> httpx.Client:
return httpx.Client(transport=httpx.MockTransport(handler))
def test_get_stops_reading_at_max_bytes_rather_than_after_the_body(
monkeypatch: pytest.MonkeyPatch) -> None:
sent: list[int] = []
def chunks() -> Iterator[bytes]:
for i in range(100):
sent.append(i)
yield b"x" * 10
monkeypatch.setattr(ats, "MAX_BYTES", 25)
with _client(lambda request: httpx.Response(200, content=chunks())) as client, \
pytest.raises(ats._TooLarge):
ats._get(client, "https://api.lever.co/v0/postings/acme", "lever", "acme")
assert len(sent) == 3 # refused on the third chunk, not the hundredth
def test_get_gives_up_at_the_deadline_however_steadily_bytes_arrive(
monkeypatch: pytest.MonkeyPatch) -> None:
clock = iter(range(0, 1000, 10))
monkeypatch.setattr(ats.time, "monotonic", lambda: float(next(clock)))
monkeypatch.setattr(ats, "DEADLINE", 25.0)
body = (b"x" for _ in range(100))
with _client(lambda request: httpx.Response(200, content=body)) as client, \
pytest.raises(httpx.ReadTimeout, match="within 25 s"):
ats._get(client, "https://api.lever.co/v0/postings/acme", "lever", "acme")
def test_get_follows_an_allowed_redirect_and_refuses_a_foreign_one() -> None:
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/v0/postings/old":
return httpx.Response(301, headers={"location": "/v0/postings/acme"})
if request.url.path == "/v0/postings/away":
return httpx.Response(302, headers={"location": "https://evil.example/"})
return httpx.Response(200, content=b"[]")
with _client(handler) as client:
assert ats._get(client, "https://api.lever.co/v0/postings/old", "lever", "acme") == b"[]"
with pytest.raises(ats._Refused):
ats._get(client, "https://api.lever.co/v0/postings/away", "lever", "acme")
def test_paging_reads_through_the_streamed_get(no_delay: None) -> None:
"""`_paged` over the real `_get`: the POST body carries the cursor, and a
429 on a later page still reaches it as an `HTTPStatusError`."""
bodies: list[Any] = []
def handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content or b"{}")
bodies.append(payload)
if payload.get("token") == "stop":
return httpx.Response(429)
page = json.loads(_page("workable-2.json" if payload else "workable.json"))
if payload:
page["total"], page["nextPage"] = 4, "stop"
return httpx.Response(200, json=page)
with _client(handler) as client:
rows, complete = ats._fetch_rows(client, "workable:exotec")
assert bodies == [{}, {"token": "WzE3ODk2ODk2MDAwMDAsNjEyNTA0MV0="}, {"token": "stop"}]
assert (len(rows), complete) == (3, False)
# ----------------------------------------------------------------- paging # ----------------------------------------------------------------- paging
def _page(name: str) -> bytes: def _page(name: str) -> bytes: