Raise the ats body cap to 96 MB, retry Greenhouse without content=true

openings.go's 8 MB cap was written against arbitrary URLs; every URL here is
on a fixed allow-list of seven ATS hosts, and the real pull of 2026-09-23
refused Anduril, Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex
for it. A board that still exceeds 96 MB now retries once without
content=true (Greenhouse only), trading the description for the listing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Sonnet 5 committed 2026-09-23 11:39:39 +02:00
1 parent 078fb8ff8f
commit 867ebe6405
3 files changed
+85 -8

No files matched your search

+35 -8
View File
@@ -5,14 +5,16 @@ unauthenticated JSON (Personio: XML) — first-party, no scraping, and a role
that drops off the board is *closed*, which is the cheapest staleness check
there is.
The field map, the redirect allow-list, the 8 MB body cap and the 10 s
timeout are ported from `openings.go` and `cmd/seedgen/main.go` in
The field map, the redirect allow-list and the 10 s timeout are ported from
`openings.go` and `cmd/seedgen/main.go` in
[jobhunt-hq](https://github.com/rembrandtreyes/jobhunt-hq) (MIT), including
`classify`'s nine functions and their keywords. Workable, SmartRecruiters,
Recruitee and Personio are new: jobhunt-hq only read Greenhouse, Lever and
Ashby, so those four are written from one real board of each host, fetched
while this file was written (2026-09-23), rather than from the Go source.
The body cap is not a straight port — see `MAX_BYTES` below.
A target is `host:slug`, e.g. `greenhouse:databricks`, `ashby:deepl`,
`lever:acme`. Every board is fetched independently; one that fails or times
out is logged and skipped, and the rest continue.
@@ -38,10 +40,17 @@ from desk.sources import (
log = logging.getLogger(__name__)
#: Matches openings.go: a board answers in 10s or it is skipped, and its body
#: is capped at 8 MB so a misbehaving host cannot hang a pull or fill memory.
#: Matches openings.go: a board answers in 10s or it is skipped. openings.go's
#: 8 MB body cap does not: it was written against arbitrary URLs, and every
#: URL fetched here is `?content=true` against a fixed allow-list of seven ATS
#: API hosts. The real run of 2026-09-23 hit it on the biggest employers —
#: Anduril at 41 MB, Anthropic, OpenAI, Databricks, Carvana, Axon and
#: Airwallex between 9 and 19 MB — and skipped their boards outright. Raised
#: to 96 MB so those boards fit; a board that still exceeds it gets one retry
#: (Greenhouse only, see `_fetch_rows`) without `?content=true`, so at least
#: the titles land.
TIMEOUT = 10.0
MAX_BYTES = 8 << 20
MAX_BYTES = 96 << 20
#: host -> (feed URL template, careers URL template). `{slug}` is filled in;
#: recruitee and personio put the slug in the hostname instead of the path.
@@ -102,6 +111,12 @@ class _Refused(Exception):
"""A board answered somewhere its allow-list does not cover."""
class _TooLarge(ValueError):
"""A response exceeded `MAX_BYTES`. A `ValueError` so a caller that only
knows the old shape still catches it; `_fetch_rows` catches it by type to
retry Greenhouse without `?content=true`."""
def _get(client: httpx.Client, url: str, host: str, slug: str, *,
method: str = "GET") -> bytes:
"""Fetch one board feed: https only, the host's own allow-list, at most
@@ -127,7 +142,7 @@ def _get(client: httpx.Client, url: str, host: str, slug: str, *,
response.raise_for_status()
body = response.content
if len(body) > MAX_BYTES:
raise ValueError(f"response too large ({len(body)} bytes)")
raise _TooLarge(f"response too large ({len(body)} bytes)")
return body
raise ValueError("too many redirects")
@@ -455,12 +470,24 @@ def careers_url(target: str) -> str:
def _fetch_rows(client: httpx.Client, target: str) -> list[Row]:
"""One board, fetched and parsed. Raises on any failure — refused host,
timeout, a feed shape none of the parsers understand."""
timeout, a feed shape none of the parsers understand.
A Greenhouse board whose `?content=true` body still exceeds `MAX_BYTES`
gets one retry without it: the description is lost, but the titles land
rather than the whole board being skipped."""
if not is_known_target(target):
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
host, _, slug = target.partition(":")
url = _BOARDS[host][0].format(slug=slug)
body = _get(client, url, host, slug, method="POST" if host == "workable" else "GET")
try:
body = _get(client, url, host, slug, method="POST" if host == "workable" else "GET")
except _TooLarge:
if host != "greenhouse" or "?" not in url:
raise
log.warning("ats: %s: body exceeded %d bytes with content=true; "
"retrying without it", target, MAX_BYTES)
url = url.split("?", 1)[0]
body = _get(client, url, host, slug, method="GET")
return _PARSERS[host](body, slug)