plugins/readers/_runner.py takes the guarded fetch, the listing (whole,
paged, one series per filter value unioned by id), completeness from READS,
the sweep of a board over the page cap across pulls, the 10% guard on a
filtered or empty="suspect" read, facets kept a day, the detail reads, the
raw an Item carries and the failure bookkeeping out of plugins/ats.py.
Ask comes from a subscription's params (ask_for, open for the reach), with
a declared key resolved into Ask.params["key"] and refused when missing.
The runtime robots.txt check goes (decided at build time, READS.robots).
plugins/search.py reads the providers of readers.PROVIDERS through the
runner and applies the saved pages param. A filtered board's close keeps
roles with a vote or an application. DETAIL_FIELDS gain places, teaser,
country and employment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every row carries places (Personio additionalOffices, Workable and Recruitee
locations[], every SuccessFactors address, Workday's own read), the list
teaser of Phenom and Oracle is kept in teaser, and the level, employment and
country a host states are mapped. Workday "N Locations" and "Country - City",
and Recruitee "Remote job", are no longer places. Phenom takes dateCreated
as posted. robots_page and ROBOTS_ANY_CASE go; READS.robots records the
decision. Signatures are feed_url(slug, ask) and page(get, slug, cursor, ask),
with an empty Ask as the default so the runner of today still calls them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each declares READS with scope=search, keeps every place and hands the excerpt over as teaser. search_params and PROVIDERS follow the declarations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plugins/readers/_contract.py holds Reads, Filters, Fields, Key, Ask, Place and
Row, and problems(), which checks a module's READS against its functions and
signatures. The package loader runs it for modules that declare READS and
lists the others with undeclared(); READERS holds every module, HOSTS the
boards and the undeclared, PROVIDERS the searches. The package docstring
documents the traits, Ask, facets, places, teaser and the robots.txt rule.
tests/test_readers_contract.py runs over every reader: READS declared (xfail
per module in UNDECLARED), declaration against functions, places="many"
and teaser fixtures, plus unit tests of the check with fake modules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every import, test and live doc follows the new path; behaviour is unchanged.
tests/test_atshosts.py becomes tests/test_readers.py. plan-readers.md and
review-ingest.md keep the old name, since they describe the state before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A role listed before its page was read kept a title-only vector, and the listing's place replaced the page's on every later pull.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).
Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.
Co-Authored-By: Claude Opus <noreply@anthropic.com>
Every ats parser now writes one human-readable pay line whether or not it
converts to yearly salary_min/salary_max/currency: hourly and daily rates,
Greenhouse's several pay zones, and a range with no currency all survive as
text instead of being dropped outright. Stored in roles.salary_text
(migration 12), replaced whole on every pull like title, so a board that
stops stating pay clears it rather than leaving a stale line. salary_state
is derived, not stored ('normalised', 'stated', or ''), and is what the
feed's new Pay filter and badge read; gates and scoring are unchanged, still
reading only the normalised columns.
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Part A: Ashby (?includeCompensation=true), Greenhouse (pay_transparency=true,
already free on the list request), Lever (salaryRange), Recruitee (salary)
and Personio (salaryInformation) all give structured pay; Workable and
SmartRecruiters do not (checked live). Every host converts to a yearly
salary_min/salary_max/currency, multiplying a monthly figure by 12 and
skipping an hourly or daily one rather than annualising a number nobody
stated. A range with no currency is ignored, and Greenhouse's own multi-zone
ranges are skipped outright since nothing says which zone the role is at.
Part B: classify() matched substrings, so " it" matched inside "Italy",
filing European sales roles as engineering; it now matches on word
boundaries (full boundary for the handful of short tokens that are
themselves the start of an unrelated word, prefix matching kept for the
stems). Title is checked before department (a hand-checked sample found
title-first right about twice as often), nothing matching returns '' rather
than "other", and a modest set of German/Dutch/Danish/Norwegian/Swedish/
French keywords covers the seed's other languages.
function is not a board field, so it gets the opposite rule from every
other raw column: extract's answer overwrites the classifier's guess
outright, and once extracted_at is set no later pull may put the guess
back. Migration 9 recomputes function for every not-yet-extracted role from
its stored title and department.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A second pull (schedule, Run button, `app job pull` in another process) could
start while one was fetching and overwrite the stored run marker, so the first
run's boards failed their stamp check or a settle closed on the other run's id
lists. `claim_run` sets `pull:run` in one conditional statement, only while it
equals `pull:settled` or is older than an hour; a refused `pull` job queues no
stage and says so in its result and a notice, and the CLI exits 1. Batches
carry the marker in their payload, the plugins stamp with it, and the settle
reads it off its own batches' payloads before releasing the run.
Co-Authored-By: Claude Opus <noreply@anthropic.com>