Commit Graph
115 Commits
Author SHA1 Message Date
WolfandClaude Opus 5.5 3462a4abe7 Read big boards within the agents' reach: countries and remote from the enabled agents, Europe for favourites, filtered at the source by six hosts
app/jobs/reach.py derives each board's reach once per pull and the pull hands it to the runner through the batch payload. SmartRecruiters, Eightfold and SuccessFactors filter one country a request, Phenom a list, Workday and Oracle by host ids from facets. Remote beside countries is a series of its own, since two facets intersect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:55:18 +02:00
WolfandClaude Opus 5.5 77b45a4b0f Read role pages in an enrich stage after scoring, best roles first
The pull reads listings only. `enrich` (app/jobs/enrich.py) takes the open
roles without text that an enabled agent ranks ungated in its best
`enrich.top` at `enrich.min_score` or above, best score first, on boards
whose reader has `detail`, within `enrich.per_board` and `enrich.per_run`,
one batch per board. `plugins.ats.read_board_details` reads them and keeps
the per-board state; a read that gave no text is read again after
`enrich.retry_days` (finding 18). Filled roles are upserted, which drops
their vectors, and embed and score follow; `cascade.step` passes over both
when nothing was filled.

The pull cascade is pull, embed, score, enrich, embed, score, extract, score,
summarise. Saving an agent and changing a company's affinity queue the
`enrich` job (enrich, embed, score) behind their rescore; a vote does not.
The stage has a switch, `stage:enrich`, over `enrich.enabled`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:53:29 +02:00
WolfandClaude Opus 5.5 1d7b286374 Merge branch 'wp-d1-hosts'
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:32:46 +02:00
WolfandClaude Opus 5.5 902089a5e2 Runner: readers read from their declaration, ats and search thin over it
plugins/readers/_runner.py takes the guarded fetch, the listing (whole,
paged, one series per filter value unioned by id), completeness from READS,
the sweep of a board over the page cap across pulls, the 10% guard on a
filtered or empty="suspect" read, facets kept a day, the detail reads, the
raw an Item carries and the failure bookkeeping out of plugins/ats.py.
Ask comes from a subscription's params (ask_for, open for the reach), with
a declared key resolved into Ask.params["key"] and refused when missing.

The runtime robots.txt check goes (decided at build time, READS.robots).
plugins/search.py reads the providers of readers.PROVIDERS through the
runner and applies the saved pages param. A filtered board's close keeps
roles with a vote or an application. DETAIL_FIELDS gain places, teaser,
country and employment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:28:33 +02:00
WolfandClaude Opus 5.5 dec5fd5009 Merge branch 'wp-d2-hosts'
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:14:55 +02:00
WolfandClaude Opus 5.5 4316624769 Declare READS for Oracle, Personio, Phenom, Pinpoint, Recruitee, Rippling, SmartRecruiters, SuccessFactors, Teamtailor, Workable and Workday
Every row carries places (Personio additionalOffices, Workable and Recruitee
locations[], every SuccessFactors address, Workday's own read), the list
teaser of Phenom and Oracle is kept in teaser, and the level, employment and
country a host states are mapped. Workday "N Locations" and "Country - City",
and Recruitee "Remote job", are no longer places. Phenom takes dateCreated
as posted. robots_page and ROBOTS_ANY_CASE go; READS.robots records the
decision. Signatures are feed_url(slug, ask) and page(get, slug, cursor, ask),
with an empty Ask as the default so the runner of today still calls them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:14:27 +02:00
WolfandClaude Opus 5.5 f2e652a8da Hosts a to l declare READS: places, teaser, stated seniority and employment
Ashby, BambooHR, Deel, Eightfold, Gem, Greenhouse, HiBob, HR Manager, Jibe,
Jobylon and Lever move to feed_url(slug, ask) and page(get, slug, cursor, ask),
declare READS with the robots decision, and drop robots_page. Rows carry
places (Ashby secondaryLocations, Lever allLocations, Greenhouse offices in
place of a Hybrid placeholder), Jobylon keeps its summary as teaser and maps
experience and employment, Eightfold reads creationTs as the posting date and
Ashby reads workplaceType. Fixtures <name>-places.* and jobylon-teaser.html
drive the conformance checks; the eleven names leave UNDECLARED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:11:53 +02:00
WolfandClaude Opus 5.5 b78c9dfc67 Search providers as reader modules: jobindex, adzuna, platsbanken
Each declares READS with scope=search, keeps every place and hands the excerpt over as teaser. search_params and PROVIDERS follow the declarations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:08:25 +02:00
WolfandClaude Opus 5.5 9f526356c0 Reader contract: READS declaration, Ask, Row with places and teaser, conformance test
plugins/readers/_contract.py holds Reads, Filters, Fields, Key, Ask, Place and
Row, and problems(), which checks a module's READS against its functions and
signatures. The package loader runs it for modules that declare READS and
lists the others with undeclared(); READERS holds every module, HOSTS the
boards and the undeclared, PROVIDERS the searches. The package docstring
documents the traits, Ask, facets, places, teaser and the robots.txt rule.

tests/test_readers_contract.py runs over every reader: READS declared (xfail
per module in UNDECLARED), declaration against functions, places="many"
and teaser fixtures, plus unit tests of the check with fake modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:55:47 +02:00
WolfandClaude Opus 5.5 73dec9028e Rename plugins/atshosts to plugins/readers
Every import, test and live doc follows the new path; behaviour is unchanged.
tests/test_atshosts.py becomes tests/test_readers.py. plan-readers.md and
review-ingest.md keep the old name, since they describe the state before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:48:30 +02:00
WolfandClaude Fable 5.1 30b36df2fb Describe every source in plugins/README.md: how it is read, what it delivers and its limits
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 22:23:43 +02:00
WolfandClaude Fable 5.1 31abad4c8e Embed a role again when its text arrives, and keep the place its own page gave
A role listed before its page was read kept a title-only vector, and the listing's place replaced the page's on every later pull.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 22:23:43 +02:00
Wolf c7c5aebdb6 Merge branch 'wp7-small-hosts' 2026-09-29 20:15:47 +02:00
Wolf f38da929c9 Merge branch 'wp4-successfactors' 2026-09-29 20:15:47 +02:00
WolfandClaude Sonnet 5.5 d80754e985 Read HR Manager's public position feed by skip and take
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:13:23 +02:00
Wolf c18c93a55e Merge branch 'wp5-oracle-phenom' 2026-09-29 20:13:16 +02:00
Wolf 0eff7c9d13 Merge branch 'wp6-json-hosts' 2026-09-29 20:12:28 +02:00
WolfandClaude Sonnet 5.5 679d9ec938 Let discovery find a SuccessFactors board from a role, landing or search link
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:12:13 +02:00
WolfandClaude Sonnet 5.5 85bfe62fec Read Eightfold career sites through the PCSX search
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:11:18 +02:00
WolfandClaude Sonnet 5.5 cf182ad921 Read a Phenom role's description from the page state when the page has no JobPosting, and fail a page with neither
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:10:52 +02:00
WolfandClaude Sonnet 5.5 27a7723a1d Read HiBob's careers boards through the company header the site sends
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:10:37 +02:00
WolfandClaude Sonnet 5.5 254645c297 Read SuccessFactors career sites by their search pages and role microdata
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:10:11 +02:00
WolfandClaude Sonnet 5.5 8ee6f9b546 Read Phenom career sites through their search widget, with the description from each role page's JSON-LD
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:09:08 +02:00
WolfandClaude Sonnet 5.5 eac483ff13 Read Rippling's boards, merging a role's places, and its pay and description from the role's record
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:08:37 +02:00
WolfandClaude Sonnet 5.5 bd4121be91 Read Jibe career sites from their jobs API
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:08:35 +02:00
WolfandClaude Sonnet 5.5 34d1f74e3a Read Oracle Recruiting Cloud boards from the REST listing, with the description from the role's own resource
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:07:56 +02:00
WolfandClaude Fable 5.1 bb8a9a3b8f Read robots.txt without case for a host that serves a path in any case, as Workday does
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 20:07:14 +02:00
WolfandClaude Sonnet 5.5 a41995da8e Read Jobylon's boards from the embed page and each role's page
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:06:16 +02:00
WolfandClaude Sonnet 5.5 1fc9668110 Read Gem boards through the job board's GraphQL API
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:05:04 +02:00
WolfandClaude Sonnet 5.5 9efc6d386c Read Workday career sites through the tenant's own list and role endpoints
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:04:23 +02:00
WolfandClaude Sonnet 5.5 68a3858355 Read Deel's job boards from the page's server payload
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:02:49 +02:00
WolfandClaude Sonnet 5.5 76187d5862 Read BambooHR boards from their careers list and role pages
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:02:45 +02:00
WolfandClaude Sonnet 5.5 0b76b16daa Read Pinpoint boards from their postings.json
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 20:00:52 +02:00
WolfandClaude Opus 5.5 2ff5046993 State the final host contract in the package docstring and the plan, with ranks, case, detail fields and robots.txt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:47:09 +02:00
WolfandClaude Opus 5.5 4fb06191cf Load the host modules when the registry reads ats.py by path without the repo root on sys.path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:44:03 +02:00
WolfandClaude Opus 5.5 60136765b2 Refuse a fetch to another port than 443, an IP address or a local host name, at every hop
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:42:58 +02:00
WolfandClaude Opus 5.5 f61caddced Match a slug whole, and compare an allowed host name without case
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:42:10 +02:00
WolfandClaude Opus 5.5 0ffd0726e9 Read never-tried roles before failed ones in detail, count a 404 as read, and take only the fields a later pull keeps
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:41:39 +02:00
WolfandClaude Opus 5.5 1e47192aba Fail a board whose robots.txt cannot be read, and follow a robots.txt redirect to the host's www. twin only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:40:14 +02:00
WolfandClaude Opus 5.5 13ce99304b Match robots.txt rules by RFC 9309, the longest rule winning, with wildcards, end anchors and case-sensitive paths
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:39:11 +02:00
WolfandClaude Opus 5.5 a9fb4397ac Order the hosts by a RANK per module, which restores the select's and the probes' order from before the modules
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:38:18 +02:00
WolfandClaude Opus 5.5 75a389d725 Keep a discovered slug's case for a host with KEEP_CASE, and check only group 1 against the hosts' own path words
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:37:26 +02:00
WolfandClaude Opus 5.5 8b14d8575b Read a role's own page once for a host with detail, and fail a board whose robots.txt disallows it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:05:44 +02:00
WolfandClaude Opus 5.5 b02b72b4aa Move the eight hosts into a module each under plugins/atshosts and build the ats plugin's tables from them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:03:37 +02:00
Wolf d887a91af0 Merge branch 'wp3-teamtailor' 2026-09-29 16:15:22 +02:00
WolfandClaude Sonnet 5.5 50edf0eeeb Read Teamtailor career sites from their jobs.rss feed, paged by offset
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:14:41 +02:00
WolfandClaude Sonnet 5.5 00e111961e Add Platsbanken as a third search provider
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:10:45 +02:00
Wolf 562fa3ab09 Merge branch 'r2-backend'
# Conflicts:
#	app/storage/schema.py
2026-09-29 15:22:03 +02:00
WolfandClaude Opus 3abb44e10a Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:12:14 +02:00
WolfandClaude Sonnet fa4e1a657f Keep a board's stated pay as salary_text, even when it does not normalise
Every ats parser now writes one human-readable pay line whether or not it
converts to yearly salary_min/salary_max/currency: hourly and daily rates,
Greenhouse's several pay zones, and a range with no currency all survive as
text instead of being dropped outright. Stored in roles.salary_text
(migration 12), replaced whole on every pull like title, so a board that
stops stating pay clears it rather than leaving a stale line. salary_state
is derived, not stored ('normalised', 'stated', or ''), and is what the
feed's new Pay filter and badge read; gates and scoring are unchanged, still
reading only the normalised columns.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:08:45 +02:00