Files
jobboard/docs/review-ingest.md
T

13 KiB

Ingest review, 2026-09-29

Review of the path from source to ranking: the source plugins, the per-role text read, embedding and scoring of roles without text, locations, dates, field coverage, and filtering at the source. Five read-only agents worked on a snapshot of the live database (33,881 open roles, schema v17, before the duplicates migration) and on one live board per host. The planner checked the main claims against the code. This review is the input for the plan that follows; it is folded into decisions.md and deleted once that plan lands.

State in one paragraph

The source layer is plugin-style: one module per host in plugins/atshosts/, a contract checked at import, and 8 parallel workers over batches of 10 boards. Agents work as intended: each role is scored against every prompt of an agent, the best one wins, and votes, favourites, gates and limits belong to each agent. The gaps are in what the modules keep from what the hosts send, in the order the text is fetched, and in the fields the filters rely on. 11,952 open roles (35%) have no text. Extract has never run, so seniority comes from the title alone and pay mostly from three hosts. The live server predates the duplicates merge and has not applied migration 18.

Findings

Ordered by impact on getting the right roles ranked. Effort: S is under a day for an agent, M is a work package.

Text

  1. SmartRecruiters and Workable have no per-role read. 7,391 roles (Bosch, Delivery Hero, Wise, Ubisoft) are ranked on the title alone. Both hosts have a per-posting endpoint. Fix: a detail in smartrecruiters.py and workable.py. M.
  2. The per-role read goes in list order, not by score. plugins/ats.py:667 takes the first 200 unread roles per board. Agent 2's ranks 2 to 11 were SAP roles without text, waiting up to four pulls. Detail reads run one after another inside the pull batch, so batches with such hosts take 300 to 440 s against 10 s for the rest, and the cascade waits 12 minutes. Fix: order the queue by best score (or gate pass for roles not yet scored), then move detail reads into a task of their own after pull, followed by embed and score, on its own schedule. M.
  3. List teasers are dropped. ats.py:703 blanks the description of every role not read in that pull, so the Phenom and Oracle teasers (1,066 roles) are lost, and feed_text is cut from the blanked value. Fix: keep the teaser in feed_text and embed from it until the full text arrives. S.
  4. Only an empty-to-filled description triggers a new embedding. embed.document reads title and description only. A hand-fetched posting (article_text), a teaser replaced by full text, or an edited posting is never embedded again, and extract ignores them too. Jobindex roles keep their 237-character teaser for good. Fix: one text rule, article_text or description or feed_text, in embed and extract; a hash of the embedded text on vectors, re-embedding when it changes; the fetch endpoint queues embed and score. S to M.
  5. Lever keeps only the intro paragraph. lever.py:102 drops lists[] and additional, about 80% of the text on Spotify's board. Recruitee drops requirements the same way. S.
  6. Extract would mark a title-only role done for good. When text arrives, upsert_role drops the vector but not extracted_at. Harmless today since extract is off; must be fixed before switching it on. S.

Roles without text in the ranking

Title-only roles are not shut out. They hold 15 of agent 2's top 30, 47 of the top 200 and 96 of the top 500. Their fit is noisy rather than inflated: on 400 roles with text, the title alone correlates at 0.72 (Spearman) with the full text, and 18 of the top 40 stay in the top 40. Findings 1 to 3 fix the cause. Until then, a "title only" mark in the feed shows which scores are guesses.

Locations

  1. Seven hosts keep one place of several. Ashby secondaryLocations (147 of 831 at OpenAI), Lever allLocations (37 of 79 at Spotify), Personio additionalOffices (116 of 320), Greenhouse offices (120 of 401 at Cloudflare), Workable locations[], Recruitee locations[], Platsbanken workplace_addresses[]. SuccessFactors' detail keeps the last address of several (successfactors.py:402) and then withholds the list's "+2 more". Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold and Jibe keep them all, joined with "; ", which the place matching reads word by word. Fix: a shared join_places() in _shared.py and every module uses it. S.
  2. Placeholders stored as places. Greenhouse "Hybrid" or "In-Office" (340 Cloudflare roles, real office in offices), Workday "N Locations" until read, empty locationsText at LEGO (192 roles, city in the URL path), Recruitee "Remote job", Workday "Country - City" which country_of does not read (63 roles). S each.
  3. A role with a place but no country is gated wrongly. The place gate has evidence whenever place is filled, the country gate has none when country is empty, so a German role at Zeiss in "Oberkochen; Jena" fails agent 2's any group. 1,262 open roles. A joined line over several countries becomes EU, which the country gate does not expand: 714 roles, 265 of them naming a Danish, Swedish or German place. Fix: fill the country from the place per town, and match EU by its places. S to M.
  4. Postings copied per location are not one role. 1,741 company-and-title groups (5,021 roles) are one posting per location; dupes.py leaves them apart on purpose. Nothing is lost, the feed is only noisier. Owner decision whether to fold them.

Remote

  1. Ashby marks hybrid roles remote. ashby.py:116 maps isRemote, which is true for hybrid roles; workplaceType is the right field. 2,537 Ashby roles are stored remote, about 23 of 550 at OpenAI really are. Every remote gate passes them. S.

Dates

  1. Eightfold and Phenom map a refresh date. Eightfold postedTs is refreshed in bulk (a whole page says today, creationTs is up to eight weeks older). Phenom postedDate is the same; dateCreated is the posting date. Fix both mappings, then trust Eightfold. S.
  2. Untrusted hosts that can be trusted. Jibe posted_date equals the page's datePosted; HR Manager's pubDate is monotone in the id; Deel's dates predate first sight by the same margin as Ashby's; Gem's detail date is plausible. Add them to TRUSTED_POSTED_HOSTS, better as a TRUST_POSTED attribute on each module so a new host states it. Pinpoint states no date. Platsbanken's local time is read as UTC. S.

Coverage and filters

Field Filled on open roles Source
Text 65% hosts; 7,391 never, 4,549 waiting for the per-role read
Seniority 43% title words only; extract never ran
Pay figures 12% Greenhouse, Ashby, Lever almost alone
Remote 25% hosts that state it, Ashby wrong
Country 94% place lookup
Permit, languages 0% extract only
  1. Seniority stays near half from titles. A hand sample of 59 roles without a level was 85% individual contributors with no level word, one manager. Better title rules add about one point. The text adds about nine: 2,937 of them state "N years of experience". Three hosts state a level and none is mapped: Personio seniority (all 320 roles), SmartRecruiters experienceLevel (about half), Recruitee experience_code; Jobylon experience. Fix: map the coarse levels (executive, director, entry, student) from those hosts, and let extract read the rest. S, plus extract on.
  2. Pay in the text but not in the fields. About 1,470 roles state a currency range in the text only, mostly Greenhouse and mostly US. Extract, or a regex with the period, would fill them. S to M.
  3. The seniority filter is only useful with "no level" on. It passes an empty level by default (store.py:912). In agent 2's top 200, "manager and up" keeps 37 known and 70 unknown and drops 93; with "no level" off it keeps 37. 14 of the top 200 are interns or working students, since the agent has no rule on level.

Losing roles

  1. Caps leave boards that never close. At MAX_PAGES (Eightfold 1,000, Workday 2,000, Workable 1,000) a read is incomplete and closes nothing, so dead roles stay open and roles beyond the cap never arrive. No attached board is near a cap today; Infineon and Microsoft are not attached because of it. Fix: filtering at the source (below), and closing an incomplete board's roles after N incomplete pulls by age.
  2. A read that returns nothing is final. ats.py:696 marks a role done when detail returns without text (Oracle closed requisitions, empty SuccessFactors blocks). Phenom raises instead and retries every pull. Fix: retry an empty read after some days. S.
  3. An empty answer closes a board. SmartRecruiters, Ashby and Gem answer a complete empty list for an unknown id; the shrink guard below covers it. S.
  4. A disabled agent keeps stale scores, shown if it is picked in the feed. Extract and summarise pick by date, not score. S each.

Jobindex and robots.txt

  1. Our Jobindex requests match its robots.txt Disallow (/jobsoegning*sort=, *page=, *geoareaid=), and the search providers read no robots.txt. This contradicts the decision that robots.txt is honoured. Owner decision: drop Jobindex, read only what robots.txt allows (no paging, no area), or accept it for this source.

Filtering at the source

Of 33,881 open roles, 25,480 are gated for agent 2. 13,558 of those sit on hosts that filter by place server-side. Greenhouse and Ashby, which hold most of the rest, ignore every place parameter.

Host Parameter Behaviour, verified live Roles saved
SmartRecruiters ?country=de one country per request; DE+SE+DK sum exactly to what we hold; remote ignored 6,210 of 7,238
SuccessFactors optionsFacetsDD_country=DK works on 4 of 4 tenants; no remote 2,880 of 3,738
Workday appliedFacets with the tenant's location facet ids read from the first answer; multi-location roles kept; remote is a separate facet on some tenants 1,765 of 2,377
Eightfold location=Sweden one country per request; multi-location roles kept; makes Infineon (1,288 to 163) and Microsoft (2,320 to 60) readable 467 at Ericsson, plus the two new boards
Phenom selected_fields.country list list unions; multi-location roles kept; no remote 903 of 1,007
Oracle selectedLocationsFacet ids, ORA_REMOTE ids per pod, small countries not always listed; unknown id returns the whole board 761 of 826
Workable, Jibe, Lever, Teamtailor country name, or exact city small boards under 1,000 together
Greenhouse, Ashby, Recruitee, the rest none 0

Design: the reach is derived at pull time from the enabled agents' any gates (countries, towns mapped to their country, remote), the union over agents, and all when any agent has a non-place member in that group. A starred company uses the require reach (Europe). A board can pin all or its own reach in subscriptions.params. Only boards above about 300 roles are filtered. A host module gets an optional queries(get, slug, reach) that returns one request series per country; _paged unions the series by id, and the read is complete only when every series is. A complete filtered read closes the roles outside the filter on purpose; a wider filter reopens them. Guards: roles with a vote, application or note are not closed; a filtered read that returns nothing or under 10% of the board counts as failed; a host that ignores the parameter returns a superset, which is safe. Remote roles outside the wanted countries are lost on the four hosts that cannot ask for remote, about 120 today. Rows from a single-country series carry that country, which also fixes finding 9 for those hosts. This is package 11 of docs/plan-hosts-wave-2.md, with the reach derived from the agents instead of stored by hand. Order: SmartRecruiters, Eightfold, Workday and SuccessFactors, Phenom and Oracle.

Plugin contract

Adding a host is one file, but the contract does not cover: the date trust (a set in app/models.py), fields beyond a fixed list (ats.py builds raw from fixed keys, so a host cannot state country, seniority or employment type), and Row as a bare dict. search.py is outside the contract: its own getter, no robots.txt check, no size cap, three providers in one file. Repeated in many modules and worth moving to _shared.py: HTML to text (16 modules), the remote word map (11, spelled differently, the cause of finding 11), pay lines (5), place joining (done by hand in 2).

Order proposed

  1. Restart the server so migration 18 applies.
  2. Text: teaser kept (3), one text rule and re-embed on change (4), extract reset (6), Lever and Recruitee bodies (5). S each, disjoint files.
  3. Locations and remote: join_places in every module (7), placeholders (8), Ashby workplaceType (11), country from place and EU in the gate (9).
  4. Dates (12, 13) and host-stated seniority (14): small per-module changes, can ride with step 3.
  5. Detail read by score in its own task (2), then SmartRecruiters and Workable detail (1).
  6. Filtering at the source.
  7. Extract on for each agent's top 500, then re-measure seniority and pay coverage before relying on those filters.