From 667e1d35f49f51ec3de4a5c5fac780ee7a46659b Mon Sep 17 00:00:00 2001 From: Wolf Date: Tue, 29 Sep 2026 23:43:19 +0200 Subject: [PATCH] Review the ingest path: text, places, dates, coverage and filtering at the source Co-Authored-By: Claude Opus 5.5 --- docs/review-ingest.md | 97 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/review-ingest.md diff --git a/docs/review-ingest.md b/docs/review-ingest.md new file mode 100644 index 0000000..fd85a6e --- /dev/null +++ b/docs/review-ingest.md @@ -0,0 +1,97 @@ +# Ingest review, 2026-09-29 + +Review of the path from source to ranking: the source plugins, the per-role text read, embedding and scoring of roles without text, locations, dates, field coverage, and filtering at the source. Five read-only agents worked on a snapshot of the live database (33,881 open roles, schema v17, before the duplicates migration) and on one live board per host. The planner checked the main claims against the code. This review is the input for the plan that follows; it is folded into `decisions.md` and deleted once that plan lands. + +## State in one paragraph + +The source layer is plugin-style: one module per host in `plugins/atshosts/`, a contract checked at import, and 8 parallel workers over batches of 10 boards. Agents work as intended: each role is scored against every prompt of an agent, the best one wins, and votes, favourites, gates and limits belong to each agent. The gaps are in what the modules keep from what the hosts send, in the order the text is fetched, and in the fields the filters rely on. 11,952 open roles (35%) have no text. Extract has never run, so seniority comes from the title alone and pay mostly from three hosts. The live server predates the duplicates merge and has not applied migration 18. + +## Findings + +Ordered by impact on getting the right roles ranked. Effort: S is under a day for an agent, M is a work package. + +### Text + +1. **SmartRecruiters and Workable have no per-role read.** 7,391 roles (Bosch, Delivery Hero, Wise, Ubisoft) are ranked on the title alone. Both hosts have a per-posting endpoint. Fix: a `detail` in `smartrecruiters.py` and `workable.py`. M. +2. **The per-role read goes in list order, not by score.** `plugins/ats.py:667` takes the first 200 unread roles per board. Agent 2's ranks 2 to 11 were SAP roles without text, waiting up to four pulls. Detail reads run one after another inside the pull batch, so batches with such hosts take 300 to 440 s against 10 s for the rest, and the cascade waits 12 minutes. Fix: order the queue by best score (or gate pass for roles not yet scored), then move detail reads into a task of their own after `pull`, followed by embed and score, on its own schedule. M. +3. **List teasers are dropped.** `ats.py:703` blanks the description of every role not read in that pull, so the Phenom and Oracle teasers (1,066 roles) are lost, and `feed_text` is cut from the blanked value. Fix: keep the teaser in `feed_text` and embed from it until the full text arrives. S. +4. **Only an empty-to-filled description triggers a new embedding.** `embed.document` reads `title` and `description` only. A hand-fetched posting (`article_text`), a teaser replaced by full text, or an edited posting is never embedded again, and extract ignores them too. Jobindex roles keep their 237-character teaser for good. Fix: one text rule, `article_text or description or feed_text`, in embed and extract; a hash of the embedded text on `vectors`, re-embedding when it changes; the fetch endpoint queues embed and score. S to M. +5. **Lever keeps only the intro paragraph.** `lever.py:102` drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way. S. +6. **Extract would mark a title-only role done for good.** When text arrives, `upsert_role` drops the vector but not `extracted_at`. Harmless today since extract is off; must be fixed before switching it on. S. + +### Roles without text in the ranking + +Title-only roles are not shut out. They hold 15 of agent 2's top 30, 47 of the top 200 and 96 of the top 500. Their fit is noisy rather than inflated: on 400 roles with text, the title alone correlates at 0.72 (Spearman) with the full text, and 18 of the top 40 stay in the top 40. Findings 1 to 3 fix the cause. Until then, a "title only" mark in the feed shows which scores are guesses. + +### Locations + +7. **Seven hosts keep one place of several.** Ashby `secondaryLocations` (147 of 831 at OpenAI), Lever `allLocations` (37 of 79 at Spotify), Personio `additionalOffices` (116 of 320), Greenhouse `offices` (120 of 401 at Cloudflare), Workable `locations[]`, Recruitee `locations[]`, Platsbanken `workplace_addresses[]`. SuccessFactors' `detail` keeps the last address of several (`successfactors.py:402`) and then withholds the list's "+2 more". Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold and Jibe keep them all, joined with "; ", which the place matching reads word by word. Fix: a shared `join_places()` in `_shared.py` and every module uses it. S. +8. **Placeholders stored as places.** Greenhouse "Hybrid" or "In-Office" (340 Cloudflare roles, real office in `offices`), Workday "N Locations" until read, empty `locationsText` at LEGO (192 roles, city in the URL path), Recruitee "Remote job", Workday "Country - City" which `country_of` does not read (63 roles). S each. +9. **A role with a place but no country is gated wrongly.** The `place` gate has evidence whenever `place` is filled, the `country` gate has none when `country` is empty, so a German role at Zeiss in "Oberkochen; Jena" fails agent 2's `any` group. 1,262 open roles. A joined line over several countries becomes `EU`, which the country gate does not expand: 714 roles, 265 of them naming a Danish, Swedish or German place. Fix: fill the country from the place per town, and match `EU` by its places. S to M. +10. **Postings copied per location are not one role.** 1,741 company-and-title groups (5,021 roles) are one posting per location; `dupes.py` leaves them apart on purpose. Nothing is lost, the feed is only noisier. Owner decision whether to fold them. + +### Remote + +11. **Ashby marks hybrid roles remote.** `ashby.py:116` maps `isRemote`, which is true for hybrid roles; `workplaceType` is the right field. 2,537 Ashby roles are stored remote, about 23 of 550 at OpenAI really are. Every remote gate passes them. S. + +### Dates + +12. **Eightfold and Phenom map a refresh date.** Eightfold `postedTs` is refreshed in bulk (a whole page says today, `creationTs` is up to eight weeks older). Phenom `postedDate` is the same; `dateCreated` is the posting date. Fix both mappings, then trust Eightfold. S. +13. **Untrusted hosts that can be trusted.** Jibe `posted_date` equals the page's `datePosted`; HR Manager's `pubDate` is monotone in the id; Deel's dates predate first sight by the same margin as Ashby's; Gem's detail date is plausible. Add them to `TRUSTED_POSTED_HOSTS`, better as a `TRUST_POSTED` attribute on each module so a new host states it. Pinpoint states no date. Platsbanken's local time is read as UTC. S. + +### Coverage and filters + +| Field | Filled on open roles | Source | +|---|---|---| +| Text | 65% | hosts; 7,391 never, 4,549 waiting for the per-role read | +| Seniority | 43% | title words only; extract never ran | +| Pay figures | 12% | Greenhouse, Ashby, Lever almost alone | +| Remote | 25% | hosts that state it, Ashby wrong | +| Country | 94% | place lookup | +| Permit, languages | 0% | extract only | + +14. **Seniority stays near half from titles.** A hand sample of 59 roles without a level was 85% individual contributors with no level word, one manager. Better title rules add about one point. The text adds about nine: 2,937 of them state "N years of experience". Three hosts state a level and none is mapped: Personio `seniority` (all 320 roles), SmartRecruiters `experienceLevel` (about half), Recruitee `experience_code`; Jobylon `experience`. Fix: map the coarse levels (executive, director, entry, student) from those hosts, and let extract read the rest. S, plus extract on. +15. **Pay in the text but not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and mostly US. Extract, or a regex with the period, would fill them. S to M. +16. **The seniority filter is only useful with "no level" on.** It passes an empty level by default (`store.py:912`). In agent 2's top 200, "manager and up" keeps 37 known and 70 unknown and drops 93; with "no level" off it keeps 37. 14 of the top 200 are interns or working students, since the agent has no rule on level. + +### Losing roles + +17. **Caps leave boards that never close.** At `MAX_PAGES` (Eightfold 1,000, Workday 2,000, Workable 1,000) a read is incomplete and closes nothing, so dead roles stay open and roles beyond the cap never arrive. No attached board is near a cap today; Infineon and Microsoft are not attached because of it. Fix: filtering at the source (below), and closing an incomplete board's roles after N incomplete pulls by age. +18. **A read that returns nothing is final.** `ats.py:696` marks a role done when `detail` returns without text (Oracle closed requisitions, empty SuccessFactors blocks). Phenom raises instead and retries every pull. Fix: retry an empty read after some days. S. +19. **An empty answer closes a board.** SmartRecruiters, Ashby and Gem answer a complete empty list for an unknown id; the shrink guard below covers it. S. +20. **A disabled agent keeps stale scores**, shown if it is picked in the feed. Extract and summarise pick by date, not score. S each. + +### Jobindex and robots.txt + +21. **Our Jobindex requests match its robots.txt `Disallow`** (`/jobsoegning*sort=`, `*page=`, `*geoareaid=`), and the search providers read no robots.txt. This contradicts the decision that robots.txt is honoured. Owner decision: drop Jobindex, read only what robots.txt allows (no paging, no area), or accept it for this source. + +## Filtering at the source + +Of 33,881 open roles, 25,480 are gated for agent 2. 13,558 of those sit on hosts that filter by place server-side. Greenhouse and Ashby, which hold most of the rest, ignore every place parameter. + +| Host | Parameter | Behaviour, verified live | Roles saved | +|---|---|---|---| +| SmartRecruiters | `?country=de` | one country per request; DE+SE+DK sum exactly to what we hold; `remote` ignored | 6,210 of 7,238 | +| SuccessFactors | `optionsFacetsDD_country=DK` | works on 4 of 4 tenants; no remote | 2,880 of 3,738 | +| Workday | `appliedFacets` with the tenant's location facet | ids read from the first answer; multi-location roles kept; remote is a separate facet on some tenants | 1,765 of 2,377 | +| Eightfold | `location=Sweden` | one country per request; multi-location roles kept; makes Infineon (1,288 to 163) and Microsoft (2,320 to 60) readable | 467 at Ericsson, plus the two new boards | +| Phenom | `selected_fields.country` list | list unions; multi-location roles kept; no remote | 903 of 1,007 | +| Oracle | `selectedLocationsFacet` ids, `ORA_REMOTE` | ids per pod, small countries not always listed; unknown id returns the whole board | 761 of 826 | +| Workable, Jibe, Lever, Teamtailor | country name, or exact city | small boards | under 1,000 together | +| Greenhouse, Ashby, Recruitee, the rest | none | | 0 | + +Design: the reach is derived at pull time from the enabled agents' `any` gates (countries, towns mapped to their country, remote), the union over agents, and `all` when any agent has a non-place member in that group. A starred company uses the `require` reach (Europe). A board can pin `all` or its own reach in `subscriptions.params`. Only boards above about 300 roles are filtered. A host module gets an optional `queries(get, slug, reach)` that returns one request series per country; `_paged` unions the series by id, and the read is complete only when every series is. A complete filtered read closes the roles outside the filter on purpose; a wider filter reopens them. Guards: roles with a vote, application or note are not closed; a filtered read that returns nothing or under 10% of the board counts as failed; a host that ignores the parameter returns a superset, which is safe. Remote roles outside the wanted countries are lost on the four hosts that cannot ask for remote, about 120 today. Rows from a single-country series carry that country, which also fixes finding 9 for those hosts. This is package 11 of `docs/plan-hosts-wave-2.md`, with the reach derived from the agents instead of stored by hand. Order: SmartRecruiters, Eightfold, Workday and SuccessFactors, Phenom and Oracle. + +## Plugin contract + +Adding a host is one file, but the contract does not cover: the date trust (a set in `app/models.py`), fields beyond a fixed list (`ats.py` builds `raw` from fixed keys, so a host cannot state country, seniority or employment type), and `Row` as a bare dict. `search.py` is outside the contract: its own getter, no robots.txt check, no size cap, three providers in one file. Repeated in many modules and worth moving to `_shared.py`: HTML to text (16 modules), the remote word map (11, spelled differently, the cause of finding 11), pay lines (5), place joining (done by hand in 2). + +## Order proposed + +1. Restart the server so migration 18 applies. +2. Text: teaser kept (3), one text rule and re-embed on change (4), extract reset (6), Lever and Recruitee bodies (5). S each, disjoint files. +3. Locations and remote: `join_places` in every module (7), placeholders (8), Ashby `workplaceType` (11), country from place and `EU` in the gate (9). +4. Dates (12, 13) and host-stated seniority (14): small per-module changes, can ride with step 3. +5. Detail read by score in its own task (2), then SmartRecruiters and Workable `detail` (1). +6. Filtering at the source. +7. Extract on for each agent's top 500, then re-measure seniority and pay coverage before relying on those filters.