Keep only the per-host filter parameters in the filter survey and name all twelve readers that declare a filter

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-10-05 09:30:01 +02:00
1 parent caed82d416
commit 1b24de6980
1 file changed
+2 -92
+2 -92
View File
@@ -1,6 +1,6 @@
# Survey: which filters the hosts take at the source
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. It covers the nine readers that declare a filter (SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken, Adzuna) and Workable, Teamtailor, Jibe and Lever, which declare none. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed. Sections 2 and 3 began as the plan for the fetch terms, which it called F2; F2 has landed (`decisions.md`, Fetch terms), and those sections now say what was recommended and what became of it.
Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. Twelve readers declare a filter in `READS.filters`: SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom, Eightfold, Jobindex, Platsbanken and Adzuna were surveyed; freehire, Jobnet and Jobbank came later and were not, and their parameters are in the reader's docstring. Workable, Teamtailor, Jibe and Lever declare none and were surveyed. Greenhouse, Ashby, Recruitee, Personio, Pinpoint, Deel, HiBob, HR Manager, Gem, Jobylon, Rippling and BambooHR take no filter at the source and were not surveyed.
Marks used in the table:
@@ -12,7 +12,7 @@ Marks used in the table:
All request counts are exact: a host with a "live" mark on a claim spent its two requests on it, so anything else about that host is `code`, `front end` or `untested`.
## 1. Filters per host
## Filters per host
Country and remote are left out; they are in the readers already. "Combine" is how a keyword and a country (or another facet) meet; "unknown" is what an unknown value does.
@@ -149,93 +149,3 @@ The environment has no key, so nothing was requested. From the public API docume
| category | `occupation-field=apaJ_2ja_LuF` with `q=python` answered 497, the first hit `Data/IT` | untested | | untested | live |
| place | `region=<län id>` (reader); `municipality=AvNB_uwa_6n6` returned a Stockholm hit | untested | | reader raises on an id outside `REGIONS` | live |
| level | `experience=false` was in the same request as `municipality` and `occupation-field`; the answer was 1 hit, so it is not separable | | | | live, inconclusive |
## 2. Towns
**How a town reaches the fetch today.** An agent's `any` gate on `place` holds names ("Malmö", "Lund", "Helsingborg", "copenhagen", "odense" in agent 2). In `reach.gate_reach`, each name goes through `gaz.lookup` and `_countries_of`, which turns it into the country the place lies in; a name the gazetteer does not know makes the whole reach everywhere. The reach is a set of ISO codes and a remote flag; it holds no town. `ask_for` narrows it to the reader's declared dimensions, so `Ask.countries` gets `SE` and `DK` and `Ask.places` is empty for every board host, since none declares `place`. The gates then keep only the towns after the pull. A town gate therefore fetches its whole country.
For agent 2 this costs nothing: its `any` group has DK and SE beside the towns, so the towns are already covered. It costs where the group has towns and no country, and the country is large (Munich as DE).
**Which hosts could take a town instead.**
- Could, by what was seen: Eightfold (`location=`, its facet values are town strings; untested), Workable (`location` objects with `city`; untested body), Lever (`location=Stockholm`, live: exact text, matches any of a role's places), Adzuna (`where`, `location0..7`), Jobindex (`geoareaid`), Platsbanken (`municipality`).
- Could, with facets: Workday if a tenant has a city facet (none seen: LEGO's is countries only, and `locationMainGroup` holds a country facet), SuccessFactors (`locationsearch=`, untested), Phenom (`location` and `locationData` in the body, untested), Jibe (a city facet is listed; the parameter is not known).
- Cannot: SmartRecruiters has `city=` and `region=` in its documentation but it is untested; Oracle lists ten countries and sites only; Teamtailor takes ids.
- Only the search providers do so today (`Filters(place="one")` on Adzuna, Jobindex and Platsbanken).
**Recommendation.** Leave towns as their country for the board hosts, and build `place` only where it saves a lot and the town is the agent's only reach:
1. The saving is small in the live data. For agent 2, the DK and SE reach already leaves at most 149 roles on any big board (Ericsson), so a town would cut a country read of one or two requests.
2. Town strings are the host's own, unlike ISO codes: each host needs a name-to-string table or a facet lookup, and a town the host spells differently returns nothing and, on the closing guard, wrongly closes roles.
3. Hosts that state their places per role (all the big ones) leave the exact match to the gates and the gazetteer, which is already right.
So: the plan was to carry `places` beside `countries` in the reach for a group that has towns and no country, and to pass them only to readers that declare `place`. That was not built. `Reach` (`app/jobs/reach.py`) carries countries and remote, a town in a gate is fetched as its country (`decisions.md`, Fetch terms), and the three search providers take their `place` from the saved params (`PARAMS`), never from a reach. Eightfold and Lever wait for a run that shows the town strings work.
## 3. What was recommended, and what landed
Landed: `query` on SmartRecruiters, SuccessFactors, Workday, Phenom, Eightfold and Oracle (`Filters(query="one")`). Not built: the rows below marked "no" or "later", and the rows for Workable, Teamtailor and Jibe, which declare no `query` yet.
### Declarations per host
| Host | `query` | `category` | `level` | `place` |
|---|---|---|---|---|
| SmartRecruiters | yes: `q`, one request per term (multi-word behaviour untested) | no: the parameter is not verified | no | no |
| Eightfold | yes: `query` | later: `filter_function`, ids learnt from `filterDef` | no: no tenant filter seen | later: `location=<town string>` after a test |
| SuccessFactors | yes: `q` | no: `department` gave a wrong answer for an unknown value and its values need script | no: the "Work Experience" facet is a tenant label | no |
| Phenom | yes: `keywords` | later: `selected_fields.category` needs the value list | no: `positionLevel` is empty | no |
| Workday | yes: `searchText` | later: `jobFamily` ids from the existing `facets` call | no | no |
| Oracle | yes: `keyword` | later: `selectedCategoriesFacet` ids from `categoriesFacet` | no | no |
| Workable | not built: `query` answered in the survey | no: the body is not settled (400) | no | no |
| Teamtailor | not built: needs a baseline request that confirms the feed filters | no | no | no |
| Jibe | not built: `keywords` | later: `categories` | no | no |
| Lever | no: no keyword parameter; boards are small and read whole | no | no | later, as above |
| Jobindex, Platsbanken, Adzuna | already `query` and `place`; `subid`, `occupation-field` and `category` are available for `category` | | | |
`level` is not worth declaring on any host: only SuccessFactors and Jibe have a facet for it, and on the tenants seen it is a tenant label or empty (Lever documents `level=` and SmartRecruiters states `experienceLevel` on each role; neither is filtered on). Seniority stays a prep matter (title rules, host word).
### What F2 built, and why in this order
The table shows how much terms can narrow big boards, from the live database of 2026-09-30:
| Board | Host | Open | In Europe | In DK or SE (today's reach) | Roles with a term in the title, Europe |
|---|---|---|---|---|---|
| Bosch | SmartRecruiters | 4,830 | 1,564 | 15 | 149 |
| Delivery Hero | SmartRecruiters | 975 | 314 | 11 | 143 |
| Coolblue | SmartRecruiters | 477 | 477 | 0 | 30 |
| Wise | SmartRecruiters | 417 | 279 | 0 | 160 |
| SAP | SuccessFactors | 807 | 365 | 6 | 128 |
| Scania | SuccessFactors | 750 | 532 | 58 | 28 |
| Vestas | SuccessFactors | 668 | 327 | 61 | 56 |
| Novo Nordisk | SuccessFactors | 428 | 138 | 93 | 45 |
| Philips | Phenom | 868 | 194 | 3 | 65 |
| Zeiss | Workday | 848 | 264 (295 with no country) | 1 | 59 |
| LEGO | Workday | 455 | 175 (45 with none) | 83 | 60 |
| Ericsson | Eightfold | 618 | 210 | 149 | 64 |
| Nokia | Oracle | 554 | 87 (54 with none) | 0 | 24 |
"Term in the title" is a count of the words head, director, VP, chief, CTO, CDO, data, AI, machine learning, forward deployed, solutions engineer or architect, professional services, research, innovation, principal, lead and manager in the title. It is a proxy for `fetch_terms`, and a lower bound on what a text filter returns: where the scope could be told, the keyword matched text and not only titles (SmartRecruiters, Eightfold and Oracle by the examples here; on SuccessFactors, Workday, Workable and Jibe it is not determined) (Ericsson: `software` matched a thesis title without it; Nokia: `engineer` matched 468 of 554; Bosch: 1,713 of 4,830). Expect a keyword read to return several times these counts, and a broad term like `engineer` to cut little.
Findings:
1. **Narrow reach needs no terms.** At DK and SE every big board is already at most 149 roles (Ericsson), and Bosch is two requests. Terms save requests only on the wide reach, which is the All countries tag; a company set to All roles takes no terms, and a favourite no longer widens the fetch (`decisions.md`).
2. **The wide reach is where terms pay.** SmartRecruiters reads Europe (33 countries) as the whole board, because `MAX_SERIES` is 8 for a `one` host: Bosch is 49 requests of 100 for 1,564 European roles. One `q` request per term (about 35% of the board for `engineer`, far less for `director` or `data`) replaces it.
3. **The order followed the size of the whole read that terms replace**, then how sure the parameter was:
1. SmartRecruiters `q`: Bosch, Delivery Hero, Wise and Coolblue, 6,700 roles in the live DB, one live-verified parameter, AND with country.
2. SuccessFactors `q`: SAP, Scania, Vestas and Novo Nordisk, 2,650 roles; verified. The reader already reads a country a request, so a term multiplies the series.
3. Workday `searchText` (Zeiss 848, LEGO 455), Phenom `keywords` (Philips 868), Eightfold `query` (Ericsson 618) and Oracle `keyword` (Nokia 554): each verified once, each a small change to the request body or URL.
4. **Category comes next, and only for Workday, Oracle, Jibe and Eightfold** (Phenom's needs its value list first). They state their own category ids in a response the reader already reads or can read (`facets`, `categoriesFacet`, `filter.categories`, `filterDef`), and the union across values is verified on Workday and Oracle. But an agent's term (`engineer`) is not a category name, so it needs a mapping from term to category per tenant that nobody has: not before terms have run on real boards.
5. **Level is not filtered at the source.** Place is filtered by the search providers; Eightfold and Lever wait for a test that proves a town string, and the rest stay on country.
### Guards the fetch terms keep
- A term filter needs the closing guards to hold: a keyword read is not a whole board, so it does not write `ats:size` (a pull no longer rewrites a size at all) and roles outside it are not closed as gone (`narrowed` in `plugins/readers/_runner.py`, `close_boards` in `app/jobs/pull.py`).
- Unknown-value behaviour differs: Lever returns nothing, Oracle returns everything, SuccessFactors returned something unrelated, Workable answered 400 to a body whose refused key is not known. The runner treats a facet value as known only when the host's own facet list stated it, as `series` does for ids.
- Keywords are text matches everywhere except Adzuna's `title_only`; the agent's own scoring still decides relevance.
### Open, and cheap to settle with one request each
- SmartRecruiters: does `q="head of data"` mean all words or any, and are `department`, `function`, `experienceLevel` accepted (one request each)?
- Teamtailor: a baseline request to confirm the feed's `query` filter.
- Workable: which key of the survey's body was refused (`department` or `location`).
- Eightfold: a `location` with a town string, and repeated `filter_function` values.
- Adzuna: everything, once the key exists.