Refresh open.md: numbers, boilerplate drift, HR Manager, empty slugs, and the review's unbuilt items

Update the stale counts, state which boilerplate copies lag and by what, record HR Manager as attached against its robots record, add Ashby to the empty-slug item, the SmartRecruiters case question and the seed lag, and add a section for what the review of 2026-09-30 does not build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-30 15:43:27 +02:00
1 parent 597ce9ca39
commit e1b465f405
1 file changed
+24 -7
+24 -7
View File
@@ -5,11 +5,11 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la
## Text
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only `summarise` reads. `embed` and `extract` read `description or teaser`, so a Jobindex role keeps being ranked on its teaser. The fix is one text rule, `article_text or description or teaser`, in embed and extract, and queuing embed and score after a fetch.
- **An edited posting is never embedded again.** The vector is dropped only when a teaser is replaced by a description. A hash of the embedded text on `vectors` would re-embed on any change.
- **An edited posting is never embedded again.** The vector is dropped only when the text ranks better than before (a title-only role gaining a teaser, or a teaser replaced by a description). A hash of the embedded text on `vectors` would re-embed on any change.
- **Extract marks a title-only role done for good.** `upsert_role` does not reset `extracted_at` when text arrives. Extract is on, so this bites. `_EXTRACT_WAITING` re-reads roles whose `seniority_model` is empty, which covers level and not pay.
- **Lever keeps only the intro paragraph.** `lever.py` reads `description` and drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way.
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 16.5% of open roles. Extract can return `salary_min` and `salary_max` but has read about 500 roles so far.
- **A title-only role has no mark in the feed.** 1,304 open roles have no text, 869 of them Eightfold. A mark would show which fits are guesses.
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 17.9% of open roles. Extract can return `salary_min` and `salary_max` but has read 554 roles so far.
- **A title-only role has no mark in the feed.** 1,180 open roles have no description, teaser or fetched posting: 677 of them Workday, 256 SmartRecruiters, 150 Workable, 57 SuccessFactors and 26 Eightfold. A mark would show which fits are guesses.
## Ranking
@@ -19,17 +19,34 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la
## Sources
- **The boilerplate moved on.** `deploy/` is at boilerplate `1679e88` and lacks `wsl-setup.sh` and `wsl-setup.ps1`; `pipeline/` and `auth/` differ from boilerplate HEAD (three languages, installable phone). Refresh them with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
- **The boilerplate moved on.** `pipeline/` equals boilerplate `a1c6f0d` once line endings are ignored: 87 files of the working tree have CRLF although the index has LF, so a plain `diff -r` shows every file as different (use `--strip-trailing-cr`). `auth/` lacks `c6a90ff`, the phone commit (`auth/README.md`, `auth/static/account.js`, `auth/templates/login.html`). `deploy/` is at `1679e88` and lacks the WSL scripts of `e1b3746` (`wsl-setup.sh`, `wsl-setup.ps1`), and `docs/release.md` lacks its `wsl.md` sentence. `desk/` is news `a275c38` less the prune and lags news by four commits (`b0e1f6f`, `f01a6bd`, `3bb6775`, `7b9c8a2`). The three languages of boilerplate `7bb7da2` are starter code in `app/`, not part of the copies. Refresh with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
- **Six hosts' dates are not trusted.** Eightfold, Jibe, Gem, Deel, HiBob and HR Manager declare `posted="listed"`, so their backlog counts as posted when first seen. Eightfold now maps `creationTs`, Jibe's `posted_date` equals the page's `datePosted`, and HR Manager's `pubDate` rises with the id; each needs one check on a live board before it declares `posted="trusted"`. Platsbanken's local time is read as UTC.
- **A mistyped SmartRecruiters or Gem slug attaches as an empty board.** Both answer an empty listing for an unknown id. `empty="suspect"` and the 10% guard keep a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board.** All three answer an empty listing for an unknown id and declare `empty="suspect"`. The 10% guard keeps a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the runner waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
- **A SuccessFactors site on a company's own domain is not discovered.** Its pages link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
- **HR Manager: decisions and data disagree.** `decisions.md` and `plugins/README.md` say no attached company uses it, but the live database has `hrmanager:teknologisk` attached with 0 open roles. Seaborg stays unattached until the owner decides whether the feed counts as allowed.
- **Stripe's places carry a "Locations" suffix.** 282 open roles have place text like "Singapore Locations" or "Dublin; Ireland Locations".
- **HR Manager is attached although its robots.txt disallows reading.** `hrmanager:teknologisk` (the Danish Technological Institute) has 7 open roles, while `READS.robots` says "disallowed by robots.txt; owner to decide". Seaborg stays unattached until the owner decides whether the feed counts as allowed.
- **Whether SmartRecruiters' identifier is case-sensitive is unsettled.** The module docstring says it is (`BoschGroup`), `decisions.md` says it is not, checked live; the live boards hold the identifiers with case. A live request with a lowercased identifier settles it.
- **The seed lags the live database.** The seed has 286 boards and the database 310: it lacks Microsoft, Infineon, Novonesis, Darktrace, Criteo, NVIDIA, HR Manager and seventeen more, and names five of them as `unread` or `robots`. `app companies seed` never replaces a board, so the seed is brought in line by hand.
- **Stripe's places carry a "Locations" suffix.** 316 open roles (285 without copies) have place text like "Singapore Locations" or "Dublin; Ireland Locations".
## The pages
- **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite.
- **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it.
- **`company_key` keeps `ø`.** Aliases make "Orsted" and "Ørsted" one company; folding the key itself would change the primary key of `companies`.
## From the review of 2026-09-30
- **Remote roles are not read from hosts that cannot filter on remote.** On SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" is asked as "DK, SE", and the filtered read closes remote roles outside them: 72 on 2026-09-29 and 30 (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3). Reading such boards whole is the owner's call; Bosch is about 70 requests a pull.
- **Provider names are hard-coded in nine places** (`cli.py`, `dupes.py`, `sources.py`, `pages.py`, `sources.html`, `sources.js`, `api.py`, `posting.py`) instead of `readers.PROVIDERS`, and `has_key` re-implements `resolve_key`.
- **`ats.py` keeps compatibility names that only tests reach for:** `_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused` and `DETAIL_FIELDS`.
- **`title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`.** Wave 2's WP9 must own `_runner.py` (`plan-hosts-wave-2.md`).
- **Some `posted="trusted"` declarations rest on the detail read** (Workday, BambooHR, Rippling) **or on a fallback the host moves** (Greenhouse `updated_at`, Phenom `postedDate`).
- **Dupes fold same-host roles with the same title and town without comparing text.** 99 open copies share under 30% of their words with the kept role.
- **Enrich never reads a role below the bar.** 349 roles without text are ranked on their title for good.
- **A pull never re-measures a board's size.** Recheck is the only way, so a board that grows past 300 roles stays read whole until someone rechecks it.
- **Fetch terms are configured on no agent**, so no board is read with terms today.
- **A starred company on a big filtered board is scored on roles the fetch closes.** All countries is the workaround.
- **`SMALL` (300), `STALE_DAYS` (21), the guard's 10% and `DETAIL_CAP` (200) are code constants**, and the pages hard-code 300 in tooltips.
- **`The todo list.md` holds items the code has done** (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.