diff --git a/docs/open.md b/docs/open.md index ccb535f..c80cd71 100644 --- a/docs/open.md +++ b/docs/open.md @@ -5,11 +5,11 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la ## Text - **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only `summarise` reads. `embed` and `extract` read `description or teaser`, so a Jobindex role keeps being ranked on its teaser. The fix is one text rule, `article_text or description or teaser`, in embed and extract, and queuing embed and score after a fetch. -- **An edited posting is never embedded again.** The vector is dropped only when a teaser is replaced by a description. A hash of the embedded text on `vectors` would re-embed on any change. +- **An edited posting is never embedded again.** The vector is dropped only when the text ranks better than before (a title-only role gaining a teaser, or a teaser replaced by a description). A hash of the embedded text on `vectors` would re-embed on any change. - **Extract marks a title-only role done for good.** `upsert_role` does not reset `extracted_at` when text arrives. Extract is on, so this bites. `_EXTRACT_WAITING` re-reads roles whose `seniority_model` is empty, which covers level and not pay. - **Lever keeps only the intro paragraph.** `lever.py` reads `description` and drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way. -- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 16.5% of open roles. Extract can return `salary_min` and `salary_max` but has read about 500 roles so far. -- **A title-only role has no mark in the feed.** 1,304 open roles have no text, 869 of them Eightfold. A mark would show which fits are guesses. +- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 17.9% of open roles. Extract can return `salary_min` and `salary_max` but has read 554 roles so far. +- **A title-only role has no mark in the feed.** 1,180 open roles have no description, teaser or fetched posting: 677 of them Workday, 256 SmartRecruiters, 150 Workable, 57 SuccessFactors and 26 Eightfold. A mark would show which fits are guesses. ## Ranking @@ -19,17 +19,34 @@ What is known and not yet fixed, checked against the code on 2026-09-30. What la ## Sources -- **The boilerplate moved on.** `deploy/` is at boilerplate `1679e88` and lacks `wsl-setup.sh` and `wsl-setup.ps1`; `pipeline/` and `auth/` differ from boilerplate HEAD (three languages, installable phone). Refresh them with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies. +- **The boilerplate moved on.** `pipeline/` equals boilerplate `a1c6f0d` once line endings are ignored: 87 files of the working tree have CRLF although the index has LF, so a plain `diff -r` shows every file as different (use `--strip-trailing-cr`). `auth/` lacks `c6a90ff`, the phone commit (`auth/README.md`, `auth/static/account.js`, `auth/templates/login.html`). `deploy/` is at `1679e88` and lacks the WSL scripts of `e1b3746` (`wsl-setup.sh`, `wsl-setup.ps1`), and `docs/release.md` lacks its `wsl.md` sentence. `desk/` is news `a275c38` less the prune and lags news by four commits (`b0e1f6f`, `f01a6bd`, `3bb6775`, `7b9c8a2`). The three languages of boilerplate `7bb7da2` are starter code in `app/`, not part of the copies. Refresh with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies. - **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them. - **Six hosts' dates are not trusted.** Eightfold, Jibe, Gem, Deel, HiBob and HR Manager declare `posted="listed"`, so their backlog counts as posted when first seen. Eightfold now maps `creationTs`, Jibe's `posted_date` equals the page's `datePosted`, and HR Manager's `pubDate` rises with the id; each needs one check on a live board before it declares `posted="trusted"`. Platsbanken's local time is read as UTC. -- **A mistyped SmartRecruiters or Gem slug attaches as an empty board.** Both answer an empty listing for an unknown id. `empty="suspect"` and the 10% guard keep a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles". +- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board.** All three answer an empty listing for an unknown id and declare `empty="suspect"`. The 10% guard keeps a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles". - **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the runner waits `PAGE_DELAY`, 0.5. A read of its board is two requests. - **A SuccessFactors site on a company's own domain is not discovered.** Its pages link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand. -- **HR Manager: decisions and data disagree.** `decisions.md` and `plugins/README.md` say no attached company uses it, but the live database has `hrmanager:teknologisk` attached with 0 open roles. Seaborg stays unattached until the owner decides whether the feed counts as allowed. -- **Stripe's places carry a "Locations" suffix.** 282 open roles have place text like "Singapore Locations" or "Dublin; Ireland Locations". +- **HR Manager is attached although its robots.txt disallows reading.** `hrmanager:teknologisk` (the Danish Technological Institute) has 7 open roles, while `READS.robots` says "disallowed by robots.txt; owner to decide". Seaborg stays unattached until the owner decides whether the feed counts as allowed. +- **Whether SmartRecruiters' identifier is case-sensitive is unsettled.** The module docstring says it is (`BoschGroup`), `decisions.md` says it is not, checked live; the live boards hold the identifiers with case. A live request with a lowercased identifier settles it. +- **The seed lags the live database.** The seed has 286 boards and the database 310: it lacks Microsoft, Infineon, Novonesis, Darktrace, Criteo, NVIDIA, HR Manager and seventeen more, and names five of them as `unread` or `robots`. `app companies seed` never replaces a board, so the seed is brought in line by hand. +- **Stripe's places carry a "Locations" suffix.** 316 open roles (285 without copies) have place text like "Singapore Locations" or "Dublin; Ireland Locations". ## The pages - **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite. - **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it. - **`company_key` keeps `ø`.** Aliases make "Orsted" and "Ørsted" one company; folding the key itself would change the primary key of `companies`. + +## From the review of 2026-09-30 + +- **Remote roles are not read from hosts that cannot filter on remote.** On SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" is asked as "DK, SE", and the filtered read closes remote roles outside them: 72 on 2026-09-29 and 30 (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3). Reading such boards whole is the owner's call; Bosch is about 70 requests a pull. +- **Provider names are hard-coded in nine places** (`cli.py`, `dupes.py`, `sources.py`, `pages.py`, `sources.html`, `sources.js`, `api.py`, `posting.py`) instead of `readers.PROVIDERS`, and `has_key` re-implements `resolve_key`. +- **`ats.py` keeps compatibility names that only tests reach for:** `_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused` and `DETAIL_FIELDS`. +- **`title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`.** Wave 2's WP9 must own `_runner.py` (`plan-hosts-wave-2.md`). +- **Some `posted="trusted"` declarations rest on the detail read** (Workday, BambooHR, Rippling) **or on a fallback the host moves** (Greenhouse `updated_at`, Phenom `postedDate`). +- **Dupes fold same-host roles with the same title and town without comparing text.** 99 open copies share under 30% of their words with the kept role. +- **Enrich never reads a role below the bar.** 349 roles without text are ranked on their title for good. +- **A pull never re-measures a board's size.** Recheck is the only way, so a board that grows past 300 roles stays read whole until someone rechecks it. +- **Fetch terms are configured on no agent**, so no board is read with terms today. +- **A starred company on a big filtered board is scored on roles the fetch closes.** All countries is the workaround. +- **`SMALL` (300), `STALE_DAYS` (21), the guard's 10% and `DETAIL_CAP` (200) are code constants**, and the pages hard-code 300 in tooltips. +- **`The todo list.md` holds items the code has done** (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.