Record the review's decisions in decisions.md and rewrite open.md as what is left: the points for the owner, what the ADRs do not say, and the structure of the plugin layer

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-10-05 10:19:01 +02:00
1 parent e8157d9509
commit e61185e4b9
2 files changed
+86 -47

No files matched your search

+73 -44
View File
@@ -1,55 +1,84 @@
# Open
What is known and not yet fixed, checked against the code on 2026-09-30. What landed is in `decisions.md`; the unbuilt readers are planned in `plan-hosts-wave-2.md`.
What is known and not yet fixed, checked against the code on 2026-10-05, the day of the architecture review. What was decided is in `decisions.md`, the target design in `documentation/`, the unbuilt readers in `plan-hosts-wave-2.md`. Counts are of the live database on that day.
## For the owner to decide
Points where the code and `ADR Fetch prep and score` disagree, or where the ADR is silent and roles are lost.
- **Remote roles on hosts that cannot filter on remote.** On 18 boards (SmartRecruiters, Phenom, SuccessFactors, Eightfold, and Workday tenants without a remote facet) the read for "DK, SE, remote" goes out as "DK, SE". The row says "remote not asked" and the remote roles already held are kept, but a new remote role elsewhere in the world gate is never fetched. ADR 1.2 ("a host that cannot filter on a part sends the board whole for that part") reads as reading such boards whole; Bosch is then about 50 requests a pull.
- **A role outside the agent's locations with remote unstated is never read by extract.** ADR 4.1 says a role can "turn out to be remote", but extract reads only the cut and the cut requires the gate. 5,016 open roles are rejected for location with remote unknown, and 805 of them say "remote" in their text. A word scan in prep costs no tokens and would let those through to extract.
- **Copies across countries.** ADR 1.4 requires the same country and overlapping towns. `dupes.fold` also merges one host's listings of one title across towns and countries when 30% of their words are shared: 2,125 open copies, about 1,550 in another country than the kept role, some of them different openings (Datadog's US federal role behind the Canberra one). The gate reads the kept role through its copies' places, so no office listing is hidden. Remote and office listings are kept apart since 2026-10-05; 176 remote listings still sit behind a remote kept role in another country, whose area alone the world gate reads.
- **Judge and languages.** ADR 4.2 names the agent's languages as an input. The judge does not read them (they only set the flag on a row), and editing them re-judges nothing. The ADR's list also omits the prompts and the must-haves, which the judge reads and which re-judge.
- **Must-haves.** A broken must-have caps career at 4, but `must_text` is set only through the API: it is not on the form and not in the ADR's Settings.
- **Show gated and Advanced rules.** ADR 5 says no order shows a role the gate rejects and that seniority and pay are filters. Show gated lists the rejected roles with the reason, on purpose; an Advanced row can gate on any of twelve fields.
- **Closing by age on boards.** ADR 1.3 closes a role no read has listed for 21 days. That holds for searches and for boards narrowed by terms. A board read that stays incomplete (a ceiling, the page cap without a sweep) closes nothing ever, and on a filtered board a role with any vote or an application never closes. Since closing is not valued, striking the rule from the ADR is the cheaper way to agree.
- **A search cut at its page cap leaves no record.** In the last pull 4 of 26 searches stopped at a cap (Jobbank and freehire at 1,000 rows, two Jobindex searches at 200). Nothing on a page says so, and Jobbank's feed is not sorted by date, so new ads can lie past the cap.
- **A row without a title, link or id is dropped by its reader, unrecorded,** while its id still counts toward completeness. `ADR Plugins` says both "never drops rows" and "title and url are required"; the runner could validate, log and mark the read incomplete.
- **The world gate is not sent to a host when an agent sets no location.** Every big board is then read whole. Nothing is lost.
## What the ADRs do not say
What someone rebuilding from `ADR Fetch prep and score` would have to guess, by section.
- **Prep has no section**, though the title names it: a host's words mapped through a table per host, a place text resolved against a gazetteer of towns, regions, countries and continents with regions across borders, remote and the remote area read from the place words, the seniority ladder, and that a text nothing maps is listed and passes the gate.
- **1.1 Fetch gate.** Every role is stored whatever the gates say, and each agent applies its gate at scoring. An unknown location is no resolved place and no country, which includes an unresolved text and a wider region alone. A role in several places needs one inside. The world gate reads a remote role by its remote area. Exclusion matches whole words of the title.
- **1.2 Reading a board.** The union is per agent the location gate, or the world gate when no location is set, then over the agents. What happens when a host filters countries but not remote. A board's own saved params win over the agents'. With no enabled agent every board is read whole. A search reads 5 pages by default and 10 at most.
- **1.3 Closing.** "Filtered read" is not defined. Switching a board or search off closes its roles; a closed role listed again reopens.
- **1.4 Copies.** The second rule, one host's listings of one title merged when their texts are alike. Groups never chain. Votes and read state move to the kept copy. How titles and company names are normalised.
- **2.1 Closeness.** The embedding model (`intfloat/multilingual-e5-small` with its query and passage prefixes) and what is embedded (title and the first 2,000 characters of the text, else the teaser). The CV is cut into chunks of at most 1,200 characters. A level is a linear clip between two cosines measured per model (prompt 0.75 to 0.88, CV 0.77 to 0.90, job like -0.14 to 0.19). With several prompts each is scaled and the best counts. The bonus divides by all four weights also when one has left the sum.
- **2.2 Filter.** The bullets carry the "and" and the "or" by layout only. A liked or saved role must still pass the gate. A like is this agent's; a save under any agent counts.
- **3 Fetch text.** Only hosts with a role page are read, Jobindex never in a pull, and the caps (200 a board, 1,000 a run, retry after 7 days) are in `ADR Plugins` only.
- **4 Model analytics.** The model of each call and the batch sizes (20, 5, 10). Extract reads the first 2,500 and last 1,500 characters and also fills pay, currency, function and work permit; it gives up after 3 unanswered tries. The judge reads up to 8,000 characters, rates career against the prompts, likes and dislikes, and the CV match is 10 times the weighted mean, 0 to 100. Summarise writes about 60 words and 3 to 8 tags, once. A stage that is off, or has no model to ask, is skipped without a notice; only Settings › Sources › Runs says so. A cut role without text is neither judged nor summarised.
- **5 Feed.** Open roles and kept copies only. Opening or voting marks a role read. The list of filters, 60 rows a page, that ★ also likes, and what a role without an embedding scores. Newest is a third order, where the glossary and the flow say two.
- **Settings.** The agent's On/Off switch and the pull's schedule (every 6 h).
Wording to settle: 2.1 "over the sum of the want feed score weights" against 5.1.3 "its weight leaves the sum"; 4 "once extract has completed the ingestion" against 4.1 "a role that enters is extracted on the next pull"; 4.1's field list against "raw and processed fields for e.g. salary"; "model analytics score" is never defined (the number is the CV match); 1.3 against `ADR Plugins` 3.2, which lets a search declare `stated`, and 2.1, which says an incomplete read closes nothing; the title is "Architecture" while the other ADR calls it by its file name.
`ADR Plugins`, read lightly since it is not final:
- **The API block already differs from the code:** `empty` is "trusted" or "suspect", not None; `Page` is a plain tuple; `Reads` has an unused `title="provisional"`; a provider must define `allowed`; a reader may import `desk`.
- **Rules the code needed and the ADR lacks:** a cursor must survive JSON, where a tuple returns as a list; a reader that sees a page come again hands it over empty with a cursor; a wrong-shaped answer is an error, never an empty list.
- **Rules the code does not keep:** "at most 8 series" is enforced only with terms or more than 8 countries; the 10% guard starts at 1 role, not 10, for filtered, narrowed and suspect reads; a 429 after page one starts a sweep; a module that breaks the contract stops the import at the first problem, and with it the app.
- **"Never maps a word"** is broken by `Place.remote`, which makes every reader decide whether "hybrid" is office, and by the national providers that state their country (Jobindex DK, Platsbanken SE).
- **"Nothing else changes"** holds for a host. A provider with a new param or a key touches the Sources form, its script, the CLI, the API body and `sources.search_params`.
- Cases 4 and 5 are handled after the plugin, so "upstream" reads the wrong way round. The constants (0.5 s, 100 pages, 96 MiB, 21 and 7 days) will go stale faster than the rules.
## Structure
From the review of the plugin layer and the inbound flow.
- **Per-target state is string keys written and read in six files** (`_runner.stamp`, `ats.py`, `pull.py`, `limits.py`, `reach.py`, `api.py`). One typed accessor in the runner would own them.
- **Detail handling is spread over five places** (`ats._board`, `_runner.read_details`, `ats.read_board_details`, `fetch_text.plan`, and none of it in `search.py`), which is why a search role's detail places are not kept across pulls.
- **`plugins/ats.py` holds three things:** the source, compatibility names only tests use (`_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused`, `DETAIL_FIELDS`), and the app's function classifier (`FUNCTIONS`, `classify`), which search roles never pass through.
- **`plugins` is not in the layering test's trees,** so `app` imports the private `_runner` in four files and single readers in two, and `models.py`, `config.py` and `dupes.py` reach the registry through a path hack.
- **The two sources build an item each their own way** (`ats.py`, `search.py`) and already differ.
- **The contract test names readers** (`POSTED_BY_DETAIL`, `SEARCH_PARAMS`), so a new reader that dates by detail edits the shared test.
- **`has_key` re-implements `resolve_key`,** and `search_params` raises `KeyError` for a provider param other than q, area, country, where, region or remote.
- **`pipeline/` only logs when a cascade's next stage fails to start** (`queue.py`); the fix lands in the boilerplate first. `pipeline/`, `auth/`, `deploy/` and `desk/` lag their upstreams.
## Text
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only `summarise` reads. `embed` and `extract` read `description or teaser`, so a Jobindex role keeps being ranked on its teaser. The fix is one text rule, `article_text or description or teaser`, in embed and extract, and queuing embed and score after a fetch.
- **An edited posting is never embedded again.** The vector is dropped only when the text ranks better than before (a title-only role gaining a teaser, or a teaser replaced by a description). A hash of the embedded text on `vectors` would re-embed on any change.
- **Extract marks a title-only role done for good.** `upsert_role` does not reset `extracted_at` when text arrives. Extract is on, so this bites. `_EXTRACT_WAITING` re-reads roles whose `seniority_model` is empty, which covers level and not pay.
- **Lever keeps only the intro paragraph.** `lever.py` reads `description` and drops `lists[]` and `additional`, about 80% of the text on Spotify's board. Recruitee drops `requirements` the same way.
- **Pay in the text is not in the fields.** About 1,470 roles state a currency range in the text only, mostly Greenhouse and US. Pay figures are filled for 17.9% of open roles. Extract can return `salary_min` and `salary_max` but has read 554 roles so far.
- **A title-only role has no mark in the feed.** 1,180 open roles have no description, teaser or fetched posting: 677 of them Workday, 256 SmartRecruiters, 150 Workable, 57 SuccessFactors and 26 Eightfold. A mark would show which fits are guesses.
## Ranking
- **The owner's agent has no rule on seniority.** Fit compares subject, not level. The ladder and the rail's From/To/Track filter exist (`LADDER` in `app/models.py`), and the level is filled for about half of open roles. A gate still names words (`principal`, `senior_manager`, `director`, `exec`); a gate on "at least a rung" would read as the rail's filter does.
- **Votes train `taste` only.** The liked-minus-passed centroid weighs 0.2 on both agents. Feeding votes into the prompt vectors is worth doing once there are a few hundred.
- **A starred company's roles crowd the top of Agent match.** With the companies weight at 0.2 a favourite's roles gain about 0.13; after starring Databricks it held 33 of the top 60 rows. Lower the weight on the agent's card, or star fewer companies.
- **A hand-fetched posting is not embedded or extracted.** "Fetch the posting" stores `article_text`, which only summarise and the judge read.
- **An edited posting is not embedded again** unless its text ranks better than before (title, teaser, description). A hash of the embedded text would catch any change.
- **Lever keeps only the intro paragraph** (`lists[]` and `additional` are dropped), and Recruitee drops `requirements`.
- **Pay stated only in the text is not in the fields** unless extract has read the role.
- **A role ranked on its title alone has no mark in the feed.**
## Sources
- **The boilerplate moved on.** `pipeline/` equals boilerplate `a1c6f0d` once line endings are ignored: 87 files of the working tree have CRLF although the index has LF, so a plain `diff -r` shows every file as different (use `--strip-trailing-cr`). `auth/` lacks `c6a90ff`, the phone commit (`auth/README.md`, `auth/static/account.js`, `auth/templates/login.html`). `deploy/` is at `1679e88` and lacks the WSL scripts of `e1b3746` (`wsl-setup.sh`, `wsl-setup.ps1`), and `docs/release.md` lacks its `wsl.md` sentence. `desk/` is news `a275c38` less the prune and lags news by four commits (`b0e1f6f`, `f01a6bd`, `3bb6775`, `7b9c8a2`). The three languages of boilerplate `7bb7da2` are starter code in `app/`, not part of the copies. Refresh with `git archive`, and decide whether `app/static/app.css` should use the kit's classes (`4573f6d`) instead of its own copies.
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
- **Six hosts' dates are not trusted.** Eightfold, Jibe, Gem, Deel, HiBob and HR Manager declare `posted="listed"`, so their backlog counts as posted when first seen. Eightfold now maps `creationTs`, Jibe's `posted_date` equals the page's `datePosted`, and HR Manager's `pubDate` rises with the id; each needs one check on a live board before it declares `posted="trusted"`. Platsbanken's local time is read as UTC.
- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board.** All three answer an empty listing for an unknown id and declare `empty="suspect"`. The 10% guard keeps a known board from closing its roles, but `set_board` keeps a new board that lists nothing, which is caught only when the person notices "0 roles".
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the runner waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
- **A SuccessFactors site on a company's own domain is not discovered.** Its pages link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
- **HR Manager is attached although its robots.txt disallows reading.** `hrmanager:teknologisk` (the Danish Technological Institute) has 7 open roles, while `READS.robots` says "disallowed by robots.txt; owner to decide". Seaborg stays unattached until the owner decides whether the feed counts as allowed.
- **Whether SmartRecruiters' identifier is case-sensitive is unsettled.** The module docstring says it is (`BoschGroup`), `decisions.md` says it is not, checked live; the live boards hold the identifiers with case. A live request with a lowercased identifier settles it.
- **The seed lags the live database.** The seed has 286 boards and the database 310: it lacks Microsoft, Infineon, Novonesis, Darktrace, Criteo, NVIDIA, HR Manager and seventeen more, and names five of them as `unread` or `robots`. `app companies seed` never replaces a board, so the seed is brought in line by hand.
- **Stripe's places carry a "Locations" suffix.** 316 open roles (285 without copies) have place text like "Singapore Locations" or "Dublin; Ireland Locations".
- **A mistyped SmartRecruiters, Gem or Ashby slug attaches as an empty board**, and nine boards have never listed a role. `set_board` keeps a board that lists nothing.
- **Six hosts' dates are not trusted** (Eightfold, Jibe, Gem, Deel, HiBob, HR Manager declare `posted="listed"`); each needs one check on a live board. Greenhouse declares `trusted` and falls back to `updated_at`.
- **The hosts added on 2026-09-29 were read from this machine only;** read one Workday and one SuccessFactors board from the deployed server.
- **robots.txt decisions wait for the owner:** SmartRecruiters' API host disallows `/` for every agent but LinkedInBot and is read meanwhile; HR Manager is attached although its record says "owner to decide". Whether SmartRecruiters' identifier is case-sensitive is unsettled.
- **`Crawl-delay` is not honoured.**
- **A SuccessFactors site on a company's own domain is not discovered.**
- **freehire states seniority and employment and the reader declares neither.** Stripe's places carry a "Locations" suffix.
- **The 10% guard compares only reads with the same filter,** so the first read after an agent's locations change is unguarded.
## The pages
- **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite.
- **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it.
- **`company_key` keeps `ø`.** Aliases make "Orsted" and "Ørsted" one company; folding the key itself would change the primary key of `companies`.
## From the review of 2026-09-30
- **Remote roles are not read from hosts that cannot filter on remote.** On SmartRecruiters, Phenom, SuccessFactors and Eightfold the reach "DK, SE, remote" is asked as "DK, SE"; since 2026-09-30 the filtered read keeps the remote roles it has and the line says "remote not asked", but new remote roles on those 21 boards are never fetched, and the 72 closed on 2026-09-29 and 30 (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3) reopen only if a read lists them. Reading such boards whole is the owner's call; Bosch is about 70 requests a pull.
- **SmartRecruiters' API host disallows `/` for every agent but LinkedInBot** (read 2026-09-30), which covers the listing and the detail requests; `READS.robots` says "owner to decide" and the eleven boards are read meanwhile. Jobylon allows `*` but disallows named job crawlers (JobrapidoBot, JoobleBot, AdzunaBot); ours is not one of them. HR Manager's record says the owner decided on 2026-09-29 to read the API host; confirm it.
- **`has_key` re-implements `resolve_key`**, and the form hard-codes the labels of Jobindex, Adzuna and Platsbanken; since 2026-10-03 the CLI, dupes, `search_params` and the other providers in the form come from `readers.PROVIDERS`.
- **A search role's detail places are not kept across pulls.** `plugins/search.py` has no `_KEPT_FROM_DETAIL`, so the next pull may overwrite the town Jobbank's `detail` added; the text is kept. Per-role detail state of a search is never pruned, since a search never lists everything.
- **Jobbank's JSON-LD `addressLocality` is sometimes not a town** ("Ørsted" on an Ørsted ad), and the feed's regions (`Øresundsregionen`, `Udlandet (øvrige)`) will show under Unmapped.
- **freehire states seniority and employment type, but the reader declares neither**, since a stated word needs a table in `app/prep/hosts/`. Jobnet keeps `utm_*` on employer links; dupes match on company, title and town, so it does not matter there.
- **`ats.py` keeps compatibility names that only tests reach for:** `_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused` and `DETAIL_FIELDS`.
- **`title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`.** Wave 2's WP9 must own `_runner.py` (`plan-hosts-wave-2.md`).
- **Some `posted="trusted"` declarations rest on the detail read** (Workday, BambooHR, Rippling) **or on a fallback the host moves** (Greenhouse `updated_at`, Phenom `postedDate`).
- **Dupes fold same-host roles with the same title and town without comparing text.** 99 open copies share under 30% of their words with the kept role.
- **Fetch text never reads a role outside the cut.** A role without text is ranked on its title until it enters the cut.
- **A pull never re-measures a board's size.** Recheck is the only way, so a board that grows past 300 roles stays read whole until someone rechecks it.
- **Fetch terms are configured on no agent**, so no board is read with terms today.
- **`SMALL` (300), `STALE_DAYS` (21), the guard's 10% and `DETAIL_CAP` (200) are code constants**, and the pages hard-code 300 in tooltips.
- **`The todo list.md` holds items the code has done** (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.
- **★ means two things on Roles:** on a row it saves the role, in the company strip it marks a favourite.
- **All locations and All roles answer without a score job id,** so the company view is stale until the rescore has run and the page is reloaded.
- **Place suggestions split every place line on each request.**
- **`company_key` keeps `ø`;** aliases make "Orsted" and "Ørsted" one company.