Files
jobboard/docs/plan-duplicates.md
T

62 lines
6.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Plan: one role per posting, across and within hosts
Planned 2026-09-29, item 1 of the backlog. This file is folded into `decisions.md` and deleted when the packages land.
## Measured on the live database, 2026-09-29
| Fact | Number |
|---|---|
| Open roles | about 34,000 |
| Groups of one `(company_key, title_key, country)` on two hosts | 13, 26 roles |
| Groups of one `(company_key, title_key, country)` on one host, more than one row | about 1,500, 3,900 roles |
| Groups of one `(company_key, title_key, place_key)` on two hosts | 0 |
The cross-host pairs are Ericsson on Platsbanken and Eightfold, GN Group on Jobindex and Workday, KMD on Jobindex and SuccessFactors, Trackunit on Jobindex and Teamtailor. The places never agree exactly: `Ballerup` against `Ballerup, Denmark`, `Lund` against `Lund, Skåne, Sweden`, `Aalborg, Copenhagen, Kolding, Aarhus, Hybrid position` against `København, Denmark; Aarhus, Denmark; ...`. So the stored cluster key, which requires an equal `place_key`, finds none of them.
The same-host groups are the larger problem. SAP lists one job under 3 to 9 requisition ids, same title, same town, often with the description on only one of them. Greenhouse, SmartRecruiters, Ashby and Workday boards do the same for MongoDB, Faculty, Delivery Hero, Databricks and others.
## Decisions
**Two open roles are one posting when company, title, country and a town agree.** Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, where either side having no town counts as a match. `places.towns` takes the first comma part of each place and every known town, so `Mountain View, California` and `San Francisco, California` share none. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each.
**A group is a set of rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, and a row with no country matches Japan and Malaysia; joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member. A vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own.
**The group is stored, not computed per query.** `roles.dup_of` holds the uid of the group's kept role, NULL on the kept role and on a role without copies. The feed, the counts and the facets add `dup_of IS NULL`, which keeps `idx_roles_posted` usable; a window function over the filtered rows would scan them all on every page.
**The kept role is the best copy.** In order: a board over a search provider (`jobindex`, `adzuna`, `platsbanken`), since a board closes reliably and carries the full text; a non-empty `description`; the earliest `posted_at`; the smallest `uid`, so the choice is stable between runs.
**Groups are rebuilt at the end of every pull, from scratch.** A rebuild reads the open roles of every `(company_key, title_key)` with more than one open role, a few thousand groups, and rewrites `dup_of` where it changed. Doing it per upsert would miss closes, which happen in bulk in `close_boards`. A closed kept role therefore hands the group to the next copy on the next pull, and a copy never outlives its posting.
**Read and judged stay read and judged.** When the kept role of a group changes, the new one takes the earliest `read_at` of the group, and each agent's vote in `role_votes` moves to it unless it already has one. An application keeps its `role_uid`.
**Copies are neither scored, extracted nor summarised.** `score_candidates`, `roles_to_embed` and the extract and summarise queues add `dup_of IS NULL`. This saves the model calls the backlog item was about.
**The reader lists the copies.** `GET /api/roles/{uid}` gains `copies`: `[{uid, host, place, original_url, posted_at}]` for every open role whose `dup_of` is this uid. A feed row gains `copies`, the count, and shows it next to the company as `×4`. The reader shows "Also posted" with a link per copy, labelled by host and place.
## Work packages
Each owns the files named and nothing else. A package that finds it needs another file stops and reports.
The storage and UI packages share files with packages E and G of `plan-feed-filters.md`, so they start after G is merged. Migration 17 is E's; this plan takes 18.
### Wave 1, now
| Package | Model | Owns |
|---|---|---|
| H, the rule | Sonnet | `app/dupes.py`, `tests/test_dupes.py` |
Done: `app/dupes.py` (`SEARCH_HOSTS`, `Copy`, `same_posting`, `keep_order`, `assign`) over `places.towns`, and `tests/test_dupes.py` with the live cases. On the live database it hides 1,080 of 33,881 open roles in 848 groups, 14 of them across hosts, in half a second. Known leftovers: Zeiss field technicians in `Remote - USA NH` and `Remote - USA WA` collapse, since a state code is not a town.
### Wave 2, after G of `plan-feed-filters.md` is merged, in parallel
| Package | Model | Owns |
|---|---|---|
| I, storage | Opus | `app/storage/store.py`, `app/storage/schema.py`, `app/models.py`, `app/web/api.py`, `app/jobs/pull.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_pull.py` |
| J, UI | Opus | `app/static/roles.js`, `app/templates/roles.html`, `app/static/app.css` |
I: migration 18 adds `roles.dup_of` with an index on it and runs the first rebuild; `Store.rebuild_dupes()` reads the candidate groups, calls `dupes.assign`, writes the changes, and carries read and votes; `close_boards`' settle calls it; `app dupes` runs it and prints the groups and rows it changed; the feed, `count_roles`, facets, tag counts, `score_candidates`, `roles_to_embed` and the stage queues exclude copies; `RoleRow.copies` and `Role.copies` as above. J builds against that contract with a stub until I lands.
### Review
The planner reads every diff, runs the suite, rebuilds on a copy of the live database and checks that SAP's four Forward Deployed Chief AI Engineering Expert rows are two feed rows, Garching with two copies and New York alone, and the 13 cross-host pairs are 13 rows, before each merge. Pages are checked at 1400 and 390 pixels.