From de156ed90f178c719f5e502fd2f8ac17edca320e Mon Sep 17 00:00:00 2001 From: Wolf Date: Tue, 29 Sep 2026 22:04:08 +0200 Subject: [PATCH] Plan one role per posting, across hosts and within one Co-Authored-By: Claude Opus 5.5 --- docs/plan-duplicates.md | 69 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 docs/plan-duplicates.md diff --git a/docs/plan-duplicates.md b/docs/plan-duplicates.md new file mode 100644 index 0000000..2db39f1 --- /dev/null +++ b/docs/plan-duplicates.md @@ -0,0 +1,69 @@ +# Plan: one role per posting, across and within hosts + +Planned 2026-09-29, item 1 of the backlog. This file is folded into `decisions.md` and deleted when the packages land. + +## Measured on the live database, 2026-09-29 + +| Fact | Number | +|---|---| +| Open roles | about 34,000 | +| Groups of one `(company_key, title_key, country)` on two hosts | 13, 26 roles | +| Groups of one `(company_key, title_key, country)` on one host, more than one row | about 1,500, 3,900 roles | +| Groups of one `(company_key, title_key, place_key)` on two hosts | 0 | + +The cross-host pairs are Ericsson on Platsbanken and Eightfold, GN Group on Jobindex and Workday, KMD on Jobindex and SuccessFactors, Trackunit on Jobindex and Teamtailor. The places never agree exactly: `Ballerup` against `Ballerup, Denmark`, `Lund` against `Lund, Skåne, Sweden`, `Aalborg, Copenhagen, Kolding, Aarhus, Hybrid position` against `København, Denmark; Aarhus, Denmark; ...`. So the stored cluster key, which requires an equal `place_key`, finds none of them. + +The same-host groups are the larger problem. SAP lists one job under 3 to 9 requisition ids, same title, same town, often with the description on only one of them. Greenhouse, SmartRecruiters, Ashby and Workday boards do the same for MongoDB, Faculty, Delivery Hero, Databricks and others. + +## Decisions + +**Two open roles are one posting when company, title, country and a town agree.** Same non-empty `company_key`, same `title_key`, same `country` (or either empty), and the sets of towns their `place` names share at least one member, where either side having no town counts as a match. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each. + +**The group is stored, not computed per query.** `roles.dup_of` holds the uid of the group's kept role, NULL on the kept role and on a role without copies. The feed, the counts and the facets add `dup_of IS NULL`, which keeps `idx_roles_posted` usable; a window function over the filtered rows would scan them all on every page. + +**The kept role is the best copy.** In order: a board over a search provider (`jobindex`, `adzuna`, `platsbanken`), since a board closes reliably and carries the full text; a non-empty `description`; the earliest `posted_at`; the smallest `uid`, so the choice is stable between runs. + +**Groups are rebuilt at the end of every pull, from scratch.** A rebuild reads the open roles of every `(company_key, title_key)` with more than one open role, a few thousand groups, and rewrites `dup_of` where it changed. Doing it per upsert would miss closes, which happen in bulk in `close_boards`. A closed kept role therefore hands the group to the next copy on the next pull, and a copy never outlives its posting. + +**Read and judged stay read and judged.** When the kept role of a group changes, the new one takes the earliest `read_at` of the group, and each agent's vote in `role_votes` moves to it unless it already has one. An application keeps its `role_uid`. + +**Copies are neither scored, extracted nor summarised.** `score_candidates`, `roles_to_embed` and the extract and summarise queues add `dup_of IS NULL`. This saves the model calls the backlog item was about. + +**The reader lists the copies.** `GET /api/roles/{uid}` gains `copies`: `[{uid, host, place, original_url, posted_at}]` for every open role whose `dup_of` is this uid. A feed row gains `copies`, the count, and shows it next to the company as `×4`. The reader shows "Also posted" with a link per copy, labelled by host and place. + +## Work packages + +Each owns the files named and nothing else. A package that finds it needs another file stops and reports. + +The storage and UI packages share files with packages E and G of `plan-feed-filters.md`, so they start after G is merged. Migration 17 is E's; this plan takes 18. + +### Wave 1, now + +| Package | Model | Owns | +|---|---|---| +| H, the rule | Sonnet | `app/dupes.py`, `tests/test_dupes.py` | + +`app/dupes.py` is pure, with no store and no SQL: + +```python +SEARCH_HOSTS: frozenset[str] # jobindex, adzuna, platsbanken +def towns(place: str, country: str) -> frozenset[str] +def same_posting(a: Copy, b: Copy) -> bool +def keep_order(c: Copy) -> tuple # sort key, smallest is kept +def assign(rows: Iterable[Copy]) -> dict[str, str | None] # uid -> dup_of +``` + +`Copy` is a frozen dataclass with `uid`, `company_key`, `title_key`, `country`, `place`, `host`, `has_description`, `posted_at`. `towns` splits on `,`, `;`, `|`, `/`, ` og `, ` och `, ` und `, ` and `, folds with `models._place_fold`, and drops country names (`models._COUNTRY_NAMES`), `Skåne`-style region words that follow a town, and words such as `hybrid`, `remote`, `position`, `mulighed for hjemmearbejde`; `København` and `Copenhagen` are one town. `assign` partitions rows by `(company_key, title_key)`, joins rows by `same_posting` transitively, and within each component points every row but the `keep_order` minimum at that minimum. The tests carry the five live cases above verbatim, Berlin against Munich, an empty place, an empty company and an empty country. + +### Wave 2, after G of `plan-feed-filters.md` is merged, in parallel + +| Package | Model | Owns | +|---|---|---| +| I, storage | Opus | `app/storage/store.py`, `app/storage/schema.py`, `app/models.py`, `app/web/api.py`, `app/jobs/pull.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_pull.py` | +| J, UI | Opus | `app/static/roles.js`, `app/templates/roles.html`, `app/static/app.css` | + +I: migration 18 adds `roles.dup_of` with an index on it and runs the first rebuild; `Store.rebuild_dupes()` reads the candidate groups, calls `dupes.assign`, writes the changes, and carries read and votes; `close_boards`' settle calls it; `app dupes` runs it and prints the groups and rows it changed; the feed, `count_roles`, facets, tag counts, `score_candidates`, `roles_to_embed` and the stage queues exclude copies; `RoleRow.copies` and `Role.copies` as above. J builds against that contract with a stub until I lands. + +### Review + +The planner reads every diff, runs the suite, rebuilds on a copy of the live database and checks that SAP's four Forward Deployed Chief AI Engineering Expert rows are two feed rows, Garching with two copies and New York alone, and the 13 cross-host pairs are 13 rows, before each merge. Pages are checked at 1400 and 390 pixels.