Files
jobboard/docs/plan-duplicates.md
T

6.2 KiB
Raw Blame History

Plan: one role per posting, across and within hosts

Planned 2026-09-29, item 1 of the backlog. This file is folded into decisions.md and deleted when the packages land.

Measured on the live database, 2026-09-29

Fact Number
Open roles about 34,000
Groups of one (company_key, title_key, country) on two hosts 13, 26 roles
Groups of one (company_key, title_key, country) on one host, more than one row about 1,500, 3,900 roles
Groups of one (company_key, title_key, place_key) on two hosts 0

The cross-host pairs are Ericsson on Platsbanken and Eightfold, GN Group on Jobindex and Workday, KMD on Jobindex and SuccessFactors, Trackunit on Jobindex and Teamtailor. The places never agree exactly: Ballerup against Ballerup, Denmark, Lund against Lund, Skåne, Sweden, Aalborg, Copenhagen, Kolding, Aarhus, Hybrid position against København, Denmark; Aarhus, Denmark; .... So the stored cluster key, which requires an equal place_key, finds none of them.

The same-host groups are the larger problem. SAP lists one job under 3 to 9 requisition ids, same title, same town, often with the description on only one of them. Greenhouse, SmartRecruiters, Ashby and Workday boards do the same for MongoDB, Faculty, Delivery Hero, Databricks and others.

Decisions

Two open roles are one posting when company, title, country and a town agree. Same non-empty company_key, same title_key, the same country (the column, or the one the place names when the column is empty or EU), and town sets from places.towns that share a member, where either side having no town counts as a match. places.towns takes the first comma part of each place and every known town, so Mountain View, California and San Francisco, California share none. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each.

A group is a set of rows that all match each other, never a chain. A row with no place matches Belgium and the Netherlands, and a row with no country matches Japan and Malaysia; joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member. A vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own.

The group is stored, not computed per query. roles.dup_of holds the uid of the group's kept role, NULL on the kept role and on a role without copies. The feed, the counts and the facets add dup_of IS NULL, which keeps idx_roles_posted usable; a window function over the filtered rows would scan them all on every page.

The kept role is the best copy. In order: a board over a search provider (jobindex, adzuna, platsbanken), since a board closes reliably and carries the full text; a non-empty description; the earliest posted_at; the smallest uid, so the choice is stable between runs.

Groups are rebuilt at the end of every pull, from scratch. A rebuild reads the open roles of every (company_key, title_key) with more than one open role, a few thousand groups, and rewrites dup_of where it changed. Doing it per upsert would miss closes, which happen in bulk in close_boards. A closed kept role therefore hands the group to the next copy on the next pull, and a copy never outlives its posting.

Read and judged stay read and judged. When the kept role of a group changes, the new one takes the earliest read_at of the group, and each agent's vote in role_votes moves to it unless it already has one. An application keeps its role_uid.

Copies are neither scored, extracted nor summarised. score_candidates, roles_to_embed and the extract and summarise queues add dup_of IS NULL. This saves the model calls the backlog item was about.

The reader lists the copies. GET /api/roles/{uid} gains copies: [{uid, host, place, original_url, posted_at}] for every open role whose dup_of is this uid. A feed row gains copies, the count, and shows it next to the company as ×4. The reader shows "Also posted" with a link per copy, labelled by host and place.

Work packages

Each owns the files named and nothing else. A package that finds it needs another file stops and reports.

The storage and UI packages share files with packages E and G of plan-feed-filters.md, so they start after G is merged. Migration 17 is E's; this plan takes 18.

Wave 1, now

Package Model Owns
H, the rule Sonnet app/dupes.py, tests/test_dupes.py

Done: app/dupes.py (SEARCH_HOSTS, Copy, same_posting, keep_order, assign) over places.towns, and tests/test_dupes.py with the live cases. On the live database it hides 1,080 of 33,881 open roles in 848 groups, 14 of them across hosts, in half a second. Known leftovers: Zeiss field technicians in Remote - USA NH and Remote - USA WA collapse, since a state code is not a town.

Wave 2, after G of plan-feed-filters.md is merged, in parallel

Package Model Owns
I, storage Opus app/storage/store.py, app/storage/schema.py, app/models.py, app/web/api.py, app/jobs/pull.py, app/cli.py, tests/test_storage.py, tests/test_web.py, tests/test_pull.py
J, UI Opus app/static/roles.js, app/templates/roles.html, app/static/app.css

I: migration 18 adds roles.dup_of with an index on it and runs the first rebuild; Store.rebuild_dupes() reads the candidate groups, calls dupes.assign, writes the changes, and carries read and votes; close_boards' settle calls it; app dupes runs it and prints the groups and rows it changed; the feed, count_roles, facets, tag counts, score_candidates, roles_to_embed and the stage queues exclude copies; RoleRow.copies and Role.copies as above. J builds against that contract with a stub until I lands.

Review

The planner reads every diff, runs the suite, rebuilds on a copy of the live database and checks that SAP's four Forward Deployed Chief AI Engineering Expert rows are two feed rows, Garching with two copies and New York alone, and the 13 cross-host pairs are 13 rows, before each merge. Pages are checked at 1400 and 390 pixels.