Regroup copies when a board or search is switched off, and fold the duplicates plan into the decisions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-29 23:11:33 +02:00
1 parent e1c197b488
commit 82b7902eff
4 files changed
+34 -65

No files matched your search

+5 -1
View File
@@ -16,7 +16,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning
## Data
**Clustering key is `(company_key, title_key, place_key)`.** newsdesk's `location_key OR title_key` collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one job under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, a row with no country matches Japan and Malaysia, and joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member; a vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own. A known miss: `Remote - USA NH` and `Remote - USA WA` merge, since a state code is not a town.
**A copy is stored as `roles.dup_of` and rebuilt after every pull.** The feed, its counts and facets, the tag and company counts, scoring, embedding, extract and summarise all add `dup_of IS NULL`, which keeps `idx_roles_posted` usable where a window function over the filtered rows would scan them all. The kept role is a board over a search provider, then one with a description, then the earliest `posted_at`, then the smallest uid. `close_boards` rebuilds from scratch in 0.3 s after its closes, and so do `set_fetch` and `remove_search` when they close roles, so a closed kept role hands its group to the next copy. Where a group changed, the kept role takes the earliest `read_at` and each agent's latest vote from its copies and the role it replaces; an unchanged group is left alone, so marking a role unread again sticks. The per-board counts on the Sources page still count copies, since they report what a board holds. A feed row shows `×N` for N postings and the reader lists them under "Also posted", newsdesk's "Also covered by"; a copy opened by its link says which role the feed lists instead.
**`place`, `country` and `remote` are separate columns.** `location` already means where an `Item`'s original lives. `remote` alone over-matches, because boards write "UK | Remote", so a gate on remote also reads `country`.