Files
jobboard/docs/plan-duplicates.md
T
2026-09-29 22:04:08 +02:00

70 lines
6.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Plan: one role per posting, across and within hosts
Planned 2026-09-29, item 1 of the backlog. This file is folded into `decisions.md` and deleted when the packages land.
## Measured on the live database, 2026-09-29
| Fact | Number |
|---|---|
| Open roles | about 34,000 |
| Groups of one `(company_key, title_key, country)` on two hosts | 13, 26 roles |
| Groups of one `(company_key, title_key, country)` on one host, more than one row | about 1,500, 3,900 roles |
| Groups of one `(company_key, title_key, place_key)` on two hosts | 0 |
The cross-host pairs are Ericsson on Platsbanken and Eightfold, GN Group on Jobindex and Workday, KMD on Jobindex and SuccessFactors, Trackunit on Jobindex and Teamtailor. The places never agree exactly: `Ballerup` against `Ballerup, Denmark`, `Lund` against `Lund, Skåne, Sweden`, `Aalborg, Copenhagen, Kolding, Aarhus, Hybrid position` against `København, Denmark; Aarhus, Denmark; ...`. So the stored cluster key, which requires an equal `place_key`, finds none of them.
The same-host groups are the larger problem. SAP lists one job under 3 to 9 requisition ids, same title, same town, often with the description on only one of them. Greenhouse, SmartRecruiters, Ashby and Workday boards do the same for MongoDB, Faculty, Delivery Hero, Databricks and others.
## Decisions
**Two open roles are one posting when company, title, country and a town agree.** Same non-empty `company_key`, same `title_key`, same `country` (or either empty), and the sets of towns their `place` names share at least one member, where either side having no town counts as a match. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each.
**The group is stored, not computed per query.** `roles.dup_of` holds the uid of the group's kept role, NULL on the kept role and on a role without copies. The feed, the counts and the facets add `dup_of IS NULL`, which keeps `idx_roles_posted` usable; a window function over the filtered rows would scan them all on every page.
**The kept role is the best copy.** In order: a board over a search provider (`jobindex`, `adzuna`, `platsbanken`), since a board closes reliably and carries the full text; a non-empty `description`; the earliest `posted_at`; the smallest `uid`, so the choice is stable between runs.
**Groups are rebuilt at the end of every pull, from scratch.** A rebuild reads the open roles of every `(company_key, title_key)` with more than one open role, a few thousand groups, and rewrites `dup_of` where it changed. Doing it per upsert would miss closes, which happen in bulk in `close_boards`. A closed kept role therefore hands the group to the next copy on the next pull, and a copy never outlives its posting.
**Read and judged stay read and judged.** When the kept role of a group changes, the new one takes the earliest `read_at` of the group, and each agent's vote in `role_votes` moves to it unless it already has one. An application keeps its `role_uid`.
**Copies are neither scored, extracted nor summarised.** `score_candidates`, `roles_to_embed` and the extract and summarise queues add `dup_of IS NULL`. This saves the model calls the backlog item was about.
**The reader lists the copies.** `GET /api/roles/{uid}` gains `copies`: `[{uid, host, place, original_url, posted_at}]` for every open role whose `dup_of` is this uid. A feed row gains `copies`, the count, and shows it next to the company as `×4`. The reader shows "Also posted" with a link per copy, labelled by host and place.
## Work packages
Each owns the files named and nothing else. A package that finds it needs another file stops and reports.
The storage and UI packages share files with packages E and G of `plan-feed-filters.md`, so they start after G is merged. Migration 17 is E's; this plan takes 18.
### Wave 1, now
| Package | Model | Owns |
|---|---|---|
| H, the rule | Sonnet | `app/dupes.py`, `tests/test_dupes.py` |
`app/dupes.py` is pure, with no store and no SQL:
```python
SEARCH_HOSTS: frozenset[str] # jobindex, adzuna, platsbanken
def towns(place: str, country: str) -> frozenset[str]
def same_posting(a: Copy, b: Copy) -> bool
def keep_order(c: Copy) -> tuple # sort key, smallest is kept
def assign(rows: Iterable[Copy]) -> dict[str, str | None] # uid -> dup_of
```
`Copy` is a frozen dataclass with `uid`, `company_key`, `title_key`, `country`, `place`, `host`, `has_description`, `posted_at`. `towns` splits on `,`, `;`, `|`, `/`, ` og `, ` och `, ` und `, ` and `, folds with `models._place_fold`, and drops country names (`models._COUNTRY_NAMES`), `Skåne`-style region words that follow a town, and words such as `hybrid`, `remote`, `position`, `mulighed for hjemmearbejde`; `København` and `Copenhagen` are one town. `assign` partitions rows by `(company_key, title_key)`, joins rows by `same_posting` transitively, and within each component points every row but the `keep_order` minimum at that minimum. The tests carry the five live cases above verbatim, Berlin against Munich, an empty place, an empty company and an empty country.
### Wave 2, after G of `plan-feed-filters.md` is merged, in parallel
| Package | Model | Owns |
|---|---|---|
| I, storage | Opus | `app/storage/store.py`, `app/storage/schema.py`, `app/models.py`, `app/web/api.py`, `app/jobs/pull.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_pull.py` |
| J, UI | Opus | `app/static/roles.js`, `app/templates/roles.html`, `app/static/app.css` |
I: migration 18 adds `roles.dup_of` with an index on it and runs the first rebuild; `Store.rebuild_dupes()` reads the candidate groups, calls `dupes.assign`, writes the changes, and carries read and votes; `close_boards`' settle calls it; `app dupes` runs it and prints the groups and rows it changed; the feed, `count_roles`, facets, tag counts, `score_candidates`, `roles_to_embed` and the stage queues exclude copies; `RoleRow.copies` and `Role.copies` as above. J builds against that contract with a stub until I lands.
### Review
The planner reads every diff, runs the suite, rebuilds on a copy of the live database and checks that SAP's four Forward Deployed Chief AI Engineering Expert rows are two feed rows, Garching with two copies and New York alone, and the 13 cross-host pairs are 13 rows, before each merge. Pages are checked at 1400 and 390 pixels.