Regroup copies when a board or search is switched off, and fold the duplicates plan into the decisions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-29 23:11:33 +02:00
1 parent e1c197b488
commit 82b7902eff
4 files changed
+34 -65

No files matched your search

+11 -3
View File
@@ -31,9 +31,13 @@ PROVIDERS = ("jobindex", "adzuna", "platsbanken")
def set_fetch(store: Store, sub_id: int, fetch: bool) -> tuple[Subscription, int]:
"""Switch one board or search on or off. Off closes its open roles and
returns how many; see the module docstring."""
returns how many; see the module docstring. A close regroups the copies
(`Store.rebuild_dupes`), so a copy of a role that closed here shows now
rather than at the next pull."""
sub = store.update_subscription(sub_id, subscribed=fetch, fetch=fetch, show=True)
closed = 0 if fetch else store.close_subscription(sub_id)
if closed:
store.rebuild_dupes()
return sub, closed
@@ -411,9 +415,13 @@ def save_search(config: Config, store: Store, provider: str, name: str,
def remove_search(store: Store, sub_id: int) -> tuple[Subscription, int]:
"""Stop running a search and hide it. Its open roles close; adding a search
of the same name again brings the row back."""
of the same name again brings the row back, and the copies regroup as in
`set_fetch`."""
sub = store.subscription(sub_id)
if sub is None or sub.source != "search":
raise KeyError(sub_id)
store.unsubscribe(sub_id)
return sub, store.close_subscription(sub_id)
closed = store.close_subscription(sub_id)
if closed:
store.rebuild_dupes()
return sub, closed
+5 -1
View File
@@ -16,7 +16,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning
## Data
**Clustering key is `(company_key, title_key, place_key)`.** newsdesk's `location_key OR title_key` collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one job under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, a row with no country matches Japan and Malaysia, and joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member; a vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own. A known miss: `Remote - USA NH` and `Remote - USA WA` merge, since a state code is not a town.
**A copy is stored as `roles.dup_of` and rebuilt after every pull.** The feed, its counts and facets, the tag and company counts, scoring, embedding, extract and summarise all add `dup_of IS NULL`, which keeps `idx_roles_posted` usable where a window function over the filtered rows would scan them all. The kept role is a board over a search provider, then one with a description, then the earliest `posted_at`, then the smallest uid. `close_boards` rebuilds from scratch in 0.3 s after its closes, and so do `set_fetch` and `remove_search` when they close roles, so a closed kept role hands its group to the next copy. Where a group changed, the kept role takes the earliest `read_at` and each agent's latest vote from its copies and the role it replaces; an unchanged group is left alone, so marking a role unread again sticks. The per-board counts on the Sources page still count copies, since they report what a board holds. A feed row shows `×N` for N postings and the reader lists them under "Also posted", newsdesk's "Also covered by"; a copy opened by its link says which role the feed lists instead.
**`place`, `country` and `remote` are separate columns.** `location` already means where an `Item`'s original lives. `remote` alone over-matches, because boards write "UK | Remote", so a gate on remote also reads `country`.
-61
View File
@@ -1,61 +0,0 @@
# Plan: one role per posting, across and within hosts
Planned 2026-09-29, item 1 of the backlog. This file is folded into `decisions.md` and deleted when the packages land.
## Measured on the live database, 2026-09-29
| Fact | Number |
|---|---|
| Open roles | about 34,000 |
| Groups of one `(company_key, title_key, country)` on two hosts | 13, 26 roles |
| Groups of one `(company_key, title_key, country)` on one host, more than one row | about 1,500, 3,900 roles |
| Groups of one `(company_key, title_key, place_key)` on two hosts | 0 |
The cross-host pairs are Ericsson on Platsbanken and Eightfold, GN Group on Jobindex and Workday, KMD on Jobindex and SuccessFactors, Trackunit on Jobindex and Teamtailor. The places never agree exactly: `Ballerup` against `Ballerup, Denmark`, `Lund` against `Lund, Skåne, Sweden`, `Aalborg, Copenhagen, Kolding, Aarhus, Hybrid position` against `København, Denmark; Aarhus, Denmark; ...`. So the stored cluster key, which requires an equal `place_key`, finds none of them.
The same-host groups are the larger problem. SAP lists one job under 3 to 9 requisition ids, same title, same town, often with the description on only one of them. Greenhouse, SmartRecruiters, Ashby and Workday boards do the same for MongoDB, Faculty, Delivery Hero, Databricks and others.
## Decisions
**Two open roles are one posting when company, title, country and a town agree.** Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, where either side having no town counts as a match. `places.towns` takes the first comma part of each place and every known town, so `Mountain View, California` and `San Francisco, California` share none. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each.
**A group is a set of rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, and a row with no country matches Japan and Malaysia; joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member. A vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own.
**The group is stored, not computed per query.** `roles.dup_of` holds the uid of the group's kept role, NULL on the kept role and on a role without copies. The feed, the counts and the facets add `dup_of IS NULL`, which keeps `idx_roles_posted` usable; a window function over the filtered rows would scan them all on every page.
**The kept role is the best copy.** In order: a board over a search provider (`jobindex`, `adzuna`, `platsbanken`), since a board closes reliably and carries the full text; a non-empty `description`; the earliest `posted_at`; the smallest `uid`, so the choice is stable between runs.
**Groups are rebuilt at the end of every pull, from scratch.** A rebuild reads the open roles of every `(company_key, title_key)` with more than one open role, a few thousand groups, and rewrites `dup_of` where it changed. Doing it per upsert would miss closes, which happen in bulk in `close_boards`. A closed kept role therefore hands the group to the next copy on the next pull, and a copy never outlives its posting.
**Read and judged stay read and judged.** When the kept role of a group changes, the new one takes the earliest `read_at` of the group, and each agent's vote in `role_votes` moves to it unless it already has one. An application keeps its `role_uid`.
**Copies are neither scored, extracted nor summarised.** `score_candidates`, `roles_to_embed` and the extract and summarise queues add `dup_of IS NULL`. This saves the model calls the backlog item was about.
**The reader lists the copies.** `GET /api/roles/{uid}` gains `copies`: `[{uid, host, place, original_url, posted_at}]` for every open role whose `dup_of` is this uid. A feed row gains `copies`, the count, and shows it next to the company as `×4`. The reader shows "Also posted" with a link per copy, labelled by host and place.
## Work packages
Each owns the files named and nothing else. A package that finds it needs another file stops and reports.
The storage and UI packages share files with packages E and G of `plan-feed-filters.md`, so they start after G is merged. Migration 17 is E's; this plan takes 18.
### Wave 1, now
| Package | Model | Owns |
|---|---|---|
| H, the rule | Sonnet | `app/dupes.py`, `tests/test_dupes.py` |
Done: `app/dupes.py` (`SEARCH_HOSTS`, `Copy`, `same_posting`, `keep_order`, `assign`) over `places.towns`, and `tests/test_dupes.py` with the live cases. On the live database it hides 1,080 of 33,881 open roles in 848 groups, 14 of them across hosts, in half a second. Known leftovers: Zeiss field technicians in `Remote - USA NH` and `Remote - USA WA` collapse, since a state code is not a town.
### Wave 2, after G of `plan-feed-filters.md` is merged, in parallel
| Package | Model | Owns |
|---|---|---|
| I, storage | Opus | `app/storage/store.py`, `app/storage/schema.py`, `app/models.py`, `app/web/api.py`, `app/jobs/pull.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_pull.py` |
| J, UI | Opus | `app/static/roles.js`, `app/templates/roles.html`, `app/static/app.css` |
I: migration 18 adds `roles.dup_of` with an index on it and runs the first rebuild; `Store.rebuild_dupes()` reads the candidate groups, calls `dupes.assign`, writes the changes, and carries read and votes; `close_boards`' settle calls it; `app dupes` runs it and prints the groups and rows it changed; the feed, `count_roles`, facets, tag counts, `score_candidates`, `roles_to_embed` and the stage queues exclude copies; `RoleRow.copies` and `Role.copies` as above. J builds against that contract with a stub until I lands.
### Review
The planner reads every diff, runs the suite, rebuilds on a copy of the live database and checks that SAP's four Forward Deployed Chief AI Engineering Expert rows are two feed rows, Garching with two copies and New York alone, and the 13 cross-host pairs are 13 rows, before each merge. Pages are checked at 1400 and 390 pixels.
+18
View File
@@ -175,3 +175,21 @@ def test_a_v17_file_gains_dup_of_with_its_first_rebuild(tmp_path: Path) -> None:
with sqlite3.connect(path) as conn:
assert conn.execute("SELECT 1 FROM sqlite_master WHERE name = 'idx_roles_dup_of'"
).fetchone() is not None
def test_switching_a_board_off_shows_its_roles_copies_at_once(store: Store) -> None:
"""A close outside a pull regroups too: the search's copy of a role whose
board is switched off is listed again without waiting for a pull."""
from app.jobs.sources import set_fetch
sub = store.add_subscription("ats", "teamtailor:trackunit", label="Trackunit")
board = role("board", company="Trackunit", title="Backend Engineer", place="Aalborg",
board="teamtailor")
search = role("search", company="Trackunit", title="Backend Engineer",
place="Aalborg, Denmark", board=None, provider="jobindex")
store.upsert_role(board, sub)
store.upsert_role(search)
store.rebuild_dupes()
assert dup_of(store)[search.uid] == board.uid
set_fetch(store, sub.id, False)
assert dup_of(store) == {board.uid: None, search.uid: None}