Group copies only when they all match, compare towns through places.towns, and record the rule in the plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-29 22:12:39 +02:00
1 parent fdb6264c98
commit b09a2f6b33
4 files changed
+90 -134

No files matched your search

+4 -12
View File
@@ -17,7 +17,9 @@ The same-host groups are the larger problem. SAP lists one job under 3 to 9 requ
## Decisions
**Two open roles are one posting when company, title, country and a town agree.** Same non-empty `company_key`, same `title_key`, same `country` (or either empty), and the sets of towns their `place` names share at least one member, where either side having no town counts as a match. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each.
**Two open roles are one posting when company, title, country and a town agree.** Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, where either side having no town counts as a match. `places.towns` takes the first comma part of each place and every known town, so `Mountain View, California` and `San Francisco, California` share none. Host does not matter, so a board's own repeats collapse like an aggregator's copy. The same title in Berlin and in Munich stays two roles, so a place filter still finds each.
**A group is a set of rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, and a row with no country matches Japan and Malaysia; joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member. A vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own.
**The group is stored, not computed per query.** `roles.dup_of` holds the uid of the group's kept role, NULL on the kept role and on a role without copies. The feed, the counts and the facets add `dup_of IS NULL`, which keeps `idx_roles_posted` usable; a window function over the filtered rows would scan them all on every page.
@@ -43,17 +45,7 @@ The storage and UI packages share files with packages E and G of `plan-feed-filt
|---|---|---|
| H, the rule | Sonnet | `app/dupes.py`, `tests/test_dupes.py` |
`app/dupes.py` is pure, with no store and no SQL:
```python
SEARCH_HOSTS: frozenset[str] # jobindex, adzuna, platsbanken
def towns(place: str, country: str) -> frozenset[str]
def same_posting(a: Copy, b: Copy) -> bool
def keep_order(c: Copy) -> tuple # sort key, smallest is kept
def assign(rows: Iterable[Copy]) -> dict[str, str | None] # uid -> dup_of
```
`Copy` is a frozen dataclass with `uid`, `company_key`, `title_key`, `country`, `place`, `host`, `has_description`, `posted_at`. `towns` splits on `,`, `;`, `|`, `/`, ` og `, ` och `, ` und `, ` and `, folds with `models._place_fold`, and drops country names (`models._COUNTRY_NAMES`), `Skåne`-style region words that follow a town, and words such as `hybrid`, `remote`, `position`, `mulighed for hjemmearbejde`; `København` and `Copenhagen` are one town. `assign` partitions rows by `(company_key, title_key)`, joins rows by `same_posting` transitively, and within each component points every row but the `keep_order` minimum at that minimum. The tests carry the five live cases above verbatim, Berlin against Munich, an empty place, an empty company and an empty country.
Done: `app/dupes.py` (`SEARCH_HOSTS`, `Copy`, `same_posting`, `keep_order`, `assign`) over `places.towns`, and `tests/test_dupes.py` with the live cases. On the live database it hides 1,080 of 33,881 open roles in 848 groups, 14 of them across hosts, in half a second. Known leftovers: Zeiss field technicians in `Remote - USA NH` and `Remote - USA WA` collapse, since a state code is not a town.
### Wave 2, after G of `plan-feed-filters.md` is merged, in parallel