"""What kind of thing is this, when it isn't an ordinary article? An outlet's feed carries more than its journalism. Golem's carries adverts ("Anzeige:") and pieces behind its paywall ("(g+)"); Hacker News marks paid links "[$]"; other feeds mark neither. There is no convention across outlets to detect, so each plugin declares how its publisher says it — as `MarkRule`s in its spec, and nowhere else. Typing a token in against one feed was how this worked before every outlet was a plugin; step 19 retired that editor, and `classify` below is the only way a mark is now found. What is general is everything after that. The outlet's token is input: it is matched, cut out of the title, and never shown. One glyph stands for a mark everywhere, so "what is this" reads the same whoever published it, instead of being a private code you learn one publisher at a time. The token is noise in a headline as well — "Anzeige: Schon 10.000-mal bestellt" is worse than the line underneath it. The vocabulary below is closed on purpose. Marks are structural — they say what a thing *is*, and the reader filters and styles them — which is a different question from what a thing is *about*; that one has tags, an open vocabulary the model can extend. Adding a mark means adding a row here and a token to the feeds that use it. """ from __future__ import annotations from dataclasses import dataclass from collections.abc import Sequence from typing import Any, Protocol from desk.models import Item @dataclass(frozen=True, slots=True) class Mark: slug: str glyph: str label: str #: What the reader is told the glyph means, on hover and to a screen reader. note: str #: Whether it is worth fetching the page for a body. False for both marks we #: have: a paywalled page hands back its gate, and an advert's body is the #: advert. Declared here rather than assumed by the backfill, so a mark that #: *does* want its body only has to say so. fetch_body: bool = False #: What an embedded frame of such an item would actually show, for the #: reader offered a control they cannot use. Here for the same reason #: `fetch_body` is: framing is a shared behaviour, and the mark is what #: knows whether a frame of it would be worth anything. See #: `OriginalView.blocked_by`. framed: str = "" MARKS: tuple[Mark, ...] = ( Mark("paywalled", "\U0001F512", "Paywalled", "Behind this outlet's paywall", framed="the outlet's paywall"), Mark("sponsored", "\U0001F4E3", "Sponsored", "An advertisement, not journalism", framed="the advert"), ) BY_SLUG = {m.slug: m for m in MARKS} #: The marks whose items the body backfill should not spend a request on. NO_BODY = frozenset(m.slug for m in MARKS if not m.fetch_body) class Rule(Protocol): @property def field(self) -> str: ... @property def method(self) -> str: ... @property def value(self) -> str: ... @property def mark(self) -> str: ... @property def strip(self) -> bool: ... def classify(item: Item, rules: Sequence[Rule]) -> Item: """Classify original inputs, retaining independent acquisition marks. Evidence lives in raw alongside the publisher categories until those acquisition inputs gain typed storage. Repeating this replaces only marks owned by the previous rule pass, never a custom acquisition hook's marks. """ previous = item.raw.get("mark_evidence", {}) original = item.source_title or item.title independent = previous.get("independent", item.marks) found: list[str] = [] evidence: list[dict[str, Any]] = [] spans: list[tuple[int, int]] = [] for rule in rules: if rule.field == "inputs.source_categories": values = item.raw.get("categories", []) matched = next((v for v in values if isinstance(v, str) and v.lower() == rule.value.lower()), None) at = 0 if matched is not None else -1 value = matched or "" else: value = item.excerpt if rule.field == "inputs.feed_text" else original lower, token = value.lower(), rule.value.lower() at = lower.find(token) if rule.method == "prefix" and not lower.startswith(token): at = -1 elif rule.method == "suffix": at = len(value) - len(rule.value) if lower.endswith(token) else -1 if at < 0: continue found.append(rule.mark) evidence.append({"field": rule.field, "method": rule.method, "value": rule.value, "mark": rule.mark, "strip": rule.strip, "matched": value[at:at + len(rule.value)]}) if rule.strip: spans.append((at, at + len(rule.value))) item.source_title = original item.title = ("".join(c for i, c in enumerate(original) if not any(start <= i < end for start, end in spans)).strip() or original ) if spans else original item.marks = sorted(set(independent) | set(found)) item.raw["mark_evidence"] = {"version": 1, "independent": list(independent), "matches": evidence} return item