The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
124 lines
5.2 KiB
Python
124 lines
5.2 KiB
Python
"""What kind of thing is this, when it isn't an ordinary article?
|
|
|
|
An outlet's feed carries more than its journalism. Golem's carries adverts
|
|
("Anzeige:") and pieces behind its paywall ("(g+)"); Hacker News marks paid
|
|
links "[$]"; other feeds mark neither. There is no convention across outlets to
|
|
detect, so each plugin declares how its publisher says it — as `MarkRule`s in
|
|
its spec, and nowhere else. Typing a token in against one feed was how this
|
|
worked before every outlet was a plugin; step 19 retired that editor, and
|
|
`classify` below is the only way a mark is now found.
|
|
|
|
What is general is everything after that. The outlet's token is input: it is
|
|
matched, cut out of the title, and never shown. One glyph stands for a mark
|
|
everywhere, so "what is this" reads the same whoever published it, instead of
|
|
being a private code you learn one publisher at a time. The token is noise in
|
|
a headline as well — "Anzeige: Schon 10.000-mal bestellt" is worse than the
|
|
line underneath it.
|
|
|
|
The vocabulary below is closed on purpose. Marks are structural — they say what
|
|
a thing *is*, and the reader filters and styles them — which is a different
|
|
question from what a thing is *about*; that one has tags, an open vocabulary the
|
|
model can extend. Adding a mark means adding a row here and a token to the feeds
|
|
that use it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass
|
|
from collections.abc import Sequence
|
|
from typing import Any, Protocol
|
|
|
|
from desk.models import Item
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class Mark:
|
|
slug: str
|
|
glyph: str
|
|
label: str
|
|
#: What the reader is told the glyph means, on hover and to a screen reader.
|
|
note: str
|
|
#: Whether it is worth fetching the page for a body. False for both marks we
|
|
#: have: a paywalled page hands back its gate, and an advert's body is the
|
|
#: advert. Declared here rather than assumed by the backfill, so a mark that
|
|
#: *does* want its body only has to say so.
|
|
fetch_body: bool = False
|
|
#: What an embedded frame of such an item would actually show, for the
|
|
#: reader offered a control they cannot use. Here for the same reason
|
|
#: `fetch_body` is: framing is a shared behaviour, and the mark is what
|
|
#: knows whether a frame of it would be worth anything. See
|
|
#: `OriginalView.blocked_by`.
|
|
framed: str = ""
|
|
|
|
|
|
MARKS: tuple[Mark, ...] = (
|
|
Mark("paywalled", "\U0001F512", "Paywalled",
|
|
"Behind this outlet's paywall", framed="the outlet's paywall"),
|
|
Mark("sponsored", "\U0001F4E3", "Sponsored",
|
|
"An advertisement, not journalism", framed="the advert"),
|
|
)
|
|
|
|
BY_SLUG = {m.slug: m for m in MARKS}
|
|
#: The marks whose items the body backfill should not spend a request on.
|
|
NO_BODY = frozenset(m.slug for m in MARKS if not m.fetch_body)
|
|
|
|
|
|
class Rule(Protocol):
|
|
@property
|
|
def field(self) -> str: ...
|
|
@property
|
|
def method(self) -> str: ...
|
|
@property
|
|
def value(self) -> str: ...
|
|
@property
|
|
def mark(self) -> str: ...
|
|
@property
|
|
def strip(self) -> bool: ...
|
|
|
|
|
|
def classify(item: Item, rules: Sequence[Rule]) -> Item:
|
|
"""Classify original inputs, retaining independent acquisition marks.
|
|
|
|
Evidence lives in raw alongside the publisher categories until those
|
|
acquisition inputs gain typed storage. Repeating this replaces only marks
|
|
owned by the previous rule pass, never a custom acquisition hook's marks.
|
|
"""
|
|
previous = item.raw.get("mark_evidence", {})
|
|
original = item.source_title or item.title
|
|
independent = previous.get("independent", item.marks)
|
|
found: list[str] = []
|
|
evidence: list[dict[str, Any]] = []
|
|
spans: list[tuple[int, int]] = []
|
|
for rule in rules:
|
|
if rule.field == "inputs.source_categories":
|
|
values = item.raw.get("categories", [])
|
|
matched = next((v for v in values if isinstance(v, str)
|
|
and v.lower() == rule.value.lower()), None)
|
|
at = 0 if matched is not None else -1
|
|
value = matched or ""
|
|
else:
|
|
value = item.excerpt if rule.field == "inputs.feed_text" else original
|
|
lower, token = value.lower(), rule.value.lower()
|
|
at = lower.find(token)
|
|
if rule.method == "prefix" and not lower.startswith(token):
|
|
at = -1
|
|
elif rule.method == "suffix":
|
|
at = len(value) - len(rule.value) if lower.endswith(token) else -1
|
|
if at < 0:
|
|
continue
|
|
found.append(rule.mark)
|
|
evidence.append({"field": rule.field, "method": rule.method,
|
|
"value": rule.value, "mark": rule.mark,
|
|
"strip": rule.strip,
|
|
"matched": value[at:at + len(rule.value)]})
|
|
if rule.strip:
|
|
spans.append((at, at + len(rule.value)))
|
|
item.source_title = original
|
|
item.title = ("".join(c for i, c in enumerate(original)
|
|
if not any(start <= i < end for start, end in spans)).strip() or original
|
|
) if spans else original
|
|
item.marks = sorted(set(independent) | set(found))
|
|
item.raw["mark_evidence"] = {"version": 1, "independent": list(independent),
|
|
"matches": evidence}
|
|
return item
|