Files
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

124 lines
5.2 KiB
Python

"""What kind of thing is this, when it isn't an ordinary article?
An outlet's feed carries more than its journalism. Golem's carries adverts
("Anzeige:") and pieces behind its paywall ("(g+)"); Hacker News marks paid
links "[$]"; other feeds mark neither. There is no convention across outlets to
detect, so each plugin declares how its publisher says it — as `MarkRule`s in
its spec, and nowhere else. Typing a token in against one feed was how this
worked before every outlet was a plugin; step 19 retired that editor, and
`classify` below is the only way a mark is now found.
What is general is everything after that. The outlet's token is input: it is
matched, cut out of the title, and never shown. One glyph stands for a mark
everywhere, so "what is this" reads the same whoever published it, instead of
being a private code you learn one publisher at a time. The token is noise in
a headline as well — "Anzeige: Schon 10.000-mal bestellt" is worse than the
line underneath it.
The vocabulary below is closed on purpose. Marks are structural — they say what
a thing *is*, and the reader filters and styles them — which is a different
question from what a thing is *about*; that one has tags, an open vocabulary the
model can extend. Adding a mark means adding a row here and a token to the feeds
that use it.
"""
from __future__ import annotations
from dataclasses import dataclass
from collections.abc import Sequence
from typing import Any, Protocol
from desk.models import Item
@dataclass(frozen=True, slots=True)
class Mark:
slug: str
glyph: str
label: str
#: What the reader is told the glyph means, on hover and to a screen reader.
note: str
#: Whether it is worth fetching the page for a body. False for both marks we
#: have: a paywalled page hands back its gate, and an advert's body is the
#: advert. Declared here rather than assumed by the backfill, so a mark that
#: *does* want its body only has to say so.
fetch_body: bool = False
#: What an embedded frame of such an item would actually show, for the
#: reader offered a control they cannot use. Here for the same reason
#: `fetch_body` is: framing is a shared behaviour, and the mark is what
#: knows whether a frame of it would be worth anything. See
#: `OriginalView.blocked_by`.
framed: str = ""
MARKS: tuple[Mark, ...] = (
Mark("paywalled", "\U0001F512", "Paywalled",
"Behind this outlet's paywall", framed="the outlet's paywall"),
Mark("sponsored", "\U0001F4E3", "Sponsored",
"An advertisement, not journalism", framed="the advert"),
)
BY_SLUG = {m.slug: m for m in MARKS}
#: The marks whose items the body backfill should not spend a request on.
NO_BODY = frozenset(m.slug for m in MARKS if not m.fetch_body)
class Rule(Protocol):
@property
def field(self) -> str: ...
@property
def method(self) -> str: ...
@property
def value(self) -> str: ...
@property
def mark(self) -> str: ...
@property
def strip(self) -> bool: ...
def classify(item: Item, rules: Sequence[Rule]) -> Item:
"""Classify original inputs, retaining independent acquisition marks.
Evidence lives in raw alongside the publisher categories until those
acquisition inputs gain typed storage. Repeating this replaces only marks
owned by the previous rule pass, never a custom acquisition hook's marks.
"""
previous = item.raw.get("mark_evidence", {})
original = item.source_title or item.title
independent = previous.get("independent", item.marks)
found: list[str] = []
evidence: list[dict[str, Any]] = []
spans: list[tuple[int, int]] = []
for rule in rules:
if rule.field == "inputs.source_categories":
values = item.raw.get("categories", [])
matched = next((v for v in values if isinstance(v, str)
and v.lower() == rule.value.lower()), None)
at = 0 if matched is not None else -1
value = matched or ""
else:
value = item.excerpt if rule.field == "inputs.feed_text" else original
lower, token = value.lower(), rule.value.lower()
at = lower.find(token)
if rule.method == "prefix" and not lower.startswith(token):
at = -1
elif rule.method == "suffix":
at = len(value) - len(rule.value) if lower.endswith(token) else -1
if at < 0:
continue
found.append(rule.mark)
evidence.append({"field": rule.field, "method": rule.method,
"value": rule.value, "mark": rule.mark,
"strip": rule.strip,
"matched": value[at:at + len(rule.value)]})
if rule.strip:
spans.append((at, at + len(rule.value)))
item.source_title = original
item.title = ("".join(c for i, c in enumerate(original)
if not any(start <= i < end for start, end in spans)).strip() or original
) if spans else original
item.marks = sorted(set(independent) | set(found))
item.raw["mark_evidence"] = {"version": 1, "independent": list(independent),
"matches": evidence}
return item