Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9d666d0937
commit
248ea1636d
36 files changed
+8393
No files matched your search
@@ -0,0 +1,123 @@
|
||||
"""What kind of thing is this, when it isn't an ordinary article?
|
||||
|
||||
An outlet's feed carries more than its journalism. Golem's carries adverts
|
||||
("Anzeige:") and pieces behind its paywall ("(g+)"); Hacker News marks paid
|
||||
links "[$]"; other feeds mark neither. There is no convention across outlets to
|
||||
detect, so each plugin declares how its publisher says it — as `MarkRule`s in
|
||||
its spec, and nowhere else. Typing a token in against one feed was how this
|
||||
worked before every outlet was a plugin; step 19 retired that editor, and
|
||||
`classify` below is the only way a mark is now found.
|
||||
|
||||
What is general is everything after that. The outlet's token is input: it is
|
||||
matched, cut out of the title, and never shown. One glyph stands for a mark
|
||||
everywhere, so "what is this" reads the same whoever published it, instead of
|
||||
being a private code you learn one publisher at a time. The token is noise in
|
||||
a headline as well — "Anzeige: Schon 10.000-mal bestellt" is worse than the
|
||||
line underneath it.
|
||||
|
||||
The vocabulary below is closed on purpose. Marks are structural — they say what
|
||||
a thing *is*, and the reader filters and styles them — which is a different
|
||||
question from what a thing is *about*; that one has tags, an open vocabulary the
|
||||
model can extend. Adding a mark means adding a row here and a token to the feeds
|
||||
that use it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from collections.abc import Sequence
|
||||
from typing import Any, Protocol
|
||||
|
||||
from desk.models import Item
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Mark:
|
||||
slug: str
|
||||
glyph: str
|
||||
label: str
|
||||
#: What the reader is told the glyph means, on hover and to a screen reader.
|
||||
note: str
|
||||
#: Whether it is worth fetching the page for a body. False for both marks we
|
||||
#: have: a paywalled page hands back its gate, and an advert's body is the
|
||||
#: advert. Declared here rather than assumed by the backfill, so a mark that
|
||||
#: *does* want its body only has to say so.
|
||||
fetch_body: bool = False
|
||||
#: What an embedded frame of such an item would actually show, for the
|
||||
#: reader offered a control they cannot use. Here for the same reason
|
||||
#: `fetch_body` is: framing is a shared behaviour, and the mark is what
|
||||
#: knows whether a frame of it would be worth anything. See
|
||||
#: `OriginalView.blocked_by`.
|
||||
framed: str = ""
|
||||
|
||||
|
||||
MARKS: tuple[Mark, ...] = (
|
||||
Mark("paywalled", "\U0001F512", "Paywalled",
|
||||
"Behind this outlet's paywall", framed="the outlet's paywall"),
|
||||
Mark("sponsored", "\U0001F4E3", "Sponsored",
|
||||
"An advertisement, not journalism", framed="the advert"),
|
||||
)
|
||||
|
||||
BY_SLUG = {m.slug: m for m in MARKS}
|
||||
#: The marks whose items the body backfill should not spend a request on.
|
||||
NO_BODY = frozenset(m.slug for m in MARKS if not m.fetch_body)
|
||||
|
||||
|
||||
class Rule(Protocol):
|
||||
@property
|
||||
def field(self) -> str: ...
|
||||
@property
|
||||
def method(self) -> str: ...
|
||||
@property
|
||||
def value(self) -> str: ...
|
||||
@property
|
||||
def mark(self) -> str: ...
|
||||
@property
|
||||
def strip(self) -> bool: ...
|
||||
|
||||
|
||||
def classify(item: Item, rules: Sequence[Rule]) -> Item:
|
||||
"""Classify original inputs, retaining independent acquisition marks.
|
||||
|
||||
Evidence lives in raw alongside the publisher categories until those
|
||||
acquisition inputs gain typed storage. Repeating this replaces only marks
|
||||
owned by the previous rule pass, never a custom acquisition hook's marks.
|
||||
"""
|
||||
previous = item.raw.get("mark_evidence", {})
|
||||
original = item.source_title or item.title
|
||||
independent = previous.get("independent", item.marks)
|
||||
found: list[str] = []
|
||||
evidence: list[dict[str, Any]] = []
|
||||
spans: list[tuple[int, int]] = []
|
||||
for rule in rules:
|
||||
if rule.field == "inputs.source_categories":
|
||||
values = item.raw.get("categories", [])
|
||||
matched = next((v for v in values if isinstance(v, str)
|
||||
and v.lower() == rule.value.lower()), None)
|
||||
at = 0 if matched is not None else -1
|
||||
value = matched or ""
|
||||
else:
|
||||
value = item.excerpt if rule.field == "inputs.feed_text" else original
|
||||
lower, token = value.lower(), rule.value.lower()
|
||||
at = lower.find(token)
|
||||
if rule.method == "prefix" and not lower.startswith(token):
|
||||
at = -1
|
||||
elif rule.method == "suffix":
|
||||
at = len(value) - len(rule.value) if lower.endswith(token) else -1
|
||||
if at < 0:
|
||||
continue
|
||||
found.append(rule.mark)
|
||||
evidence.append({"field": rule.field, "method": rule.method,
|
||||
"value": rule.value, "mark": rule.mark,
|
||||
"strip": rule.strip,
|
||||
"matched": value[at:at + len(rule.value)]})
|
||||
if rule.strip:
|
||||
spans.append((at, at + len(rule.value)))
|
||||
item.source_title = original
|
||||
item.title = ("".join(c for i, c in enumerate(original)
|
||||
if not any(start <= i < end for start, end in spans)).strip() or original
|
||||
) if spans else original
|
||||
item.marks = sorted(set(independent) | set(found))
|
||||
item.raw["mark_evidence"] = {"version": 1, "independent": list(independent),
|
||||
"matches": evidence}
|
||||
return item
|
||||
Reference in new issue
Block a user