The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1013 lines
45 KiB
Python
1013 lines
45 KiB
Python
"""What a plugin declares about itself, as types rather than as convention.
|
|
|
|
A plugin is still one file with a `Source` subclass in it — `registry.py`
|
|
finds it and `base.py` is the contract for *fetching*. This is the other half,
|
|
and it is declarative: instead of a plugin calling a model, cutting a
|
|
transcript or emitting markup, it says which shared tools acquire its content,
|
|
what the shared summariser and tagger should do with it, and how its fields
|
|
are laid out. The application owns the doing. See `docs/plugin-rework.md`.
|
|
|
|
class HackerNews(Source):
|
|
name = "hacker_news"
|
|
spec = PluginSpec(api_version=1, id=name, label="Hacker News", ...)
|
|
|
|
Nothing executes a spec yet. This step introduces the contract and its
|
|
validation so that later steps have one thing to implement against, rather
|
|
than each inventing its own declaration and then reconciling them.
|
|
|
|
Three rules hold this together:
|
|
|
|
*Versioned.* `api_version` is the whole contract, not a part of it, which is
|
|
why every type it needs is declared here at once. A field a later step starts
|
|
*reading* costs nothing; a field a later step has to *add* means every plugin
|
|
written against version 1 was written against a contract that did not say what
|
|
it claimed to say.
|
|
|
|
*Validated twice, for two different reasons.* What is wrong on its face — a
|
|
step that needs one that isn't there, a cycle, a step writing over a summary —
|
|
is a pydantic error raised where the plugin is declared, so a bad spec fails
|
|
at import and `registry._load_file` logs it against that file and keeps every
|
|
other plugin. What can only be judged by the app — is `transcribe_media` a
|
|
tool this application has, is `segment` a kind it lays out — is `problems()`,
|
|
because `desk` has no list of either and inventing one here would be the
|
|
contract deciding what an app on it supports. See `Vocabulary`.
|
|
|
|
*Undeclared is a legitimate state.* Every plugin that exists today declares no
|
|
spec, and `spec_of` gives it one marked `adapted`: honest about the identity
|
|
it does have and silent about everything it never said. An adapted spec is
|
|
never invalid, because there is nothing in it that a plugin author wrote.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
from dataclasses import dataclass
|
|
from typing import Any, Literal
|
|
|
|
from pydantic import model_validator
|
|
|
|
from desk.models import Model
|
|
from desk.tools import failures as failures_tool
|
|
from desk.tools import marks as marks_tool
|
|
|
|
#: The contract this module defines. A plugin naming any other number is not
|
|
#: refused because it is wrong but because nothing here knows what it means.
|
|
API_VERSION = 1
|
|
|
|
#: A plugin id, and a target id: stable machine names. Not the displayed
|
|
#: outlet name — that is `label`, it is written for people, and it changes.
|
|
#: An id that changed would orphan every stored item that recorded it.
|
|
_MACHINE_NAME = re.compile(r"^[a-z][a-z0-9_]*$")
|
|
|
|
#: Where an acquisition step is allowed to write. The plugin and its steps own
|
|
#: inputs; the shared summariser and tagger own outputs. A step that could
|
|
#: write `outputs.summary` would be a plugin generating its own summary
|
|
#: through the back door, which is the one thing this contract exists to stop.
|
|
_OWNED_BY_ACQUISITION = "inputs."
|
|
|
|
#: The marks are a shared vocabulary with shared glyphs and shared filters; a
|
|
#: plugin declares the *evidence* for one, never a mark of its own.
|
|
_MARK_SLUGS = frozenset(marks_tool.BY_SLUG)
|
|
|
|
#: What a failed step can be called. Shared for the same reason the marks are:
|
|
#: the categories are what the runner *produces*, so a policy naming one they
|
|
#: do not contain is a policy that matches nothing, and the place to find that
|
|
#: out is the plugin file rather than an evening of retries that never happen.
|
|
_FAILURES = failures_tool.CATEGORIES
|
|
|
|
|
|
# --------------------------------------------------------------- acquisition
|
|
|
|
class RetryPolicy(Model):
|
|
"""When a failed acquisition step is worth running again.
|
|
|
|
Per step, not per plugin and not per pull: a transcript that was not ready
|
|
an hour ago and an article whose host timed out are different waits, and
|
|
retrying the whole outlet to get one of them is how a source ends up
|
|
fetched four times an evening.
|
|
"""
|
|
|
|
#: Counting the first attempt, so 1 means "no retry" and reads as it says.
|
|
max_attempts: int = 1
|
|
delay_seconds: int = 0
|
|
#: Multiplied into the delay for each further attempt. 1.0 is a fixed wait.
|
|
backoff: float = 1.0
|
|
#: Which failures are worth another attempt — `timeout`, `server_error`.
|
|
#: Empty retries none of them, which is what `max_attempts=1` already says.
|
|
#: A consent wall is deliberately not in any app's list by default: it is
|
|
#: the same answer however many times it is asked for. The names are
|
|
#: `desk.tools.failures.CATEGORIES`; a misspelt one is refused here rather
|
|
#: than becoming a policy that quietly matches nothing.
|
|
on: list[str] = []
|
|
#: Whether the reader offers a person the retry this policy would not take
|
|
#: on its own.
|
|
manual: bool = False
|
|
|
|
@model_validator(mode="after")
|
|
def _bounded(self) -> RetryPolicy:
|
|
if not 1 <= self.max_attempts <= 10:
|
|
raise ValueError(
|
|
f"max_attempts is {self.max_attempts}; it counts the first "
|
|
"attempt, so it must be between 1 and 10")
|
|
if self.delay_seconds < 0 or self.backoff < 1.0:
|
|
raise ValueError("a retry delay cannot be negative and backoff "
|
|
"cannot shrink it")
|
|
unknown = sorted(set(self.on) - _FAILURES)
|
|
if unknown:
|
|
raise ValueError(
|
|
f"retry on {', '.join(unknown)}: not a failure category "
|
|
f"({', '.join(sorted(_FAILURES))})")
|
|
return self
|
|
|
|
|
|
class Step(Model):
|
|
"""One named piece of acquisition: a shared tool, and what it reads and
|
|
writes.
|
|
|
|
`inputs` maps the tool's own parameters to where their values come from;
|
|
`outputs` maps what it produced to the item fields it lands in. Both sides
|
|
named, so a reader of the plugin can see what a step will change without
|
|
knowing the tool, and so the runner can refuse a step that would write
|
|
somewhere it does not own.
|
|
"""
|
|
|
|
id: str
|
|
tool: str
|
|
inputs: dict[str, str] = {}
|
|
outputs: dict[str, str] = {}
|
|
#: Step ids in the same profile that must succeed first. The extraction of
|
|
#: an article needs the destination URL a discovery hook worked out.
|
|
needs: list[str] = []
|
|
#: A field path that must have a value for this step to run at all. Empty
|
|
#: runs it always.
|
|
when: str = ""
|
|
retry: RetryPolicy = RetryPolicy()
|
|
|
|
@model_validator(mode="after")
|
|
def _writes_only_inputs(self) -> Step:
|
|
if not _MACHINE_NAME.match(self.id):
|
|
raise ValueError(f"step id {self.id!r} is not a machine name")
|
|
for produced, destination in self.outputs.items():
|
|
if not destination.startswith(_OWNED_BY_ACQUISITION):
|
|
raise ValueError(
|
|
f"step {self.id!r} writes {produced!r} to {destination!r}; "
|
|
"an acquisition step may only write `inputs.*` — outputs "
|
|
"belong to the shared summariser and tagger")
|
|
return self
|
|
|
|
|
|
class Discovery(Model):
|
|
"""How a plugin finds items in the first place.
|
|
|
|
Separate from acquisition because it answers a different question: not
|
|
"what is in this item" but "which items are there". `plugin` means the
|
|
class's own `fetch`, which is what every plugin does today.
|
|
"""
|
|
|
|
tool: Literal["plugin", "rss", "api"] = "plugin"
|
|
#: One fixed feed, for a plugin whose whole catalogue is a single address.
|
|
#: A plugin with more than one declares them in `targets` instead, and the
|
|
#: discovery helper reads them from there; two competing lists of feeds is
|
|
#: the thing this must not become.
|
|
feed_url: str = ""
|
|
options: dict[str, Any] = {}
|
|
|
|
|
|
class RSS(Discovery):
|
|
"""`discovery=RSS(feed_url=...)`, which is the common case spelled out."""
|
|
|
|
tool: Literal["plugin", "rss", "api"] = "rss"
|
|
|
|
|
|
# ---------------------------------------------------------------- the tasks
|
|
|
|
class SummaryPolicy(Model):
|
|
"""What the shared summariser should do with this profile's items.
|
|
|
|
Structured parameters settle length and eligibility; `instructions` is
|
|
editorial and cannot change the schema, the fields written, or whether an
|
|
item is eligible at all. It is added to the same model call rather than
|
|
being a second pass over the answer.
|
|
"""
|
|
|
|
#: Required, and required even when false: a profile that says nothing
|
|
#: about summarising is a profile whose author has not decided, and the
|
|
#: cost of guessing is a model bill or a queue of items silently skipped.
|
|
enabled: bool
|
|
#: In order; the first field with usable content is the one summarised.
|
|
#: Not concatenated — the same words twice is not more context.
|
|
input_fields: list[str] = []
|
|
#: Whether a partial article — a feed's first paragraph — may be summarised
|
|
#: as though it were the piece.
|
|
require_complete: bool = True
|
|
#: Below this, the item is skipped with a reason rather than summarised.
|
|
#: A long article and a two-minute segment want different floors.
|
|
min_words: int = 0
|
|
target_words: int = 140
|
|
instructions: str = ""
|
|
|
|
@model_validator(mode="after")
|
|
def _has_something_to_read(self) -> SummaryPolicy:
|
|
return _needs_inputs(self, "summary")
|
|
|
|
|
|
class TaggingPolicy(Model):
|
|
"""What the shared tagger should do with this profile's items.
|
|
|
|
The vocabulary stays central: a plugin adds emphasis, never a tag
|
|
namespace of its own. Prefer acquired content as the input rather than the
|
|
summary, so tagging does not silently wait on a summary that failed.
|
|
|
|
Naming a field says *which* text is read and never how much of it. An app
|
|
that picks from a fixed list of labels may read the opening of it and no
|
|
more — newsdesk does, at four hundred characters — so naming a transcript
|
|
here is not a request to send a transcript.
|
|
"""
|
|
|
|
enabled: bool
|
|
input_fields: list[str] = []
|
|
#: Read for context but not as the thing being tagged: the title, the
|
|
#: publisher's own categories.
|
|
context_fields: list[str] = []
|
|
instructions: str = ""
|
|
|
|
@model_validator(mode="after")
|
|
def _has_something_to_read(self) -> TaggingPolicy:
|
|
return _needs_inputs(self, "tagging")
|
|
|
|
|
|
def _needs_inputs[T: SummaryPolicy | TaggingPolicy](policy: T, name: str) -> T:
|
|
"""An enabled task with no input field is a task that will look at every
|
|
item and find nothing to read — reported here rather than as an empty
|
|
result for the whole archive."""
|
|
if policy.enabled and not policy.input_fields:
|
|
raise ValueError(f"{name} is enabled but names no input_fields")
|
|
return policy
|
|
|
|
|
|
class Tasks(Model):
|
|
"""The shared tasks, both declared explicitly. See `SummaryPolicy.enabled`
|
|
for why neither has a default."""
|
|
|
|
summary: SummaryPolicy
|
|
tagging: TaggingPolicy
|
|
|
|
|
|
# ---------------------------------------------------------- how it is shown
|
|
|
|
class Section(Model):
|
|
"""One band of a view: a standard field, bound to a shared component.
|
|
|
|
Not markup. Ordering the list is what moves a player above or below the
|
|
links, and that is the whole of the layout language on purpose — safe
|
|
links, responsive behaviour, loading and error states belong to the
|
|
renderer, and every plugin that could express them would express them
|
|
differently.
|
|
"""
|
|
|
|
component: str
|
|
field: str
|
|
label: str = ""
|
|
#: For a `links` section: show only links with this role — `reading`,
|
|
#: `discussion`, `source`.
|
|
role: str = ""
|
|
#: For a `media` section: which media reference, when an item has several.
|
|
media: str = ""
|
|
collapsed: bool = False
|
|
|
|
|
|
class View(Model):
|
|
"""One of the reader's exclusive views, as this profile fills it in."""
|
|
|
|
label: str
|
|
sections: list[Section] = []
|
|
#: Field paths that must have a value for this view to be offered. A
|
|
#: Summary view with no summary is disabled and says why, rather than
|
|
#: rendering an empty panel or a placeholder dressed as an abstract.
|
|
requires: list[str] = []
|
|
#: What to show here when `requires` cannot be met: another view, with its
|
|
#: own label, its own requirement and its own sections. The control keeps
|
|
#: its place in the drawer and renames itself to say what is actually in
|
|
#: it, which is the difference between a reader who sees the feed's own
|
|
#: two sentences and one who sees a greyed-out button.
|
|
#:
|
|
#: The queue has done this since it existed — `QueueLayout.preview` is an
|
|
#: ordered list and the first non-empty field wins. A row that falls
|
|
#: through to the feed's text while the pane behind it refuses to show
|
|
#: anything is the same item answering two different ways.
|
|
instead: View | None = None
|
|
|
|
def chain(self) -> list[View]:
|
|
"""This view and every fallback under it, best first."""
|
|
return [self, *(self.instead.chain() if self.instead else [])]
|
|
|
|
|
|
class QueueLayout(Model):
|
|
"""The row, which is a decision rather than a read: a title, the best
|
|
preview available, and whatever metadata makes the decision."""
|
|
|
|
#: In order, first non-empty wins. A generated summary, then the feed's
|
|
#: own text — and a placeholder is never one of these.
|
|
preview: list[str] = []
|
|
metadata: list[str] = []
|
|
#: Whether the row names the host its item points at, beside the outlet.
|
|
#: True where the outlet is not the publisher — an aggregator's front page
|
|
#: — because there the host is half of what the decision to read is made
|
|
#: on, and "Hacker News" alone does not say whether this is a kernel
|
|
#: mailing list or a newspaper. False everywhere the two are the same
|
|
#: name twice.
|
|
destination: bool = False
|
|
|
|
|
|
class OriginalView(Model):
|
|
"""The destination itself, embedded. Hidden on mobile, and never shown
|
|
over an outlet that refuses framing however this is labelled."""
|
|
|
|
field: str
|
|
label: str = "Original"
|
|
#: The marks whose items this view is not offered for. Per item, because
|
|
#: the mark is: a piece behind the paywall frames the paywall, and the
|
|
#: piece beside it in the same feed frames the article. The layout does
|
|
#: not change — the control stays where it is, greyed, with the reason on
|
|
#: it — because which items are readable is not a fact about the outlet.
|
|
#: An outlet where *nothing* is readable is a `Wall`, which is a different
|
|
#: thing and has a different flow. Mark slugs, from the shared vocabulary.
|
|
blocked_by: list[str] = ["paywalled"]
|
|
|
|
@model_validator(mode="after")
|
|
def _known_marks(self) -> OriginalView:
|
|
unknown = sorted(set(self.blocked_by) - _MARK_SLUGS)
|
|
if unknown:
|
|
raise ValueError(
|
|
f"blocked_by names {', '.join(repr(m) for m in unknown)}, "
|
|
f"which is not among the shared marks "
|
|
f"({', '.join(sorted(_MARK_SLUGS))})")
|
|
return self
|
|
|
|
|
|
class ExternalLink(Model):
|
|
"""The new-tab action beside the views. Not a view: it leaves."""
|
|
|
|
field: str
|
|
label: str = ""
|
|
|
|
|
|
class Presentation(Model):
|
|
"""Declared once per profile, never per stored item.
|
|
|
|
The drawer is fixed and the content is not. Every item is read through the
|
|
same four controls in the same order, whatever produced it:
|
|
|
|
- `summary`, the short version: an abstract, or the feed's own teaser
|
|
- `text`, the whole words: an article, a transcript, a post
|
|
- `original`, the thing itself inside the pane: the page framed, or the
|
|
recording playing
|
|
- `external`, the way out to a new tab
|
|
|
|
A plugin decides which of them it fills, what each is called, what sits
|
|
in each and what each falls back to. It never adds a fifth: a reader
|
|
moving from a newspaper to a podcast to a front-page link keeps one hand
|
|
position, and a sticky choice of view means the same thing on every item.
|
|
|
|
`header` is what holds whichever view is on: where an item points and
|
|
where it is discussed. An aggregator's destination is not its own outlet,
|
|
and a reader in the Text view of a Hacker News link is owed both names
|
|
without leaving it.
|
|
"""
|
|
|
|
queue: QueueLayout = QueueLayout()
|
|
#: Shown above the pane in every view, including one that cannot be
|
|
#: shown. `links` sections only: `inputs.original_url` is the item's own
|
|
#: destination, `inputs.links` with a `role` is the typed links. A section
|
|
#: with nothing in it drops out, so a show whose notes link nothing has no
|
|
#: header rather than an empty one.
|
|
header: list[Section] = []
|
|
default_view: str = ""
|
|
views: dict[str, View] = {}
|
|
original: OriginalView | None = None
|
|
external: ExternalLink | None = None
|
|
|
|
@model_validator(mode="after")
|
|
def _default_is_a_view(self) -> Presentation:
|
|
if self.default_view and self.default_view not in self.views:
|
|
raise ValueError(
|
|
f"default_view is {self.default_view!r}, which is not one of "
|
|
f"the declared views ({', '.join(sorted(self.views)) or 'none'})")
|
|
return self
|
|
|
|
@model_validator(mode="after")
|
|
def _header_is_links(self) -> Presentation:
|
|
wrong = sorted({s.component for s in self.header} - {"links"})
|
|
if wrong:
|
|
raise ValueError(
|
|
f"the header holds links, and this one declares "
|
|
f"{', '.join(repr(c) for c in wrong)}; prose and media belong "
|
|
"in a view")
|
|
return self
|
|
|
|
|
|
#: What an ordinary published article looks like in a reader, as a preset a
|
|
#: plugin can name instead of restating. The shapes are the contract's; which
|
|
#: of them an application actually renders is its own business, and
|
|
#: `problems()` is where the two meet.
|
|
#:
|
|
#: Summary before Text is the order the shell has, not a preference: a reader
|
|
#: who wants the piece presses Text, and the view they were last in is
|
|
#: remembered. A profile that wants its own arrangement declares one rather
|
|
#: than editing this.
|
|
ARTICLE_READER = Presentation(
|
|
queue=QueueLayout(preview=["outputs.summary", "inputs.feed_text"],
|
|
metadata=["outlet"]),
|
|
#: The page this item is, named by its host. What a reader in the Text
|
|
#: view is owed without leaving it: which page these words came from.
|
|
header=[Section(component="links", field="inputs.original_url")],
|
|
default_view="text",
|
|
views={
|
|
# An outlet that is never summarised, or one whose article we could
|
|
# not fetch, still published two sentences in its feed. They are the
|
|
# shortest version of the piece we have, which is what this control is
|
|
# for, so it shows them rather than greying out — and says "RSS feed",
|
|
# because they are the publisher's own teaser and calling a teaser a
|
|
# summary is how a reader comes to think our summaries are terrible.
|
|
"summary": View(label="Summary", requires=["outputs.summary"], sections=[
|
|
Section(component="prose", field="outputs.summary"),
|
|
], instead=View(label="RSS feed", requires=["inputs.feed_text"],
|
|
sections=[
|
|
Section(component="prose", field="inputs.feed_text"),
|
|
])),
|
|
"text": View(label="Text", requires=["inputs.article_text"], sections=[
|
|
Section(component="prose", field="inputs.article_text"),
|
|
Section(component="links", field="inputs.links", label="Links"),
|
|
]),
|
|
},
|
|
original=OriginalView(field="inputs.original_url"),
|
|
external=ExternalLink(field="inputs.original_url"),
|
|
)
|
|
|
|
|
|
#: What a social post looks like, as the preset a network names. A post is
|
|
#: its own text and a link out of it: there is no article to extract and no
|
|
#: abstract to wait for, so the profile declares one view rather than
|
|
#: offering a Summary that could never become ready.
|
|
#:
|
|
#: `inputs.feed_text` is the post itself and not a teaser for something
|
|
#: else — the same field an outlet's feed fills, read here as what the
|
|
#: network published rather than as a stub of an article behind it.
|
|
#:
|
|
#: The article section below it is the link post: a post that pointed at
|
|
#: something and had it fetched shows both, in the order they were written.
|
|
#: It is optional and a post without one simply has one band — which is why
|
|
#: `requires` names the post text and not the article.
|
|
POST_READER = Presentation(
|
|
# The same order every row is built in, and not a narrower claim: the
|
|
# queue falls through an abstract to the feed's own text whatever the
|
|
# profile, and a post declaring only the second half would be describing
|
|
# its own policy rather than the queue. There is never a summary here to
|
|
# come first.
|
|
queue=QueueLayout(preview=["outputs.summary", "inputs.feed_text"],
|
|
metadata=["outlet"]),
|
|
#: The page this item is, named by its host. What a reader in the Text
|
|
#: view is owed without leaving it: which page these words came from.
|
|
header=[Section(component="links", field="inputs.original_url")],
|
|
default_view="text",
|
|
views={
|
|
"text": View(label="Post", requires=["inputs.feed_text"], sections=[
|
|
Section(component="prose", field="inputs.feed_text"),
|
|
Section(component="prose", field="inputs.article_text",
|
|
label="What it linked to"),
|
|
Section(component="links", field="inputs.links", label="Links"),
|
|
]),
|
|
},
|
|
original=OriginalView(field="inputs.original_url"),
|
|
external=ExternalLink(field="inputs.original_url"),
|
|
)
|
|
|
|
|
|
#: What an outlet we cannot read at all looks like: the text its feed gave us,
|
|
#: and the way out to the page. The flow a profile declaring a `Wall` wants —
|
|
#: there is no article to put under a Text control, no abstract will ever be
|
|
#: written, and the destination frames the gate, so none of the three is
|
|
#: offered. `ARTICLE_READER` with two permanently disabled buttons says the
|
|
#: same thing, and says it again on every item for as long as the wall stands.
|
|
#:
|
|
#: The feed's text sits in the summary slot, where `ARTICLE_READER` puts it
|
|
#: when an abstract is missing, so that the control is in the same place and
|
|
#: keeps the same sticky choice as a reader moves between outlets.
|
|
FEED_READER = Presentation(
|
|
queue=QueueLayout(preview=["outputs.summary", "inputs.feed_text"],
|
|
metadata=["outlet"]),
|
|
#: The page this item is, named by its host. What a reader in the Text
|
|
#: view is owed without leaving it: which page these words came from.
|
|
header=[Section(component="links", field="inputs.original_url")],
|
|
default_view="summary",
|
|
views={
|
|
"summary": View(label="RSS feed", requires=["inputs.feed_text"],
|
|
sections=[
|
|
Section(component="prose", field="inputs.feed_text"),
|
|
Section(component="links", field="inputs.links", label="Links"),
|
|
]),
|
|
},
|
|
external=ExternalLink(field="inputs.original_url"),
|
|
)
|
|
|
|
|
|
#: What a link to somebody else's page looks like: an aggregator's front page,
|
|
#: where the item is a pointer and the outlet is not the publisher. The views
|
|
#: are `ARTICLE_READER`'s, because what is behind the link is an article, or
|
|
#: fails to be one, like any other. The header is what differs: the host the
|
|
#: link goes to and the thread it is discussed in, above every view — so both
|
|
#: are there when the page would not extract, which is exactly when a reader
|
|
#: needs them, and the Text view is the article and nothing else.
|
|
LINK_READER = Presentation(
|
|
queue=QueueLayout(preview=["outputs.summary", "inputs.feed_text"],
|
|
metadata=["outlet"], destination=True),
|
|
header=[
|
|
Section(component="links", field="inputs.original_url"),
|
|
Section(component="links", field="inputs.links", role="discussion"),
|
|
],
|
|
default_view="text",
|
|
views={
|
|
"summary": View(label="Summary", requires=["outputs.summary"], sections=[
|
|
Section(component="prose", field="outputs.summary"),
|
|
], instead=View(label="RSS feed", requires=["inputs.feed_text"],
|
|
sections=[
|
|
Section(component="prose", field="inputs.feed_text"),
|
|
])),
|
|
"text": View(label="Text", requires=["inputs.article_text"], sections=[
|
|
Section(component="prose", field="inputs.article_text"),
|
|
]),
|
|
},
|
|
original=OriginalView(field="inputs.original_url"),
|
|
external=ExternalLink(field="inputs.original_url"),
|
|
)
|
|
|
|
|
|
#: What a stretch of a recording looks like: the player's strip, then what
|
|
#: was said. Summary, Transcript, Watch — the same three places as an article,
|
|
#: under the names of what is in them.
|
|
#:
|
|
#: No fallback to `inputs.feed_text` under Summary, which is what an article's
|
|
#: does: a segment's feed text is the *episode's* whole timestamped outline,
|
|
#: so falling through to it would put the running order under every story cut
|
|
#: out of it. The transcript is a view you choose and not the thing under the
|
|
#: abstract, because it arrives as captions with the names mangled.
|
|
#:
|
|
#: The articles the hosts were reading are the header, so they stay in reach
|
|
#: while the recording plays. A show whose notes link nothing has no header,
|
|
#: and needs no second preset for it.
|
|
RECORDING_READER = Presentation(
|
|
queue=QueueLayout(preview=["outputs.summary", "inputs.feed_text"],
|
|
metadata=["outlet", "inputs.media"]),
|
|
header=[Section(component="links", field="inputs.links", role="reading")],
|
|
default_view="summary",
|
|
views={
|
|
"summary": View(label="Summary", requires=["outputs.summary"], sections=[
|
|
Section(component="media", field="inputs.media"),
|
|
Section(component="prose", field="outputs.summary"),
|
|
]),
|
|
"text": View(label="Transcript", requires=["inputs.transcript"],
|
|
sections=[
|
|
Section(component="media", field="inputs.media"),
|
|
Section(component="prose", field="inputs.transcript"),
|
|
]),
|
|
},
|
|
# Watch, not Original: the destination is the recording, playing at this
|
|
# segment's own offset.
|
|
original=OriginalView(field="inputs.media", label="Watch"),
|
|
external=ExternalLink(field="inputs.original_url"),
|
|
)
|
|
|
|
|
|
# -------------------------------------------------------------- the targets
|
|
|
|
class Target(Model):
|
|
"""One feed or one account, as the plugin declares it.
|
|
|
|
`id` is the archive's name for it and never the URL or the label: a feed
|
|
that moves, or a handle that renames itself, must stay the same thing it
|
|
was, with its items, votes and read state on it.
|
|
"""
|
|
|
|
id: str
|
|
label: str = ""
|
|
#: The address, handle or query this target is. Empty for an editable
|
|
#: target, which gets its locator when someone adds it.
|
|
locator: str = ""
|
|
section: str = ""
|
|
profile: str = "default"
|
|
#: What this target does before anyone touches the switches. Seeded once;
|
|
#: a later reload must not reassert them over a choice someone made.
|
|
fetch: bool = True
|
|
show: bool = True
|
|
|
|
|
|
class TargetSpec(Model):
|
|
"""Whether the plugin's targets are a fixed catalogue or a list a person
|
|
adds to — a plugin capability, not a default for every outlet.
|
|
|
|
A publisher declares its own feeds and there is no Add box: another
|
|
masthead's feed is another plugin, with that publisher's own extraction,
|
|
marks and retries. A social network declares no accounts and the box is
|
|
the point.
|
|
"""
|
|
|
|
mode: Literal["catalog", "editable"]
|
|
#: What one of these *is*: `rss_feed`, `account`. The app decides which
|
|
#: types it can add and validate.
|
|
kind: str
|
|
defaults: list[Target] = []
|
|
#: Names the check a typed value must pass before it is accepted —
|
|
#: `bluesky_handle`. Empty accepts anything the app's own type check does.
|
|
validator: str = ""
|
|
default_fetch: bool = True
|
|
default_show: bool = True
|
|
|
|
@model_validator(mode="after")
|
|
def _ids_are_stable_and_unique(self) -> TargetSpec:
|
|
seen = [t.id for t in self.defaults]
|
|
if len(seen) != len(set(seen)):
|
|
raise ValueError("two targets share an id: "
|
|
f"{sorted({i for i in seen if seen.count(i) > 1})}")
|
|
for target in self.defaults:
|
|
if not _MACHINE_NAME.match(target.id):
|
|
raise ValueError(f"target id {target.id!r} is not a machine name")
|
|
if self.mode == "catalog" and not self.defaults:
|
|
raise ValueError("a catalog declares its targets, and this one "
|
|
"declares none; use mode='editable' for a plugin "
|
|
"whose targets are added by a person")
|
|
return self
|
|
|
|
|
|
# ----------------------------------------------------------------- the wall
|
|
|
|
class Wall(Model):
|
|
"""What an outlet answers with instead of its articles — all of them.
|
|
|
|
Not a mark, and the difference is the whole reason this exists. A mark is
|
|
a fact about one item: this piece is behind the paywall, that one is an
|
|
advert. The archive filters on it, refuses the embed for it, and leaves
|
|
everything else about the outlet alone, because the piece next to it in
|
|
the same feed is an ordinary readable article. A wall is a fact about the
|
|
plugin: there is no article to fetch on any URL it produces, no abstract
|
|
will ever be written from one, and the destination frames the gate rather
|
|
than the piece. A walled outlet is not an outlet where everything is
|
|
paywalled — it is one we cannot read at all, and what it needs is a
|
|
different flow rather than the ordinary one with its controls greyed out
|
|
on every item for ever.
|
|
|
|
Declared rather than detected because the consequence is the *layout*.
|
|
`desk.tools.walls` learns a wall from three identical bodies and stops
|
|
fetching, which is right, is about a host, and arrives long after the
|
|
reading pane has had to decide what to offer. A plugin that learns to get
|
|
past its wall — a browser running the CMP's JavaScript for a
|
|
tracking-consent gate — deletes this declaration and its profile is an
|
|
ordinary article profile again, flow included.
|
|
"""
|
|
|
|
#: In the vocabulary the runner already reports failures in, so a declared
|
|
#: wall and a detected one are the same word on the row: `consent_wall`
|
|
#: for a tracking-consent gate, `denied` for a login or a bot check.
|
|
kind: Literal["consent_wall", "denied"]
|
|
#: Why, in words. For whoever reads the plugin, and for the reader, who is
|
|
#: owed an answer to "where is the rest of this".
|
|
reason: str = ""
|
|
|
|
@model_validator(mode="after")
|
|
def _says_why(self) -> Wall:
|
|
if not self.reason.strip():
|
|
raise ValueError(
|
|
"a wall needs a reason: it is what the reading pane tells "
|
|
"somebody who can see there is no article here")
|
|
return self
|
|
|
|
|
|
# ---------------------------------------------------------------- the marks
|
|
|
|
class MarkRule(Model):
|
|
"""How this outlet says "paid" or "advert" in its own text.
|
|
|
|
The vocabulary and the glyphs stay shared; only where the evidence is
|
|
found moves into the plugin. `strip` cuts a matched token out of the
|
|
displayed title — the original is kept, because a title with its token
|
|
already cut off cannot be reclassified later.
|
|
"""
|
|
|
|
#: Where to look: a field path, or a publisher category list.
|
|
field: str
|
|
method: Literal["token", "prefix", "suffix", "contains", "category"]
|
|
value: str
|
|
mark: str
|
|
strip: bool = False
|
|
|
|
@model_validator(mode="after")
|
|
def _known_mark(self) -> MarkRule:
|
|
if self.field not in {"title", "inputs.source_title", "inputs.feed_text",
|
|
"inputs.source_categories"}:
|
|
raise ValueError(f"unsupported mark input {self.field!r}")
|
|
if not self.value.strip():
|
|
raise ValueError("a mark rule needs a non-empty value")
|
|
categories = self.field == "inputs.source_categories"
|
|
if categories != (self.method == "category"):
|
|
raise ValueError("category matching requires source_categories")
|
|
if self.strip and self.field not in {"title", "inputs.source_title"}:
|
|
raise ValueError("only a title token may be stripped")
|
|
if self.mark not in _MARK_SLUGS:
|
|
raise ValueError(
|
|
f"mark {self.mark!r} is not one of the shared marks "
|
|
f"({', '.join(sorted(_MARK_SLUGS))}); a plugin declares the "
|
|
"evidence for a mark, not a mark of its own")
|
|
return self
|
|
|
|
|
|
# ------------------------------------------------------------- the contract
|
|
|
|
class Profile(Model):
|
|
"""One shape of content a plugin produces, end to end.
|
|
|
|
A show that publishes both whole episodes and cut segments declares two of
|
|
these rather than being two plugins, and each item records which one it
|
|
is. That recorded name is how an item is laid out and processed for the
|
|
rest of its life — never a guess made later from whether it happens to
|
|
have a video key, or from what its outlet is called.
|
|
"""
|
|
|
|
#: `article`, `post`, `segment` in newsdesk. A free string here on
|
|
#: purpose: which shapes exist is the app's business — see `Item.kind` —
|
|
#: and the app's `Vocabulary` is what checks it.
|
|
kind: str
|
|
acquisition: list[Step] = []
|
|
#: None inherits the plugin rules; an explicit list replaces them.
|
|
marks: list[MarkRule] | None = None
|
|
#: What this outlet answers with instead of its articles, when it does.
|
|
#: None is the ordinary case and says nothing; a `Wall` says there is no
|
|
#: article behind any of these URLs, which is a different flow and not a
|
|
#: worse version of this one. See `Wall`, and `FEED_READER` for the flow.
|
|
#:
|
|
#: The acquisition step stays declared beside it on purpose: it is what
|
|
#: would run if the wall came down, and running it is how `desk.tools.walls`
|
|
#: finds out that it has not.
|
|
wall: Wall | None = None
|
|
tasks: Tasks
|
|
presentation: Presentation = Presentation()
|
|
|
|
@model_validator(mode="after")
|
|
def _a_wall_is_not_a_view(self) -> Profile:
|
|
"""A walled profile may not offer the destination as a view.
|
|
|
|
The page frames perfectly well — a consent gate is a real page, and
|
|
`framing` has no way to tell it from an article — so this is not
|
|
something the app can find out for itself. And it is true of every
|
|
item from this outlet, for as long as the wall stands, which makes it
|
|
a layout decision rather than a control to grey out one item at a
|
|
time. The way to the page is `ExternalLink`: in a tab of their own a
|
|
reader can accept the gate and read the piece, which is the whole
|
|
difference between useless and the way out.
|
|
"""
|
|
original = self.presentation.original
|
|
if self.wall and original and original.field.split(".")[-1] != "media":
|
|
raise ValueError(
|
|
f"this profile declares a {self.wall.kind} and an Original "
|
|
"view of the destination, which would frame the wall on every "
|
|
"item; declare the way out as `external` instead — see "
|
|
"`FEED_READER`")
|
|
return self
|
|
|
|
@model_validator(mode="after")
|
|
def _acquisition_is_a_graph(self) -> Profile:
|
|
steps = {s.id: s for s in self.acquisition}
|
|
if len(steps) != len(self.acquisition):
|
|
raise ValueError("two acquisition steps share an id")
|
|
for step in self.acquisition:
|
|
for needed in step.needs:
|
|
if needed not in steps:
|
|
raise ValueError(
|
|
f"step {step.id!r} needs {needed!r}, which is not a "
|
|
f"step of this profile ({', '.join(steps) or 'none'})")
|
|
_no_cycles(steps)
|
|
return self
|
|
|
|
|
|
def _no_cycles(steps: dict[str, Step]) -> None:
|
|
"""Refuse a dependency loop where it is written rather than where it
|
|
hangs. Depth-first with three colours: an edge back into the path being
|
|
walked is the cycle, and the path is the useful half of the message."""
|
|
walking: list[str] = []
|
|
done: set[str] = set()
|
|
|
|
def walk(current: str) -> None:
|
|
if current in done:
|
|
return
|
|
if current in walking:
|
|
loop = " -> ".join([*walking[walking.index(current):], current])
|
|
raise ValueError(f"acquisition steps depend on each other: {loop}")
|
|
walking.append(current)
|
|
for needed in steps[current].needs:
|
|
walk(needed)
|
|
walking.pop()
|
|
done.add(current)
|
|
|
|
for step_id in steps:
|
|
walk(step_id)
|
|
|
|
|
|
class PluginSpec(Model):
|
|
"""Everything a plugin declares. One version, one id, one set of profiles.
|
|
|
|
Structural faults raise here, where the plugin is written. What only the
|
|
application can judge is `problems()`.
|
|
"""
|
|
|
|
api_version: int
|
|
#: Stable and machine-readable. Conventionally the class's `name`, which
|
|
#: is already the config key and already on every stored item.
|
|
id: str
|
|
#: The plugin's own revision. Bumping it makes outputs produced under the
|
|
#: old one recognisably stale without regenerating the archive.
|
|
version: str = "1"
|
|
#: What a person is shown. Free to change; `id` is not.
|
|
label: str = ""
|
|
discovery: Discovery = Discovery()
|
|
targets: TargetSpec | None = None
|
|
marks: list[MarkRule] = []
|
|
profiles: dict[str, Profile]
|
|
#: True when nothing in the plugin file declared this and `spec_of` read
|
|
#: it off the class. Such a plugin runs exactly as it did before the
|
|
#: contract existed; nothing about it is a claim its author made.
|
|
adapted: bool = False
|
|
|
|
@model_validator(mode="after")
|
|
def _coherent(self) -> PluginSpec:
|
|
if self.api_version != API_VERSION:
|
|
raise ValueError(
|
|
f"plugin {self.id!r} declares api_version {self.api_version}; "
|
|
f"this application implements version {API_VERSION}")
|
|
if not _MACHINE_NAME.match(self.id):
|
|
raise ValueError(
|
|
f"plugin id {self.id!r} is not a machine name: lowercase "
|
|
"letters, digits and underscores, starting with a letter")
|
|
if not self.profiles:
|
|
raise ValueError(f"plugin {self.id!r} declares no profiles; every "
|
|
"plugin has at least a `default`")
|
|
for name, target in _targets_of(self):
|
|
if target.profile not in self.profiles:
|
|
raise ValueError(
|
|
f"target {name!r} names profile {target.profile!r}, which "
|
|
f"this plugin does not declare "
|
|
f"({', '.join(sorted(self.profiles))})")
|
|
return self
|
|
|
|
|
|
def _targets_of(spec: PluginSpec) -> list[tuple[str, Target]]:
|
|
return [(t.id, t) for t in (spec.targets.defaults if spec.targets else [])]
|
|
|
|
|
|
# ------------------------------------------------------- what the app allows
|
|
|
|
@dataclass(frozen=True)
|
|
class Vocabulary:
|
|
"""The names one application answers to, against which a spec is checked.
|
|
|
|
Deliberately not a constant in this module. `desk` does not know which
|
|
acquisition tools an app has wired up, which kinds it lays out, or which
|
|
components it renders — and a list here would be this contract deciding
|
|
what an application on it supports, which is the opposite of the point.
|
|
An empty set means the app does not restrict that name, which is the
|
|
right answer for `fields` until the typed item fields exist.
|
|
"""
|
|
|
|
tools: frozenset[str] = frozenset()
|
|
kinds: frozenset[str] = frozenset()
|
|
components: frozenset[str] = frozenset()
|
|
views: frozenset[str] = frozenset()
|
|
fields: frozenset[str] = frozenset()
|
|
#: Target types the app can store and validate: `rss_feed`, `account`.
|
|
target_kinds: frozenset[str] = frozenset()
|
|
#: The named value checks the app implements for a target somebody types
|
|
#: in — `bluesky_handle`. A plugin naming one the app has not written is
|
|
#: asking for a guarantee nothing would enforce, which is worse than
|
|
#: declaring none. Empty does not restrict it.
|
|
validators: frozenset[str] = frozenset()
|
|
#: The preview order the app's queue can actually produce, in order, with
|
|
#: the leading group dropped. A queue row is one column in a list of
|
|
#: hundreds, so what it can show is the app's to say — an app whose
|
|
#: preview is a generated column can produce exactly one order, and a
|
|
#: profile declaring another has to be told rather than quietly given the
|
|
#: one order there is. Empty does not restrict it.
|
|
queue_preview: tuple[str, ...] = ()
|
|
|
|
|
|
def problems(spec: PluginSpec, vocabulary: Vocabulary) -> list[str]:
|
|
"""Every reason this application cannot run this plugin, in one list.
|
|
|
|
All of them, not the first: a plugin naming three tools an app does not
|
|
have should say so once rather than over three edits. An empty list means
|
|
it can be loaded.
|
|
|
|
An adapted spec is always empty — see `spec_of`. Nothing in one was
|
|
declared, so there is nothing in it to be wrong.
|
|
"""
|
|
if spec.adapted:
|
|
return []
|
|
|
|
found: list[str] = []
|
|
for name, profile in sorted(spec.profiles.items()):
|
|
where = f"{spec.id}.{name}"
|
|
found += _unknown(vocabulary.kinds, [profile.kind],
|
|
f"{where}: kind", "kinds this application lays out")
|
|
found += _unknown(vocabulary.tools, [s.tool for s in profile.acquisition],
|
|
f"{where}: acquisition tool", "tools it can run")
|
|
found += _unknown(
|
|
vocabulary.components,
|
|
[s.component for view in profile.presentation.views.values()
|
|
for shown in view.chain() for s in shown.sections]
|
|
+ [s.component for s in profile.presentation.header],
|
|
f"{where}: component", "components it renders")
|
|
found += _unknown(vocabulary.views, list(profile.presentation.views),
|
|
f"{where}: view", "views its reader has")
|
|
found += _unknown(vocabulary.fields, _fields_named(profile),
|
|
f"{where}: field", "fields an item has")
|
|
declared = tuple(name.split(".")[-1]
|
|
for name in profile.presentation.queue.preview)
|
|
if vocabulary.queue_preview and declared and declared != vocabulary.queue_preview:
|
|
found.append(
|
|
f"{where}: queue preview {' then '.join(declared)} is not an "
|
|
f"order this application's queue can produce "
|
|
f"({' then '.join(vocabulary.queue_preview)})")
|
|
if spec.targets:
|
|
found += _unknown(vocabulary.target_kinds, [spec.targets.kind],
|
|
f"{spec.id}: target kind",
|
|
"target types it can store")
|
|
found += _unknown(vocabulary.validators,
|
|
[v for v in [spec.targets.validator] if v],
|
|
f"{spec.id}: target validator",
|
|
"value checks it implements")
|
|
return found
|
|
|
|
|
|
def _unknown(allowed: frozenset[str], used: list[str], what: str,
|
|
supported: str) -> list[str]:
|
|
"""One diagnostic per unrecognised name, naming what was expected. An
|
|
empty `allowed` is "this app does not restrict these" and never a
|
|
vocabulary in which every name is wrong."""
|
|
if not allowed:
|
|
return []
|
|
return [f"{what} {name!r} is not among the {supported} "
|
|
f"({', '.join(sorted(allowed))})"
|
|
for name in sorted(set(used)) if name not in allowed]
|
|
|
|
|
|
def _fields_named(profile: Profile) -> list[str]:
|
|
"""Every item field this profile binds, from any of the places one can be
|
|
named. The leading group is dropped: `inputs.transcript` is the transcript
|
|
wherever it is read from."""
|
|
named = [*profile.tasks.summary.input_fields,
|
|
*profile.tasks.tagging.input_fields,
|
|
*profile.tasks.tagging.context_fields,
|
|
*profile.presentation.queue.preview,
|
|
*profile.presentation.queue.metadata,
|
|
*[section.field for section in profile.presentation.header]]
|
|
for step in profile.acquisition:
|
|
named += [*step.outputs.values(), *([step.when] if step.when else [])]
|
|
for view in profile.presentation.views.values():
|
|
for shown in view.chain():
|
|
named += [section.field for section in shown.sections]
|
|
named += shown.requires
|
|
return [name.split(".")[-1] for name in named if name]
|
|
|
|
|
|
# ------------------------------------------------ plugins that declare none
|
|
|
|
def spec_of(source: type) -> PluginSpec:
|
|
"""The plugin's own spec, or one adapted from the class it already is.
|
|
|
|
Every plugin in this codebase is undeclared today, and most will still be
|
|
after the first few are converted, so this is the ordinary path and not a
|
|
fallback. It reads only what the class actually states — its name, its
|
|
description, and whether its subscriptions are feeds or handles — and
|
|
states nothing else on the author's behalf. `adapted` is how everything
|
|
downstream tells the difference between a plugin that declared no
|
|
instructions and one that declared none *deliberately*.
|
|
|
|
Read off the class itself and never off a parent: a show that subclasses
|
|
`PodcastSource` is its own plugin with its own id and its own archive, and
|
|
inheriting a declaration would quietly file its items under the class it
|
|
borrowed from. A subclass that wants its parent's acquisition says so.
|
|
|
|
The one profile it carries is `default` with both tasks disabled. That is
|
|
not a claim that the plugin is not summarised or tagged — it is today,
|
|
by the pipeline, exactly as before. It is this contract saying it has been
|
|
told nothing about either, which is the truth.
|
|
"""
|
|
declared = source.__dict__.get("spec")
|
|
if isinstance(declared, PluginSpec):
|
|
return declared
|
|
name = getattr(source, "name", "") or ""
|
|
return PluginSpec(
|
|
api_version=API_VERSION, id=name, version="0",
|
|
label=getattr(source, "default_outlet", "") or name,
|
|
profiles={"default": Profile(
|
|
kind="", tasks=Tasks(summary=SummaryPolicy(enabled=False),
|
|
tagging=TaggingPolicy(enabled=False)))},
|
|
adapted=True,
|
|
)
|