The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
15 KiB
desk.sources — the plugins
What it is
"Source" and "plugin" are the same thing. One .py file with a Source subclass in it. It answers one question: where do items come from — fetch, prepare, yield. Nothing else: no store, no queue, no config beyond its own options.
| Contract | base.py — Source, Item, Suggestion, CheckResult |
| Discovery | registry.available() — every module in desk/sources/, plus every *.py in the dirs under plugin_paths: in config.yaml (leading _ skipped) |
| Built-ins | bluesky, mastodon, x, podcast. RssSource is here too and is not a plugin: it is shared, which the registry passes over, and a publisher subclasses it |
| Packages | httpx fetches, feedparser parses what a feed answered with. No plugin framework, no entry points |
| Door | from desk.sources import Source, Item, build, available, PodcastSource |
| Override | later definition wins — a user plugin named bluesky replaces the built-in, with a warning naming both |
The three steps
- Fetch —
fetch()is a generator; hit the network, yield as you go. Put a timeout on every request you make — see Limits. - Prepare — normalise into
Item. Over-fetching is fine: the runner dedups onuid = sha1(source + source_id). - Output —
Items. The runner fillssource, applies the default outlet, and stores them.
Item(source_id, location, title, excerpt="", body="", author="", published_at=now, subscription="", kind="article", marks=[], raw={}, links=[]). published_at must be an aware datetime; it is stored as UTC. A non-empty body means "don't scrape this page" — that's how a podcast segment keeps its transcript. A post whose location equals raw["post_url"] links nothing and is not fetched either. The source's own labels go in raw under categories, which the tagger reads as a weak hint. ItemLink, exported through the same door, carries an absolute HTTP(S) URL, title and reading/discussion/source role separately from the item's primary destination.
A plugin may register how to read its own URLs. getters = (Getter("bluesky_api", posts.bluesky, posts.is_bluesky, claims=True),) on the class says that a bsky.app post link, from whichever outlet, is read through that function rather than as a web page, while this plugin is active. It gets the page only; which link an item points to is decided in fetch. See docs/architecture.md §1.1.
A plugin declares what should be done with its items, and the app does it. spec.py is that declaration — PluginSpec, its profiles, and a SummaryPolicy and TaggingPolicy per profile. Both shared jobs now run off one: a profile says whether its items are summarised and tagged, which stored text each task reads in order of preference, how short is too short for an abstract and how long one should be, and what this outlet wants said or emphasised about its own items. Those instructions travel with each item in a mixed batch and never apply to another outlet's. See newsdesk/jobs/summarise.py and newsdesk/jobs/tagging.py.
An outlet is a plugin. There is no generic "Any RSS feed" source to paste a masthead into — step 19 retired it, because a feed belongs to a publisher and a publisher has extraction, marks, retries and a summary floor of its own. RssSource is the fetching those plugins share: subclass it, give it a name and a spec, declare the feeds. plugins/ars_technica.py is the smallest one, a catalogue of a single feed. A feed nobody has written a plugin for keeps its archive and its Show switch and is simply not fetched; the v28 migration names each one in the log.
Declaring nothing is not declaring no. A plugin with no spec gets an adapted one with both tasks off, and the two jobs read that silence differently on purpose. Nothing undeclared has ever been summarised, so the summariser takes it as no and loses nothing. Tagging has run over every row in the archive since long before this contract, so it takes it as today's behaviour — the acquired text, then the feed's own, then an abstract as a last resort — and only a profile that says enabled=False out loud is left untagged. A converted outlet must therefore say enabled=True where its items were already tagged; Guardian and Hacker News say so. Which field a tagging policy names decides which text and not how much of it: the tagger reads the opening few hundred characters whatever is named, because it is choosing from a closed list of ~35 labels and a whole transcript would be thirty times the tokens for the same answer.
A profile also declares how its items are read. The drawer is fixed: Summary, Text, Original and a new-tab action, in that order, for every plugin. The content is the plugin's: which slots it fills, what to call them (a segment's text is its Transcript, its original is Watch), and an ordered list of sections per view, each prose, links or media bound to a named field. It cannot add a fourth view, emit markup, or overrule an outlet that refuses framing. header holds links shown above every view: the host of the page the item is, the discussion thread of an aggregator's link, the articles a podcast segment discusses. QueueLayout.destination adds that host to the queue row too, and is for a plugin whose outlet is not the publisher. Name a preset rather than restating one: ARTICLE_READER for a publisher's own article, LINK_READER for a link to someone else's page, FEED_READER for a walled outlet, POST_READER for a social post, RECORDING_READER for a recording's segment. The table in docs/architecture.md §1.4 says which flow each is. A view may declare what to show instead when its own field is empty, with its own label: ARTICLE_READER's Summary control falls through to the feed's own text and calls itself RSS feed, which is what the queue row beside it has always done. The rename is half of it: a publisher's teaser under a control marked Summary reads as our abstract of the piece, and as a poor one.
A mark and a wall are different facts, and only one of them changes the flow. framing asks whether the browser will render a page in a frame, and the page behind a paywalled item renders perfectly — it is the paywall. OriginalView.blocked_by names the marks that refuse the embed for the items carrying one (paywalled by default); the layout does not change, because the piece beside it in the same feed is an ordinary article, and the control greys out with the mark's own words (Mark.framed). A Wall on a profile is the other thing: there is no article behind any URL this outlet publishes, no abstract will ever be written from one, and the destination frames the gate — so the flow differs rather than the ordinary one standing there with two permanently dead controls. FEED_READER is that flow, the feed's own text and the way out, and a walled profile declaring an Original view of its destination is refused where it is written. Its reason rides on ReaderModel.note as the one line in the bar, since there is no greyed button left to hover. A plugin that learns to get past its wall deletes the declaration and is an ordinary article profile again. The server resolves the declaration against one item and serves the answer; the browser renders it and decides nothing. See newsdesk/web/reader.py.
A show subclasses PodcastSource, which finds the episode, retrieves its captions once and cuts them at the marks the show published. plugins/lastweekin_ai.py is the model: a catalog of one feed, segment_profile() — the machinery's own declaration of what it produces, exported through this door for a show to reuse — and nothing of its own beyond how this show numbers an episode and what it wants said about its segments. Each segment carries the articles its outline linked as links with the reading role, so the reading pane and the summariser read them from the same field an outlet's links live in.
A show whose site already publishes the cut needs none of that. plugins/ai_daily_brief.py subclasses Source directly: the site lists its editions in /agent.json, each as JSON nuggets and a minute-marked transcript, so the plugin makes one segment per nugget from two requests, with no feed, no captions and no summariser.
RSS outlets subclass RssSource. Its item_from_entry(entry, url, label, author) hook refines one parsed entry while shared code owns HTTP, feed parsing, limits and failure reporting. plugins/hacker_news.py uses it to preserve publisher destinations and HN discussion links while keeping points/comment metadata out of preview text. Guardian needs no override. Both declare fixed catalogs; unknown custom feeds remain supported through generic RSS.
There is no outlet and no section. Who publishes a thing is a fact about the subscription it came from, held once on the subscription row and joined when something needs it — so subscription is the only attribution an item carries. They used to be fields, copied onto every item, and a rename left four hundred rows disagreeing with the feed they came from until somebody walked the table.
What a plugin declares about failing
spec.py is the other half of the contract: not where items come from but what should be done with them. Step.retry is the part of it that runs today.
Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"},
retry=RetryPolicy(max_attempts=2, delay_seconds=300, backoff=1.0,
on=["timeout", "unreachable", "server_error", "rate_limited"],
manual=True))
Per step and per plugin, which is the whole reason it is declared. A transcript that was not ready an hour ago and a host that timed out are different waits, and retrying the whole outlet to get one of them is how a source ends up fetched four times an evening. max_attempts counts the first attempt, so the default of 1 is "no retry" and is what every plugin that declares nothing gets — behaviour unchanged from before any of this existed.
on names failure categories from desk.tools.failures, and a name that is not one of them is refused where the plugin is written rather than becoming a policy that silently matches nothing. Name the ones that could answer differently next time. A consent wall is a category and belongs in nobody's list: it is the same answer however many times it is asked for. Where the server named its own wait — a Retry-After — that is honoured in place of delay_seconds, but it buys no extra attempt: the bound is the plugin's and the wait is the world's.
manual is whether the reader offers a person the retry the policy would not take on its own. The app still checks that the offer means anything: the plugin must be active, and the item must actually have been tried and come back empty. A target's Fetch switch is not checked for a manual retry — Fetch governs the evening's automatic acquisition, not a repair somebody asked for.
Minimum example
# plugins/mysource.py (or desk/sources/mysource.py)
from desk.sources import Source, Item
class MySource(Source):
name = "mysource" # the key under `sources:` in config.yaml
description = "One line shown in the Sources tab."
subscription_field = "accounts" # or "feeds" for URL-shaped subscriptions
def fetch(self):
for target, label in self.targets(): # subscriptions from config/UI
yield Item(source_id="abc123", location="https://example.com/x",
title="Headline", excerpt="Summary", subscription=target)
# config.yaml
plugin_paths: ["plugins"]
sources:
mysource:
enabled: true
accounts: ["someone"]
Optional, both defaulting to nothing: discover() -> [Suggestion] (accounts worth following) and check() -> [CheckResult] (does each subscription still resolve — free requests only, it runs for every source at once).
What a plugin is given
self.options— its own config block. By pull time the subscription list has been replaced with the fetch-enabled rows from the Sources tab.self.state— aMutableMapping[str, str]backed by the database, for what it must remember between pulls (a spend counter, which episodes are done). Driven directly it's a plain dict.
self.state is one namespace shared by every plugin, not one per source. Nothing prefixes your keys for you, so prefix them yourself: x.py writes x_spend_2026_09, and PodcastSource derives its done-marker key from the subclass name. A bare key like last_seen is a collision waiting for the second plugin that wants one, and the symptom is one source silently reading another's bookkeeping.
Helpers worth knowing: strip_html, parse_timestamp, entry_time, split_label, outlet_name, MULTIWORD_OUTLETS, http_failure, USER_AGENT, check_feed.
Two of those read names, and which one you want depends on who wrote the string. split_label reads a label you wrote and cuts it into outlet and section — "Guardian Tech" into the two — with MULTIWORD_OUTLETS as the list of names that contain a space themselves, so "Hacker News Best" does not come out as an outlet called Hacker. outlet_name reads a name a server gave for itself and cuts the tagline off it, because both a feed's <title> and a site's own title tend to arrive as the SEO version.
Limits
- One file, no state but
self.state. No store, no queue, no cross-source access. That's the deal that keeps a new source one file. - A failed target and an empty one are different answers. Yielding nothing means there was nothing new; raising means something is wrong and you are owed a notice about it —
desk/notices.pydecides from the exception whether it reads as the world misbehaving or as a bug of ours. A source with several subscriptions gets both at once by yielding everything that worked and raising once at the end for what didn't, which is what stops one dead feed costing you the other nineteen.rss.FeedsFailedis that exception. fetchis synchronous and blocking. Runs in the worker's thread. Async, rate-limiting and timeouts are yours. A slow source holds the pull; a source that fetches without a deadline holds it for ever, because cancellation is cooperative and Kill cannot reach a thread parked in a socket read. The case to watch for is a library that does its own fetching:feedparser.parse(url)andrequestswith notimeout=both wait indefinitely by default, sorss.pyfetches the bytes itself and hands the parser a document.- No sandbox, no versioning. A plugin file is imported and trusted; a plugin that raises on import is logged and skipped, one that raises inside
fetchfails that pull's source. namemust be unique and set — noname, not discovered.- Discovery is flat:
*.pyin a directory, not packages, not installed distributions. - A source can't create jobs. It yields items; what happens next (body, vectors, tags) is the app's ingest job.