Files
jobboard/desk/sources/sources.md
T
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

111 lines
15 KiB
Markdown

# `desk.sources` — the plugins
## What it is
**"Source" and "plugin" are the same thing.** One `.py` file with a `Source` subclass in it. It answers one question: *where do items come from* — fetch, prepare, yield. Nothing else: no store, no queue, no config beyond its own options.
| | |
|---|---|
| Contract | `base.py` — `Source`, `Item`, `Suggestion`, `CheckResult` |
| Discovery | `registry.available()` — every module in `desk/sources/`, plus every `*.py` in the dirs under `plugin_paths:` in config.yaml (leading `_` skipped) |
| Built-ins | `bluesky`, `mastodon`, `x`, `podcast`. `RssSource` is here too and is **not** a plugin: it is `shared`, which the registry passes over, and a publisher subclasses it |
| Packages | `httpx` fetches, `feedparser` parses what a feed answered with. No plugin framework, no entry points |
| Door | `from desk.sources import Source, Item, build, available, PodcastSource` |
| Override | later definition wins — a user plugin named `bluesky` replaces the built-in, with a warning naming both |
## The three steps
1. **Fetch** — `fetch()` is a generator; hit the network, yield as you go. Put a timeout on every request you make — see Limits.
2. **Prepare** — normalise into `Item`. Over-fetching is fine: the runner dedups on `uid = sha1(source + source_id)`.
3. **Output** — `Item`s. The runner fills `source`, applies the default outlet, and stores them.
`Item(source_id, location, title, excerpt="", body="", author="", published_at=now, subscription="", kind="article", marks=[], raw={}, links=[])`. `published_at` must be an aware datetime; it is stored as UTC. A non-empty `body` means "don't scrape this page" — that's how a podcast segment keeps its transcript. A post whose `location` equals `raw["post_url"]` links nothing and is not fetched either. The source's own labels go in `raw` under `categories`, which the tagger reads as a weak hint. `ItemLink`, exported through the same door, carries an absolute HTTP(S) URL, title and `reading`/`discussion`/`source` role separately from the item's primary destination.
**A plugin may register how to read its own URLs.** `getters = (Getter("bluesky_api", posts.bluesky, posts.is_bluesky, claims=True),)` on the class says that a `bsky.app` post link, from whichever outlet, is read through that function rather than as a web page, while this plugin is active. It gets the page only; which link an item points to is decided in `fetch`. See `docs/architecture.md` §1.1.
**A plugin declares what should be done with its items, and the app does it.** `spec.py` is that declaration — `PluginSpec`, its profiles, and a `SummaryPolicy` and `TaggingPolicy` per profile. Both shared jobs now run off one: a profile says whether its items are summarised and tagged, which stored text each task reads in order of preference, how short is too short for an abstract and how long one should be, and what this outlet wants said or emphasised about its own items. Those instructions travel with each item in a mixed batch and never apply to another outlet's. See `newsdesk/jobs/summarise.py` and `newsdesk/jobs/tagging.py`.
**An outlet is a plugin.** There is no generic "Any RSS feed" source to paste a masthead into — step 19 retired it, because a feed belongs to a publisher and a publisher has extraction, marks, retries and a summary floor of its own. `RssSource` is the fetching those plugins share: subclass it, give it a `name` and a `spec`, declare the feeds. `plugins/ars_technica.py` is the smallest one, a catalogue of a single feed. A feed nobody has written a plugin for keeps its archive and its Show switch and is simply not fetched; the v28 migration names each one in the log.
**Declaring nothing is not declaring no.** A plugin with no `spec` gets an adapted one with both tasks off, and the two jobs read that silence differently on purpose. Nothing undeclared has ever been summarised, so the summariser takes it as no and loses nothing. Tagging has run over every row in the archive since long before this contract, so it takes it as *today's behaviour* — the acquired text, then the feed's own, then an abstract as a last resort — and only a profile that says `enabled=False` out loud is left untagged. A converted outlet must therefore say `enabled=True` where its items were already tagged; Guardian and Hacker News say so. Which field a tagging policy names decides *which* text and not how much of it: the tagger reads the opening few hundred characters whatever is named, because it is choosing from a closed list of ~35 labels and a whole transcript would be thirty times the tokens for the same answer.
**A profile also declares how its items are read.** The drawer is fixed: Summary, Text, Original and a new-tab action, in that order, for every plugin. The content is the plugin's: which slots it fills, what to call them (a segment's text is its Transcript, its original is Watch), and an ordered list of sections per view, each `prose`, `links` or `media` bound to a named field. It cannot add a fourth view, emit markup, or overrule an outlet that refuses framing. `header` holds links shown above every view: the host of the page the item is, the discussion thread of an aggregator's link, the articles a podcast segment discusses. `QueueLayout.destination` adds that host to the queue row too, and is for a plugin whose outlet is not the publisher. Name a preset rather than restating one: `ARTICLE_READER` for a publisher's own article, `LINK_READER` for a link to someone else's page, `FEED_READER` for a walled outlet, `POST_READER` for a social post, `RECORDING_READER` for a recording's segment. The table in `docs/architecture.md` §1.4 says which flow each is. A view may declare what to show `instead` when its own field is empty, with its own label: `ARTICLE_READER`'s Summary control falls through to the feed's own text and calls itself RSS feed, which is what the queue row beside it has always done. The rename is half of it: a publisher's teaser under a control marked Summary reads as our abstract of the piece, and as a poor one.
**A mark and a wall are different facts, and only one of them changes the flow.** `framing` asks whether the browser will render a page in a frame, and the page behind a paywalled item renders perfectly — it is the paywall. `OriginalView.blocked_by` names the marks that refuse the embed for the items carrying one (`paywalled` by default); the layout does not change, because the piece beside it in the same feed is an ordinary article, and the control greys out with the mark's own words (`Mark.framed`). A `Wall` on a profile is the other thing: there is no article behind any URL this outlet publishes, no abstract will ever be written from one, and the destination frames the gate — so the flow differs rather than the ordinary one standing there with two permanently dead controls. `FEED_READER` is that flow, the feed's own text and the way out, and a walled profile declaring an Original view of its destination is refused where it is written. Its reason rides on `ReaderModel.note` as the one line in the bar, since there is no greyed button left to hover. A plugin that learns to get past its wall deletes the declaration and is an ordinary article profile again. The server resolves the declaration against one item and serves the answer; the browser renders it and decides nothing. See `newsdesk/web/reader.py`.
A show subclasses `PodcastSource`, which finds the episode, retrieves its captions once and cuts them at the marks the show published. `plugins/lastweekin_ai.py` is the model: a catalog of one feed, `segment_profile()` — the machinery's own declaration of what it produces, exported through this door for a show to reuse — and nothing of its own beyond how this show numbers an episode and what it wants said about its segments. Each segment carries the articles its outline linked as `links` with the `reading` role, so the reading pane and the summariser read them from the same field an outlet's links live in.
A show whose site already publishes the cut needs none of that. `plugins/ai_daily_brief.py` subclasses `Source` directly: the site lists its editions in `/agent.json`, each as JSON nuggets and a minute-marked transcript, so the plugin makes one segment per nugget from two requests, with no feed, no captions and no summariser.
RSS outlets subclass `RssSource`. Its `item_from_entry(entry, url, label, author)` hook refines one parsed entry while shared code owns HTTP, feed parsing, limits and failure reporting. `plugins/hacker_news.py` uses it to preserve publisher destinations and HN discussion links while keeping points/comment metadata out of preview text. Guardian needs no override. Both declare fixed catalogs; unknown custom feeds remain supported through generic RSS.
**There is no `outlet` and no `section`.** Who publishes a thing is a fact about the subscription it came from, held once on the subscription row and joined when something needs it — so `subscription` is the only attribution an item carries. They used to be fields, copied onto every item, and a rename left four hundred rows disagreeing with the feed they came from until somebody walked the table.
## What a plugin declares about failing
`spec.py` is the other half of the contract: not where items come from but what should be done with them. `Step.retry` is the part of it that runs today.
```python
Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"},
retry=RetryPolicy(max_attempts=2, delay_seconds=300, backoff=1.0,
on=["timeout", "unreachable", "server_error", "rate_limited"],
manual=True))
```
**Per step and per plugin, which is the whole reason it is declared.** A transcript that was not ready an hour ago and a host that timed out are different waits, and retrying the whole outlet to get one of them is how a source ends up fetched four times an evening. `max_attempts` counts the first attempt, so the default of 1 is "no retry" and is what every plugin that declares nothing gets — behaviour unchanged from before any of this existed.
**`on` names failure categories from `desk.tools.failures`**, and a name that is not one of them is refused where the plugin is written rather than becoming a policy that silently matches nothing. Name the ones that could answer differently next time. A consent wall is a category and belongs in nobody's list: it is the same answer however many times it is asked for. Where the server named its own wait — a `Retry-After` — that is honoured in place of `delay_seconds`, but it buys no extra attempt: the bound is the plugin's and the wait is the world's.
**`manual` is whether the reader offers a person the retry** the policy would not take on its own. The app still checks that the offer means anything: the plugin must be active, and the item must actually have been tried and come back empty. A target's Fetch switch is *not* checked for a manual retry — Fetch governs the evening's automatic acquisition, not a repair somebody asked for.
## Minimum example
```python
# plugins/mysource.py (or desk/sources/mysource.py)
from desk.sources import Source, Item
class MySource(Source):
name = "mysource" # the key under `sources:` in config.yaml
description = "One line shown in the Sources tab."
subscription_field = "accounts" # or "feeds" for URL-shaped subscriptions
def fetch(self):
for target, label in self.targets(): # subscriptions from config/UI
yield Item(source_id="abc123", location="https://example.com/x",
title="Headline", excerpt="Summary", subscription=target)
```
```yaml
# config.yaml
plugin_paths: ["plugins"]
sources:
mysource:
enabled: true
accounts: ["someone"]
```
Optional, both defaulting to nothing: `discover() -> [Suggestion]` (accounts worth following) and `check() -> [CheckResult]` (does each subscription still resolve — free requests only, it runs for every source at once).
## What a plugin is given
- `self.options` — its own config block. By pull time the subscription list has been replaced with the fetch-enabled rows from the Sources tab.
- `self.state` — a `MutableMapping[str, str]` backed by the database, for what it must remember between pulls (a spend counter, which episodes are done). Driven directly it's a plain dict.
**`self.state` is one namespace shared by every plugin**, not one per source. Nothing prefixes your keys for you, so prefix them yourself: `x.py` writes `x_spend_2026_09`, and `PodcastSource` derives its done-marker key from the subclass name. A bare key like `last_seen` is a collision waiting for the second plugin that wants one, and the symptom is one source silently reading another's bookkeeping.
Helpers worth knowing: `strip_html`, `parse_timestamp`, `entry_time`, `split_label`, `outlet_name`, `MULTIWORD_OUTLETS`, `http_failure`, `USER_AGENT`, `check_feed`.
Two of those read names, and which one you want depends on who wrote the string. `split_label` reads a label *you* wrote and cuts it into outlet and section — "Guardian Tech" into the two — with `MULTIWORD_OUTLETS` as the list of names that contain a space themselves, so "Hacker News Best" does not come out as an outlet called Hacker. `outlet_name` reads a name a *server* gave for itself and cuts the tagline off it, because both a feed's `<title>` and a site's own title tend to arrive as the SEO version.
## Limits
- **One file, no state but `self.state`.** No store, no queue, no cross-source access. That's the deal that keeps a new source one file.
- **A failed target and an empty one are different answers.** Yielding nothing means there was nothing new; raising means something is wrong and you are owed a notice about it — `desk/notices.py` decides from the exception whether it reads as the world misbehaving or as a bug of ours. A source with several subscriptions gets both at once by yielding everything that worked and raising once at the end for what didn't, which is what stops one dead feed costing you the other nineteen. `rss.FeedsFailed` is that exception.
- **`fetch` is synchronous and blocking.** Runs in the worker's thread. Async, rate-limiting and *timeouts* are yours. A slow source holds the pull; a source that fetches without a deadline holds it for ever, because cancellation is cooperative and Kill cannot reach a thread parked in a socket read. The case to watch for is a library that does its own fetching: `feedparser.parse(url)` and `requests` with no `timeout=` both wait indefinitely by default, so `rss.py` fetches the bytes itself and hands the parser a document.
- **No sandbox, no versioning.** A plugin file is imported and trusted; a plugin that raises on import is logged and skipped, one that raises inside `fetch` fails that pull's source.
- **`name` must be unique and set** — no `name`, not discovered.
- **Discovery is flat**: `*.py` in a directory, not packages, not installed distributions.
- **A source can't create jobs.** It yields items; what happens next (body, vectors, tags) is the app's ingest job.