Files
jobboard/desk/sources/podcast.py
T
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

547 lines
26 KiB
Python

"""A news podcast, read as one item per story rather than one per episode.
The general half. `desk/tools/video/` below knows how to cut a video into
chapters; this knows the things that are true of a *show*: it has a feed, it
publishes on a cycle, its episodes are somewhere on YouTube, and its stories
belong in your queue next to the RSS copies of those same articles.
`plugins/lastweekin_ai.py` above knows one show.
That three-layer split is not decoration. A show that publishes a timestamped
outline works here with nothing show-specific involved — Dwarkesh and Latent
Space both do. What varies between shows is smaller than it looks, and is
declared as two methods rather than guessed at:
video_key(entry) where this show puts its video. The default reads
the feed body, which is the common case and is
wrong for several of the biggest shows.
episode_number(title) how this show writes an episode number.
Where each piece comes from, for one episode:
the feed title, date, the audio enclosure, the video link,
and the show notes as HTML — one fetch, everything
the notes chapter marks, sometimes with the article links attached,
which is what makes a segment a story rather than a topic
YouTube the transcript, with timings
video.cut() one Segment per chapter
this file one Item per Segment
**The subscription is just a feed.** `feeds:` is the same config key `rss`
uses, so a show imports into the `subscriptions` table like any other
subscription and gets add/remove/fetch/show, the Sources tab and the sidebar
for free — none of which had to know podcasts exist.
Config, for any show that is not one somebody has written a plugin for:
sources:
podcast:
feeds:
- url: https://api.substack.com/feed/podcast/69345.rss
label: Dwarkesh
episodes_per_feed: 2
Two things this deliberately does not do. It does not summarise: `text` starts
as the opening of the transcript and is replaced when a summary lands. What the
hosts made of a story is the one thing here that no feed already gives you, and
so the only part of this worth a model — which makes it a separate job, whose
failure costs you the abstracts and not the pull. See
newsdesk/jobs/summarise.py.
And it does not store the episode as an item, or as anything else: an episode
is a date and a video id, both of which live on every segment it produced, so
"every segment of episode 255, in order" is already a query and needs no
entity behind it.
**A show with no outline gets nothing stored, on purpose.** That claim above
used to hold only by luck: with no marks anywhere, `cut` hands back the whole
episode as one segment — right for a link you added by hand, wrong here — and
this stored it, so a two-hour recording arrived as a single eleven-thousand-word
item whose standfirst was the first two sentences of the intro and which then
queued itself for a summariser expecting one story. Now `_one_episode` stops
before the transcript is even fetched and says so in the log. Only shows that
publish a timestamped outline work here, which was always true and is now also
enforced.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from typing import Any
import feedparser
import httpx
from desk.sources.base import USER_AGENT, CheckResult, Item, Source, entry_time
from desk.sources.rss import check_feed
from desk.sources.spec import (
API_VERSION, RECORDING_READER, RSS, PluginSpec, Profile, RetryPolicy, Step,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools.http import session
from desk.tools.video import Segment, Video, cut, marks, media, opening, youtube
log = logging.getLogger(__name__)
#: "LWiAI Podcast #255 - Gemini 3.7, ..." -> 255. Every show numbers its
#: episodes somewhere; the title is where they all agree to put it.
_NUMBER = re.compile(r"(?:#|episode\s+|ep\.?\s*)(\d{1,5})\b", re.IGNORECASE)
def segment_profile(instructions: str = "", tags: str = "",
target_words: int = 70) -> Profile:
"""What a show that publishes a timestamped outline produces, declared.
A reusable typed preset, so that a plugin for one show is its feed, its
numbering and its editorial line — and not a second copy of this
machinery's description. `plugins/lastweekin_ai.py` is the model.
The two steps are what `_one_episode` below does, in the contract's words:
retrieve the recording's captions *once*, then cut them at the marks the
show published. Nothing executes them yet — the runner is a later step of
`docs/plugin-rework.md` — and they are written now because they are the
part a future reader has to be able to check this file against.
`instructions` is the one thing a show is expected to fill in: what its
own segments want said about them. What is general about summarising
speech — write what they made of it, keep the disagreement, do not quote a
mangled caption back — belongs to the shared task and is written once in
`newsdesk/jobs/summarise.py`, not per show.
"""
return Profile(
# One profile, because one thing is stored: the stories, not the
# episodes they came out of. An episode is a date and a video id, both
# of which are on every segment it produced.
kind="segment",
acquisition=[
Step(id="captions", tool="retrieve_captions",
inputs={"media": "inputs.media"},
outputs={"transcript": "inputs.transcript"},
# Auto-captions on a two-hour upload can take hours to
# appear, so absence is a wait and not a failure. Today that
# wait is "the next pull, for ever", because the episode is
# simply not marked done; a bound is what this declares.
retry=RetryPolicy(max_attempts=4, delay_seconds=3600,
backoff=2.0, on=["not_ready"])),
Step(id="segments", tool="cut_segments", needs=["captions"],
# The marks come from the show's own notes, which arrive in
# the feed entry — HTML, so each story keeps the article link
# beside its timestamp.
inputs={"transcript": "inputs.transcript",
"marks": "inputs.feed_text"},
outputs={"transcript": "inputs.transcript",
"links": "inputs.links"}),
],
tasks=Tasks(
summary=SummaryPolicy(
enabled=True, input_fields=["inputs.transcript"],
require_complete=True,
# No floor: a two-sentence housekeeping segment gets one flat
# sentence saying so, which is worth having and is what the
# shared prompt already asks for.
min_words=0, target_words=target_words,
instructions=instructions),
tagging=TaggingPolicy(
# The transcript, not the abstract: tagging a segment must not
# quietly wait on a summariser that failed. Naming it decides
# *which* text and not how much of it — the tagger reads the
# opening few hundred characters of whatever a profile names,
# because it is choosing from a closed list of labels and a
# two-hour transcript would be thirty times the tokens for
# the same answer. See `newsdesk/jobs/tagging.py`.
enabled=True, input_fields=["inputs.transcript"],
context_fields=["title", "inputs.links"], instructions=tags),
),
# The player first, then what the hosts made of it; the articles
# they were reading stay above it in every view. See the preset.
presentation=RECORDING_READER,
)
class PodcastSource(Source):
name = "podcast"
description = "A show's feed, cut into one item per story discussed."
subscription_field = "feeds"
default_outlet = "Podcast"
#: The generic source declares the same profile it hands its subclasses,
#: with no editorial line of its own: it is any show, and it cannot know
#: what one of them wants said. Its targets are editable because they are
#: whatever you put under `podcast:` in config.yaml — a show with a plugin
#: of its own declares a catalogue instead.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Podcast",
discovery=RSS(),
targets=TargetSpec(mode="editable", kind="rss_feed", defaults=[]),
profiles={"default": segment_profile()},
)
# -------------------------------------------------------- what a show is
#
# The two things that differ between shows, as methods rather than as
# config knobs — because neither has a value you could write down. Finding
# the video means knowing where that show puts it, and numbering an episode
# means knowing how that show writes a number. A subclass that knows one
# show answers both; everything else in this file is the same for all of
# them.
def video_key(self, entry: Any) -> str:
"""The YouTube id for this episode, or "" if there isn't one to find.
**The one place a show's publishing habits really bite.** The default
reads the feed body, which works when the show embeds or links its own
video there — Last Week in AI, Dwarkesh, Latent Space all do.
It is not general, and it fails in a way worth naming: The Vergecast,
Hard Fork, TWiT, All-In and Lex Fridman all publish full video to
YouTube and none of them mentions it in the feed, so all five come back
empty here. Fixing that means matching a feed entry against a channel's
uploads — which title, which date, whose channel — and that is a
per-show judgement, not a parser. This is where it goes when someone
wants one of those shows: override this, return the id, and every other
line in this file already works.
"""
try:
return youtube.video_id(_body(entry))
except ValueError:
return ""
def episode_number(self, title: str) -> int | None:
"""Which episode this is, from its title, or None if it doesn't say.
Every show numbers its episodes and the title is where they all agree
to put it — but not in one format. `#255` and `Episode 42` and `Ep. 7`
are the common family and are what the default reads; `TWiT 1100` and
`SN 1094` are not, and an interview show may not number at all.
Nothing downstream depends on the answer: it is stored on the item for
display and for "every segment of episode 255, in order". So the honest
default is a regex that catches most of them and returns None rather
than guessing when it doesn't, and a subclass overrides it when its
show writes numbers its own way.
"""
found = _NUMBER.search(title or "")
return int(found.group(1)) if found else None
# ------------------------------------------------------------------ fetch
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("episodes_per_feed", 2))
with session(None, timeout=40) as http:
for entry in self.options.get("feeds", []):
url = entry["url"] if isinstance(entry, dict) else entry
label = _label(entry, url)
try:
yield from self._one_feed(http, url, label, limit)
except youtube.Blocked:
# The one failure that is not this show's. YouTube refused
# the machine, so every other show here would be refused
# too, and each attempt is one more request from an address
# already taken for a bot. Out of `fetch` altogether: the
# runner keeps what was yielded and turns this into a
# notice, which is what the log line below never was.
raise
except Exception:
# One dead show must not cost the others their episodes.
log.exception("podcast feed %s failed", url)
def _one_feed(self, http: httpx.Client, url: str, label: str,
limit: int) -> Iterator[Item]:
response = http.get(url, headers={"User-Agent": USER_AGENT})
response.raise_for_status()
# The bytes, not `.text`. A feed declares its own encoding in its XML
# declaration and plenty of them are not UTF-8; handing the parser a
# string that httpx has already decoded by its own guess throws that
# declaration away, and what comes back is an episode title with a
# mojibake apostrophe in it. `check_feed` parses the bytes too.
feed = feedparser.parse(response.content)
if not feed.entries:
log.warning("no episodes in %s", url)
return
for entry in feed.entries[:limit]:
episode = self._episode(entry, label)
if episode is None:
continue
if self._already_done(episode.key):
log.debug("%s: already ingested", episode.title)
continue
try:
yield from self._one_episode(http, episode, entry, label)
except youtube.Blocked:
raise # not this episode's failure — see `fetch`
except Exception:
log.exception("episode %s failed; continuing the feed", episode.key)
def _one_episode(self, http: httpx.Client | None, episode: Video, entry: Any,
label: str) -> Iterator[Item]:
# The show's own notes first: same outline as the video description,
# but the links survived. Then the description, then the comments.
chapters = marks.best(episode.notes)
if not chapters:
described = youtube.details(episode.key, client=http)
chapters = marks.best(described["description"])
episode.duration = episode.duration or described["duration"]
if not chapters and self.options.get("read_comments", True):
chapters = marks.in_comments(youtube.comments(episode.key, client=http))
if not chapters:
# Nothing published an outline: not the notes, not the description,
# not a listener in the comments. `cut` would hand back the whole
# episode as one segment — correct of it, and what a hand-added
# link wants — but this source's contract is one item per *story*,
# and quietly substituting one item per episode is a different
# product wearing its name.
#
# It is also unusable downstream in a way that is easy to miss. The
# item's `text` would be `opening(transcript)`, the first two
# sentences of a two-hour recording, which is what the ranker and
# the tag filters would then judge it on; and eleven thousand words
# would go to a summariser whose prompt promises it a single story
# with the article the hosts were reading. Better to store nothing and say
# why than to store one wall and three wrong things about it.
#
# Marked done regardless. Reaching here means the episode was
# fetched and read successfully; there was simply nothing to cut
# along. A show that never publishes an outline would otherwise
# re-download a full transcript on every pull, for ever, to throw
# it away again — which is the waste `_already_done` exists to stop.
log.warning(
"%s: no timestamped outline in the notes, the description or "
"the comments, so there are no stories to cut it into — "
"storing nothing for this episode", episode.title)
self._mark_done(episode.key)
return
try:
cues = youtube.transcript(episode.key, client=http)
except youtube.Blocked:
# A `TranscriptUnavailable` too, and the one that must not be
# treated as one here. Waiting is the answer to captions that are
# still being made; to a refusal it is the same nothing on every
# pull, at INFO, with the episode never marked done and nobody
# told. Still not marked done, so the pull after the address is
# clean reads it — but it leaves as a failure, and reaches the
# screen. `details` above raises the same thing the same way.
raise
except youtube.TranscriptUnavailable as exc:
# Not done, so no marker is written and the next pull tries again.
# Auto-captions on a two-hour upload can take hours to appear, and
# an episode without them is worth waiting for rather than
# ingesting as twenty titles with nothing under them.
log.info("%s: %s", episode.title, exc)
return
segments = cut(episode, cues, chapters)
log.info("%s: %d segments", episode.title, len(segments))
for segment in segments:
yield _item(segment, entry, label,
number=self.episode_number(episode.title))
self._mark_done(episode.key)
def check(self) -> list[CheckResult]:
"""A show is a feed, so the feed check is the right one."""
return [check_feed(url, label) for url, label in self.targets()]
def _episode(self, entry: Any, show: str) -> Video | None:
"""A feed entry as a `Video`. None when there is no video to read.
A show whose video this cannot find is not a failure — see
`video_key`, which is the part a subclass replaces. Saying so once,
quietly, beats failing per episode.
"""
key = self.video_key(entry)
if not key:
# Deliberately not "audio-only shows need local transcription",
# which is what this used to say and which was wrong for the five
# biggest shows it fires on: they all post video, they just do not
# say so in the feed. Naming the actual condition is what stops the
# next person going and building a transcriber they did not need.
log.info("no video id in the feed entry for %r — the show may still "
"publish one; see PodcastSource.video_key",
(entry.get("title") or "")[:60])
return None
return Video(
ref=media.youtube(key),
title=(entry.get("title") or "").strip(),
url=entry.get("link") or "",
author=entry.get("author") or show,
published_at=entry_time(entry),
notes=_body(entry),
)
# ------------------------------------------------------------ bookkeeping
def _already_done(self, key: str) -> bool:
"""Whether this episode has been through here successfully.
In the `state` table rather than in a table of its own, because that is
genuinely all the state an episode has: it is done or it is not. An
episode is not a thing you read, so it needs no row of its own, and one
key in a table that already exists needs no migration.
The key is written only on success, so an episode whose captions
YouTube had not finished generating is left unmarked and comes round
again on the next pull.
Without this, a weekly pull would re-fetch and re-cut every transcript
in the window on every run and then discard the lot at the dedup check
— correct, and several wasted seconds per pull.
"""
return bool(self.state.get(self._done_key(key)))
def _mark_done(self, key: str) -> None:
self.state[self._done_key(key)] = datetime.now(UTC).isoformat()
def _done_key(self, key: str) -> str:
"""Namespaced by the plugin, not hard-coded to `podcast`.
Two shows can publish the same video id — a guest episode cross-posted,
a network re-run — and with one shared prefix the second would find the
first's marker and skip an episode it had never read. One subclass per
show means one namespace per show for free.
"""
return f"{self.name}:done:{key}"
# ------------------------------------------------------------------ internals
def _label(entry: Any, url: str) -> str:
"""What one configured show is called. Which outlet that stands for is the
subscription's business — see `Item.subscription`."""
if not isinstance(entry, dict):
return url
return entry.get("label") or url
def _item(segment: Segment, entry: Any, label: str,
number: int | None = None) -> Item:
"""One segment, as the row newsdesk stores.
Three of the field choices are not obvious:
`body` is the transcript, which is both where it belongs and how this
tells `fetch_bodies` not to go and scrape a YouTube page for one. It
is not the `excerpt`, because the excerpt is what the queue row renders,
what the ranker's document is built from and what search indexes, and
sixteen hundred words of speech would swamp all three.
`excerpt` is a placeholder until a summary is written over it. It is the
opening of the transcript rather than empty, because an item with no
excerpt is an empty row in the queue, and the ranker and the tagger both
read it. It used to be accompanied by a `summarised: False` flag, which
was this source telling the summariser that the row was waiting for it.
Nothing needs telling now: the profile declares that these items are
summarised, and an empty `summary` column is the whole of "not yet".
`published_at` is the episode's stamp plus the segment's offset, so twenty
items published in the same instant still sort in the order they were said.
`links` is the reason a segment is a story rather than a topic: the
articles the outline attached to this stretch, typed and roled, so the
reading pane and the summariser read them from the same field an outlet's
links live in rather than from this plugin's own corner of `raw`.
"""
episode = segment.video
return Item(
source_id=segment.external_id,
location=segment.url(),
title=segment.title,
excerpt=opening(segment.transcript),
body=segment.transcript,
author=episode.author,
published_at=episode.published_at + timedelta(seconds=segment.start),
subscription=label,
links=segment.reading(),
# Not `article`, which is what every one of these used to be stored as
# — twenty-three transcripts filed as prose. What the pipeline made,
# not what carried it: this arrived over RSS and left as segments.
kind="segment",
raw={
# The chapter this segment sits under. Per *item* — it changes
# down the episode — so it is here rather than on the
# subscription's `section`, which names a branch of an outlet and
# is the same for everything that subscription hands over.
"heading": segment.heading,
# Where it plays and what it linked to: the record every stored
# segment carries, defined once on `Segment`.
**segment.record(),
"show": label,
"episode": number,
"episode_title": episode.title,
"episode_url": episode.url,
"episode_published": episode.published_at.isoformat(),
"audio_url": _enclosure(entry),
},
)
def _body(entry: Any) -> str:
"""The richest form of the notes the feed offers."""
for content in entry.get("content") or []:
if value := content.get("value"):
return value
return entry.get("summary") or ""
def _enclosure(entry: Any) -> str:
return next(
(e.get("href", "") for e in entry.get("enclosures") or []
if (e.get("type") or "").startswith("audio")),
"",
)
# ----------------------------------------------------------------- dry run
def main(argv: list[str] | None = None) -> int:
"""`python -m desk.sources.podcast <feed url>` — what a pull would store.
Runs the plugin with no store behind it and prints the items instead of
writing them. This is the thing to run when a show changes the shape of
its notes: everything else in the pipeline is downstream of these fields.
"""
import argparse
import sys
parser = argparse.ArgumentParser(prog="python -m desk.sources.podcast")
parser.add_argument("feed", help="the show's RSS feed")
parser.add_argument("--episodes", type=int, default=1)
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)-7s %(message)s")
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
source = PodcastSource({
"feeds": [args.feed], "episodes_per_feed": args.episodes,
})
items = list(source.fetch())
if not items:
print("no items — run with -v to see why", file=sys.stderr)
return 1
print(f"{len(items)} items from {args.feed}\n")
heading = None
for item in items:
if item.raw["heading"] != heading:
heading = item.raw["heading"]
print(f" {heading or '(unfiled)'}")
stamp = marks.hhmmss(item.raw["start"])
print(f" {stamp:>8} {item.title[:80]}")
print(f" {item.location}")
for link in item.links:
print(f" → {link.url}")
print(f" {len(item.body.split()):,} words · {item.excerpt[:90]}")
print()
return 0
if __name__ == "__main__":
raise SystemExit(main())