Prune desk to what roledesk uses

Drops news's own sources (Bluesky, Mastodon, X, podcast) and the tools only
they use (video, posts, walls), with the PodcastSource and segment_profile
exports. scrape, article, checks, browser and framing stay: base.py imports
scrape's Getter at module level and scrape imports the other four, so
dropping them would mean editing the plugin contract. desk/README.md records
the source of truth, the prune and how to refresh. All 371 tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-23 12:33:58 +02:00
1 parent 0627b74f03
commit 24508dd6fc
15 files changed
+44 -2986

No files matched your search

+23
View File
@@ -0,0 +1,23 @@
# desk
The plugin contract (`desk.sources`), the model task (`desk.modeltask`), the settings sections (`desk.config`), the shared models and notices, and the tools roledesk uses: `agent`, `embedding`, `http`, `failures`, `marks`.
## Source of truth
`../news/desk`. This folder is a downstream copy. A fix lands in news first and is copied here, otherwise the next refresh reverts it. The commit that brought each copy in says which news commit it came from (`from news/desk at <sha>`).
## What was pruned, and why
news's own sources and the tools only they use: `sources/bluesky.py`, `mastodon.py`, `x.py`, `podcast.py`, and `tools/video/`, `posts.py`, `walls.py`. roledesk's one plugin is `plugins/ats.py`, which imports none of them. `sources/__init__.py` drops the `PodcastSource` and `segment_profile` exports with them, and its and `tools/__init__.py`'s docstrings say what is left.
Kept although nothing in roledesk calls them: `tools/scrape.py`, because `sources/base.py` imports its `Getter` at module level, and `tools/article.py`, `checks.py`, `browser.py` and `framing.py`, because `scrape.py` imports them. Pruning them would mean editing the contract itself. `tools/credentials.py` stays because `agent.py` imports it. `sources/sources.md` and `tools/tools.md` are news's and still describe the full set.
## Refreshing
```bash
git -C ../news archive --format=tar HEAD desk | tar -x -C .
rm -rf desk/sources/bluesky.py desk/sources/mastodon.py desk/sources/x.py desk/sources/podcast.py desk/tools/video desk/tools/posts.py desk/tools/walls.py
git diff desk/sources/__init__.py desk/tools/__init__.py
```
The archive overwrites the two `__init__.py` files. If news did not change them, `git checkout -- desk/sources/__init__.py desk/tools/__init__.py` restores the pruned versions; if it did, merge its change into them by hand. Then run `uv run pytest` and commit with `from news/desk at <sha>` in the body.
+9 -10
View File
@@ -17,19 +17,21 @@ source the same way.
shared tools that acquire its content, what the shared shared tools that acquire its content, what the shared
summariser and tagger should do with it, how it is laid out summariser and tagger should do with it, how it is laid out
registry.py finds the plugins on disk and builds the configured ones registry.py finds the plugins on disk and builds the configured ones
bluesky.py, mastodon.py, x.py, podcast.py the built-in ones
rss.py feed fetching, which a publisher's plugin subclasses rss.py feed fetching, which a publisher's plugin subclasses
Declaring a `spec` is optional in the contract and every plugin installed here Declaring a `spec` is optional in the contract and every plugin installed here
declares one: `spec_of` adapts a plugin that declares none, and such a plugin declares one: `spec_of` adapts a plugin that declares none, and such a plugin
runs exactly as it always has, but nothing about it is a claim its author made. runs exactly as it always has, but nothing about it is a claim its author made.
`PodcastSource` and `RssSource` are exported next to `Source` because that is `RssSource` is exported next to `Source` because that is how a publisher's
how a publisher's plugin is written: subclass the class that already does the plugin is written: subclass the class that already does the fetching, give it
fetching, give it a name and a `spec`, and add nothing else. a name and a `spec`, and add nothing else. `RssSource` is `shared` and is
`plugins/lastweekin_ai.py` and `plugins/guardian.py` are the two models. therefore not a plugin itself — there is no generic "Any RSS feed" source to
`RssSource` is `shared` and is therefore not a plugin itself — there is no paste a masthead into.
generic "Any RSS feed" source to paste a masthead into.
roledesk's copy has no built-in sources: news's Bluesky, Mastodon, X and
podcast plugins are pruned (see `desk/README.md`), and roledesk's one plugin
is `plugins/ats.py`.
""" """
from __future__ import annotations from __future__ import annotations
@@ -41,7 +43,6 @@ from desk.sources.base import (
MULTIWORD_OUTLETS, USER_AGENT, CheckResult, Getter, Source, Suggestion, MULTIWORD_OUTLETS, USER_AGENT, CheckResult, Getter, Source, Suggestion,
entry_time, http_failure, outlet_name, parse_timestamp, split_label, strip_html, entry_time, http_failure, outlet_name, parse_timestamp, split_label, strip_html,
) )
from desk.sources.podcast import PodcastSource, segment_profile
from desk.sources.registry import available from desk.sources.registry import available
from desk.sources.rss import RssSource, check_feed from desk.sources.rss import RssSource, check_feed
from desk.sources.spec import ( from desk.sources.spec import (
@@ -71,7 +72,6 @@ __all__ = [
"MarkRule", "MarkRule",
"OriginalView", "OriginalView",
"PluginSpec", "PluginSpec",
"PodcastSource",
"Presentation", "Presentation",
"Profile", "Profile",
"QueueLayout", "QueueLayout",
@@ -99,7 +99,6 @@ __all__ = [
"outlet_name", "outlet_name",
"parse_timestamp", "parse_timestamp",
"problems", "problems",
"segment_profile",
"spec_of", "spec_of",
"split_label", "split_label",
"strip_html", "strip_html",
-242
View File
@@ -1,242 +0,0 @@
"""Bluesky via the public AppView. No auth, no key, no bill.
Public app.bsky.* endpoints can be called directly against
https://public.api.bsky.app, which is what this plugin does. Endpoints used:
app.bsky.feed.getAuthorFeed - posts by one account
app.bsky.graph.getFollows - who an account follows (for discovery)
app.bsky.graph.getSuggestedFollowsByActor - accounts similar to a given one
"""
from __future__ import annotations
import logging
from collections import defaultdict
from collections.abc import Iterator
from typing import Any
import httpx
from desk.sources.base import (
USER_AGENT, CheckResult, Getter, Item, Source, Suggestion, http_failure,
parse_timestamp,
)
from desk.sources.spec import (
API_VERSION, POST_READER, PluginSpec, Profile, Step, SummaryPolicy,
TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools import posts
log = logging.getLogger(__name__)
API = "https://public.api.bsky.app/xrpc"
class BlueskySource(Source):
name = "bluesky"
description = "Posts from Bluesky accounts (free public API)."
default_outlet = "Bluesky"
#: A `bsky.app/profile/.../post/...` link from any outlet is read through
#: the same public API, not as the web app's script shell.
getters = (Getter("bluesky_api", posts.bluesky, posts.is_bluesky, claims=True),)
#: A network declares no accounts, because which ones you follow are
#: yours. `mode="editable"` is that sentence in the contract, and it is
#: what puts an Add box under this plugin and nothing but Fetch and Show
#: under a publisher's — see §8 of docs/plugin-rework.md.
#:
#: Both policies are declared as these items already behave rather than
#: as anyone would design them fresh, which is step 11's rule that
#: silence is not a decision. Posts are tagged today and have been for as
#: long as there has been a tagger; a post is already the length an
#: abstract would be, so summarising is off and says so.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Bluesky",
targets=TargetSpec(mode="editable", kind="account", defaults=[],
validator="bluesky_handle"),
profiles={"default": Profile(
kind="post",
# The fetching a link post has always had: `links_only` keeps the
# ones that point somewhere, and where they point is extracted
# into the field the reader shows under the post itself. A post
# linking to nothing carries its own permalink as `location` and
# as `raw.post_url`, and the body fetch skips it.
acquisition=[Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"})],
tasks=Tasks(
summary=SummaryPolicy(enabled=False),
tagging=TaggingPolicy(
enabled=True, input_fields=["inputs.feed_text"],
context_fields=["author"]),
),
presentation=POST_READER,
)},
)
def __init__(self, options: dict[str, Any] | None = None) -> None:
super().__init__(options)
self.client = httpx.Client(
timeout=20,
headers={"User-Agent": USER_AGENT},
)
def _get(self, method: str, **params) -> dict[str, Any]:
r = self.client.get(f"{API}/{method}", params=params)
r.raise_for_status()
return r.json()
# ------------------------------------------------------------- ingest
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("posts_per_account", 40))
skip_replies = self.options.get("skip_replies", True)
links_only = self.options.get("links_only", False)
for handle, label in self.targets():
try:
data = self._get(
"app.bsky.feed.getAuthorFeed",
actor=handle,
limit=min(limit, 100),
filter="posts_no_replies" if skip_replies else "posts_with_replies",
)
except httpx.HTTPError as exc:
log.warning("bluesky: %s failed (%s)", handle, exc)
continue
for entry in data.get("feed", []):
post = entry.get("post") or {}
record = post.get("record") or {}
author = post.get("author") or {}
text = record.get("text", "")
link = self._external_link(post)
if links_only and not link:
continue
rkey = (post.get("uri") or "").rsplit("/", 1)[-1]
yield Item(
source_id=post.get("uri", ""),
location=link
or f"https://bsky.app/profile/{author.get('handle', handle)}/post/{rkey}",
title="", # base class derives one from the text
excerpt=text,
author=author.get("displayName") or author.get("handle", handle),
published_at=parse_timestamp(record.get("createdAt", "")),
subscription=label,
kind="post",
raw={
"likes": post.get("likeCount", 0),
"reposts": post.get("repostCount", 0),
"post_url": f"https://bsky.app/profile/"
f"{author.get('handle', handle)}/post/{rkey}",
},
)
@staticmethod
def _external_link(post: dict[str, Any]) -> str:
"""Prefer the article a post links to over the post itself."""
embed = post.get("embed") or {}
external = embed.get("external") or {}
if external.get("uri"):
return external["uri"]
for facet in (post.get("record") or {}).get("facets", []) or []:
for feature in facet.get("features", []):
if feature.get("$type", "").endswith("#link") and feature.get("uri"):
return feature["uri"]
return ""
# --------------------------------------------------------------- check
def check(self) -> list[CheckResult]:
return [self._check_one(handle) for handle, _ in self.targets()]
def _check_one(self, handle: str) -> CheckResult:
try:
data = self._get("app.bsky.actor.getProfile", actor=handle)
except httpx.HTTPError as exc:
return CheckResult("bluesky", handle, False, http_failure(exc))
return CheckResult(
"bluesky", handle, True,
f"{data.get('postsCount', '?')} posts, "
f"{data.get('followersCount', '?')} followers",
)
# ----------------------------------------------------------- discovery
def discover(self) -> list[Suggestion]:
settings = self.options.get("discover") or {}
if not settings.get("enabled", True):
return []
seeds = [target for target, _ in self.targets()][: settings.get("max_seeds", 25)]
if not seeds:
return []
following = {s.lower() for s in seeds}
# Fractional, because signal 2 is weighted above signal 1 — so not a
# Counter, whose values typeshed (correctly) says are ints.
counts: defaultdict[str, float] = defaultdict(float)
meta: dict[str, dict[str, str]] = {}
# Signal 1: accounts that several of your accounts follow.
for seed in seeds:
for account in self._paged_follows(seed):
handle = account.get("handle", "").lower()
if not handle or handle in following:
continue
counts[handle] += 1
meta.setdefault(handle, {
"display_name": account.get("displayName", ""),
"description": (account.get("description") or "")[:160],
})
# Signal 2: Bluesky's own "similar to this account" endpoint.
for seed in seeds[:10]:
try:
data = self._get("app.bsky.graph.getSuggestedFollowsByActor", actor=seed)
except httpx.HTTPError:
continue
for account in data.get("suggestions", []):
handle = account.get("handle", "").lower()
if not handle or handle in following:
continue
counts[handle] += 1.5 # weight it slightly above a raw co-follow
meta.setdefault(handle, {
"display_name": account.get("displayName", ""),
"description": (account.get("description") or "")[:160],
})
threshold = settings.get("min_overlap", 3)
out = []
ranked = sorted(counts.items(), key=lambda kv: -kv[1])[:200]
for handle, score in ranked:
if score < threshold:
break
info = meta.get(handle, {})
out.append(Suggestion(
handle=handle,
display_name=info.get("display_name", ""),
reason=f"followed by {int(score)} of the accounts you follow",
score=float(score),
url=f"https://bsky.app/profile/{handle}",
))
return out
def _paged_follows(self, actor: str, cap: int = 400) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
cursor: str | None = None
while len(out) < cap:
params = {"actor": actor, "limit": 100}
if cursor:
params["cursor"] = cursor
try:
data = self._get("app.bsky.graph.getFollows", **params)
except httpx.HTTPError as exc:
log.warning("bluesky: follows for %s failed (%s)", actor, exc)
break
out.extend(data.get("follows", []))
cursor = data.get("cursor")
if not cursor:
break
return out
-196
View File
@@ -1,196 +0,0 @@
"""Mastodon via each instance's public REST API.
Uses /api/v1/accounts/lookup, /api/v1/accounts/{id}/statuses and
/api/v1/accounts/{id}/following. Public statuses are readable without a token
on Mastodon 2.7.0 and later, so no app registration is needed for public
accounts. Instances that run in whitelist mode will 401; those get skipped.
Accounts are written as user@instance.tld.
"""
from __future__ import annotations
import logging
import time
from collections import Counter
from collections.abc import Iterator
from typing import Any
import httpx
from desk.sources.base import (
USER_AGENT, CheckResult, Getter, Item, Source, Suggestion, http_failure,
parse_timestamp, strip_html,
)
from desk.sources.spec import (
API_VERSION, POST_READER, PluginSpec, Profile, Step, SummaryPolicy,
TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools import posts
log = logging.getLogger(__name__)
class MastodonSource(Source):
name = "mastodon"
description = "Public posts from Mastodon accounts (no token needed)."
default_outlet = "Mastodon"
#: A post link on any instance (`/@user/123`) is read through that
#: instance's API. The shape is not proof of Mastodon; a host that does not
#: answer the API in JSON gets the ordinary path.
getters = (Getter("mastodon_api", posts.mastodon, posts.is_mastodon, claims=True),)
#: The same declaration Bluesky makes, for the same reasons — read that
#: one. What differs is the handle: `user@instance.tld` names the server
#: as well as the person, so the check is its own.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Mastodon",
targets=TargetSpec(mode="editable", kind="account", defaults=[],
validator="mastodon_handle"),
profiles={"default": Profile(
kind="post",
acquisition=[Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"})],
tasks=Tasks(
summary=SummaryPolicy(enabled=False),
tagging=TaggingPolicy(
enabled=True, input_fields=["inputs.feed_text"],
context_fields=["author"]),
),
presentation=POST_READER,
)},
)
def __init__(self, options: dict[str, Any] | None = None) -> None:
super().__init__(options)
self.client = httpx.Client(
timeout=20,
follow_redirects=True,
headers={"User-Agent": USER_AGENT},
)
self._delay = float(self.options.get("request_delay", 0.3))
@staticmethod
def _split(acct: str) -> tuple[str, str]:
acct = acct.lstrip("@")
user, _, host = acct.partition("@")
return user, host
def _get(self, host: str, path: str, **params) -> Any:
time.sleep(self._delay)
r = self.client.get(f"https://{host}/api/v1/{path}", params=params)
r.raise_for_status()
return r.json()
def _lookup(self, acct: str) -> dict[str, Any] | None:
user, host = self._split(acct)
if not host:
log.warning("mastodon: %s is missing an instance (use user@instance)", acct)
return None
try:
return self._get(host, "accounts/lookup", acct=user)
except httpx.HTTPError as exc:
log.warning("mastodon: lookup %s failed (%s)", acct, exc)
return None
# ------------------------------------------------------------- ingest
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("posts_per_account", 40))
for acct, label in self.targets():
account = self._lookup(acct)
if not account:
continue
_, host = self._split(acct)
try:
statuses = self._get(
host,
f"accounts/{account['id']}/statuses",
limit=min(limit, 40),
exclude_replies=str(self.options.get("skip_replies", True)).lower(),
exclude_reblogs=str(self.options.get("skip_boosts", True)).lower(),
)
except httpx.HTTPError as exc:
log.warning("mastodon: statuses for %s failed (%s)", acct, exc)
continue
for status in statuses:
card = status.get("card") or {}
text = strip_html(status.get("content", ""))
yield Item(
source_id=status.get("uri") or status.get("url", ""),
location=card.get("url") or status.get("url", ""),
title=card.get("title", ""),
excerpt=text,
author=(status.get("account") or {}).get("display_name") or acct,
published_at=parse_timestamp(status.get("created_at", "")),
subscription=label,
kind="post",
raw={
"boosts": status.get("reblogs_count", 0),
"favourites": status.get("favourites_count", 0),
"post_url": status.get("url", ""),
},
)
# --------------------------------------------------------------- check
def check(self) -> list[CheckResult]:
return [self._check_one(acct) for acct, _ in self.targets()]
def _check_one(self, acct: str) -> CheckResult:
user, host = self._split(acct)
if not host:
return CheckResult("mastodon", acct, False,
"missing instance (use user@instance)")
try:
data = self._get(host, "accounts/lookup", acct=user)
except httpx.HTTPError as exc:
return CheckResult("mastodon", acct, False, http_failure(exc))
return CheckResult("mastodon", acct, True,
f"{data.get('statuses_count', '?')} posts")
# ----------------------------------------------------------- discovery
def discover(self) -> list[Suggestion]:
settings = self.options.get("discover") or {}
if not settings.get("enabled", True):
return []
seeds = [target for target, _ in self.targets()]
following = {s.lstrip("@").lower() for s in seeds}
counts: Counter[str] = Counter()
meta: dict[str, str] = {}
for acct in seeds:
account = self._lookup(acct)
if not account:
continue
_, host = self._split(acct)
try:
# Many accounts hide this list; a 401/403 just means skip.
follows = self._get(host, f"accounts/{account['id']}/following", limit=80)
except httpx.HTTPError:
continue
for other in follows:
handle = other.get("acct", "")
if "@" not in handle:
handle = f"{handle}@{host}"
if handle.lower() in following:
continue
counts[handle] += 1
meta.setdefault(handle, other.get("display_name", ""))
threshold = settings.get("min_overlap", 2)
return [
Suggestion(
handle=handle,
display_name=meta.get(handle, ""),
reason=f"followed by {n} of the accounts you follow",
score=float(n),
url=f"https://{handle.split('@')[-1]}/@{handle.split('@')[0]}",
)
for handle, n in counts.most_common(200)
if n >= threshold
]
-546
View File
@@ -1,546 +0,0 @@
"""A news podcast, read as one item per story rather than one per episode.
The general half. `desk/tools/video/` below knows how to cut a video into
chapters; this knows the things that are true of a *show*: it has a feed, it
publishes on a cycle, its episodes are somewhere on YouTube, and its stories
belong in your queue next to the RSS copies of those same articles.
`plugins/lastweekin_ai.py` above knows one show.
That three-layer split is not decoration. A show that publishes a timestamped
outline works here with nothing show-specific involved — Dwarkesh and Latent
Space both do. What varies between shows is smaller than it looks, and is
declared as two methods rather than guessed at:
video_key(entry) where this show puts its video. The default reads
the feed body, which is the common case and is
wrong for several of the biggest shows.
episode_number(title) how this show writes an episode number.
Where each piece comes from, for one episode:
the feed title, date, the audio enclosure, the video link,
and the show notes as HTML — one fetch, everything
the notes chapter marks, sometimes with the article links attached,
which is what makes a segment a story rather than a topic
YouTube the transcript, with timings
video.cut() one Segment per chapter
this file one Item per Segment
**The subscription is just a feed.** `feeds:` is the same config key `rss`
uses, so a show imports into the `subscriptions` table like any other
subscription and gets add/remove/fetch/show, the Sources tab and the sidebar
for free — none of which had to know podcasts exist.
Config, for any show that is not one somebody has written a plugin for:
sources:
podcast:
feeds:
- url: https://api.substack.com/feed/podcast/69345.rss
label: Dwarkesh
episodes_per_feed: 2
Two things this deliberately does not do. It does not summarise: `text` starts
as the opening of the transcript and is replaced when a summary lands. What the
hosts made of a story is the one thing here that no feed already gives you, and
so the only part of this worth a model — which makes it a separate job, whose
failure costs you the abstracts and not the pull. See
newsdesk/jobs/summarise.py.
And it does not store the episode as an item, or as anything else: an episode
is a date and a video id, both of which live on every segment it produced, so
"every segment of episode 255, in order" is already a query and needs no
entity behind it.
**A show with no outline gets nothing stored, on purpose.** That claim above
used to hold only by luck: with no marks anywhere, `cut` hands back the whole
episode as one segment — right for a link you added by hand, wrong here — and
this stored it, so a two-hour recording arrived as a single eleven-thousand-word
item whose standfirst was the first two sentences of the intro and which then
queued itself for a summariser expecting one story. Now `_one_episode` stops
before the transcript is even fetched and says so in the log. Only shows that
publish a timestamped outline work here, which was always true and is now also
enforced.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from typing import Any
import feedparser
import httpx
from desk.sources.base import USER_AGENT, CheckResult, Item, Source, entry_time
from desk.sources.rss import check_feed
from desk.sources.spec import (
API_VERSION, RECORDING_READER, RSS, PluginSpec, Profile, RetryPolicy, Step,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools.http import session
from desk.tools.video import Segment, Video, cut, marks, media, opening, youtube
log = logging.getLogger(__name__)
#: "LWiAI Podcast #255 - Gemini 3.7, ..." -> 255. Every show numbers its
#: episodes somewhere; the title is where they all agree to put it.
_NUMBER = re.compile(r"(?:#|episode\s+|ep\.?\s*)(\d{1,5})\b", re.IGNORECASE)
def segment_profile(instructions: str = "", tags: str = "",
target_words: int = 70) -> Profile:
"""What a show that publishes a timestamped outline produces, declared.
A reusable typed preset, so that a plugin for one show is its feed, its
numbering and its editorial line — and not a second copy of this
machinery's description. `plugins/lastweekin_ai.py` is the model.
The two steps are what `_one_episode` below does, in the contract's words:
retrieve the recording's captions *once*, then cut them at the marks the
show published. Nothing executes them yet — the runner is a later step of
`docs/plugin-rework.md` — and they are written now because they are the
part a future reader has to be able to check this file against.
`instructions` is the one thing a show is expected to fill in: what its
own segments want said about them. What is general about summarising
speech — write what they made of it, keep the disagreement, do not quote a
mangled caption back — belongs to the shared task and is written once in
`newsdesk/jobs/summarise.py`, not per show.
"""
return Profile(
# One profile, because one thing is stored: the stories, not the
# episodes they came out of. An episode is a date and a video id, both
# of which are on every segment it produced.
kind="segment",
acquisition=[
Step(id="captions", tool="retrieve_captions",
inputs={"media": "inputs.media"},
outputs={"transcript": "inputs.transcript"},
# Auto-captions on a two-hour upload can take hours to
# appear, so absence is a wait and not a failure. Today that
# wait is "the next pull, for ever", because the episode is
# simply not marked done; a bound is what this declares.
retry=RetryPolicy(max_attempts=4, delay_seconds=3600,
backoff=2.0, on=["not_ready"])),
Step(id="segments", tool="cut_segments", needs=["captions"],
# The marks come from the show's own notes, which arrive in
# the feed entry — HTML, so each story keeps the article link
# beside its timestamp.
inputs={"transcript": "inputs.transcript",
"marks": "inputs.feed_text"},
outputs={"transcript": "inputs.transcript",
"links": "inputs.links"}),
],
tasks=Tasks(
summary=SummaryPolicy(
enabled=True, input_fields=["inputs.transcript"],
require_complete=True,
# No floor: a two-sentence housekeeping segment gets one flat
# sentence saying so, which is worth having and is what the
# shared prompt already asks for.
min_words=0, target_words=target_words,
instructions=instructions),
tagging=TaggingPolicy(
# The transcript, not the abstract: tagging a segment must not
# quietly wait on a summariser that failed. Naming it decides
# *which* text and not how much of it — the tagger reads the
# opening few hundred characters of whatever a profile names,
# because it is choosing from a closed list of labels and a
# two-hour transcript would be thirty times the tokens for
# the same answer. See `newsdesk/jobs/tagging.py`.
enabled=True, input_fields=["inputs.transcript"],
context_fields=["title", "inputs.links"], instructions=tags),
),
# The player first, then what the hosts made of it; the articles
# they were reading stay above it in every view. See the preset.
presentation=RECORDING_READER,
)
class PodcastSource(Source):
name = "podcast"
description = "A show's feed, cut into one item per story discussed."
subscription_field = "feeds"
default_outlet = "Podcast"
#: The generic source declares the same profile it hands its subclasses,
#: with no editorial line of its own: it is any show, and it cannot know
#: what one of them wants said. Its targets are editable because they are
#: whatever you put under `podcast:` in config.yaml — a show with a plugin
#: of its own declares a catalogue instead.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Podcast",
discovery=RSS(),
targets=TargetSpec(mode="editable", kind="rss_feed", defaults=[]),
profiles={"default": segment_profile()},
)
# -------------------------------------------------------- what a show is
#
# The two things that differ between shows, as methods rather than as
# config knobs — because neither has a value you could write down. Finding
# the video means knowing where that show puts it, and numbering an episode
# means knowing how that show writes a number. A subclass that knows one
# show answers both; everything else in this file is the same for all of
# them.
def video_key(self, entry: Any) -> str:
"""The YouTube id for this episode, or "" if there isn't one to find.
**The one place a show's publishing habits really bite.** The default
reads the feed body, which works when the show embeds or links its own
video there — Last Week in AI, Dwarkesh, Latent Space all do.
It is not general, and it fails in a way worth naming: The Vergecast,
Hard Fork, TWiT, All-In and Lex Fridman all publish full video to
YouTube and none of them mentions it in the feed, so all five come back
empty here. Fixing that means matching a feed entry against a channel's
uploads — which title, which date, whose channel — and that is a
per-show judgement, not a parser. This is where it goes when someone
wants one of those shows: override this, return the id, and every other
line in this file already works.
"""
try:
return youtube.video_id(_body(entry))
except ValueError:
return ""
def episode_number(self, title: str) -> int | None:
"""Which episode this is, from its title, or None if it doesn't say.
Every show numbers its episodes and the title is where they all agree
to put it — but not in one format. `#255` and `Episode 42` and `Ep. 7`
are the common family and are what the default reads; `TWiT 1100` and
`SN 1094` are not, and an interview show may not number at all.
Nothing downstream depends on the answer: it is stored on the item for
display and for "every segment of episode 255, in order". So the honest
default is a regex that catches most of them and returns None rather
than guessing when it doesn't, and a subclass overrides it when its
show writes numbers its own way.
"""
found = _NUMBER.search(title or "")
return int(found.group(1)) if found else None
# ------------------------------------------------------------------ fetch
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("episodes_per_feed", 2))
with session(None, timeout=40) as http:
for entry in self.options.get("feeds", []):
url = entry["url"] if isinstance(entry, dict) else entry
label = _label(entry, url)
try:
yield from self._one_feed(http, url, label, limit)
except youtube.Blocked:
# The one failure that is not this show's. YouTube refused
# the machine, so every other show here would be refused
# too, and each attempt is one more request from an address
# already taken for a bot. Out of `fetch` altogether: the
# runner keeps what was yielded and turns this into a
# notice, which is what the log line below never was.
raise
except Exception:
# One dead show must not cost the others their episodes.
log.exception("podcast feed %s failed", url)
def _one_feed(self, http: httpx.Client, url: str, label: str,
limit: int) -> Iterator[Item]:
response = http.get(url, headers={"User-Agent": USER_AGENT})
response.raise_for_status()
# The bytes, not `.text`. A feed declares its own encoding in its XML
# declaration and plenty of them are not UTF-8; handing the parser a
# string that httpx has already decoded by its own guess throws that
# declaration away, and what comes back is an episode title with a
# mojibake apostrophe in it. `check_feed` parses the bytes too.
feed = feedparser.parse(response.content)
if not feed.entries:
log.warning("no episodes in %s", url)
return
for entry in feed.entries[:limit]:
episode = self._episode(entry, label)
if episode is None:
continue
if self._already_done(episode.key):
log.debug("%s: already ingested", episode.title)
continue
try:
yield from self._one_episode(http, episode, entry, label)
except youtube.Blocked:
raise # not this episode's failure — see `fetch`
except Exception:
log.exception("episode %s failed; continuing the feed", episode.key)
def _one_episode(self, http: httpx.Client | None, episode: Video, entry: Any,
label: str) -> Iterator[Item]:
# The show's own notes first: same outline as the video description,
# but the links survived. Then the description, then the comments.
chapters = marks.best(episode.notes)
if not chapters:
described = youtube.details(episode.key, client=http)
chapters = marks.best(described["description"])
episode.duration = episode.duration or described["duration"]
if not chapters and self.options.get("read_comments", True):
chapters = marks.in_comments(youtube.comments(episode.key, client=http))
if not chapters:
# Nothing published an outline: not the notes, not the description,
# not a listener in the comments. `cut` would hand back the whole
# episode as one segment — correct of it, and what a hand-added
# link wants — but this source's contract is one item per *story*,
# and quietly substituting one item per episode is a different
# product wearing its name.
#
# It is also unusable downstream in a way that is easy to miss. The
# item's `text` would be `opening(transcript)`, the first two
# sentences of a two-hour recording, which is what the ranker and
# the tag filters would then judge it on; and eleven thousand words
# would go to a summariser whose prompt promises it a single story
# with the article the hosts were reading. Better to store nothing and say
# why than to store one wall and three wrong things about it.
#
# Marked done regardless. Reaching here means the episode was
# fetched and read successfully; there was simply nothing to cut
# along. A show that never publishes an outline would otherwise
# re-download a full transcript on every pull, for ever, to throw
# it away again — which is the waste `_already_done` exists to stop.
log.warning(
"%s: no timestamped outline in the notes, the description or "
"the comments, so there are no stories to cut it into — "
"storing nothing for this episode", episode.title)
self._mark_done(episode.key)
return
try:
cues = youtube.transcript(episode.key, client=http)
except youtube.Blocked:
# A `TranscriptUnavailable` too, and the one that must not be
# treated as one here. Waiting is the answer to captions that are
# still being made; to a refusal it is the same nothing on every
# pull, at INFO, with the episode never marked done and nobody
# told. Still not marked done, so the pull after the address is
# clean reads it — but it leaves as a failure, and reaches the
# screen. `details` above raises the same thing the same way.
raise
except youtube.TranscriptUnavailable as exc:
# Not done, so no marker is written and the next pull tries again.
# Auto-captions on a two-hour upload can take hours to appear, and
# an episode without them is worth waiting for rather than
# ingesting as twenty titles with nothing under them.
log.info("%s: %s", episode.title, exc)
return
segments = cut(episode, cues, chapters)
log.info("%s: %d segments", episode.title, len(segments))
for segment in segments:
yield _item(segment, entry, label,
number=self.episode_number(episode.title))
self._mark_done(episode.key)
def check(self) -> list[CheckResult]:
"""A show is a feed, so the feed check is the right one."""
return [check_feed(url, label) for url, label in self.targets()]
def _episode(self, entry: Any, show: str) -> Video | None:
"""A feed entry as a `Video`. None when there is no video to read.
A show whose video this cannot find is not a failure — see
`video_key`, which is the part a subclass replaces. Saying so once,
quietly, beats failing per episode.
"""
key = self.video_key(entry)
if not key:
# Deliberately not "audio-only shows need local transcription",
# which is what this used to say and which was wrong for the five
# biggest shows it fires on: they all post video, they just do not
# say so in the feed. Naming the actual condition is what stops the
# next person going and building a transcriber they did not need.
log.info("no video id in the feed entry for %r — the show may still "
"publish one; see PodcastSource.video_key",
(entry.get("title") or "")[:60])
return None
return Video(
ref=media.youtube(key),
title=(entry.get("title") or "").strip(),
url=entry.get("link") or "",
author=entry.get("author") or show,
published_at=entry_time(entry),
notes=_body(entry),
)
# ------------------------------------------------------------ bookkeeping
def _already_done(self, key: str) -> bool:
"""Whether this episode has been through here successfully.
In the `state` table rather than in a table of its own, because that is
genuinely all the state an episode has: it is done or it is not. An
episode is not a thing you read, so it needs no row of its own, and one
key in a table that already exists needs no migration.
The key is written only on success, so an episode whose captions
YouTube had not finished generating is left unmarked and comes round
again on the next pull.
Without this, a weekly pull would re-fetch and re-cut every transcript
in the window on every run and then discard the lot at the dedup check
— correct, and several wasted seconds per pull.
"""
return bool(self.state.get(self._done_key(key)))
def _mark_done(self, key: str) -> None:
self.state[self._done_key(key)] = datetime.now(UTC).isoformat()
def _done_key(self, key: str) -> str:
"""Namespaced by the plugin, not hard-coded to `podcast`.
Two shows can publish the same video id — a guest episode cross-posted,
a network re-run — and with one shared prefix the second would find the
first's marker and skip an episode it had never read. One subclass per
show means one namespace per show for free.
"""
return f"{self.name}:done:{key}"
# ------------------------------------------------------------------ internals
def _label(entry: Any, url: str) -> str:
"""What one configured show is called. Which outlet that stands for is the
subscription's business — see `Item.subscription`."""
if not isinstance(entry, dict):
return url
return entry.get("label") or url
def _item(segment: Segment, entry: Any, label: str,
number: int | None = None) -> Item:
"""One segment, as the row newsdesk stores.
Three of the field choices are not obvious:
`body` is the transcript, which is both where it belongs and how this
tells `fetch_bodies` not to go and scrape a YouTube page for one. It
is not the `excerpt`, because the excerpt is what the queue row renders,
what the ranker's document is built from and what search indexes, and
sixteen hundred words of speech would swamp all three.
`excerpt` is a placeholder until a summary is written over it. It is the
opening of the transcript rather than empty, because an item with no
excerpt is an empty row in the queue, and the ranker and the tagger both
read it. It used to be accompanied by a `summarised: False` flag, which
was this source telling the summariser that the row was waiting for it.
Nothing needs telling now: the profile declares that these items are
summarised, and an empty `summary` column is the whole of "not yet".
`published_at` is the episode's stamp plus the segment's offset, so twenty
items published in the same instant still sort in the order they were said.
`links` is the reason a segment is a story rather than a topic: the
articles the outline attached to this stretch, typed and roled, so the
reading pane and the summariser read them from the same field an outlet's
links live in rather than from this plugin's own corner of `raw`.
"""
episode = segment.video
return Item(
source_id=segment.external_id,
location=segment.url(),
title=segment.title,
excerpt=opening(segment.transcript),
body=segment.transcript,
author=episode.author,
published_at=episode.published_at + timedelta(seconds=segment.start),
subscription=label,
links=segment.reading(),
# Not `article`, which is what every one of these used to be stored as
# — twenty-three transcripts filed as prose. What the pipeline made,
# not what carried it: this arrived over RSS and left as segments.
kind="segment",
raw={
# The chapter this segment sits under. Per *item* — it changes
# down the episode — so it is here rather than on the
# subscription's `section`, which names a branch of an outlet and
# is the same for everything that subscription hands over.
"heading": segment.heading,
# Where it plays and what it linked to: the record every stored
# segment carries, defined once on `Segment`.
**segment.record(),
"show": label,
"episode": number,
"episode_title": episode.title,
"episode_url": episode.url,
"episode_published": episode.published_at.isoformat(),
"audio_url": _enclosure(entry),
},
)
def _body(entry: Any) -> str:
"""The richest form of the notes the feed offers."""
for content in entry.get("content") or []:
if value := content.get("value"):
return value
return entry.get("summary") or ""
def _enclosure(entry: Any) -> str:
return next(
(e.get("href", "") for e in entry.get("enclosures") or []
if (e.get("type") or "").startswith("audio")),
"",
)
# ----------------------------------------------------------------- dry run
def main(argv: list[str] | None = None) -> int:
"""`python -m desk.sources.podcast <feed url>` — what a pull would store.
Runs the plugin with no store behind it and prints the items instead of
writing them. This is the thing to run when a show changes the shape of
its notes: everything else in the pipeline is downstream of these fields.
"""
import argparse
import sys
parser = argparse.ArgumentParser(prog="python -m desk.sources.podcast")
parser.add_argument("feed", help="the show's RSS feed")
parser.add_argument("--episodes", type=int, default=1)
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)-7s %(message)s")
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
source = PodcastSource({
"feeds": [args.feed], "episodes_per_feed": args.episodes,
})
items = list(source.fetch())
if not items:
print("no items — run with -v to see why", file=sys.stderr)
return 1
print(f"{len(items)} items from {args.feed}\n")
heading = None
for item in items:
if item.raw["heading"] != heading:
heading = item.raw["heading"]
print(f" {heading or '(unfiled)'}")
stamp = marks.hhmmss(item.raw["start"])
print(f" {stamp:>8} {item.title[:80]}")
print(f" {item.location}")
for link in item.links:
print(f" → {link.url}")
print(f" {len(item.body.split()):,} words · {item.excerpt[:90]}")
print()
return 0
if __name__ == "__main__":
raise SystemExit(main())
-203
View File
@@ -1,203 +0,0 @@
"""X (Twitter) via the official API v2. Off by default, because it costs money.
X moved to pay-per-usage billing. As published on the official pricing page,
post reads are $0.005 per resource and user reads are $0.010 per resource,
with pay-per-usage plans capped at 3 million post reads per billing cycle.
Reads of your *own* account's data ("Owned Reads") are $0.001 per resource.
So a config with 30 accounts x 20 posts x 4 polls a day is roughly
30*20*4*30 = 72,000 reads/month = about $360/month. Check the estimate this
plugin prints before you enable it.
This plugin therefore:
- is disabled unless you set `enabled: true`
- refuses to run without `budget_usd_per_month`
- tracks spend in the `state` table and stops when the budget is gone
- caches user IDs so it never pays for a lookup twice
Source for rates: https://docs.x.com/x-api/getting-started/pricing
"""
from __future__ import annotations
from urllib.parse import urlsplit
import logging
import os
from collections.abc import Iterator
from datetime import UTC, datetime
from typing import Any
import httpx
from desk.sources.base import USER_AGENT, Item, Source, Suggestion, parse_timestamp
from desk.sources.spec import (
API_VERSION, POST_READER, PluginSpec, Profile, Step, SummaryPolicy,
TaggingPolicy, TargetSpec, Tasks,
)
log = logging.getLogger(__name__)
API = "https://api.x.com/2"
POST_READ_USD = 0.005
USER_READ_USD = 0.010
def _is_internal(url: str) -> bool:
"""Match the host boundary so linux.com remains an external article."""
try:
host = (urlsplit(url).hostname or "").lower().rstrip(".")
except ValueError:
return False
return any(host == domain or host.endswith("." + domain)
for domain in ("x.com", "twitter.com"))
class XSource(Source):
name = "x"
default_outlet = "X"
description = "X/Twitter timelines (paid API — off by default)."
#: Declared like the other two networks. The budget, the token and the
#: refusal to run without either stay in the code above: they are what
#: this plugin *does*, and none of them is a thing the shared jobs or the
#: reader need to be told about.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="X",
targets=TargetSpec(mode="editable", kind="account", defaults=[],
validator="x_handle"),
profiles={"default": Profile(
kind="post",
acquisition=[Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"})],
tasks=Tasks(
summary=SummaryPolicy(enabled=False),
tagging=TaggingPolicy(
enabled=True, input_fields=["inputs.feed_text"],
context_fields=["author"]),
),
presentation=POST_READER,
)},
)
def __init__(self, options: dict[str, Any] | None = None) -> None:
super().__init__(options)
token_env = self.options.get("bearer_token_env", "X_BEARER_TOKEN")
self.token = os.environ.get(token_env, "")
self.client = httpx.Client(
timeout=25,
headers={
"Authorization": f"Bearer {self.token}",
"User-Agent": USER_AGENT,
},
)
@property
def enabled(self) -> bool:
return bool(self.options.get("enabled", False))
def estimate_monthly_usd(self, polls_per_day: int = 4) -> float:
n = len(self.options.get("accounts", []))
per_poll = n * int(self.options.get("posts_per_account", 10))
return per_poll * polls_per_day * 30 * POST_READ_USD
# ------------------------------------------------------------ budget
def _spent(self) -> float:
month = datetime.now(UTC).strftime("%Y-%m")
return float(self.state.get(f"x_spend_{month}", "0") or 0)
def _charge(self, usd: float) -> None:
month = datetime.now(UTC).strftime("%Y-%m")
self.state[f"x_spend_{month}"] = str(self._spent() + usd)
def _user_id(self, handle: str) -> str | None:
cache_key = f"x_userid_{handle.lower()}"
cached = self.state.get(cache_key)
if cached:
return cached
r = self.client.get(f"{API}/users/by/username/{handle}")
if r.status_code != 200:
log.warning("x: lookup %s returned %s", handle, r.status_code)
return None
self._charge(USER_READ_USD)
uid = (r.json().get("data") or {}).get("id")
if uid:
self.state[cache_key] = uid
return uid
# ------------------------------------------------------------- ingest
def fetch(self) -> Iterator[Item]:
if not self.token:
log.error("x: no bearer token in $%s",
self.options.get("bearer_token_env", "X_BEARER_TOKEN"))
return
budget = self.options.get("budget_usd_per_month")
if budget is None:
log.error("x: refusing to run without `budget_usd_per_month` in config")
return
per_account = int(self.options.get("posts_per_account", 10))
exclude = ",".join(self.options.get("exclude", ["retweets", "replies"]))
for handle, label in self.targets():
spent = self._spent()
if spent >= float(budget):
log.warning("x: monthly budget of $%.2f exhausted ($%.2f spent), stopping",
float(budget), spent)
return
uid = self._user_id(handle)
if not uid:
continue
r = self.client.get(
f"{API}/users/{uid}/tweets",
params={
"max_results": max(5, min(per_account, 100)),
"exclude": exclude,
"tweet.fields": "created_at,entities,public_metrics",
},
)
if r.status_code != 200:
log.warning("x: timeline %s returned %s: %s",
handle, r.status_code, r.text[:200])
continue
payload = r.json()
posts = payload.get("data", []) or []
self._charge(len(posts) * POST_READ_USD)
for post in posts:
urls = ((post.get("entities") or {}).get("urls") or [])
external = next(
(u.get("expanded_url") for u in urls
if u.get("expanded_url")
and not _is_internal(u["expanded_url"])),
"",
)
yield Item(
source_id=post["id"],
location=external or f"https://x.com/{handle}/status/{post['id']}",
title="",
excerpt=post.get("text", ""),
author=handle,
published_at=parse_timestamp(post.get("created_at", "")),
subscription=label,
kind="post",
raw={
"metrics": post.get("public_metrics", {}),
"post_url": f"https://x.com/{handle}/status/{post['id']}",
},
)
def discover(self) -> list[Suggestion]:
# The follow-graph endpoints are billed per user read, which makes
# graph-walking on X expensive enough that it is not worth doing
# automatically. Use Bluesky/Mastodon discovery instead.
return []
# `check` is left at the default too: a user lookup is a billed read, and
# a check job must never cost money.
+12 -6
View File
@@ -7,14 +7,20 @@ if it would make sense in a different application, it is a tool.
agent.py who answers when something needs a model: the Claude API agent.py who answers when something needs a model: the Claude API
with a key, the `claude` CLI on a subscription, or nobody with a key, the `claude` CLI on a subscription, or nobody
credentials.py the token `agent` reads from the app's database
embedding.py text in, vector out embedding.py text in, vector out
framing.py will this page load inside an iframe?
http.py one shared HTTP client per call chain http.py one shared HTTP client per call chain
failures.py what went wrong, in one word a retry policy can use
marks.py what kind of thing is this, if not an article? marks.py what kind of thing is this, if not an article?
video/ a recording, read as text with timings on it scrape.py an article out of a URL: fetch, check, convert.
walls.py was that the article, or the page shown instead of it? Kept because `desk.sources.base` imports its `Getter`;
`article.py`, `checks.py`, `browser.py` and `framing.py`
stay only because `scrape.py` imports them
Nothing is imported here on purpose. `video` pulls in httpx and the caption roledesk's copy prunes news's `video/`, `posts.py` and `walls.py`, which only
parsers, `embedding` a sentence-transformer, `agent` an SDK; each is paid for news's own sources use (see `desk/README.md`).
by the caller that wants it and by nobody else.
Nothing is imported here on purpose. `embedding` pulls in a
sentence-transformer, `agent` an SDK; each is paid for by the caller that
wants it and by nobody else.
""" """
-193
View File
@@ -1,193 +0,0 @@
"""A Mastodon or Bluesky post read through its network's public API.
Both web apps serve a script shell to a plain request, so `checks` names these
getters before `browser`: the API answers in a fraction of a second, needs no
Chromium and no login, and returns the post rather than a rendering of it.
Each getter builds a small HTML page from the answer, so `scrape` checks and
converts it exactly as it would a publisher's page.
from desk.tools import posts
posts.is_mastodon("https://hachyderm.io/@simontatham/117201594980991062")
page = posts.mastodon(url) # checks.Page
page = posts.bluesky("https://bsky.app/profile/bsky.app/post/3mv3shqdfuc2e")
The Mastodon and Bluesky plugins register them with `scrape` as the
`mastodon_api` and `bluesky_api` getters for their URLs; see `Source.getters`.
**A thread is read as far as its author continued it.** A post a link points
to is often the first of several by the same person; the replies by anyone
else are not part of it.
X is not here: its API bills every read and needs a token, and x.com shows a
browser a login wall.
GET https://{host}/api/v1/statuses/{id} Mastodon, any instance
GET https://{host}/api/v1/statuses/{id}/context
GET https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread
"""
from __future__ import annotations
import html
import re
from typing import Any
from urllib.parse import urlsplit
import httpx
from desk.tools.checks import Page
TIMEOUT = 20.0
#: Says what is asking, the way `desk.sources.base.USER_AGENT` does for the
#: plugins; a tool sits below `sources` and cannot import it.
USER_AGENT = "newsdesk (+https://github.com/; reading a linked post)"
BLUESKY_API = "https://public.api.bsky.app/xrpc"
#: `/@user/123` is what an instance links; `/users/user/statuses/123` is the
#: ActivityPub id some links carry instead.
_MASTODON_PATH = re.compile(r"^/(?:@[^/]+|users/[^/]+/statuses)/(\d+)/?$")
_BLUESKY_PATH = re.compile(r"^/profile/([^/]+)/post/([a-z0-9]+)/?$")
#: How deep a Bluesky thread is asked for. A self-thread longer than this is
#: cut, which is rare and costs the tail rather than the post.
THREAD_DEPTH = 20
def _get(url: str, params: dict[str, Any] | None = None) -> httpx.Response:
# Module-level, so a test replaces this one function.
return httpx.get(url, params=params, timeout=TIMEOUT,
headers={"User-Agent": USER_AGENT}, follow_redirects=True)
# ---------------------------------------------------------------- Mastodon
def _mastodon_id(url: str) -> tuple[str, str] | None:
parts = urlsplit(url)
found = _MASTODON_PATH.match(parts.path)
return (parts.netloc, found.group(1)) if parts.scheme == "https" and found else None
def is_mastodon(url: str) -> bool:
"""Whether `url` has a Mastodon post's shape. Any host: an instance is
recognised by the check that found its web app's shell, not by name."""
return _mastodon_id(url) is not None
def mastodon(url: str) -> Page | None:
"""The post at `url`, and its author's continuation, from the instance's API."""
host, status_id = _mastodon_id(url) or ("", "")
answer = _get(f"https://{host}/api/v1/statuses/{status_id}")
if not _json(answer):
# Not an instance: `/@user/123` is a Mastodon address's shape, and
# another site can have it. Nothing to say about the page, so the
# ordinary path runs.
return None
if answer.status_code != 200:
return Page(answer.status_code, "", url, dict(answer.headers))
status = answer.json()
author = status.get("account") or {}
thread = [status]
context = _get(f"https://{host}/api/v1/statuses/{status_id}/context")
if context.status_code == 200 and _json(context):
# Descendants come in thread order; the author's continuation is the
# chain of their replies, each to the one before.
for reply in context.json().get("descendants") or []:
if ((reply.get("account") or {}).get("id") == author.get("id")
and reply.get("in_reply_to_id") == thread[-1].get("id")):
thread.append(reply)
name = author.get("display_name") or author.get("acct") or host
card = status.get("card") or {}
return Page(200, _page(
title=f"{name}: {_first_words(_text(status.get('content', '')))}",
byline=f"{name} (@{author.get('acct', '')})",
published=status.get("created_at", ""),
paragraphs=[s.get("content", "") for s in thread],
link=(card.get("url", ""), card.get("title", "")),
alts=[m.get("description") or "" for s in thread
for m in s.get("media_attachments") or []],
escaped=False), url)
# ---------------------------------------------------------------- Bluesky
def _bluesky_uri(url: str) -> str:
parts = urlsplit(url)
found = _BLUESKY_PATH.match(parts.path)
if parts.netloc not in {"bsky.app", "www.bsky.app"} or not found:
return ""
return f"at://{found.group(1)}/app.bsky.feed.post/{found.group(2)}"
def is_bluesky(url: str) -> bool:
return bool(_bluesky_uri(url))
def bluesky(url: str) -> Page | None:
"""The post at `url`, and its author's continuation, from Bluesky's AppView."""
answer = _get(f"{BLUESKY_API}/app.bsky.feed.getPostThread",
{"uri": _bluesky_uri(url), "depth": THREAD_DEPTH, "parentHeight": 0})
if answer.status_code != 200:
# A post that is gone is a 400 `NotFound` here, not a 404.
gone = answer.status_code == 400 and "NotFound" in answer.text
return Page(404 if gone else answer.status_code, "", url, dict(answer.headers))
thread = answer.json().get("thread") or {}
post = thread.get("post") or {}
author = post.get("author") or {}
posts = [post]
node = thread
while True:
# The author's own reply, if there is one, is the next part of the thread.
nxt = next((r for r in node.get("replies") or []
if ((r.get("post") or {}).get("author") or {}).get("did") == author.get("did")),
None)
if nxt is None:
break
posts.append(nxt["post"])
node = nxt
record = post.get("record") or {}
external = ((post.get("embed") or {}).get("external") or {})
name = author.get("displayName") or author.get("handle", "")
return Page(200, _page(
title=f"{name}: {_first_words(record.get('text', ''))}",
byline=f"{name} (@{author.get('handle', '')})",
published=record.get("createdAt", ""),
paragraphs=[(p.get("record") or {}).get("text", "") for p in posts],
link=(external.get("uri", ""), external.get("title", "")),
alts=[image.get("alt") or "" for p in posts
for image in (p.get("embed") or {}).get("images") or []],
escaped=True), url)
# ---------------------------------------------------------------- the page
def _page(*, title: str, byline: str, published: str, paragraphs: list[str],
link: tuple[str, str], alts: list[str], escaped: bool) -> str:
"""An article page for the converter. Mastodon's content is HTML already;
Bluesky's is text, escaped here and split into paragraphs on blank lines."""
if escaped:
paragraphs = ["".join(f"<p>{html.escape(block).replace(chr(10), '<br>')}</p>"
for block in re.split(r"\n\s*\n", text) if block.strip())
for text in paragraphs]
body = "\n".join(paragraphs)
url, label = link
if url:
body += f'\n<p><a href="{html.escape(url)}">{html.escape(label or url)}</a></p>'
body += "".join(f"\n<p>[Image: {html.escape(alt)}]</p>" for alt in alts if alt.strip())
return (f"<!DOCTYPE html><html><head><title>{html.escape(title)}</title></head><body>"
f"<article><header><p>{html.escape(byline)}</p>"
f"<time datetime=\"{html.escape(published)}\">{html.escape(published)}</time></header>"
f"\n{body}\n</article></body></html>")
def _json(answer: httpx.Response) -> bool:
return "json" in answer.headers.get("content-type", "")
def _text(markup: str) -> str:
return html.unescape(re.sub(r"<[^>]+>", " ", markup))
def _first_words(text: str, count: int = 8) -> str:
words = text.split()
return " ".join(words[:count]) + ("…" if len(words) > count else "")
-103
View File
@@ -1,103 +0,0 @@
"""Video, read as text with timings on it.
The generic half of watching things so you don't have to. Give it a video and
it gives back a list of `Segment`s — each one a stretch of the recording with a
title, the words spoken in it, and a link that plays it from that second.
read() a video -> Video + Segment[] the whole thing, one call
youtube captions, metadata, comments where both inputs come from
marks an outline -> chapter marks description, notes, comments
media a MediaRef -> a playable link YouTube today, a file later
segment cues + marks -> Segment[] the cut, and transcript text
**This package knows nothing about newsdesk.** No store, no queue, no config,
no `Item`. That is what makes it reusable, and it is the seam the design is
built on: `desk/sources/podcast.py` sits on top and adds the things that
*are* specific to a show — its feed, its weekly cadence, the article links in
its notes, and turning segments into rows.
Two decisions worth knowing before reading further.
**A video is a `MediaRef`, not a URL.** Today every ref is a YouTube id; the
reading pane's link is computed from it on the way out. Point one at a file on
disk in config and every link to it changes, with nothing rewritten and no
item's identity touched. See `media.py`.
**Transcripts come from whoever can supply one.** `youtube-transcript-api`
first, and a built-in fetcher against YouTube's own player endpoint when the
library can't get the track. Transcribing audio locally (yt-dlp plus
faster-whisper) is the third backend and the one that makes local video work;
it is not written yet, and `read` is shaped so that adding it changes nothing
above.
Try it:
python -m desk.tools.video https://www.youtube.com/watch?v=GUYEofBebDc
"""
from __future__ import annotations
import logging
import httpx
from desk.tools.video import marks, media, youtube
from desk.tools.http import session
from desk.tools.video.marks import Chapter, Link
from desk.tools.video.media import MediaRef
from desk.tools.video.segment import Segment, Video, cut, opening, paragraphs
from desk.tools.video.youtube import Blocked, Cue, TranscriptUnavailable
log = logging.getLogger(__name__)
__all__ = [
"Blocked", "Chapter", "Cue", "Link", "MediaRef", "Segment",
"TranscriptUnavailable", "Video", "cut", "describe", "marks", "media",
"opening", "paragraphs", "read", "youtube",
]
def read(url: str, *, comments: bool = True,
client: httpx.Client | None = None) -> tuple[Video, list[Segment]]:
"""One video, cut into its chapters. The whole package in one call.
`comments` is the only cost worth thinking about: it is an extra request to
the watch page and is only consulted when the description has no outline in
it, so for a show that publishes chapter marks it never happens.
"""
with session(client, timeout=40) as http:
video = describe(url, client=http)
try:
cues = youtube.transcript(video.key, client=http)
except TranscriptUnavailable as exc:
# Not fatal on its own. The outline still gives every chapter, its
# links and its timecode; only the quoted words are missing, and
# auto-captions usually appear within a few hours of an upload.
log.warning("%s", exc)
cues = []
chapters = marks.best(video.notes)
if not chapters and comments:
chapters = marks.in_comments(youtube.comments(video.key, client=http))
return video, cut(video, cues, chapters)
def describe(url: str, *, client: httpx.Client | None = None) -> Video:
"""What a video says about itself: title, author, length, description.
YouTube only, for now. A local file would come back with its stem as the
title and no notes, which is why `Video.notes` is allowed to be empty and
why `read` has a comments fallback at all.
"""
key = youtube.video_id(url)
with session(client) as http:
meta = youtube.details(key, client=http)
return Video(
ref=media.youtube(key),
title=meta["title"],
url=media.youtube(key).at(0),
author=meta["author"],
duration=meta["duration"],
notes=meta["description"],
)
-120
View File
@@ -1,120 +0,0 @@
"""Cut a video up and print what came out.
python -m desk.tools.video <youtube url>
python -m desk.tools.video <youtube url> --json > segments.json
Videos only. A podcast episode — where the chapter marks and the article links
live on the show's own page rather than on YouTube — goes through the layer
above:
python -m desk.sources.podcast <episode url | feed url>
"""
from __future__ import annotations
import argparse
import dataclasses
import json
import logging
import sys
import desk.tools.video as video_pkg
from desk.tools.video import marks
from desk.tools.video.segment import Segment, Video
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m desk.tools.video",
description="Cut a video into one item per chapter mark.",
)
parser.add_argument("url", help="a YouTube link or bare video id")
parser.add_argument("--json", action="store_true",
help="dump everything, transcripts included, as JSON")
parser.add_argument("--chars", type=int, default=160,
help="how much transcript to preview per segment (default 160)")
parser.add_argument("--no-comments", action="store_true",
help="don't fall back to chapter marks typed in the comments")
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(
level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)-7s %(message)s",
)
# Windows still opens a console in the system codepage, and a video title
# with a "ñ" in it — or the arrows this prints — is an unhandled
# UnicodeEncodeError half way through the report rather than a bad glyph.
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
try:
video, segments = video_pkg.read(args.url, comments=not args.no_comments)
except Exception as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
if args.json:
json.dump(dump(video, segments), sys.stdout, indent=2, default=str)
print()
else:
report(video, segments, args.chars)
return 0
def report(video: Video, segments: list[Segment], chars: int) -> None:
"""The human-readable view. Also used by the podcast module."""
words = sum(len(s.transcript.split()) for s in segments)
length = video.duration or (segments[-1].end if segments else 0)
print(f"{video.title}")
print(f" {video.author or '—'} {video.ref.kind}:{video.ref.key}"
f" {marks.hhmmss(length)}")
print(f" {len(segments)} segments, {words:,} words of transcript")
print()
heading = None
for segment in segments:
if segment.heading != heading:
heading = segment.heading
print(f" {heading or '(unfiled)'}")
preview = segment.transcript[:chars]
if len(segment.transcript) > chars:
preview += "…"
print(f" {segment.timecode:>8} {segment.title[:78]}")
print(f" {marks.hhmmss(segment.duration)} long · {segment.url()}")
for link in segment.links:
print(f" → {link.url}")
print(f" {preview or '(no transcript under this mark)'}")
print()
def dump(video: Video, segments: list[Segment]) -> dict[str, object]:
"""The shape a stored item is built from, plus the fields a dry run wants
to eyeball. The stored half is `Segment.record()` and is defined there;
desk/sources/podcast.py turns the rest into a row."""
return {
"video": {
"ref": dataclasses.asdict(video.ref),
"title": video.title,
"url": video.url,
"author": video.author,
"published_at": video.published_at,
"duration": video.duration,
},
"segments": [
{
**s.record(),
"external_id": s.external_id,
"title": s.title,
"heading": s.heading,
"timecode": s.timecode,
"url": s.url(),
"transcript": s.transcript,
}
for s in segments
],
}
if __name__ == "__main__":
raise SystemExit(main())
-265
View File
@@ -1,265 +0,0 @@
"""Chapter marks: a video's own table of contents, wherever it was written.
The important discovery about this kind of video is that **the segmentation
already exists and somebody else did it**. Last Week in AI ships every episode
with a timestamped outline — a rubric, then one line per story, each with the
time it starts and a link to the article being discussed:
Tools & Apps
(00:03:32) Google announces Gemini 3.7 Flash ... - Ars Technica
(00:13:11) SpaceXAI Releases Grok 4.6 ... - MarkTechPost
That is a better set of boundaries than any topic-segmentation model would
produce, for nothing, with the source links attached. So this module parses it
rather than inferring it. An LLM is still wanted downstream — to summarise what
was *said* in each stretch — but not to work out where one stretch ends.
**Three places to find that outline**, in descending order of usefulness, all
parsed by the same function because they are the same convention written in
different markup:
1. **The show's own page or feed body** — HTML, so the links are real
`<a href>` and each chapter arrives with the article it is about. Only a
show with a website has this, which is why the podcast source hands it in
rather than this module going to look.
2. **The video description** — plain text. Usually the same outline with the
links flattened away.
3. **The comments** — see `in_comments`. When a show publishes no marks at
all, a listener has often typed some out; YouTube itself reads those to
draw its chapter markers.
`best` takes them in that order and returns the first that is real. Nothing
here is specific to one show beyond the shape "timestamp, then a title,
optionally under a heading", which is the near-universal convention.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Iterable
from dataclasses import dataclass, field
from html.parser import HTMLParser
log = logging.getLogger(__name__)
#: `(00:03:32)`, `00:03:32`, `3:32` — at the start of the line, which is what
#: makes it a chapter mark rather than a time mentioned in a sentence.
TIMESTAMP = re.compile(
# The character class ends with hyphen, en dash and em dash on purpose:
# a show separates the time from the title with whichever of the three its
# editor's keyboard produced, and means the same thing by all of them.
r"^\(?\s*(\d{1,2}):([0-5]\d)(?::([0-5]\d))?\s*\)?[\s\-–—:]*" # noqa: RUF001
)
_URL = re.compile(r"https?://[^\s<>\"')\]]+")
#: Tags that end a line of the outline. Everything else is inline and stays
#: on the line it was found in, which is what keeps `<span>(00:03:32) </span>`
#: attached to the `<a>` that follows it.
_BLOCKS = frozenset({
"li", "p", "br", "div", "tr", "ul", "ol", "h1", "h2", "h3", "h4", "blockquote",
})
@dataclass(frozen=True, slots=True)
class Link:
"""One article the hosts were talking about."""
url: str
title: str = ""
@dataclass(slots=True)
class Chapter:
"""One entry in the outline: where a story starts and what it is about."""
start: int # seconds into the episode
title: str
heading: str = "" # the rubric it sits under
links: list[Link] = field(default_factory=list)
@property
def url(self) -> str:
"""The article being discussed, if the notes linked one."""
return self.links[0].url if self.links else ""
def parse(body: str) -> list[Chapter]:
"""Every chapter mark in a set of show notes, in the order they play.
Takes HTML or plain text and works out which it was given. Returns `[]`
for notes with no outline in them at all, which is a real answer — plenty
of shows don't write one, and the caller falls back to treating the episode
as a single undivided item rather than guessing at boundaries.
"""
lines = _html_lines(body) if "<" in body and ">" in body else _text_lines(body)
chapters: list[Chapter] = []
heading = ""
for text, links in lines:
match = TIMESTAMP.match(text)
if match is None:
# A heading is only a heading once the outline has started;
# before that every line is preamble — the blurb, the sponsors,
# the "Timestamps:" label itself.
if chapters and _is_heading(text, links):
heading = text.rstrip(":")
continue
title = text[match.end():].strip(" -–—|") # noqa: RUF001 — as above
# Prefer the anchor's own text: a line that links two follow-ups reads
# "First story | The Verge + Second story", and the first link's text
# is the story, where the whole line is the story plus its footnote.
if links and links[0].title:
title = links[0].title
chapters.append(Chapter(
start=_seconds(match), title=title, heading=heading, links=links,
))
if not chapters:
log.debug("no chapter marks found in %d chars of notes", len(body))
return chapters
def best(*candidates: str, minimum: int = 3) -> list[Chapter]:
"""The first candidate that yields a real outline.
Chapter marks come from up to three places and they are not equally good,
so the order they are passed in is the order of preference — for a podcast
that is: the show's own notes as HTML (links intact), then the YouTube
description (same outline, links flattened away), then the comments.
`minimum` is what stops a stray "at 3:20 he says" in a blurb becoming a
one-chapter outline that suppresses the good source behind it. Three marks
is the point where a line of text stops being a coincidence.
"""
for text in candidates:
found = parse(text)
if len(found) >= minimum:
return found
return []
def in_comments(comments: Iterable[str], minimum: int = 3) -> list[Chapter]:
"""The outline somebody typed into the comments, if anybody did.
Read as the fallback for shows that publish no chapter marks of their own,
where it is common for a listener to write one out — YouTube itself treats
such a comment as good enough to draw chapter markers from.
Each comment is parsed on its own rather than the lot being concatenated.
Joined together, an outline comment and thirty unrelated replies produce
one bogus outline interleaved with chatter; separately, the outline is the
comment with the most marks in it and the chatter scores zero.
"""
outlines = [parse(comment) for comment in comments]
outlines = [found for found in outlines if len(found) >= minimum]
if not outlines:
return []
richest = max(outlines, key=len)
log.info("took %d chapter marks from a comment", len(richest))
return richest
def hhmmss(seconds: float) -> str:
"""3932 -> '1:05:32'. What a timecode looks like to a reader."""
total = int(seconds)
hours, rest = divmod(total, 3600)
minutes, secs = divmod(rest, 60)
return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}"
# ------------------------------------------------------------------ internals
def _seconds(match: re.Match[str]) -> int:
a, b, c = match.group(1), match.group(2), match.group(3)
if c is None: # mm:ss
return int(a) * 60 + int(b)
return int(a) * 3600 + int(b) * 60 + int(c)
def _is_heading(text: str, links: list[Link]) -> bool:
"""A rubric like 'Policy & Safety', not a sentence and not a story.
Deliberately strict. Anything that gets through becomes the heading for
every chapter after it, so a false positive mislabels the rest of the
episode — where a false negative just leaves a few items unfiled.
"""
return (
not links
and len(text) <= 60
and len(text.split()) <= 6
and not text.endswith((".", "!", "?"))
)
def _text_lines(body: str) -> list[tuple[str, list[Link]]]:
"""Plain-text notes — a YouTube description. Bare URLs become links."""
lines = []
for raw in body.splitlines():
line = raw.strip()
if not line:
continue
links = [Link(url=u) for u in _URL.findall(line)]
lines.append((_URL.sub("", line).strip(), links))
return lines
def _html_lines(body: str) -> list[tuple[str, list[Link]]]:
parser = _Outline()
parser.feed(body)
parser.close()
return parser.lines
class _Outline(HTMLParser):
"""HTML to (line, links) pairs, without pulling in a parser library.
A full HTML tree would be nicer and is not worth a dependency for this:
the only structure that matters is where the line breaks fall and which
anchors landed on which line. `convert_charrefs` (on by default) means
entities arrive already decoded, so `Tools &amp; Apps` is `Tools & Apps`
by the time it reaches `handle_data`.
"""
def __init__(self) -> None:
super().__init__()
self.lines: list[tuple[str, list[Link]]] = []
self._text: list[str] = []
self._links: list[Link] = []
self._href = ""
self._anchor: list[str] = []
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if tag in _BLOCKS:
self._flush()
elif tag == "a":
self._href = dict(attrs).get("href") or ""
self._anchor = []
def handle_endtag(self, tag: str) -> None:
if tag in _BLOCKS:
self._flush()
elif tag == "a":
if self._href:
self._links.append(
Link(url=self._href, title=_squash("".join(self._anchor)))
)
self._href, self._anchor = "", []
def handle_data(self, data: str) -> None:
self._text.append(data)
if self._href:
self._anchor.append(data)
def _flush(self) -> None:
text = _squash("".join(self._text))
if text or self._links:
self.lines.append((text, self._links))
self._text, self._links = [], []
def close(self) -> None:
super().close()
self._flush()
def _squash(text: str) -> str:
return re.sub(r"\s+", " ", text or "").strip()
-113
View File
@@ -1,113 +0,0 @@
"""Where a video actually lives, and how to link into the middle of it.
Every segment newsdesk stores has to answer one question: *press play at
1:08:27 — where?* Today the answer is a YouTube URL. It will not always be.
The same episode may sit on disk, ripped or recorded; a show may take its
uploads down; a video may only ever have existed as a local file.
So nothing in this codebase stores a YouTube URL as a fact. It stores a
`MediaRef` — a kind and a key — and asks this module for a link when it needs
one. Swapping YouTube for a copy on disk is then a line of config rather than
an UPDATE across every item that mentioned it.
video:
local_media:
GUYEofBebDc: D:/media/lwiai/255.mp4
`resolve` applies that map. Everything downstream keeps working because it was
never holding a URL in the first place.
The one thing this deliberately does not do is *find* local copies. Matching
"is this file the same episode as that video" by filename or duration is a
guess, and a wrong guess points a citation at the wrong recording. The mapping
is written by a person, or it does not exist.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path, PurePath
#: A local file, playable in the reading pane's <video> element. Anything else
#: on disk (mkv, wav) still resolves; the pane just may not play it.
PLAYABLE = frozenset({".mp4", ".m4v", ".webm", ".ogg", ".mp3", ".m4a", ".wav"})
@dataclass(frozen=True, slots=True)
class MediaRef:
"""A video, identified by where it is rather than by a URL.
`kind` is what the key means: `youtube` is an 11-character video id,
`spotify` is an episode id, `file` is a path on this machine, `url` is
anything else somebody handed us. `key` is stable — it is what
`Segment.external_id` is built from — so re-pointing a video at a local
copy must *not* change it. That is why `resolve` returns a separate
playback ref instead of rewriting this one.
"""
kind: str
key: str
@property
def playable_locally(self) -> bool:
return self.kind == "file" and PurePath(self.key).suffix.lower() in PLAYABLE
def at(self, seconds: float = 0) -> str:
"""A link that opens this video at that moment.
The media fragment (`#t=`) is the same idea in every case: it is what
YouTube's `&t=` and Spotify's `?t=` do, what a `<video>` element
honours natively, and what most desktop players understand from a
`file://` URL.
"""
offset = max(0, int(seconds))
if self.kind == "youtube":
return f"https://www.youtube.com/watch?v={self.key}&t={offset}s"
if self.kind == "spotify":
return f"https://open.spotify.com/episode/{self.key}?t={offset}"
if self.kind == "file":
return Path(self.key).absolute().as_uri() + (f"#t={offset}" if offset else "")
return self.key + (f"#t={offset}" if offset else "")
def embed(self, seconds: float = 0) -> str:
"""The player a reading pane can frame at that moment, or "".
Only the hosts that publish a player built for framing have one: a
YouTube watch page and a Spotify episode page both refuse to be
framed, and each has an `/embed/` form that does not. A local file
needs none — the pane plays it in a `<video>` element — and an
arbitrary URL is not known to allow it.
"""
offset = max(0, int(seconds))
if self.kind == "youtube":
return f"https://www.youtube.com/embed/{self.key}?start={offset}"
if self.kind == "spotify":
return f"https://open.spotify.com/embed/episode/{self.key}?t={offset}"
return ""
def youtube(video_id: str) -> MediaRef:
return MediaRef(kind="youtube", key=video_id)
def local(path: str | Path) -> MediaRef:
return MediaRef(kind="file", key=str(path))
def resolve(ref: MediaRef, overrides: dict[str, str] | None = None) -> MediaRef:
"""The ref to *play*, after applying the local-media map.
Keyed on the original key, so a mapping written once keeps working however
the item was ingested. Returns the ref unchanged when there is no override
or the file has since gone — a stale mapping should fall back to the
original rather than hand out a link to a file that isn't there.
"""
if not overrides:
return ref
replacement = overrides.get(ref.key)
if not replacement:
return ref
path = Path(replacement).expanduser()
if not path.exists():
return ref
return local(path)
-261
View File
@@ -1,261 +0,0 @@
"""A video, and the pieces it breaks into.
This is the generic half and it knows nothing about podcasts, feeds or
newsdesk's tables. A `Video` is something with a transcript and a duration; a
`Segment` is a span of one, with a title, a start, an end and the words spoken
inside it. Cutting one into the other needs two inputs and no model:
Video --+-- transcript -> timed cues -----+
| +-> cut() -> Segment[]
+-- chapter marks (marks.best) ----+
What supplies those inputs is somebody else's problem, and deliberately so.
YouTube supplies both today. A show's own web page supplies better marks, with
the article links attached — that is what `desk/sources/podcast.py` does,
by passing its own marks in. A local recording plus a Whisper transcript would
supply both without touching this file.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass, field
from datetime import UTC, datetime
from typing import Any
from desk.models import ItemLink
from desk.tools.video import marks as marks_mod
from desk.tools.video import youtube
from desk.tools.video.marks import Chapter, Link
from desk.tools.video.media import MediaRef
log = logging.getLogger(__name__)
@dataclass(slots=True)
class Video:
"""One video, before anything has been done to it."""
ref: MediaRef
title: str = ""
#: The page it was published on, when that is a different thing from the
#: video itself — a Substack post, a show's episode page. Empty for a video
#: that is only a video.
url: str = ""
author: str = ""
published_at: datetime = field(default_factory=lambda: datetime.now(UTC))
duration: float = 0.0
#: Whatever text came with it and might carry chapter marks: a YouTube
#: description, a feed body, show notes as HTML.
notes: str = ""
@property
def key(self) -> str:
"""What identifies this video for the life of the archive.
The media *ref's* key, not a URL, and never the resolved playback
location — pointing an item at a local copy must not change its
identity or every segment of that video re-ingests as a new item.
"""
return self.ref.key
@dataclass(slots=True)
class Segment:
"""One span of a video: the unit that becomes a newsdesk item."""
video: Video
index: int
start: int
end: int
title: str
heading: str = "" # the rubric it was filed under
links: list[Link] = field(default_factory=list)
transcript: str = "" # verbatim, this span only
@property
def external_id(self) -> str:
"""Stable across re-ingests of the same video.
Keyed on the video and the start time rather than the position in the
list, so a show that inserts a chapter it forgot doesn't renumber —
and therefore doesn't duplicate — everything after it. Re-cutting
against corrected timestamps *does* produce new ids, which is the
honest outcome: they are different spans of the recording.
"""
return f"{self.video.key}#t={self.start}"
def url(self, overrides: dict[str, str] | None = None) -> str:
"""A link that plays this segment, wherever the video currently lives.
Resolved on the way out rather than stored, which is the whole point of
`media.MediaRef`: mapping a YouTube id to a file on disk changes every
link in the archive and rewrites nothing.
"""
from desk.tools.video import media
return media.resolve(self.video.ref, overrides).at(self.start)
@property
def duration(self) -> int:
return max(0, self.end - self.start)
@property
def timecode(self) -> str:
return marks_mod.hhmmss(self.start)
def record(self) -> dict[str, Any]:
"""The segment as a JSON-able dict: what newsdesk stores in `items.raw`.
One definition of the layout, so the two writers (a podcast segment,
a video added by hand) and the readers (the reading pane, the
summariser's query) agree on the keys. `media` is the reference and
never a URL — see `media.py`; `duration` is denormalised so a queue row
can show a length chip without arithmetic across two JSON extractions.
The articles are not in here: they are a typed collection of their own
now — see `reading`. `raw` is what no shared field has a name for, and
a link the whole reader understands is not that.
"""
return {
"media": {"kind": self.video.ref.kind, "key": self.video.ref.key},
"start": self.start,
"end": self.end,
"duration": self.duration,
"index": self.index,
}
def reading(self) -> list[ItemLink]:
"""What the hosts were reading, as links the whole app understands.
The outline's links with a role on them, which is the difference
between "a URL this show happened to print" and "the article this
stretch is about": the reader lists them under the player, the
summariser names them in the prompt, and neither has to know a podcast
produced them.
A line of show notes is whatever an editor typed, so a relative href
or a `mailto:` is an ordinary outcome rather than a fault — dropped
with a note, because one malformed link in the outline must not cost
the segment it sits on.
"""
out = []
for link in self.links:
try:
out.append(ItemLink(url=link.url, title=link.title, role="reading"))
except ValueError:
log.debug("%s: dropping unusable outline link %r",
self.external_id, link.url[:120])
return out
def cut(video: Video, cues: list[youtube.Cue],
chapters: list[Chapter] | None = None) -> list[Segment]:
"""Cut a video into one segment per chapter mark.
A chapter runs until the next one starts; the last one runs to the end of
the recording. That is the only boundary rule, and it is exact, because the
marks were written by the people who recorded it.
With no chapter marks at all the video comes back as a single segment
holding the whole transcript. That is deliberately not a failure: an
unsegmented video is still worth having, and inventing boundaries where
nobody published any is a job for a model and a later stage, not for a
slicing function.
"""
chapters = chapters if chapters is not None else marks_mod.parse(video.notes)
end_of_video = int(video.duration or (cues[-1].end if cues else 0))
if not chapters:
return [Segment(
video=video, index=0, start=0, end=end_of_video,
title=video.title, transcript=youtube.text_between(cues, 0),
)]
ordered = sorted(chapters, key=lambda c: c.start)
out = []
for index, chapter in enumerate(ordered):
following = (ordered[index + 1].start if index + 1 < len(ordered)
else end_of_video)
# A show that mistypes a timestamp can put one mark after the next.
# Clamping keeps the slice empty rather than reversed, and leaves the
# neighbouring segments intact.
end = max(chapter.start, following)
out.append(Segment(
video=video,
index=index,
start=chapter.start,
end=end,
title=chapter.title,
heading=chapter.heading,
links=chapter.links,
transcript=youtube.text_between(cues, chapter.start, end or None),
))
empty = sum(1 for s in out if not s.transcript)
if empty:
log.warning("%s: %d of %d segments have no transcript under them",
video.key, empty, len(out))
return out
# ------------------------------------------------------ transcript as text
_SENTENCE_END = re.compile(r"(?<=[.!?])\s+")
def opening(transcript: str, sentences: int = 2, cap: int = 400) -> str:
"""The first sentence or two, as a stand-in for a summary that isn't
written yet. An empty `text` is an empty row in the queue."""
if not transcript:
return ""
parts = _SENTENCE_END.split(transcript.strip())
return " ".join(parts[:sentences])[:cap]
def paragraphs(text: str, target: int = 700) -> str:
"""Break a captions stream into paragraphs a reading pane can render.
Captions arrive as one unbroken run, and the reader splits a body on blank
lines — so without this a forty-minute talk is one paragraph. Sentences
are grouped until roughly `target` characters, then a break: arbitrary,
but readable, and the alternative is a wall.
Auto-generated captions often have no punctuation at all, so one
"sentence" can be the whole talk. Those are broken on a word boundary
instead — worse than a real sentence end, better than the wall.
"""
text = " ".join((text or "").split())
if not text:
return ""
out: list[str] = []
current: list[str] = []
length = 0
for sentence in _SENTENCE_END.split(text):
for piece in _words(sentence, target * 2):
current.append(piece)
length += len(piece) + 1
if length >= target:
out.append(" ".join(current))
current, length = [], 0
if current:
out.append(" ".join(current))
return "\n\n".join(out)
def _words(run: str, limit: int) -> list[str]:
"""Split an over-long run at spaces, keeping every piece under `limit`."""
if len(run) <= limit:
return [run]
pieces: list[str] = []
current: list[str] = []
length = 0
for word in run.split(" "):
if current and length + len(word) + 1 > limit:
pieces.append(" ".join(current))
current, length = [], 0
current.append(word)
length += len(word) + 1
if current:
pieces.append(" ".join(current))
return pieces
-600
View File
@@ -1,600 +0,0 @@
"""Captions, with their timings, for one YouTube video.
YouTube has always published a machine-readable transcript for anything with
captions — including the auto-generated ones, which is most podcasts. It is the
cheapest transcript in existence: somebody else already paid for the ASR.
Getting at it is the fiddly part, and it is fiddly for a reason worth writing
down. The caption URLs embedded in a `watch` page are signed for a *browser*
session, and fetching one from anywhere else now returns `200 OK` with an empty
body — no error, no redirect, nothing to catch. So the obvious implementation
(scrape the watch page, follow `baseUrl`) fails silently, which is the worst
way for an ingest step to fail.
What works is asking the same endpoint the mobile app asks. `/youtubei/v1/player`
with a mobile client context hands back a player response whose caption URLs are
not tied to a browser session, and those fetch fine. That is the whole trick,
and it is why this module talks to an undocumented endpoint instead of scraping
a page: the scrape is the fragile one.
Two clients are tried in order, because they fail differently — `IOS` returns
JSON3 when asked and `ANDROID` returns SRV3 whatever you ask for, so both
formats are parsed. If Google breaks both, the fallbacks are `yt-dlp`, which
tracks this arms race full-time, and transcribing the audio yourself. Neither
is in here — `__init__.py` says where they would go. Until one of them is
written, a break in both clients raises `TranscriptUnavailable` rather than
storing an empty transcript: loudly, which is the whole point of the paragraph
above.
**A refusal is not an absence.** YouTube gates by address: from a VPN exit or
a datacentre range a client is answered `LOGIN_REQUIRED` ("Sign in to confirm
you're not a bot") for every video, captioned or not. Measured on 2026-09-21,
one machine, one video, seven clients, each asked bare and again with the
`visitorData` a watch page hands out:
VPN on `IOS`, `ANDROID`, `WEB`, `MWEB`, `TVHTML5` and `ANDROID_VR` all
get the bot check, and `WEB_EMBEDDED_PLAYER` an error. No tracks
from any of them, and `visitorData` changes none of it.
`youtube-transcript-api` raises `RequestBlocked`.
VPN off `IOS` and `ANDROID` return tracks that fetch. The rest are no
use even here: `WEB`, `MWEB` and `WEB_EMBEDDED_PLAYER` are
unplayable without a PO token, and `TVHTML5` and `ANDROID_VR`
are answered with the bot check from a clean address too.
So there is no anonymous client that gets past the gate and none to add: the
fix is on the network. That case raises `Blocked`, a `TranscriptUnavailable` a
caller can tell apart, because "wait for the captions" is the wrong answer to
it and would be given for ever.
"""
from __future__ import annotations
import html
import json
import logging
import re
import xml.etree.ElementTree as ET
from collections.abc import Iterator
from dataclasses import dataclass
from typing import Any
from urllib.parse import parse_qs, urlsplit
import httpx
from desk.tools.http import session
log = logging.getLogger(__name__)
INNERTUBE = "https://www.youtube.com/youtubei/v1/player?prettyPrint=false"
#: Only for the watch page, which is read for comments. A caption fetch
#: deliberately identifies as a mobile app instead — see CLIENTS.
BROWSER_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
#: Tried in order. Each is (client context, the User-Agent that goes with it) —
#: they have to match, or the response comes back playable but captionless.
CLIENTS = (
(
{"clientName": "IOS", "clientVersion": "20.10.4",
"deviceModel": "iPhone16,2", "hl": "en"},
"com.google.ios.youtube/20.10.4 (iPhone16,2; U; CPU iOS 18_3_2 like Mac OS X)",
),
(
{"clientName": "ANDROID", "clientVersion": "20.10.38",
"androidSdkVersion": 30, "hl": "en"},
"com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip",
),
)
_VIDEO_ID = re.compile(r"^[\w-]{11}$")
#: Every shape a YouTube link comes in, including the one Substack embeds
#: (`<div class="youtube-wrap" id="youtube2-VIDEOID">`).
_IN_TEXT = re.compile(
r"(?:youtu\.be/|youtube\.com/(?:watch\?(?:[^\"'\s]*&)?v=|embed/|live/|shorts/)"
r"|youtube2-)([\w-]{11})"
)
class TranscriptUnavailable(Exception):
"""No captions to be had — yet, or at all.
Raised rather than returning `[]` because the two cases an ingest has to
tell apart both look like an empty list: a video with no captions, and a
video whose captions are still being generated. The message says which,
and the caller decides whether to retry on the next pull.
"""
class Blocked(TranscriptUnavailable):
"""YouTube refused this machine, whatever the video has.
A subclass, so a caller that only wants "no captions, carry on" — a link
added by hand, a repair — is unchanged. A caller that waits for captions
has to catch this first: a refusal is not lifted by waiting, it answers
every video the same way, and retrying it quietly is how a show stores
nothing for a fortnight with nobody told. See `_refused`.
"""
@dataclass(frozen=True, slots=True)
class Cue:
"""One caption line: when it starts, how long it lasts, what was said."""
start: float
duration: float
text: str
@property
def end(self) -> float:
return self.start + self.duration
def video_id(value: str) -> str:
"""The 11-character id, from a bare id, any YouTube URL, or a blob of HTML.
Deliberately tolerant about its input: the id arrives from three different
places in this pipeline — typed by hand, in a feed entry's link, and buried
in the HTML of a Substack post — and each of them would otherwise need its
own extraction.
"""
value = (value or "").strip()
if _VIDEO_ID.match(value):
return value
if match := _IN_TEXT.search(value):
return match.group(1)
# A URL shape the pattern above doesn't cover, but with ?v= in the query.
found = parse_qs(urlsplit(value).query).get("v")
if found and _VIDEO_ID.match(found[0]):
return found[0]
raise ValueError(f"no YouTube video id in {value[:120]!r}")
def transcript(video: str, *, language: str = "en",
client: httpx.Client | None = None) -> list[Cue]:
"""Every caption cue for one video, in order.
`language` is matched as a prefix, so "en" accepts "en-US" — which matters,
because a channel that uploads a real transcript files it under a regional
code while the auto-generated one is plain "en".
Two backends, tried in that order. `youtube-transcript-api` first — it is
a dependency, and it is somebody's full-time job to keep up with YouTube's
changes. The built-in fetcher below when the library can't get the track,
so one broken path doesn't cost the transcript.
"""
cues = _via_library(video, language)
if cues is not None:
return cues
return _via_innertube(video, language, client)
#: Seconds one `requests` call inside `youtube-transcript-api` gets. Roughly
#: what `desk/tools/http.py` gives an httpx call, for the same reason.
LIBRARY_TIMEOUT = 30
def _timed_client() -> Any:
"""A `requests` session with a deadline on it, for a library that owns its HTTP.
`youtube-transcript-api` does its own fetching, on `requests`, and a
`requests` call with no `timeout=` waits on a socket read for ever. What
that costs is not a slow transcript. The work runs in a worker thread;
cancellation is cooperative, so a thread parked in a read is one the Kill
button cannot reach, and a restart re-queues the job to park on the same
video. The library takes a session of ours through `http_client`, which is
its own supported way of being told this; a `requests.Session` has no
timeout of its own to set, so the default goes on one method down.
**This is why there is no `socket.setdefaulttimeout` anywhere.** A
process-wide default is the tempting one-liner for library code that hides
its HTTP, and it is the wrong trade here: it would be inherited by every
socket the process opens, uvicorn's listener and each websocket included,
to buy a deadline for call sites that can each be told one directly. There
were two. The other was feedparser, and it is handled the same way — by
fetching where a timeout can be set; see `desk/sources/rss.py`.
Typed as `Any` because `requests` is only imported inside the guard below.
It is a `requests.Session`.
"""
import requests
class Timed(requests.Session):
def request(self, *args: Any, **kwargs: Any) -> Any:
kwargs.setdefault("timeout", LIBRARY_TIMEOUT)
return super().request(*args, **kwargs)
return Timed()
def _via_library(video: str, language: str) -> list[Cue] | None:
"""`youtube-transcript-api`. None means "not this way".
Every failure is None rather than an exception, including a video with no
captions at all: the fallback below raises the error that gets reported, so
there is one message for "no transcript" instead of two that depend on
which packages happen to be present.
"""
try:
from youtube_transcript_api import YouTubeTranscriptApi
client = _timed_client()
except ImportError:
return None
try:
tracks = YouTubeTranscriptApi(http_client=client).list(video)
wanted = [language, f"{language}-US", f"{language}-GB"]
try:
track = tracks.find_manually_created_transcript(wanted)
except Exception:
track = tracks.find_generated_transcript(wanted)
fetched = track.fetch()
except Exception as exc:
log.debug("youtube-transcript-api could not fetch %s: %s", video, exc)
return None
cues = [
Cue(start=c.start, duration=c.duration, text=clean)
for c in fetched
if (clean := _clean(c.text))
]
log.info("%s: %d cues via youtube-transcript-api", video, len(cues))
return cues or None
def _via_innertube(video: str, language: str,
client: httpx.Client | None) -> list[Cue]:
with session(client) as http:
player = _player(video, http)
track = _pick(_tracks(player), language)
if track is None:
raise TranscriptUnavailable(
f"{video}: no {language} captions. "
"Auto-captions can take a few hours to appear on a long upload."
)
# `&fmt=json3` is a request, not a promise: the ANDROID client answers
# it with SRV3 anyway. Both are parsed, so it doesn't matter which
# arrives — see `_cues`.
response = http.get(track["baseUrl"] + "&fmt=json3",
headers={"User-Agent": player.get("_caption_agent", BROWSER_UA)})
if _throttled(response):
raise Blocked(_refusal(video, [f"captions: HTTP {response.status_code}"]))
response.raise_for_status()
body = response.text
if not body.strip():
raise TranscriptUnavailable(
f"{video}: caption track fetched empty. YouTube does this when "
"the request doesn't look like the client that asked for it."
)
cues = _cues(body)
log.info("%s: %d cues, %.0f minutes", video, len(cues),
(cues[-1].end / 60) if cues else 0)
return cues
def details(video: str, *, client: httpx.Client | None = None) -> dict[str, Any]:
"""Title, author, length and description for a video.
The description is here because for a show that publishes no newsletter
it is the only place the chapter marks live — `marks.parse` reads it as
plain text. Everything else is the metadata an episode needs when YouTube
is the only source there is.
"""
with session(client) as http:
player = _player(video, http, require_captions=False)
video_details = player.get("videoDetails", {})
return {
"title": video_details.get("title", ""),
"author": video_details.get("author", ""),
"duration": float(video_details.get("lengthSeconds") or 0),
"description": video_details.get("shortDescription", ""),
}
def comments(video: str, *, limit: int = 100,
client: httpx.Client | None = None) -> list[str]:
"""Top-level comments, newest-and-most-liked first, as plain text.
Here for one reason: when a show does not publish an outline, somebody in
the comments usually has. A viewer typing out "0:00 intro / 3:32 Gemini /
13:11 Grok" is doing the same work the show notes would have done, and
YouTube itself treats such a comment as authoritative enough to draw
chapter markers from it. `marks.parse` reads these exactly as it reads a
description — see `marks.best`.
Returns `[]` rather than raising when comments are off, or when YouTube
changes the shape again. A missing fallback is not worth failing an ingest
that already has a transcript.
"""
try:
with session(client) as http:
page = http.get(f"https://www.youtube.com/watch?v={video}",
headers={"User-Agent": BROWSER_UA,
"Accept-Language": "en-US,en"}).text
context = _json_after(page, '"INNERTUBE_CONTEXT":')
initial = _json_after(page, "ytInitialData = ")
if not context or not initial:
return []
out: list[str] = []
# Several continuations live in the page (the related rail, the
# description); only one of them is the comment feed, and which
# index that is moves around. Trying them in order and keeping the
# first that yields comments is cheaper than pinning a path
# through a structure Google reshuffles.
for token in _continuations(initial)[:4]:
response = http.post(
"https://www.youtube.com/youtubei/v1/next?prettyPrint=false",
json={"context": context, "continuation": token},
headers={"Content-Type": "application/json",
"User-Agent": BROWSER_UA},
)
out = _comment_texts(response.json())
if out:
break
except Exception as exc:
log.debug("could not read comments for %s: %s", video, exc)
return []
log.info("%s: %d comments read", video, len(out))
return out[:limit]
def text_between(cues: list[Cue], start: float, end: float | None = None) -> str:
"""The words spoken in a span, as one paragraph.
A cue counts as inside the span when it *starts* inside it. Overlap would
be the other defensible rule, but it double-counts the sentence that
straddles a boundary — and these boundaries come from chapter marks a human
typed, so a sentence of drift at the seam is already there and doesn't need
doubling.
"""
return " ".join(
cue.text for cue in cues
if cue.start >= start and (end is None or cue.start < end)
).strip()
# ------------------------------------------------- internals: the watch page
def _json_after(page: str, marker: str) -> dict[str, Any] | None:
"""The JSON object that follows `marker` in a watch page.
Brace-matched rather than matched with a regex. The two blobs this reads —
`ytInitialData` and `INNERTUBE_CONTEXT` — are hundreds of kilobytes of
nested JSON containing every bracket character inside string values, and a
non-greedy regex stops at the first `}` inside a caption or a title. This
walks the string properly, which is about fifteen lines and always right.
"""
start = page.find(marker)
if start < 0:
return None
start = page.find("{", start + len(marker))
if start < 0:
return None
depth, in_string, escaped = 0, False, False
for index in range(start, len(page)):
char = page[index]
if in_string:
if escaped:
escaped = False
elif char == "\\":
escaped = True
elif char == '"':
in_string = False
continue
if char == '"':
in_string = True
elif char == "{":
depth += 1
elif char == "}":
depth -= 1
if depth == 0:
try:
return json.loads(page[start:index + 1])
except ValueError:
return None
return None
def _walk(node: Any, key: str) -> Iterator[Any]:
"""Every value stored under `key`, anywhere in a nested structure."""
if isinstance(node, dict):
if key in node:
yield node[key]
for value in node.values():
yield from _walk(value, key)
elif isinstance(node, list):
for value in node:
yield from _walk(value, key)
def _continuations(initial: dict[str, Any]) -> list[str]:
return [
token for renderer in _walk(initial, "continuationItemRenderer")
if (token := (renderer.get("continuationEndpoint", {})
.get("continuationCommand", {}) or {}).get("token"))
]
def _comment_texts(response: dict[str, Any]) -> list[str]:
"""Comment bodies out of a `next` response.
Comments arrive as `commentEntityPayload` entities in a flat batch beside
the renderers that position them, so reading them needs no understanding of
the layout at all — which is the part that keeps changing.
"""
out = []
for payload in _walk(response, "commentEntityPayload"):
text = (payload.get("properties", {}).get("content", {}) or {}).get("content", "")
if text:
out.append(text)
return out
# ------------------------------------------------------------------ internals
def _player(video: str, http: httpx.Client,
*, require_captions: bool = True) -> dict[str, Any]:
"""The player response, from whichever client answers with captions.
`require_captions` is what separates the two callers. A transcript wants
the first client that actually offers caption tracks; metadata wants the
first that answers at all, and would otherwise fail on a video that plays
fine and simply has no captions yet.
Every client's answer is kept for the error, not only the last one's: the
two fail differently, and which of them said what is the first thing
anyone asks when this breaks.
"""
said: list[str] = []
refused = False
for context, agent in CLIENTS:
name = context["clientName"]
payload = {"context": {"client": context}, "videoId": video,
"contentCheckOk": True, "racyCheckOk": True}
try:
response = http.post(
INNERTUBE, json=payload,
headers={"Content-Type": "application/json", "User-Agent": agent},
)
if _throttled(response):
refused = True
said.append(f"{name}: HTTP {response.status_code}")
continue
data = response.json()
except (httpx.HTTPError, ValueError) as exc:
said.append(f"{name}: {exc}")
continue
playability = data.get("playabilityStatus", {})
status = playability.get("status", "?")
if _tracks(data) or (not require_captions and status == "OK"):
# Caption URLs are bound to the client that obtained the player.
data["_caption_agent"] = agent
return data
refused = refused or _refused(playability)
reason = playability.get("reason") or ""
said.append(f"{name}: {status}" + (f" ({reason})" if reason else "")
+ ", no caption tracks")
log.debug("innertube %s", said[-1])
if refused:
raise Blocked(_refusal(video, said))
raise TranscriptUnavailable(
f"{video}: {'; '.join(said) or 'no client returned captions'}")
def _refused(playability: dict[str, Any]) -> bool:
"""Whether a player response is YouTube declining to serve this machine.
The status alone, not the reason beside it. The bot check reads "Sign in
to confirm you're not a bot" today, and a match on that sentence would go
back to reporting a refusal as captions not being ready on the day it is
reworded — silently, which is the failure this exists to end. The status
is the stable half, and it is sufficient: nothing asked anonymously is
ever let past `LOGIN_REQUIRED` by being asked again later.
That also takes in a private or age-gated video, where the refusal is
about the video rather than the address. It is still not a wait, and the
reason is quoted in the error for whoever reads it.
"""
return playability.get("status") == "LOGIN_REQUIRED"
def _throttled(response: httpx.Response) -> bool:
"""The same refusal, made in HTTP: a 403, a 429, or Google's `/sorry/`
interstitial, which arrives as a redirect to a captcha page."""
return response.status_code in (403, 429) or "/sorry/" in response.url.path
def _refusal(video: str, said: list[str]) -> str:
"""`Blocked`'s sentence. "YouTube refused" is what `desk/notices.py` reads
to tell this from a feed that failed, so the wording is load-bearing."""
return (f"{video}: YouTube refused this address and wants a sign-in — "
+ "; ".join(said))
def _tracks(player: dict[str, Any]) -> list[dict[str, Any]]:
return (player.get("captions", {})
.get("playerCaptionsTracklistRenderer", {})
.get("captionTracks", []) or [])
def _pick(tracks: list[dict[str, Any]], language: str) -> dict[str, Any] | None:
"""Best track for a language: a real transcript over an auto-generated one.
A channel that uploads its own captions has punctuation, speaker turns and
correctly spelled names; ASR has none of those and spells "Qwen" six ways.
Worth preferring, and never worth requiring — most shows only have ASR.
"""
wanted = [t for t in tracks
if (t.get("languageCode") or "").lower().startswith(language.lower())]
wanted.sort(key=lambda t: t.get("kind") == "asr")
return wanted[0] if wanted else None
def _cues(body: str) -> list[Cue]:
"""Parse whichever of YouTube's three caption formats turned up."""
body = body.strip()
if body.startswith("{"):
return _from_json3(json.loads(body))
root = ET.fromstring(body)
# SRV1 is `transcript > text[start,dur]`; SRV3 is `timedtext > body > p[t,d]`.
if root.tag == "transcript":
return _from_srv1(root)
return _from_srv3(root)
def _from_json3(data: dict[str, Any]) -> list[Cue]:
cues = []
for event in data.get("events", []):
text = "".join(seg.get("utf8", "") for seg in event.get("segs", []))
clean = _clean(text)
if not clean:
continue
cues.append(Cue(
start=event.get("tStartMs", 0) / 1000,
duration=event.get("dDurationMs", 0) / 1000,
text=clean,
))
return cues
def _from_srv1(root: ET.Element) -> list[Cue]:
cues = []
for node in root.iter("text"):
clean = _clean(node.text or "")
if not clean:
continue
cues.append(Cue(
start=float(node.get("start", 0)),
duration=float(node.get("dur", 0)),
text=clean,
))
return cues
def _from_srv3(root: ET.Element) -> list[Cue]:
cues = []
for node in root.iter("p"):
clean = _clean("".join(node.itertext()))
if not clean:
continue
cues.append(Cue(
start=int(node.get("t", 0)) / 1000,
duration=int(node.get("d", 0)) / 1000,
text=clean,
))
return cues
def _clean(text: str) -> str:
"""Entities out, whitespace collapsed, sound cues dropped.
Unescaped twice on purpose: SRV1 sends `&amp;#39;` — an entity for an
entity — because the caption text was escaped once when it was stored and
once again on the way into the XML.
"""
text = html.unescape(html.unescape(text or ""))
text = re.sub(r"\[(?:Music|Applause|Laughter)\]", " ", text)
return re.sub(r"\s+", " ", text).strip()
-128
View File
@@ -1,128 +0,0 @@
"""Did we fetch the article, or the page the outlet shows instead of it?
Some sites answer every article URL with the same thing: a consent gate, an
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
faithfully — it is real text on a real page — and we store it as the body. That
is worse than storing nothing, because the reader prefers the body to the feed's
summary, so a wall doesn't merely fail to add anything: it *hides* the two
readable sentences the feed already gave us.
The tell is repetition. Different articles cannot have the same body; a wall
always does. That is the whole detector — no keyword list, no per-site rules,
nothing to keep up to date. It learns golem.de's German consent page and a
Cloudflare challenge with the same line of code, and it costs no extra
requests, because the evidence is a by-product of a fetch we were making
anyway.
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
requests to one host earns a throttle or challenge page, that page is served to
whichever two articles asked next, and the two fingerprint the same. Refetching
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
was about our request rate and not about the outlet. A wall repeats on every
article, so a third body costs one extra fetch per walled host, once; a throttle
page that repeats three consecutive times is the site answering nothing else.
The verdict is per host, because the wall is: the site serves it, it isn't
chosen per article. Once a host is known to answer with one, we stop fetching
from it altogether — which makes pulls faster, not just more correct. That is
also why the count matters: `known` short-circuits the request, so a wall
verdict is the last evidence a host ever gets to give. Nothing can disprove it
except `forget`.
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
in a test), because a tool has no store of its own to remember things in. The
one way this is wrong is an outlet that really does publish the same body
`STRIKES` times running; the cost is that its articles fall back to their feed
summaries until someone calls `forget`, which is what the re-check button in
the Sources tab is for.
"""
from __future__ import annotations
import hashlib
import logging
from collections.abc import Mapping, MutableMapping
from desk.tools.framing import host_of
log = logging.getLogger(__name__)
#: Sits alongside `embed:` in the same table. The two verdicts answer different
#: questions about a host — will it let us frame it, will it give up its text —
#: and neither implies the other.
CACHE_PREFIX = "body:"
_WALL = "wall:"
REASON = "every article comes back as the same page"
#: How many consecutive identical bodies settle it. See the module docstring:
#: two was one throttle page away from retiring a working outlet for good.
STRIKES = 3
def _fingerprint(body: str) -> str:
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
def _seen(value: str) -> tuple[str, int]:
"""`(fingerprint, how many in a row)` from a stored value.
A bare fingerprint is one sighting, which is what every row written before
`STRIKES` existed holds — so an upgraded database is read correctly and
needs no migration of its own.
"""
mark, _, run = value.partition("*")
return mark, (int(run) if run.isdigit() else 1)
def known(cache: Mapping[str, str], url: str) -> str:
"""Why this host's bodies aren't worth fetching, or "" if they are.
Asked before the request, so a settled verdict costs nothing at all. Takes
a read-only mapping, because this one only ever reads — which lets the
Sources tab hand over the snapshot it already loaded rather than the live
table, and says in the signature that opening a panel changes nothing.
"""
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
"""Record a body we just extracted. Returns a reason if it proves a wall.
Only the previous body from the host is kept and how many articles running
have handed it over — not every body ever seen. A wall repeats on the
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
and one key per host beats a growing set per host.
One distinct body resets the run to one. That is what makes the count mean
"in a row" rather than "ever", and it is the whole difference between a
site that answers with a gate and a site that answered twice from a cache.
"""
host = host_of(url)
if not host or not body.strip():
return ""
key = CACHE_PREFIX + host
stored = cache.get(key, "")
if stored.startswith(_WALL):
return stored[len(_WALL):]
mark = _fingerprint(body)
seen, run = _seen(stored)
run = run + 1 if seen == mark else 1
if run >= STRIKES:
cache[key] = _WALL + REASON
log.info("%s serves a wall, not articles: %s (%d in a row)",
host, REASON, run)
return REASON
cache[key] = mark if run == 1 else f"{mark}*{run}"
return ""
def forget(cache: MutableMapping[str, str], url: str) -> None:
"""Drop what we learned about a host, so the next fetch decides again.
Outlets change their minds — a wall goes up for a quarter and comes down
again — and a verdict with no way back would be a permanent decision made
on two HTTP responses.
"""
cache.pop(CACHE_PREFIX + host_of(url), None)