Prune desk to what roledesk uses
Drops news's own sources (Bluesky, Mastodon, X, podcast) and the tools only they use (video, posts, walls), with the PodcastSource and segment_profile exports. scrape, article, checks, browser and framing stay: base.py imports scrape's Getter at module level and scrape imports the other four, so dropping them would mean editing the plugin contract. desk/README.md records the source of truth, the prune and how to refresh. All 371 tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0627b74f03
commit
24508dd6fc
15 files changed
+44
-2986
No files matched your search
+12
-6
@@ -7,14 +7,20 @@ if it would make sense in a different application, it is a tool.
|
||||
|
||||
agent.py who answers when something needs a model: the Claude API
|
||||
with a key, the `claude` CLI on a subscription, or nobody
|
||||
credentials.py the token `agent` reads from the app's database
|
||||
embedding.py text in, vector out
|
||||
framing.py will this page load inside an iframe?
|
||||
http.py one shared HTTP client per call chain
|
||||
failures.py what went wrong, in one word a retry policy can use
|
||||
marks.py what kind of thing is this, if not an article?
|
||||
video/ a recording, read as text with timings on it
|
||||
walls.py was that the article, or the page shown instead of it?
|
||||
scrape.py an article out of a URL: fetch, check, convert.
|
||||
Kept because `desk.sources.base` imports its `Getter`;
|
||||
`article.py`, `checks.py`, `browser.py` and `framing.py`
|
||||
stay only because `scrape.py` imports them
|
||||
|
||||
Nothing is imported here on purpose. `video` pulls in httpx and the caption
|
||||
parsers, `embedding` a sentence-transformer, `agent` an SDK; each is paid for
|
||||
by the caller that wants it and by nobody else.
|
||||
roledesk's copy prunes news's `video/`, `posts.py` and `walls.py`, which only
|
||||
news's own sources use (see `desk/README.md`).
|
||||
|
||||
Nothing is imported here on purpose. `embedding` pulls in a
|
||||
sentence-transformer, `agent` an SDK; each is paid for by the caller that
|
||||
wants it and by nobody else.
|
||||
"""
|
||||
@@ -1,193 +0,0 @@
|
||||
"""A Mastodon or Bluesky post read through its network's public API.
|
||||
|
||||
Both web apps serve a script shell to a plain request, so `checks` names these
|
||||
getters before `browser`: the API answers in a fraction of a second, needs no
|
||||
Chromium and no login, and returns the post rather than a rendering of it.
|
||||
Each getter builds a small HTML page from the answer, so `scrape` checks and
|
||||
converts it exactly as it would a publisher's page.
|
||||
|
||||
from desk.tools import posts
|
||||
|
||||
posts.is_mastodon("https://hachyderm.io/@simontatham/117201594980991062")
|
||||
page = posts.mastodon(url) # checks.Page
|
||||
page = posts.bluesky("https://bsky.app/profile/bsky.app/post/3mv3shqdfuc2e")
|
||||
|
||||
The Mastodon and Bluesky plugins register them with `scrape` as the
|
||||
`mastodon_api` and `bluesky_api` getters for their URLs; see `Source.getters`.
|
||||
|
||||
**A thread is read as far as its author continued it.** A post a link points
|
||||
to is often the first of several by the same person; the replies by anyone
|
||||
else are not part of it.
|
||||
|
||||
X is not here: its API bills every read and needs a token, and x.com shows a
|
||||
browser a login wall.
|
||||
|
||||
GET https://{host}/api/v1/statuses/{id} Mastodon, any instance
|
||||
GET https://{host}/api/v1/statuses/{id}/context
|
||||
GET https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
from desk.tools.checks import Page
|
||||
|
||||
TIMEOUT = 20.0
|
||||
#: Says what is asking, the way `desk.sources.base.USER_AGENT` does for the
|
||||
#: plugins; a tool sits below `sources` and cannot import it.
|
||||
USER_AGENT = "newsdesk (+https://github.com/; reading a linked post)"
|
||||
BLUESKY_API = "https://public.api.bsky.app/xrpc"
|
||||
|
||||
#: `/@user/123` is what an instance links; `/users/user/statuses/123` is the
|
||||
#: ActivityPub id some links carry instead.
|
||||
_MASTODON_PATH = re.compile(r"^/(?:@[^/]+|users/[^/]+/statuses)/(\d+)/?$")
|
||||
_BLUESKY_PATH = re.compile(r"^/profile/([^/]+)/post/([a-z0-9]+)/?$")
|
||||
|
||||
#: How deep a Bluesky thread is asked for. A self-thread longer than this is
|
||||
#: cut, which is rare and costs the tail rather than the post.
|
||||
THREAD_DEPTH = 20
|
||||
|
||||
|
||||
def _get(url: str, params: dict[str, Any] | None = None) -> httpx.Response:
|
||||
# Module-level, so a test replaces this one function.
|
||||
return httpx.get(url, params=params, timeout=TIMEOUT,
|
||||
headers={"User-Agent": USER_AGENT}, follow_redirects=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- Mastodon
|
||||
|
||||
def _mastodon_id(url: str) -> tuple[str, str] | None:
|
||||
parts = urlsplit(url)
|
||||
found = _MASTODON_PATH.match(parts.path)
|
||||
return (parts.netloc, found.group(1)) if parts.scheme == "https" and found else None
|
||||
|
||||
|
||||
def is_mastodon(url: str) -> bool:
|
||||
"""Whether `url` has a Mastodon post's shape. Any host: an instance is
|
||||
recognised by the check that found its web app's shell, not by name."""
|
||||
return _mastodon_id(url) is not None
|
||||
|
||||
|
||||
def mastodon(url: str) -> Page | None:
|
||||
"""The post at `url`, and its author's continuation, from the instance's API."""
|
||||
host, status_id = _mastodon_id(url) or ("", "")
|
||||
answer = _get(f"https://{host}/api/v1/statuses/{status_id}")
|
||||
if not _json(answer):
|
||||
# Not an instance: `/@user/123` is a Mastodon address's shape, and
|
||||
# another site can have it. Nothing to say about the page, so the
|
||||
# ordinary path runs.
|
||||
return None
|
||||
if answer.status_code != 200:
|
||||
return Page(answer.status_code, "", url, dict(answer.headers))
|
||||
status = answer.json()
|
||||
author = status.get("account") or {}
|
||||
thread = [status]
|
||||
context = _get(f"https://{host}/api/v1/statuses/{status_id}/context")
|
||||
if context.status_code == 200 and _json(context):
|
||||
# Descendants come in thread order; the author's continuation is the
|
||||
# chain of their replies, each to the one before.
|
||||
for reply in context.json().get("descendants") or []:
|
||||
if ((reply.get("account") or {}).get("id") == author.get("id")
|
||||
and reply.get("in_reply_to_id") == thread[-1].get("id")):
|
||||
thread.append(reply)
|
||||
name = author.get("display_name") or author.get("acct") or host
|
||||
card = status.get("card") or {}
|
||||
return Page(200, _page(
|
||||
title=f"{name}: {_first_words(_text(status.get('content', '')))}",
|
||||
byline=f"{name} (@{author.get('acct', '')})",
|
||||
published=status.get("created_at", ""),
|
||||
paragraphs=[s.get("content", "") for s in thread],
|
||||
link=(card.get("url", ""), card.get("title", "")),
|
||||
alts=[m.get("description") or "" for s in thread
|
||||
for m in s.get("media_attachments") or []],
|
||||
escaped=False), url)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- Bluesky
|
||||
|
||||
def _bluesky_uri(url: str) -> str:
|
||||
parts = urlsplit(url)
|
||||
found = _BLUESKY_PATH.match(parts.path)
|
||||
if parts.netloc not in {"bsky.app", "www.bsky.app"} or not found:
|
||||
return ""
|
||||
return f"at://{found.group(1)}/app.bsky.feed.post/{found.group(2)}"
|
||||
|
||||
|
||||
def is_bluesky(url: str) -> bool:
|
||||
return bool(_bluesky_uri(url))
|
||||
|
||||
|
||||
def bluesky(url: str) -> Page | None:
|
||||
"""The post at `url`, and its author's continuation, from Bluesky's AppView."""
|
||||
answer = _get(f"{BLUESKY_API}/app.bsky.feed.getPostThread",
|
||||
{"uri": _bluesky_uri(url), "depth": THREAD_DEPTH, "parentHeight": 0})
|
||||
if answer.status_code != 200:
|
||||
# A post that is gone is a 400 `NotFound` here, not a 404.
|
||||
gone = answer.status_code == 400 and "NotFound" in answer.text
|
||||
return Page(404 if gone else answer.status_code, "", url, dict(answer.headers))
|
||||
thread = answer.json().get("thread") or {}
|
||||
post = thread.get("post") or {}
|
||||
author = post.get("author") or {}
|
||||
posts = [post]
|
||||
node = thread
|
||||
while True:
|
||||
# The author's own reply, if there is one, is the next part of the thread.
|
||||
nxt = next((r for r in node.get("replies") or []
|
||||
if ((r.get("post") or {}).get("author") or {}).get("did") == author.get("did")),
|
||||
None)
|
||||
if nxt is None:
|
||||
break
|
||||
posts.append(nxt["post"])
|
||||
node = nxt
|
||||
record = post.get("record") or {}
|
||||
external = ((post.get("embed") or {}).get("external") or {})
|
||||
name = author.get("displayName") or author.get("handle", "")
|
||||
return Page(200, _page(
|
||||
title=f"{name}: {_first_words(record.get('text', ''))}",
|
||||
byline=f"{name} (@{author.get('handle', '')})",
|
||||
published=record.get("createdAt", ""),
|
||||
paragraphs=[(p.get("record") or {}).get("text", "") for p in posts],
|
||||
link=(external.get("uri", ""), external.get("title", "")),
|
||||
alts=[image.get("alt") or "" for p in posts
|
||||
for image in (p.get("embed") or {}).get("images") or []],
|
||||
escaped=True), url)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the page
|
||||
|
||||
def _page(*, title: str, byline: str, published: str, paragraphs: list[str],
|
||||
link: tuple[str, str], alts: list[str], escaped: bool) -> str:
|
||||
"""An article page for the converter. Mastodon's content is HTML already;
|
||||
Bluesky's is text, escaped here and split into paragraphs on blank lines."""
|
||||
if escaped:
|
||||
paragraphs = ["".join(f"<p>{html.escape(block).replace(chr(10), '<br>')}</p>"
|
||||
for block in re.split(r"\n\s*\n", text) if block.strip())
|
||||
for text in paragraphs]
|
||||
body = "\n".join(paragraphs)
|
||||
url, label = link
|
||||
if url:
|
||||
body += f'\n<p><a href="{html.escape(url)}">{html.escape(label or url)}</a></p>'
|
||||
body += "".join(f"\n<p>[Image: {html.escape(alt)}]</p>" for alt in alts if alt.strip())
|
||||
return (f"<!DOCTYPE html><html><head><title>{html.escape(title)}</title></head><body>"
|
||||
f"<article><header><p>{html.escape(byline)}</p>"
|
||||
f"<time datetime=\"{html.escape(published)}\">{html.escape(published)}</time></header>"
|
||||
f"\n{body}\n</article></body></html>")
|
||||
|
||||
|
||||
def _json(answer: httpx.Response) -> bool:
|
||||
return "json" in answer.headers.get("content-type", "")
|
||||
|
||||
|
||||
def _text(markup: str) -> str:
|
||||
return html.unescape(re.sub(r"<[^>]+>", " ", markup))
|
||||
|
||||
|
||||
def _first_words(text: str, count: int = 8) -> str:
|
||||
words = text.split()
|
||||
return " ".join(words[:count]) + ("…" if len(words) > count else "")
|
||||
@@ -1,103 +0,0 @@
|
||||
"""Video, read as text with timings on it.
|
||||
|
||||
The generic half of watching things so you don't have to. Give it a video and
|
||||
it gives back a list of `Segment`s — each one a stretch of the recording with a
|
||||
title, the words spoken in it, and a link that plays it from that second.
|
||||
|
||||
read() a video -> Video + Segment[] the whole thing, one call
|
||||
youtube captions, metadata, comments where both inputs come from
|
||||
marks an outline -> chapter marks description, notes, comments
|
||||
media a MediaRef -> a playable link YouTube today, a file later
|
||||
segment cues + marks -> Segment[] the cut, and transcript text
|
||||
|
||||
**This package knows nothing about newsdesk.** No store, no queue, no config,
|
||||
no `Item`. That is what makes it reusable, and it is the seam the design is
|
||||
built on: `desk/sources/podcast.py` sits on top and adds the things that
|
||||
*are* specific to a show — its feed, its weekly cadence, the article links in
|
||||
its notes, and turning segments into rows.
|
||||
|
||||
Two decisions worth knowing before reading further.
|
||||
|
||||
**A video is a `MediaRef`, not a URL.** Today every ref is a YouTube id; the
|
||||
reading pane's link is computed from it on the way out. Point one at a file on
|
||||
disk in config and every link to it changes, with nothing rewritten and no
|
||||
item's identity touched. See `media.py`.
|
||||
|
||||
**Transcripts come from whoever can supply one.** `youtube-transcript-api`
|
||||
first, and a built-in fetcher against YouTube's own player endpoint when the
|
||||
library can't get the track. Transcribing audio locally (yt-dlp plus
|
||||
faster-whisper) is the third backend and the one that makes local video work;
|
||||
it is not written yet, and `read` is shaped so that adding it changes nothing
|
||||
above.
|
||||
|
||||
Try it:
|
||||
|
||||
python -m desk.tools.video https://www.youtube.com/watch?v=GUYEofBebDc
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
||||
from desk.tools.video import marks, media, youtube
|
||||
from desk.tools.http import session
|
||||
from desk.tools.video.marks import Chapter, Link
|
||||
from desk.tools.video.media import MediaRef
|
||||
from desk.tools.video.segment import Segment, Video, cut, opening, paragraphs
|
||||
from desk.tools.video.youtube import Blocked, Cue, TranscriptUnavailable
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
__all__ = [
|
||||
"Blocked", "Chapter", "Cue", "Link", "MediaRef", "Segment",
|
||||
"TranscriptUnavailable", "Video", "cut", "describe", "marks", "media",
|
||||
"opening", "paragraphs", "read", "youtube",
|
||||
]
|
||||
|
||||
|
||||
def read(url: str, *, comments: bool = True,
|
||||
client: httpx.Client | None = None) -> tuple[Video, list[Segment]]:
|
||||
"""One video, cut into its chapters. The whole package in one call.
|
||||
|
||||
`comments` is the only cost worth thinking about: it is an extra request to
|
||||
the watch page and is only consulted when the description has no outline in
|
||||
it, so for a show that publishes chapter marks it never happens.
|
||||
"""
|
||||
with session(client, timeout=40) as http:
|
||||
video = describe(url, client=http)
|
||||
try:
|
||||
cues = youtube.transcript(video.key, client=http)
|
||||
except TranscriptUnavailable as exc:
|
||||
# Not fatal on its own. The outline still gives every chapter, its
|
||||
# links and its timecode; only the quoted words are missing, and
|
||||
# auto-captions usually appear within a few hours of an upload.
|
||||
log.warning("%s", exc)
|
||||
cues = []
|
||||
|
||||
chapters = marks.best(video.notes)
|
||||
if not chapters and comments:
|
||||
chapters = marks.in_comments(youtube.comments(video.key, client=http))
|
||||
|
||||
return video, cut(video, cues, chapters)
|
||||
|
||||
|
||||
def describe(url: str, *, client: httpx.Client | None = None) -> Video:
|
||||
"""What a video says about itself: title, author, length, description.
|
||||
|
||||
YouTube only, for now. A local file would come back with its stem as the
|
||||
title and no notes, which is why `Video.notes` is allowed to be empty and
|
||||
why `read` has a comments fallback at all.
|
||||
"""
|
||||
key = youtube.video_id(url)
|
||||
with session(client) as http:
|
||||
meta = youtube.details(key, client=http)
|
||||
return Video(
|
||||
ref=media.youtube(key),
|
||||
title=meta["title"],
|
||||
url=media.youtube(key).at(0),
|
||||
author=meta["author"],
|
||||
duration=meta["duration"],
|
||||
notes=meta["description"],
|
||||
)
|
||||
@@ -1,120 +0,0 @@
|
||||
"""Cut a video up and print what came out.
|
||||
|
||||
python -m desk.tools.video <youtube url>
|
||||
python -m desk.tools.video <youtube url> --json > segments.json
|
||||
|
||||
Videos only. A podcast episode — where the chapter marks and the article links
|
||||
live on the show's own page rather than on YouTube — goes through the layer
|
||||
above:
|
||||
|
||||
python -m desk.sources.podcast <episode url | feed url>
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import dataclasses
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
|
||||
import desk.tools.video as video_pkg
|
||||
from desk.tools.video import marks
|
||||
from desk.tools.video.segment import Segment, Video
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="python -m desk.tools.video",
|
||||
description="Cut a video into one item per chapter mark.",
|
||||
)
|
||||
parser.add_argument("url", help="a YouTube link or bare video id")
|
||||
parser.add_argument("--json", action="store_true",
|
||||
help="dump everything, transcripts included, as JSON")
|
||||
parser.add_argument("--chars", type=int, default=160,
|
||||
help="how much transcript to preview per segment (default 160)")
|
||||
parser.add_argument("--no-comments", action="store_true",
|
||||
help="don't fall back to chapter marks typed in the comments")
|
||||
parser.add_argument("-v", "--verbose", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO if args.verbose else logging.WARNING,
|
||||
format="%(levelname)-7s %(message)s",
|
||||
)
|
||||
# Windows still opens a console in the system codepage, and a video title
|
||||
# with a "ñ" in it — or the arrows this prints — is an unhandled
|
||||
# UnicodeEncodeError half way through the report rather than a bad glyph.
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
try:
|
||||
video, segments = video_pkg.read(args.url, comments=not args.no_comments)
|
||||
except Exception as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.json:
|
||||
json.dump(dump(video, segments), sys.stdout, indent=2, default=str)
|
||||
print()
|
||||
else:
|
||||
report(video, segments, args.chars)
|
||||
return 0
|
||||
|
||||
|
||||
def report(video: Video, segments: list[Segment], chars: int) -> None:
|
||||
"""The human-readable view. Also used by the podcast module."""
|
||||
words = sum(len(s.transcript.split()) for s in segments)
|
||||
length = video.duration or (segments[-1].end if segments else 0)
|
||||
print(f"{video.title}")
|
||||
print(f" {video.author or '—'} {video.ref.kind}:{video.ref.key}"
|
||||
f" {marks.hhmmss(length)}")
|
||||
print(f" {len(segments)} segments, {words:,} words of transcript")
|
||||
print()
|
||||
|
||||
heading = None
|
||||
for segment in segments:
|
||||
if segment.heading != heading:
|
||||
heading = segment.heading
|
||||
print(f" {heading or '(unfiled)'}")
|
||||
preview = segment.transcript[:chars]
|
||||
if len(segment.transcript) > chars:
|
||||
preview += "…"
|
||||
print(f" {segment.timecode:>8} {segment.title[:78]}")
|
||||
print(f" {marks.hhmmss(segment.duration)} long · {segment.url()}")
|
||||
for link in segment.links:
|
||||
print(f" → {link.url}")
|
||||
print(f" {preview or '(no transcript under this mark)'}")
|
||||
print()
|
||||
|
||||
|
||||
def dump(video: Video, segments: list[Segment]) -> dict[str, object]:
|
||||
"""The shape a stored item is built from, plus the fields a dry run wants
|
||||
to eyeball. The stored half is `Segment.record()` and is defined there;
|
||||
desk/sources/podcast.py turns the rest into a row."""
|
||||
return {
|
||||
"video": {
|
||||
"ref": dataclasses.asdict(video.ref),
|
||||
"title": video.title,
|
||||
"url": video.url,
|
||||
"author": video.author,
|
||||
"published_at": video.published_at,
|
||||
"duration": video.duration,
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
**s.record(),
|
||||
"external_id": s.external_id,
|
||||
"title": s.title,
|
||||
"heading": s.heading,
|
||||
"timecode": s.timecode,
|
||||
"url": s.url(),
|
||||
"transcript": s.transcript,
|
||||
}
|
||||
for s in segments
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,265 +0,0 @@
|
||||
"""Chapter marks: a video's own table of contents, wherever it was written.
|
||||
|
||||
The important discovery about this kind of video is that **the segmentation
|
||||
already exists and somebody else did it**. Last Week in AI ships every episode
|
||||
with a timestamped outline — a rubric, then one line per story, each with the
|
||||
time it starts and a link to the article being discussed:
|
||||
|
||||
Tools & Apps
|
||||
(00:03:32) Google announces Gemini 3.7 Flash ... - Ars Technica
|
||||
(00:13:11) SpaceXAI Releases Grok 4.6 ... - MarkTechPost
|
||||
|
||||
That is a better set of boundaries than any topic-segmentation model would
|
||||
produce, for nothing, with the source links attached. So this module parses it
|
||||
rather than inferring it. An LLM is still wanted downstream — to summarise what
|
||||
was *said* in each stretch — but not to work out where one stretch ends.
|
||||
|
||||
**Three places to find that outline**, in descending order of usefulness, all
|
||||
parsed by the same function because they are the same convention written in
|
||||
different markup:
|
||||
|
||||
1. **The show's own page or feed body** — HTML, so the links are real
|
||||
`<a href>` and each chapter arrives with the article it is about. Only a
|
||||
show with a website has this, which is why the podcast source hands it in
|
||||
rather than this module going to look.
|
||||
2. **The video description** — plain text. Usually the same outline with the
|
||||
links flattened away.
|
||||
3. **The comments** — see `in_comments`. When a show publishes no marks at
|
||||
all, a listener has often typed some out; YouTube itself reads those to
|
||||
draw its chapter markers.
|
||||
|
||||
`best` takes them in that order and returns the first that is real. Nothing
|
||||
here is specific to one show beyond the shape "timestamp, then a title,
|
||||
optionally under a heading", which is the near-universal convention.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass, field
|
||||
from html.parser import HTMLParser
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: `(00:03:32)`, `00:03:32`, `3:32` — at the start of the line, which is what
|
||||
#: makes it a chapter mark rather than a time mentioned in a sentence.
|
||||
TIMESTAMP = re.compile(
|
||||
# The character class ends with hyphen, en dash and em dash on purpose:
|
||||
# a show separates the time from the title with whichever of the three its
|
||||
# editor's keyboard produced, and means the same thing by all of them.
|
||||
r"^\(?\s*(\d{1,2}):([0-5]\d)(?::([0-5]\d))?\s*\)?[\s\-–—:]*" # noqa: RUF001
|
||||
)
|
||||
_URL = re.compile(r"https?://[^\s<>\"')\]]+")
|
||||
#: Tags that end a line of the outline. Everything else is inline and stays
|
||||
#: on the line it was found in, which is what keeps `<span>(00:03:32) </span>`
|
||||
#: attached to the `<a>` that follows it.
|
||||
_BLOCKS = frozenset({
|
||||
"li", "p", "br", "div", "tr", "ul", "ol", "h1", "h2", "h3", "h4", "blockquote",
|
||||
})
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Link:
|
||||
"""One article the hosts were talking about."""
|
||||
|
||||
url: str
|
||||
title: str = ""
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Chapter:
|
||||
"""One entry in the outline: where a story starts and what it is about."""
|
||||
|
||||
start: int # seconds into the episode
|
||||
title: str
|
||||
heading: str = "" # the rubric it sits under
|
||||
links: list[Link] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
"""The article being discussed, if the notes linked one."""
|
||||
return self.links[0].url if self.links else ""
|
||||
|
||||
|
||||
def parse(body: str) -> list[Chapter]:
|
||||
"""Every chapter mark in a set of show notes, in the order they play.
|
||||
|
||||
Takes HTML or plain text and works out which it was given. Returns `[]`
|
||||
for notes with no outline in them at all, which is a real answer — plenty
|
||||
of shows don't write one, and the caller falls back to treating the episode
|
||||
as a single undivided item rather than guessing at boundaries.
|
||||
"""
|
||||
lines = _html_lines(body) if "<" in body and ">" in body else _text_lines(body)
|
||||
|
||||
chapters: list[Chapter] = []
|
||||
heading = ""
|
||||
for text, links in lines:
|
||||
match = TIMESTAMP.match(text)
|
||||
if match is None:
|
||||
# A heading is only a heading once the outline has started;
|
||||
# before that every line is preamble — the blurb, the sponsors,
|
||||
# the "Timestamps:" label itself.
|
||||
if chapters and _is_heading(text, links):
|
||||
heading = text.rstrip(":")
|
||||
continue
|
||||
title = text[match.end():].strip(" -–—|") # noqa: RUF001 — as above
|
||||
# Prefer the anchor's own text: a line that links two follow-ups reads
|
||||
# "First story | The Verge + Second story", and the first link's text
|
||||
# is the story, where the whole line is the story plus its footnote.
|
||||
if links and links[0].title:
|
||||
title = links[0].title
|
||||
chapters.append(Chapter(
|
||||
start=_seconds(match), title=title, heading=heading, links=links,
|
||||
))
|
||||
|
||||
if not chapters:
|
||||
log.debug("no chapter marks found in %d chars of notes", len(body))
|
||||
return chapters
|
||||
|
||||
|
||||
def best(*candidates: str, minimum: int = 3) -> list[Chapter]:
|
||||
"""The first candidate that yields a real outline.
|
||||
|
||||
Chapter marks come from up to three places and they are not equally good,
|
||||
so the order they are passed in is the order of preference — for a podcast
|
||||
that is: the show's own notes as HTML (links intact), then the YouTube
|
||||
description (same outline, links flattened away), then the comments.
|
||||
|
||||
`minimum` is what stops a stray "at 3:20 he says" in a blurb becoming a
|
||||
one-chapter outline that suppresses the good source behind it. Three marks
|
||||
is the point where a line of text stops being a coincidence.
|
||||
"""
|
||||
for text in candidates:
|
||||
found = parse(text)
|
||||
if len(found) >= minimum:
|
||||
return found
|
||||
return []
|
||||
|
||||
|
||||
def in_comments(comments: Iterable[str], minimum: int = 3) -> list[Chapter]:
|
||||
"""The outline somebody typed into the comments, if anybody did.
|
||||
|
||||
Read as the fallback for shows that publish no chapter marks of their own,
|
||||
where it is common for a listener to write one out — YouTube itself treats
|
||||
such a comment as good enough to draw chapter markers from.
|
||||
|
||||
Each comment is parsed on its own rather than the lot being concatenated.
|
||||
Joined together, an outline comment and thirty unrelated replies produce
|
||||
one bogus outline interleaved with chatter; separately, the outline is the
|
||||
comment with the most marks in it and the chatter scores zero.
|
||||
"""
|
||||
outlines = [parse(comment) for comment in comments]
|
||||
outlines = [found for found in outlines if len(found) >= minimum]
|
||||
if not outlines:
|
||||
return []
|
||||
richest = max(outlines, key=len)
|
||||
log.info("took %d chapter marks from a comment", len(richest))
|
||||
return richest
|
||||
|
||||
|
||||
def hhmmss(seconds: float) -> str:
|
||||
"""3932 -> '1:05:32'. What a timecode looks like to a reader."""
|
||||
total = int(seconds)
|
||||
hours, rest = divmod(total, 3600)
|
||||
minutes, secs = divmod(rest, 60)
|
||||
return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ internals
|
||||
|
||||
def _seconds(match: re.Match[str]) -> int:
|
||||
a, b, c = match.group(1), match.group(2), match.group(3)
|
||||
if c is None: # mm:ss
|
||||
return int(a) * 60 + int(b)
|
||||
return int(a) * 3600 + int(b) * 60 + int(c)
|
||||
|
||||
|
||||
def _is_heading(text: str, links: list[Link]) -> bool:
|
||||
"""A rubric like 'Policy & Safety', not a sentence and not a story.
|
||||
|
||||
Deliberately strict. Anything that gets through becomes the heading for
|
||||
every chapter after it, so a false positive mislabels the rest of the
|
||||
episode — where a false negative just leaves a few items unfiled.
|
||||
"""
|
||||
return (
|
||||
not links
|
||||
and len(text) <= 60
|
||||
and len(text.split()) <= 6
|
||||
and not text.endswith((".", "!", "?"))
|
||||
)
|
||||
|
||||
|
||||
def _text_lines(body: str) -> list[tuple[str, list[Link]]]:
|
||||
"""Plain-text notes — a YouTube description. Bare URLs become links."""
|
||||
lines = []
|
||||
for raw in body.splitlines():
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
continue
|
||||
links = [Link(url=u) for u in _URL.findall(line)]
|
||||
lines.append((_URL.sub("", line).strip(), links))
|
||||
return lines
|
||||
|
||||
|
||||
def _html_lines(body: str) -> list[tuple[str, list[Link]]]:
|
||||
parser = _Outline()
|
||||
parser.feed(body)
|
||||
parser.close()
|
||||
return parser.lines
|
||||
|
||||
|
||||
class _Outline(HTMLParser):
|
||||
"""HTML to (line, links) pairs, without pulling in a parser library.
|
||||
|
||||
A full HTML tree would be nicer and is not worth a dependency for this:
|
||||
the only structure that matters is where the line breaks fall and which
|
||||
anchors landed on which line. `convert_charrefs` (on by default) means
|
||||
entities arrive already decoded, so `Tools & Apps` is `Tools & Apps`
|
||||
by the time it reaches `handle_data`.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.lines: list[tuple[str, list[Link]]] = []
|
||||
self._text: list[str] = []
|
||||
self._links: list[Link] = []
|
||||
self._href = ""
|
||||
self._anchor: list[str] = []
|
||||
|
||||
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
|
||||
if tag in _BLOCKS:
|
||||
self._flush()
|
||||
elif tag == "a":
|
||||
self._href = dict(attrs).get("href") or ""
|
||||
self._anchor = []
|
||||
|
||||
def handle_endtag(self, tag: str) -> None:
|
||||
if tag in _BLOCKS:
|
||||
self._flush()
|
||||
elif tag == "a":
|
||||
if self._href:
|
||||
self._links.append(
|
||||
Link(url=self._href, title=_squash("".join(self._anchor)))
|
||||
)
|
||||
self._href, self._anchor = "", []
|
||||
|
||||
def handle_data(self, data: str) -> None:
|
||||
self._text.append(data)
|
||||
if self._href:
|
||||
self._anchor.append(data)
|
||||
|
||||
def _flush(self) -> None:
|
||||
text = _squash("".join(self._text))
|
||||
if text or self._links:
|
||||
self.lines.append((text, self._links))
|
||||
self._text, self._links = [], []
|
||||
|
||||
def close(self) -> None:
|
||||
super().close()
|
||||
self._flush()
|
||||
|
||||
|
||||
def _squash(text: str) -> str:
|
||||
return re.sub(r"\s+", " ", text or "").strip()
|
||||
@@ -1,113 +0,0 @@
|
||||
"""Where a video actually lives, and how to link into the middle of it.
|
||||
|
||||
Every segment newsdesk stores has to answer one question: *press play at
|
||||
1:08:27 — where?* Today the answer is a YouTube URL. It will not always be.
|
||||
The same episode may sit on disk, ripped or recorded; a show may take its
|
||||
uploads down; a video may only ever have existed as a local file.
|
||||
|
||||
So nothing in this codebase stores a YouTube URL as a fact. It stores a
|
||||
`MediaRef` — a kind and a key — and asks this module for a link when it needs
|
||||
one. Swapping YouTube for a copy on disk is then a line of config rather than
|
||||
an UPDATE across every item that mentioned it.
|
||||
|
||||
video:
|
||||
local_media:
|
||||
GUYEofBebDc: D:/media/lwiai/255.mp4
|
||||
|
||||
`resolve` applies that map. Everything downstream keeps working because it was
|
||||
never holding a URL in the first place.
|
||||
|
||||
The one thing this deliberately does not do is *find* local copies. Matching
|
||||
"is this file the same episode as that video" by filename or duration is a
|
||||
guess, and a wrong guess points a citation at the wrong recording. The mapping
|
||||
is written by a person, or it does not exist.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path, PurePath
|
||||
|
||||
#: A local file, playable in the reading pane's <video> element. Anything else
|
||||
#: on disk (mkv, wav) still resolves; the pane just may not play it.
|
||||
PLAYABLE = frozenset({".mp4", ".m4v", ".webm", ".ogg", ".mp3", ".m4a", ".wav"})
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class MediaRef:
|
||||
"""A video, identified by where it is rather than by a URL.
|
||||
|
||||
`kind` is what the key means: `youtube` is an 11-character video id,
|
||||
`spotify` is an episode id, `file` is a path on this machine, `url` is
|
||||
anything else somebody handed us. `key` is stable — it is what
|
||||
`Segment.external_id` is built from — so re-pointing a video at a local
|
||||
copy must *not* change it. That is why `resolve` returns a separate
|
||||
playback ref instead of rewriting this one.
|
||||
"""
|
||||
|
||||
kind: str
|
||||
key: str
|
||||
|
||||
@property
|
||||
def playable_locally(self) -> bool:
|
||||
return self.kind == "file" and PurePath(self.key).suffix.lower() in PLAYABLE
|
||||
|
||||
def at(self, seconds: float = 0) -> str:
|
||||
"""A link that opens this video at that moment.
|
||||
|
||||
The media fragment (`#t=`) is the same idea in every case: it is what
|
||||
YouTube's `&t=` and Spotify's `?t=` do, what a `<video>` element
|
||||
honours natively, and what most desktop players understand from a
|
||||
`file://` URL.
|
||||
"""
|
||||
offset = max(0, int(seconds))
|
||||
if self.kind == "youtube":
|
||||
return f"https://www.youtube.com/watch?v={self.key}&t={offset}s"
|
||||
if self.kind == "spotify":
|
||||
return f"https://open.spotify.com/episode/{self.key}?t={offset}"
|
||||
if self.kind == "file":
|
||||
return Path(self.key).absolute().as_uri() + (f"#t={offset}" if offset else "")
|
||||
return self.key + (f"#t={offset}" if offset else "")
|
||||
|
||||
def embed(self, seconds: float = 0) -> str:
|
||||
"""The player a reading pane can frame at that moment, or "".
|
||||
|
||||
Only the hosts that publish a player built for framing have one: a
|
||||
YouTube watch page and a Spotify episode page both refuse to be
|
||||
framed, and each has an `/embed/` form that does not. A local file
|
||||
needs none — the pane plays it in a `<video>` element — and an
|
||||
arbitrary URL is not known to allow it.
|
||||
"""
|
||||
offset = max(0, int(seconds))
|
||||
if self.kind == "youtube":
|
||||
return f"https://www.youtube.com/embed/{self.key}?start={offset}"
|
||||
if self.kind == "spotify":
|
||||
return f"https://open.spotify.com/embed/episode/{self.key}?t={offset}"
|
||||
return ""
|
||||
|
||||
|
||||
def youtube(video_id: str) -> MediaRef:
|
||||
return MediaRef(kind="youtube", key=video_id)
|
||||
|
||||
|
||||
def local(path: str | Path) -> MediaRef:
|
||||
return MediaRef(kind="file", key=str(path))
|
||||
|
||||
|
||||
def resolve(ref: MediaRef, overrides: dict[str, str] | None = None) -> MediaRef:
|
||||
"""The ref to *play*, after applying the local-media map.
|
||||
|
||||
Keyed on the original key, so a mapping written once keeps working however
|
||||
the item was ingested. Returns the ref unchanged when there is no override
|
||||
or the file has since gone — a stale mapping should fall back to the
|
||||
original rather than hand out a link to a file that isn't there.
|
||||
"""
|
||||
if not overrides:
|
||||
return ref
|
||||
replacement = overrides.get(ref.key)
|
||||
if not replacement:
|
||||
return ref
|
||||
path = Path(replacement).expanduser()
|
||||
if not path.exists():
|
||||
return ref
|
||||
return local(path)
|
||||
@@ -1,261 +0,0 @@
|
||||
"""A video, and the pieces it breaks into.
|
||||
|
||||
This is the generic half and it knows nothing about podcasts, feeds or
|
||||
newsdesk's tables. A `Video` is something with a transcript and a duration; a
|
||||
`Segment` is a span of one, with a title, a start, an end and the words spoken
|
||||
inside it. Cutting one into the other needs two inputs and no model:
|
||||
|
||||
Video --+-- transcript -> timed cues -----+
|
||||
| +-> cut() -> Segment[]
|
||||
+-- chapter marks (marks.best) ----+
|
||||
|
||||
What supplies those inputs is somebody else's problem, and deliberately so.
|
||||
YouTube supplies both today. A show's own web page supplies better marks, with
|
||||
the article links attached — that is what `desk/sources/podcast.py` does,
|
||||
by passing its own marks in. A local recording plus a Whisper transcript would
|
||||
supply both without touching this file.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from desk.models import ItemLink
|
||||
from desk.tools.video import marks as marks_mod
|
||||
from desk.tools.video import youtube
|
||||
from desk.tools.video.marks import Chapter, Link
|
||||
from desk.tools.video.media import MediaRef
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Video:
|
||||
"""One video, before anything has been done to it."""
|
||||
|
||||
ref: MediaRef
|
||||
title: str = ""
|
||||
#: The page it was published on, when that is a different thing from the
|
||||
#: video itself — a Substack post, a show's episode page. Empty for a video
|
||||
#: that is only a video.
|
||||
url: str = ""
|
||||
author: str = ""
|
||||
published_at: datetime = field(default_factory=lambda: datetime.now(UTC))
|
||||
duration: float = 0.0
|
||||
#: Whatever text came with it and might carry chapter marks: a YouTube
|
||||
#: description, a feed body, show notes as HTML.
|
||||
notes: str = ""
|
||||
|
||||
@property
|
||||
def key(self) -> str:
|
||||
"""What identifies this video for the life of the archive.
|
||||
|
||||
The media *ref's* key, not a URL, and never the resolved playback
|
||||
location — pointing an item at a local copy must not change its
|
||||
identity or every segment of that video re-ingests as a new item.
|
||||
"""
|
||||
return self.ref.key
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Segment:
|
||||
"""One span of a video: the unit that becomes a newsdesk item."""
|
||||
|
||||
video: Video
|
||||
index: int
|
||||
start: int
|
||||
end: int
|
||||
title: str
|
||||
heading: str = "" # the rubric it was filed under
|
||||
links: list[Link] = field(default_factory=list)
|
||||
transcript: str = "" # verbatim, this span only
|
||||
|
||||
@property
|
||||
def external_id(self) -> str:
|
||||
"""Stable across re-ingests of the same video.
|
||||
|
||||
Keyed on the video and the start time rather than the position in the
|
||||
list, so a show that inserts a chapter it forgot doesn't renumber —
|
||||
and therefore doesn't duplicate — everything after it. Re-cutting
|
||||
against corrected timestamps *does* produce new ids, which is the
|
||||
honest outcome: they are different spans of the recording.
|
||||
"""
|
||||
return f"{self.video.key}#t={self.start}"
|
||||
|
||||
def url(self, overrides: dict[str, str] | None = None) -> str:
|
||||
"""A link that plays this segment, wherever the video currently lives.
|
||||
|
||||
Resolved on the way out rather than stored, which is the whole point of
|
||||
`media.MediaRef`: mapping a YouTube id to a file on disk changes every
|
||||
link in the archive and rewrites nothing.
|
||||
"""
|
||||
from desk.tools.video import media
|
||||
return media.resolve(self.video.ref, overrides).at(self.start)
|
||||
|
||||
@property
|
||||
def duration(self) -> int:
|
||||
return max(0, self.end - self.start)
|
||||
|
||||
@property
|
||||
def timecode(self) -> str:
|
||||
return marks_mod.hhmmss(self.start)
|
||||
|
||||
def record(self) -> dict[str, Any]:
|
||||
"""The segment as a JSON-able dict: what newsdesk stores in `items.raw`.
|
||||
|
||||
One definition of the layout, so the two writers (a podcast segment,
|
||||
a video added by hand) and the readers (the reading pane, the
|
||||
summariser's query) agree on the keys. `media` is the reference and
|
||||
never a URL — see `media.py`; `duration` is denormalised so a queue row
|
||||
can show a length chip without arithmetic across two JSON extractions.
|
||||
|
||||
The articles are not in here: they are a typed collection of their own
|
||||
now — see `reading`. `raw` is what no shared field has a name for, and
|
||||
a link the whole reader understands is not that.
|
||||
"""
|
||||
return {
|
||||
"media": {"kind": self.video.ref.kind, "key": self.video.ref.key},
|
||||
"start": self.start,
|
||||
"end": self.end,
|
||||
"duration": self.duration,
|
||||
"index": self.index,
|
||||
}
|
||||
|
||||
def reading(self) -> list[ItemLink]:
|
||||
"""What the hosts were reading, as links the whole app understands.
|
||||
|
||||
The outline's links with a role on them, which is the difference
|
||||
between "a URL this show happened to print" and "the article this
|
||||
stretch is about": the reader lists them under the player, the
|
||||
summariser names them in the prompt, and neither has to know a podcast
|
||||
produced them.
|
||||
|
||||
A line of show notes is whatever an editor typed, so a relative href
|
||||
or a `mailto:` is an ordinary outcome rather than a fault — dropped
|
||||
with a note, because one malformed link in the outline must not cost
|
||||
the segment it sits on.
|
||||
"""
|
||||
out = []
|
||||
for link in self.links:
|
||||
try:
|
||||
out.append(ItemLink(url=link.url, title=link.title, role="reading"))
|
||||
except ValueError:
|
||||
log.debug("%s: dropping unusable outline link %r",
|
||||
self.external_id, link.url[:120])
|
||||
return out
|
||||
|
||||
|
||||
def cut(video: Video, cues: list[youtube.Cue],
|
||||
chapters: list[Chapter] | None = None) -> list[Segment]:
|
||||
"""Cut a video into one segment per chapter mark.
|
||||
|
||||
A chapter runs until the next one starts; the last one runs to the end of
|
||||
the recording. That is the only boundary rule, and it is exact, because the
|
||||
marks were written by the people who recorded it.
|
||||
|
||||
With no chapter marks at all the video comes back as a single segment
|
||||
holding the whole transcript. That is deliberately not a failure: an
|
||||
unsegmented video is still worth having, and inventing boundaries where
|
||||
nobody published any is a job for a model and a later stage, not for a
|
||||
slicing function.
|
||||
"""
|
||||
chapters = chapters if chapters is not None else marks_mod.parse(video.notes)
|
||||
end_of_video = int(video.duration or (cues[-1].end if cues else 0))
|
||||
|
||||
if not chapters:
|
||||
return [Segment(
|
||||
video=video, index=0, start=0, end=end_of_video,
|
||||
title=video.title, transcript=youtube.text_between(cues, 0),
|
||||
)]
|
||||
|
||||
ordered = sorted(chapters, key=lambda c: c.start)
|
||||
out = []
|
||||
for index, chapter in enumerate(ordered):
|
||||
following = (ordered[index + 1].start if index + 1 < len(ordered)
|
||||
else end_of_video)
|
||||
# A show that mistypes a timestamp can put one mark after the next.
|
||||
# Clamping keeps the slice empty rather than reversed, and leaves the
|
||||
# neighbouring segments intact.
|
||||
end = max(chapter.start, following)
|
||||
out.append(Segment(
|
||||
video=video,
|
||||
index=index,
|
||||
start=chapter.start,
|
||||
end=end,
|
||||
title=chapter.title,
|
||||
heading=chapter.heading,
|
||||
links=chapter.links,
|
||||
transcript=youtube.text_between(cues, chapter.start, end or None),
|
||||
))
|
||||
|
||||
empty = sum(1 for s in out if not s.transcript)
|
||||
if empty:
|
||||
log.warning("%s: %d of %d segments have no transcript under them",
|
||||
video.key, empty, len(out))
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------ transcript as text
|
||||
|
||||
_SENTENCE_END = re.compile(r"(?<=[.!?])\s+")
|
||||
|
||||
|
||||
def opening(transcript: str, sentences: int = 2, cap: int = 400) -> str:
|
||||
"""The first sentence or two, as a stand-in for a summary that isn't
|
||||
written yet. An empty `text` is an empty row in the queue."""
|
||||
if not transcript:
|
||||
return ""
|
||||
parts = _SENTENCE_END.split(transcript.strip())
|
||||
return " ".join(parts[:sentences])[:cap]
|
||||
|
||||
|
||||
def paragraphs(text: str, target: int = 700) -> str:
|
||||
"""Break a captions stream into paragraphs a reading pane can render.
|
||||
|
||||
Captions arrive as one unbroken run, and the reader splits a body on blank
|
||||
lines — so without this a forty-minute talk is one paragraph. Sentences
|
||||
are grouped until roughly `target` characters, then a break: arbitrary,
|
||||
but readable, and the alternative is a wall.
|
||||
|
||||
Auto-generated captions often have no punctuation at all, so one
|
||||
"sentence" can be the whole talk. Those are broken on a word boundary
|
||||
instead — worse than a real sentence end, better than the wall.
|
||||
"""
|
||||
text = " ".join((text or "").split())
|
||||
if not text:
|
||||
return ""
|
||||
out: list[str] = []
|
||||
current: list[str] = []
|
||||
length = 0
|
||||
for sentence in _SENTENCE_END.split(text):
|
||||
for piece in _words(sentence, target * 2):
|
||||
current.append(piece)
|
||||
length += len(piece) + 1
|
||||
if length >= target:
|
||||
out.append(" ".join(current))
|
||||
current, length = [], 0
|
||||
if current:
|
||||
out.append(" ".join(current))
|
||||
return "\n\n".join(out)
|
||||
|
||||
|
||||
def _words(run: str, limit: int) -> list[str]:
|
||||
"""Split an over-long run at spaces, keeping every piece under `limit`."""
|
||||
if len(run) <= limit:
|
||||
return [run]
|
||||
pieces: list[str] = []
|
||||
current: list[str] = []
|
||||
length = 0
|
||||
for word in run.split(" "):
|
||||
if current and length + len(word) + 1 > limit:
|
||||
pieces.append(" ".join(current))
|
||||
current, length = [], 0
|
||||
current.append(word)
|
||||
length += len(word) + 1
|
||||
if current:
|
||||
pieces.append(" ".join(current))
|
||||
return pieces
|
||||
@@ -1,600 +0,0 @@
|
||||
"""Captions, with their timings, for one YouTube video.
|
||||
|
||||
YouTube has always published a machine-readable transcript for anything with
|
||||
captions — including the auto-generated ones, which is most podcasts. It is the
|
||||
cheapest transcript in existence: somebody else already paid for the ASR.
|
||||
|
||||
Getting at it is the fiddly part, and it is fiddly for a reason worth writing
|
||||
down. The caption URLs embedded in a `watch` page are signed for a *browser*
|
||||
session, and fetching one from anywhere else now returns `200 OK` with an empty
|
||||
body — no error, no redirect, nothing to catch. So the obvious implementation
|
||||
(scrape the watch page, follow `baseUrl`) fails silently, which is the worst
|
||||
way for an ingest step to fail.
|
||||
|
||||
What works is asking the same endpoint the mobile app asks. `/youtubei/v1/player`
|
||||
with a mobile client context hands back a player response whose caption URLs are
|
||||
not tied to a browser session, and those fetch fine. That is the whole trick,
|
||||
and it is why this module talks to an undocumented endpoint instead of scraping
|
||||
a page: the scrape is the fragile one.
|
||||
|
||||
Two clients are tried in order, because they fail differently — `IOS` returns
|
||||
JSON3 when asked and `ANDROID` returns SRV3 whatever you ask for, so both
|
||||
formats are parsed. If Google breaks both, the fallbacks are `yt-dlp`, which
|
||||
tracks this arms race full-time, and transcribing the audio yourself. Neither
|
||||
is in here — `__init__.py` says where they would go. Until one of them is
|
||||
written, a break in both clients raises `TranscriptUnavailable` rather than
|
||||
storing an empty transcript: loudly, which is the whole point of the paragraph
|
||||
above.
|
||||
|
||||
**A refusal is not an absence.** YouTube gates by address: from a VPN exit or
|
||||
a datacentre range a client is answered `LOGIN_REQUIRED` ("Sign in to confirm
|
||||
you're not a bot") for every video, captioned or not. Measured on 2026-09-21,
|
||||
one machine, one video, seven clients, each asked bare and again with the
|
||||
`visitorData` a watch page hands out:
|
||||
|
||||
VPN on `IOS`, `ANDROID`, `WEB`, `MWEB`, `TVHTML5` and `ANDROID_VR` all
|
||||
get the bot check, and `WEB_EMBEDDED_PLAYER` an error. No tracks
|
||||
from any of them, and `visitorData` changes none of it.
|
||||
`youtube-transcript-api` raises `RequestBlocked`.
|
||||
VPN off `IOS` and `ANDROID` return tracks that fetch. The rest are no
|
||||
use even here: `WEB`, `MWEB` and `WEB_EMBEDDED_PLAYER` are
|
||||
unplayable without a PO token, and `TVHTML5` and `ANDROID_VR`
|
||||
are answered with the bot check from a clean address too.
|
||||
|
||||
So there is no anonymous client that gets past the gate and none to add: the
|
||||
fix is on the network. That case raises `Blocked`, a `TranscriptUnavailable` a
|
||||
caller can tell apart, because "wait for the captions" is the wrong answer to
|
||||
it and would be given for ever.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import xml.etree.ElementTree as ET
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
from desk.tools.http import session
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
INNERTUBE = "https://www.youtube.com/youtubei/v1/player?prettyPrint=false"
|
||||
|
||||
#: Only for the watch page, which is read for comments. A caption fetch
|
||||
#: deliberately identifies as a mobile app instead — see CLIENTS.
|
||||
BROWSER_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
|
||||
|
||||
#: Tried in order. Each is (client context, the User-Agent that goes with it) —
|
||||
#: they have to match, or the response comes back playable but captionless.
|
||||
CLIENTS = (
|
||||
(
|
||||
{"clientName": "IOS", "clientVersion": "20.10.4",
|
||||
"deviceModel": "iPhone16,2", "hl": "en"},
|
||||
"com.google.ios.youtube/20.10.4 (iPhone16,2; U; CPU iOS 18_3_2 like Mac OS X)",
|
||||
),
|
||||
(
|
||||
{"clientName": "ANDROID", "clientVersion": "20.10.38",
|
||||
"androidSdkVersion": 30, "hl": "en"},
|
||||
"com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip",
|
||||
),
|
||||
)
|
||||
|
||||
_VIDEO_ID = re.compile(r"^[\w-]{11}$")
|
||||
#: Every shape a YouTube link comes in, including the one Substack embeds
|
||||
#: (`<div class="youtube-wrap" id="youtube2-VIDEOID">`).
|
||||
_IN_TEXT = re.compile(
|
||||
r"(?:youtu\.be/|youtube\.com/(?:watch\?(?:[^\"'\s]*&)?v=|embed/|live/|shorts/)"
|
||||
r"|youtube2-)([\w-]{11})"
|
||||
)
|
||||
|
||||
|
||||
class TranscriptUnavailable(Exception):
|
||||
"""No captions to be had — yet, or at all.
|
||||
|
||||
Raised rather than returning `[]` because the two cases an ingest has to
|
||||
tell apart both look like an empty list: a video with no captions, and a
|
||||
video whose captions are still being generated. The message says which,
|
||||
and the caller decides whether to retry on the next pull.
|
||||
"""
|
||||
|
||||
|
||||
class Blocked(TranscriptUnavailable):
|
||||
"""YouTube refused this machine, whatever the video has.
|
||||
|
||||
A subclass, so a caller that only wants "no captions, carry on" — a link
|
||||
added by hand, a repair — is unchanged. A caller that waits for captions
|
||||
has to catch this first: a refusal is not lifted by waiting, it answers
|
||||
every video the same way, and retrying it quietly is how a show stores
|
||||
nothing for a fortnight with nobody told. See `_refused`.
|
||||
"""
|
||||
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Cue:
|
||||
"""One caption line: when it starts, how long it lasts, what was said."""
|
||||
|
||||
start: float
|
||||
duration: float
|
||||
text: str
|
||||
|
||||
@property
|
||||
def end(self) -> float:
|
||||
return self.start + self.duration
|
||||
|
||||
|
||||
def video_id(value: str) -> str:
|
||||
"""The 11-character id, from a bare id, any YouTube URL, or a blob of HTML.
|
||||
|
||||
Deliberately tolerant about its input: the id arrives from three different
|
||||
places in this pipeline — typed by hand, in a feed entry's link, and buried
|
||||
in the HTML of a Substack post — and each of them would otherwise need its
|
||||
own extraction.
|
||||
"""
|
||||
value = (value or "").strip()
|
||||
if _VIDEO_ID.match(value):
|
||||
return value
|
||||
if match := _IN_TEXT.search(value):
|
||||
return match.group(1)
|
||||
# A URL shape the pattern above doesn't cover, but with ?v= in the query.
|
||||
found = parse_qs(urlsplit(value).query).get("v")
|
||||
if found and _VIDEO_ID.match(found[0]):
|
||||
return found[0]
|
||||
raise ValueError(f"no YouTube video id in {value[:120]!r}")
|
||||
|
||||
|
||||
def transcript(video: str, *, language: str = "en",
|
||||
client: httpx.Client | None = None) -> list[Cue]:
|
||||
"""Every caption cue for one video, in order.
|
||||
|
||||
`language` is matched as a prefix, so "en" accepts "en-US" — which matters,
|
||||
because a channel that uploads a real transcript files it under a regional
|
||||
code while the auto-generated one is plain "en".
|
||||
|
||||
Two backends, tried in that order. `youtube-transcript-api` first — it is
|
||||
a dependency, and it is somebody's full-time job to keep up with YouTube's
|
||||
changes. The built-in fetcher below when the library can't get the track,
|
||||
so one broken path doesn't cost the transcript.
|
||||
"""
|
||||
cues = _via_library(video, language)
|
||||
if cues is not None:
|
||||
return cues
|
||||
return _via_innertube(video, language, client)
|
||||
|
||||
|
||||
#: Seconds one `requests` call inside `youtube-transcript-api` gets. Roughly
|
||||
#: what `desk/tools/http.py` gives an httpx call, for the same reason.
|
||||
LIBRARY_TIMEOUT = 30
|
||||
|
||||
|
||||
def _timed_client() -> Any:
|
||||
"""A `requests` session with a deadline on it, for a library that owns its HTTP.
|
||||
|
||||
`youtube-transcript-api` does its own fetching, on `requests`, and a
|
||||
`requests` call with no `timeout=` waits on a socket read for ever. What
|
||||
that costs is not a slow transcript. The work runs in a worker thread;
|
||||
cancellation is cooperative, so a thread parked in a read is one the Kill
|
||||
button cannot reach, and a restart re-queues the job to park on the same
|
||||
video. The library takes a session of ours through `http_client`, which is
|
||||
its own supported way of being told this; a `requests.Session` has no
|
||||
timeout of its own to set, so the default goes on one method down.
|
||||
|
||||
**This is why there is no `socket.setdefaulttimeout` anywhere.** A
|
||||
process-wide default is the tempting one-liner for library code that hides
|
||||
its HTTP, and it is the wrong trade here: it would be inherited by every
|
||||
socket the process opens, uvicorn's listener and each websocket included,
|
||||
to buy a deadline for call sites that can each be told one directly. There
|
||||
were two. The other was feedparser, and it is handled the same way — by
|
||||
fetching where a timeout can be set; see `desk/sources/rss.py`.
|
||||
|
||||
Typed as `Any` because `requests` is only imported inside the guard below.
|
||||
It is a `requests.Session`.
|
||||
"""
|
||||
import requests
|
||||
|
||||
class Timed(requests.Session):
|
||||
def request(self, *args: Any, **kwargs: Any) -> Any:
|
||||
kwargs.setdefault("timeout", LIBRARY_TIMEOUT)
|
||||
return super().request(*args, **kwargs)
|
||||
|
||||
return Timed()
|
||||
|
||||
|
||||
def _via_library(video: str, language: str) -> list[Cue] | None:
|
||||
"""`youtube-transcript-api`. None means "not this way".
|
||||
|
||||
Every failure is None rather than an exception, including a video with no
|
||||
captions at all: the fallback below raises the error that gets reported, so
|
||||
there is one message for "no transcript" instead of two that depend on
|
||||
which packages happen to be present.
|
||||
"""
|
||||
try:
|
||||
from youtube_transcript_api import YouTubeTranscriptApi
|
||||
|
||||
client = _timed_client()
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
try:
|
||||
tracks = YouTubeTranscriptApi(http_client=client).list(video)
|
||||
wanted = [language, f"{language}-US", f"{language}-GB"]
|
||||
try:
|
||||
track = tracks.find_manually_created_transcript(wanted)
|
||||
except Exception:
|
||||
track = tracks.find_generated_transcript(wanted)
|
||||
fetched = track.fetch()
|
||||
except Exception as exc:
|
||||
log.debug("youtube-transcript-api could not fetch %s: %s", video, exc)
|
||||
return None
|
||||
|
||||
cues = [
|
||||
Cue(start=c.start, duration=c.duration, text=clean)
|
||||
for c in fetched
|
||||
if (clean := _clean(c.text))
|
||||
]
|
||||
log.info("%s: %d cues via youtube-transcript-api", video, len(cues))
|
||||
return cues or None
|
||||
|
||||
|
||||
def _via_innertube(video: str, language: str,
|
||||
client: httpx.Client | None) -> list[Cue]:
|
||||
with session(client) as http:
|
||||
player = _player(video, http)
|
||||
track = _pick(_tracks(player), language)
|
||||
if track is None:
|
||||
raise TranscriptUnavailable(
|
||||
f"{video}: no {language} captions. "
|
||||
"Auto-captions can take a few hours to appear on a long upload."
|
||||
)
|
||||
# `&fmt=json3` is a request, not a promise: the ANDROID client answers
|
||||
# it with SRV3 anyway. Both are parsed, so it doesn't matter which
|
||||
# arrives — see `_cues`.
|
||||
response = http.get(track["baseUrl"] + "&fmt=json3",
|
||||
headers={"User-Agent": player.get("_caption_agent", BROWSER_UA)})
|
||||
if _throttled(response):
|
||||
raise Blocked(_refusal(video, [f"captions: HTTP {response.status_code}"]))
|
||||
response.raise_for_status()
|
||||
body = response.text
|
||||
if not body.strip():
|
||||
raise TranscriptUnavailable(
|
||||
f"{video}: caption track fetched empty. YouTube does this when "
|
||||
"the request doesn't look like the client that asked for it."
|
||||
)
|
||||
cues = _cues(body)
|
||||
log.info("%s: %d cues, %.0f minutes", video, len(cues),
|
||||
(cues[-1].end / 60) if cues else 0)
|
||||
return cues
|
||||
|
||||
|
||||
def details(video: str, *, client: httpx.Client | None = None) -> dict[str, Any]:
|
||||
"""Title, author, length and description for a video.
|
||||
|
||||
The description is here because for a show that publishes no newsletter
|
||||
it is the only place the chapter marks live — `marks.parse` reads it as
|
||||
plain text. Everything else is the metadata an episode needs when YouTube
|
||||
is the only source there is.
|
||||
"""
|
||||
with session(client) as http:
|
||||
player = _player(video, http, require_captions=False)
|
||||
video_details = player.get("videoDetails", {})
|
||||
return {
|
||||
"title": video_details.get("title", ""),
|
||||
"author": video_details.get("author", ""),
|
||||
"duration": float(video_details.get("lengthSeconds") or 0),
|
||||
"description": video_details.get("shortDescription", ""),
|
||||
}
|
||||
|
||||
|
||||
def comments(video: str, *, limit: int = 100,
|
||||
client: httpx.Client | None = None) -> list[str]:
|
||||
"""Top-level comments, newest-and-most-liked first, as plain text.
|
||||
|
||||
Here for one reason: when a show does not publish an outline, somebody in
|
||||
the comments usually has. A viewer typing out "0:00 intro / 3:32 Gemini /
|
||||
13:11 Grok" is doing the same work the show notes would have done, and
|
||||
YouTube itself treats such a comment as authoritative enough to draw
|
||||
chapter markers from it. `marks.parse` reads these exactly as it reads a
|
||||
description — see `marks.best`.
|
||||
|
||||
Returns `[]` rather than raising when comments are off, or when YouTube
|
||||
changes the shape again. A missing fallback is not worth failing an ingest
|
||||
that already has a transcript.
|
||||
"""
|
||||
try:
|
||||
with session(client) as http:
|
||||
page = http.get(f"https://www.youtube.com/watch?v={video}",
|
||||
headers={"User-Agent": BROWSER_UA,
|
||||
"Accept-Language": "en-US,en"}).text
|
||||
context = _json_after(page, '"INNERTUBE_CONTEXT":')
|
||||
initial = _json_after(page, "ytInitialData = ")
|
||||
if not context or not initial:
|
||||
return []
|
||||
out: list[str] = []
|
||||
# Several continuations live in the page (the related rail, the
|
||||
# description); only one of them is the comment feed, and which
|
||||
# index that is moves around. Trying them in order and keeping the
|
||||
# first that yields comments is cheaper than pinning a path
|
||||
# through a structure Google reshuffles.
|
||||
for token in _continuations(initial)[:4]:
|
||||
response = http.post(
|
||||
"https://www.youtube.com/youtubei/v1/next?prettyPrint=false",
|
||||
json={"context": context, "continuation": token},
|
||||
headers={"Content-Type": "application/json",
|
||||
"User-Agent": BROWSER_UA},
|
||||
)
|
||||
out = _comment_texts(response.json())
|
||||
if out:
|
||||
break
|
||||
except Exception as exc:
|
||||
log.debug("could not read comments for %s: %s", video, exc)
|
||||
return []
|
||||
log.info("%s: %d comments read", video, len(out))
|
||||
return out[:limit]
|
||||
|
||||
|
||||
def text_between(cues: list[Cue], start: float, end: float | None = None) -> str:
|
||||
"""The words spoken in a span, as one paragraph.
|
||||
|
||||
A cue counts as inside the span when it *starts* inside it. Overlap would
|
||||
be the other defensible rule, but it double-counts the sentence that
|
||||
straddles a boundary — and these boundaries come from chapter marks a human
|
||||
typed, so a sentence of drift at the seam is already there and doesn't need
|
||||
doubling.
|
||||
"""
|
||||
return " ".join(
|
||||
cue.text for cue in cues
|
||||
if cue.start >= start and (end is None or cue.start < end)
|
||||
).strip()
|
||||
|
||||
|
||||
# ------------------------------------------------- internals: the watch page
|
||||
|
||||
def _json_after(page: str, marker: str) -> dict[str, Any] | None:
|
||||
"""The JSON object that follows `marker` in a watch page.
|
||||
|
||||
Brace-matched rather than matched with a regex. The two blobs this reads —
|
||||
`ytInitialData` and `INNERTUBE_CONTEXT` — are hundreds of kilobytes of
|
||||
nested JSON containing every bracket character inside string values, and a
|
||||
non-greedy regex stops at the first `}` inside a caption or a title. This
|
||||
walks the string properly, which is about fifteen lines and always right.
|
||||
"""
|
||||
start = page.find(marker)
|
||||
if start < 0:
|
||||
return None
|
||||
start = page.find("{", start + len(marker))
|
||||
if start < 0:
|
||||
return None
|
||||
|
||||
depth, in_string, escaped = 0, False, False
|
||||
for index in range(start, len(page)):
|
||||
char = page[index]
|
||||
if in_string:
|
||||
if escaped:
|
||||
escaped = False
|
||||
elif char == "\\":
|
||||
escaped = True
|
||||
elif char == '"':
|
||||
in_string = False
|
||||
continue
|
||||
if char == '"':
|
||||
in_string = True
|
||||
elif char == "{":
|
||||
depth += 1
|
||||
elif char == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
try:
|
||||
return json.loads(page[start:index + 1])
|
||||
except ValueError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _walk(node: Any, key: str) -> Iterator[Any]:
|
||||
"""Every value stored under `key`, anywhere in a nested structure."""
|
||||
if isinstance(node, dict):
|
||||
if key in node:
|
||||
yield node[key]
|
||||
for value in node.values():
|
||||
yield from _walk(value, key)
|
||||
elif isinstance(node, list):
|
||||
for value in node:
|
||||
yield from _walk(value, key)
|
||||
|
||||
|
||||
def _continuations(initial: dict[str, Any]) -> list[str]:
|
||||
return [
|
||||
token for renderer in _walk(initial, "continuationItemRenderer")
|
||||
if (token := (renderer.get("continuationEndpoint", {})
|
||||
.get("continuationCommand", {}) or {}).get("token"))
|
||||
]
|
||||
|
||||
|
||||
def _comment_texts(response: dict[str, Any]) -> list[str]:
|
||||
"""Comment bodies out of a `next` response.
|
||||
|
||||
Comments arrive as `commentEntityPayload` entities in a flat batch beside
|
||||
the renderers that position them, so reading them needs no understanding of
|
||||
the layout at all — which is the part that keeps changing.
|
||||
"""
|
||||
out = []
|
||||
for payload in _walk(response, "commentEntityPayload"):
|
||||
text = (payload.get("properties", {}).get("content", {}) or {}).get("content", "")
|
||||
if text:
|
||||
out.append(text)
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ internals
|
||||
|
||||
def _player(video: str, http: httpx.Client,
|
||||
*, require_captions: bool = True) -> dict[str, Any]:
|
||||
"""The player response, from whichever client answers with captions.
|
||||
|
||||
`require_captions` is what separates the two callers. A transcript wants
|
||||
the first client that actually offers caption tracks; metadata wants the
|
||||
first that answers at all, and would otherwise fail on a video that plays
|
||||
fine and simply has no captions yet.
|
||||
|
||||
Every client's answer is kept for the error, not only the last one's: the
|
||||
two fail differently, and which of them said what is the first thing
|
||||
anyone asks when this breaks.
|
||||
"""
|
||||
said: list[str] = []
|
||||
refused = False
|
||||
for context, agent in CLIENTS:
|
||||
name = context["clientName"]
|
||||
payload = {"context": {"client": context}, "videoId": video,
|
||||
"contentCheckOk": True, "racyCheckOk": True}
|
||||
try:
|
||||
response = http.post(
|
||||
INNERTUBE, json=payload,
|
||||
headers={"Content-Type": "application/json", "User-Agent": agent},
|
||||
)
|
||||
if _throttled(response):
|
||||
refused = True
|
||||
said.append(f"{name}: HTTP {response.status_code}")
|
||||
continue
|
||||
data = response.json()
|
||||
except (httpx.HTTPError, ValueError) as exc:
|
||||
said.append(f"{name}: {exc}")
|
||||
continue
|
||||
playability = data.get("playabilityStatus", {})
|
||||
status = playability.get("status", "?")
|
||||
if _tracks(data) or (not require_captions and status == "OK"):
|
||||
# Caption URLs are bound to the client that obtained the player.
|
||||
data["_caption_agent"] = agent
|
||||
return data
|
||||
refused = refused or _refused(playability)
|
||||
reason = playability.get("reason") or ""
|
||||
said.append(f"{name}: {status}" + (f" ({reason})" if reason else "")
|
||||
+ ", no caption tracks")
|
||||
log.debug("innertube %s", said[-1])
|
||||
if refused:
|
||||
raise Blocked(_refusal(video, said))
|
||||
raise TranscriptUnavailable(
|
||||
f"{video}: {'; '.join(said) or 'no client returned captions'}")
|
||||
|
||||
|
||||
def _refused(playability: dict[str, Any]) -> bool:
|
||||
"""Whether a player response is YouTube declining to serve this machine.
|
||||
|
||||
The status alone, not the reason beside it. The bot check reads "Sign in
|
||||
to confirm you're not a bot" today, and a match on that sentence would go
|
||||
back to reporting a refusal as captions not being ready on the day it is
|
||||
reworded — silently, which is the failure this exists to end. The status
|
||||
is the stable half, and it is sufficient: nothing asked anonymously is
|
||||
ever let past `LOGIN_REQUIRED` by being asked again later.
|
||||
|
||||
That also takes in a private or age-gated video, where the refusal is
|
||||
about the video rather than the address. It is still not a wait, and the
|
||||
reason is quoted in the error for whoever reads it.
|
||||
"""
|
||||
return playability.get("status") == "LOGIN_REQUIRED"
|
||||
|
||||
|
||||
def _throttled(response: httpx.Response) -> bool:
|
||||
"""The same refusal, made in HTTP: a 403, a 429, or Google's `/sorry/`
|
||||
interstitial, which arrives as a redirect to a captcha page."""
|
||||
return response.status_code in (403, 429) or "/sorry/" in response.url.path
|
||||
|
||||
|
||||
def _refusal(video: str, said: list[str]) -> str:
|
||||
"""`Blocked`'s sentence. "YouTube refused" is what `desk/notices.py` reads
|
||||
to tell this from a feed that failed, so the wording is load-bearing."""
|
||||
return (f"{video}: YouTube refused this address and wants a sign-in — "
|
||||
+ "; ".join(said))
|
||||
|
||||
|
||||
def _tracks(player: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
return (player.get("captions", {})
|
||||
.get("playerCaptionsTracklistRenderer", {})
|
||||
.get("captionTracks", []) or [])
|
||||
|
||||
|
||||
def _pick(tracks: list[dict[str, Any]], language: str) -> dict[str, Any] | None:
|
||||
"""Best track for a language: a real transcript over an auto-generated one.
|
||||
|
||||
A channel that uploads its own captions has punctuation, speaker turns and
|
||||
correctly spelled names; ASR has none of those and spells "Qwen" six ways.
|
||||
Worth preferring, and never worth requiring — most shows only have ASR.
|
||||
"""
|
||||
wanted = [t for t in tracks
|
||||
if (t.get("languageCode") or "").lower().startswith(language.lower())]
|
||||
wanted.sort(key=lambda t: t.get("kind") == "asr")
|
||||
return wanted[0] if wanted else None
|
||||
|
||||
|
||||
def _cues(body: str) -> list[Cue]:
|
||||
"""Parse whichever of YouTube's three caption formats turned up."""
|
||||
body = body.strip()
|
||||
if body.startswith("{"):
|
||||
return _from_json3(json.loads(body))
|
||||
root = ET.fromstring(body)
|
||||
# SRV1 is `transcript > text[start,dur]`; SRV3 is `timedtext > body > p[t,d]`.
|
||||
if root.tag == "transcript":
|
||||
return _from_srv1(root)
|
||||
return _from_srv3(root)
|
||||
|
||||
|
||||
def _from_json3(data: dict[str, Any]) -> list[Cue]:
|
||||
cues = []
|
||||
for event in data.get("events", []):
|
||||
text = "".join(seg.get("utf8", "") for seg in event.get("segs", []))
|
||||
clean = _clean(text)
|
||||
if not clean:
|
||||
continue
|
||||
cues.append(Cue(
|
||||
start=event.get("tStartMs", 0) / 1000,
|
||||
duration=event.get("dDurationMs", 0) / 1000,
|
||||
text=clean,
|
||||
))
|
||||
return cues
|
||||
|
||||
|
||||
def _from_srv1(root: ET.Element) -> list[Cue]:
|
||||
cues = []
|
||||
for node in root.iter("text"):
|
||||
clean = _clean(node.text or "")
|
||||
if not clean:
|
||||
continue
|
||||
cues.append(Cue(
|
||||
start=float(node.get("start", 0)),
|
||||
duration=float(node.get("dur", 0)),
|
||||
text=clean,
|
||||
))
|
||||
return cues
|
||||
|
||||
|
||||
def _from_srv3(root: ET.Element) -> list[Cue]:
|
||||
cues = []
|
||||
for node in root.iter("p"):
|
||||
clean = _clean("".join(node.itertext()))
|
||||
if not clean:
|
||||
continue
|
||||
cues.append(Cue(
|
||||
start=int(node.get("t", 0)) / 1000,
|
||||
duration=int(node.get("d", 0)) / 1000,
|
||||
text=clean,
|
||||
))
|
||||
return cues
|
||||
|
||||
|
||||
def _clean(text: str) -> str:
|
||||
"""Entities out, whitespace collapsed, sound cues dropped.
|
||||
|
||||
Unescaped twice on purpose: SRV1 sends `&#39;` — an entity for an
|
||||
entity — because the caption text was escaped once when it was stored and
|
||||
once again on the way into the XML.
|
||||
"""
|
||||
text = html.unescape(html.unescape(text or ""))
|
||||
text = re.sub(r"\[(?:Music|Applause|Laughter)\]", " ", text)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
@@ -1,128 +0,0 @@
|
||||
"""Did we fetch the article, or the page the outlet shows instead of it?
|
||||
|
||||
Some sites answer every article URL with the same thing: a consent gate, an
|
||||
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
|
||||
faithfully — it is real text on a real page — and we store it as the body. That
|
||||
is worse than storing nothing, because the reader prefers the body to the feed's
|
||||
summary, so a wall doesn't merely fail to add anything: it *hides* the two
|
||||
readable sentences the feed already gave us.
|
||||
|
||||
The tell is repetition. Different articles cannot have the same body; a wall
|
||||
always does. That is the whole detector — no keyword list, no per-site rules,
|
||||
nothing to keep up to date. It learns golem.de's German consent page and a
|
||||
Cloudflare challenge with the same line of code, and it costs no extra
|
||||
requests, because the evidence is a by-product of a fetch we were making
|
||||
anyway.
|
||||
|
||||
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
|
||||
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
|
||||
requests to one host earns a throttle or challenge page, that page is served to
|
||||
whichever two articles asked next, and the two fingerprint the same. Refetching
|
||||
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
|
||||
was about our request rate and not about the outlet. A wall repeats on every
|
||||
article, so a third body costs one extra fetch per walled host, once; a throttle
|
||||
page that repeats three consecutive times is the site answering nothing else.
|
||||
|
||||
The verdict is per host, because the wall is: the site serves it, it isn't
|
||||
chosen per article. Once a host is known to answer with one, we stop fetching
|
||||
from it altogether — which makes pulls faster, not just more correct. That is
|
||||
also why the count matters: `known` short-circuits the request, so a wall
|
||||
verdict is the last evidence a host ever gets to give. Nothing can disprove it
|
||||
except `forget`.
|
||||
|
||||
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
|
||||
in a test), because a tool has no store of its own to remember things in. The
|
||||
one way this is wrong is an outlet that really does publish the same body
|
||||
`STRIKES` times running; the cost is that its articles fall back to their feed
|
||||
summaries until someone calls `forget`, which is what the re-check button in
|
||||
the Sources tab is for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
from collections.abc import Mapping, MutableMapping
|
||||
|
||||
from desk.tools.framing import host_of
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: Sits alongside `embed:` in the same table. The two verdicts answer different
|
||||
#: questions about a host — will it let us frame it, will it give up its text —
|
||||
#: and neither implies the other.
|
||||
CACHE_PREFIX = "body:"
|
||||
_WALL = "wall:"
|
||||
REASON = "every article comes back as the same page"
|
||||
|
||||
#: How many consecutive identical bodies settle it. See the module docstring:
|
||||
#: two was one throttle page away from retiring a working outlet for good.
|
||||
STRIKES = 3
|
||||
|
||||
|
||||
def _fingerprint(body: str) -> str:
|
||||
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _seen(value: str) -> tuple[str, int]:
|
||||
"""`(fingerprint, how many in a row)` from a stored value.
|
||||
|
||||
A bare fingerprint is one sighting, which is what every row written before
|
||||
`STRIKES` existed holds — so an upgraded database is read correctly and
|
||||
needs no migration of its own.
|
||||
"""
|
||||
mark, _, run = value.partition("*")
|
||||
return mark, (int(run) if run.isdigit() else 1)
|
||||
|
||||
|
||||
def known(cache: Mapping[str, str], url: str) -> str:
|
||||
"""Why this host's bodies aren't worth fetching, or "" if they are.
|
||||
|
||||
Asked before the request, so a settled verdict costs nothing at all. Takes
|
||||
a read-only mapping, because this one only ever reads — which lets the
|
||||
Sources tab hand over the snapshot it already loaded rather than the live
|
||||
table, and says in the signature that opening a panel changes nothing.
|
||||
"""
|
||||
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
|
||||
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
|
||||
|
||||
|
||||
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
|
||||
"""Record a body we just extracted. Returns a reason if it proves a wall.
|
||||
|
||||
Only the previous body from the host is kept and how many articles running
|
||||
have handed it over — not every body ever seen. A wall repeats on the
|
||||
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
|
||||
and one key per host beats a growing set per host.
|
||||
|
||||
One distinct body resets the run to one. That is what makes the count mean
|
||||
"in a row" rather than "ever", and it is the whole difference between a
|
||||
site that answers with a gate and a site that answered twice from a cache.
|
||||
"""
|
||||
host = host_of(url)
|
||||
if not host or not body.strip():
|
||||
return ""
|
||||
key = CACHE_PREFIX + host
|
||||
stored = cache.get(key, "")
|
||||
if stored.startswith(_WALL):
|
||||
return stored[len(_WALL):]
|
||||
mark = _fingerprint(body)
|
||||
seen, run = _seen(stored)
|
||||
run = run + 1 if seen == mark else 1
|
||||
if run >= STRIKES:
|
||||
cache[key] = _WALL + REASON
|
||||
log.info("%s serves a wall, not articles: %s (%d in a row)",
|
||||
host, REASON, run)
|
||||
return REASON
|
||||
cache[key] = mark if run == 1 else f"{mark}*{run}"
|
||||
return ""
|
||||
|
||||
|
||||
def forget(cache: MutableMapping[str, str], url: str) -> None:
|
||||
"""Drop what we learned about a host, so the next fetch decides again.
|
||||
|
||||
Outlets change their minds — a wall goes up for a quarter and comes down
|
||||
again — and a verdict with no way back would be a permanent decision made
|
||||
on two HTTP responses.
|
||||
"""
|
||||
cache.pop(CACHE_PREFIX + host_of(url), None)
|
||||
Reference in new issue
Block a user