Bring in desk from news

The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-23 12:29:17 +02:00
1 parent 9d666d0937
commit 248ea1636d
36 files changed
+8393

No files matched your search

+20
View File
@@ -0,0 +1,20 @@
"""Reusable machinery with no opinion about news.
A tool is something a pipeline stage picks up and puts down: it takes plain
inputs, returns plain outputs, and knows nothing about subscriptions, feeds,
or what a job is for. That is the test for whether a module belongs here —
if it would make sense in a different application, it is a tool.
agent.py who answers when something needs a model: the Claude API
with a key, the `claude` CLI on a subscription, or nobody
embedding.py text in, vector out
framing.py will this page load inside an iframe?
http.py one shared HTTP client per call chain
marks.py what kind of thing is this, if not an article?
video/ a recording, read as text with timings on it
walls.py was that the article, or the page shown instead of it?
Nothing is imported here on purpose. `video` pulls in httpx and the caption
parsers, `embedding` a sentence-transformer, `agent` an SDK; each is paid for
by the caller that wants it and by nobody else.
"""
+461
View File
@@ -0,0 +1,461 @@
"""Who answers when a job needs a model.
There is one question this module exists to answer — *ask Claude this, get JSON
back* — and two ways to answer it. They are interchangeable, so a handler
asks without knowing which one ran:
api One call to the Claude API. Wants ANTHROPIC_API_KEY (or an
`ant auth login` profile). No local install, no subprocess,
per-token billing.
claude_code The Claude Agent SDK drives the `claude` CLI already on this
machine, on the subscription it is already signed in with.
No key, nothing to bill. This is what the server runs on
today, authenticated by CLAUDE_CODE_OAUTH_TOKEN.
`backend()` answers `""` when neither can run, and that is a failure rather
than a third way: the caller says so with a notice naming the reason — no
key, no CLI, `backend: off` — and the batch fails without asking anybody.
Nothing waits for a person to answer it by hand.
The seam is deliberately narrow: `ask_json` takes a system prompt, a payload
and a JSON schema, and returns parsed JSON. Both backends take the *same*
schema — the Agent SDK accepts the Messages API's `json_schema` output format
and hands the parsed object back on `ResultMessage.structured_output` — so
neither path is the degraded one, and moving between them is a config line
rather than a second implementation.
Both also *stream*, and hand the text out through `on_partial` as it arrives.
Nothing here does anything with it — this module still knows only "JSON in,
JSON out" — but a batch of twenty-five articles is one request that takes the
best part of a minute, and the answers inside it are finished one at a time
long before the request is. Passing the pieces up is what lets the layer that
knows what an answer looks like count them, and a progress bar move per
article instead of per batch.
What the `claude_code` backend deliberately does not get: no tools
(`tools=[]`), no filesystem settings (`setting_sources=None`, so no CLAUDE.md,
no skills, no MCP servers leak into a tagging call), no session to resume. One
text-in, JSON-out turn — the same shape as the API call it stands in for.
`thinking` is the one knob a caller turns, and it is stated to both backends
rather than left to either. It has to be: the CLI is built for agent work and
thinks unprompted, the Messages API thinks only when asked, so "say nothing"
means opposite things on the two sides. Saying `disabled` out loud is what
makes them the interchangeable pair this module claims — and it is worth
saying, because on a batch of twenty-five headlines the CLI's default cost
128s against 17s for the same tags. What to think about is the task's business
(`ModelTaskConfig.thinking`); making both backends agree on it is this one's.
"""
from __future__ import annotations
import importlib.util
import json
import logging
import os
from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
from typing import Any
from desk.config import AgentConfig
from desk.tools import credentials
log = logging.getLogger(__name__)
#: The backends `backend()` can name, best first. `auto` walks this list.
BACKENDS = ("api", "claude_code")
#: Where the answer goes as it arrives, a piece at a time. Called with raw
#: text, not parsed anything — half a JSON object is not a JSON object, and
#: what to make of a partial answer is the caller's business, not this
#: module's.
Partial = Callable[[str], None]
class AgentError(Exception):
"""The backend was there and the call still didn't produce an answer.
Raised for every failure mode of both paths — no SDK installed, no CLI on
PATH, an API error, a timeout, a response that isn't the shape asked for.
Callers treat it as "no answer this time", which for a job means either
moving on to the next batch or handing off to a human-driven agent.
"""
@dataclass(slots=True)
class Reply:
"""One answer, and enough about where it came from to say so in the UI."""
data: dict[str, Any]
backend: str = ""
model: str = ""
#: Filled by the `claude_code` path only, which is the reverse of what you
#: would expect: the SDK's `ResultMessage` prices the turn, so a
#: subscription that is billed nothing for it still reports a figure, while
#: the API path — the one actually billed — leaves this at zero because the
#: Messages response does not price itself. An estimate of what the run
#: would have cost on the API, then, and never a bill. `usage` is the field
#: to trust if you want to price it yourself.
cost_usd: float = 0.0
usage: dict[str, Any] = field(default_factory=dict)
# ------------------------------------------------------------ which backend
def has_api_credentials() -> bool:
"""Whether the API path can run unattended.
An unset ANTHROPIC_API_KEY isn't proof there are no credentials — the SDK
also reads ANTHROPIC_AUTH_TOKEN and an `ant auth login` profile — but env
vars are the only part we can check without importing the SDK.
"""
return any(os.environ.get(name) for name in (
"ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_PROFILE",
))
def has_claude_code() -> bool:
"""Whether the Agent SDK is here, and so whether there is a CLI to drive.
The SDK bundles its own `claude`, so having the package is enough — a
binary on PATH is a bonus, not a requirement, and checking for one would
wrongly rule out a machine where the SDK is the only install.
Not proof the call will work — the CLI may be signed out — but it is the
check that can be made in microseconds, which is what a job asking "can I
do this myself?" needs. A signed-out CLI surfaces as an `AgentError` on
the call, and the caller falls back exactly as if this had returned False.
"""
return importlib.util.find_spec("claude_agent_sdk") is not None
def backend(settings: AgentConfig) -> str:
"""Which backend will answer, or `""` for nobody.
`agent.backend: auto` prefers the API when there are credentials, because
that is the unattended path and the one production will use; it falls back
to the local Claude Code subscription, which is the one that costs nothing
while you are building. Naming a backend skips the detection — including
`off`, which is how you make everything that wants a model fail.
"""
want = settings.backend
if want == "off":
return ""
if want in BACKENDS:
return want
if has_api_credentials():
return "api"
if has_claude_code():
return "claude_code"
return ""
# ------------------------------------------------------------------ the ask
def _thinking(setting: str) -> Any:
"""One `thinking` setting in the form both SDKs take.
Returns `Any` on purpose. Each SDK declares its own TypedDict for the same
two-word object — `ThinkingConfigDisabledParam` on one side,
`ThinkingConfigDisabled` on the other — and neither can be named here
without importing an SDK this module is careful to keep optional. One
honest `Any` at the seam beats a hard dependency at the top of the file.
Anything that is not a mode we offer is off, rather than an error. This is
reached from config pydantic has already constrained to the same two words,
so a third one arriving here would be a bug in our own plumbing — and the
honest answer to that is the same one the default gives, not a failed run
the model would happily have answered.
"""
return {"type": "adaptive"} if setting == "adaptive" else {"type": "disabled"}
def ask_json(
settings: AgentConfig,
*,
system: str,
question: dict[str, Any],
schema: dict[str, Any],
model: str = "",
using: str = "",
thinking: str = "off",
on_partial: Partial | None = None,
) -> Reply:
"""Ask Claude `question`, constrained to `schema`, and parse the answer.
`system` is expected to be byte-identical across the calls in a run — it is
what the API path caches. `question` is the part that changes. `using`
forces a backend for one call; otherwise `backend()` decides. `thinking` is
`off` or `adaptive`, and defaults to off because the caller who wants it is
rarer than the caller who would be surprised to be waiting for it.
`on_partial` is called with each piece of the answer as it arrives, in
order, on the thread this was called on. It is a courtesy, not a contract:
a backend that cannot stream simply never calls it, and the return value
is the same either way.
"""
name = using or backend(settings)
model = model or settings.model
if thinking == "adaptive" and model.startswith(
("claude-haiku-4-5", "claude-sonnet-4-5", "claude-opus-4-5")):
# These models support manual extended thinking only; fail before spend.
# https://platform.claude.com/docs/en/build-with-claude/extended-thinking
raise AgentError(f"{model} does not support adaptive thinking; set thinking: off "
"or choose a model supporting adaptive thinking")
if not name:
raise AgentError("no model backend is available")
if name == "api":
return _ask_api(settings, system, question, schema, model, thinking, on_partial)
if name == "claude_code":
return _ask_claude_code(settings, system, question, schema, model, thinking,
on_partial)
raise AgentError(f"unknown backend {name!r}")
def _ask_api(settings: AgentConfig, system: str, question: dict[str, Any],
schema: dict[str, Any], model: str, thinking: str = "off",
on_partial: Partial | None = None) -> Reply:
try:
import anthropic
except ImportError as exc:
raise AgentError(
"the anthropic SDK isn't installed — `uv pip install anthropic`"
) from exc
try:
client = anthropic.Anthropic(timeout=settings.timeout)
# Streamed rather than awaited whole, always — not only when someone
# passed `on_partial`. One code path is worth more than the few lines a
# non-streaming branch would save, and a minute-long request held open
# by a single response is the shape that trips idle timeouts in
# whatever sits between here and the API.
with client.messages.stream(
model=model,
max_tokens=settings.max_tokens,
system=[{
"type": "text",
"text": system,
# The instructions are byte-identical across every batch and
# every run; only the payload below changes.
"cache_control": {"type": "ephemeral"},
}],
messages=[{"role": "user", "content": _dumps(question)}],
thinking=_thinking(thinking),
output_config={"format": {"type": "json_schema", "schema": schema}},
) as stream:
for event in stream:
if on_partial is not None and (piece := _delta(event)):
on_partial(piece)
response = stream.get_final_message()
except (anthropic.APIError, TypeError) as exc:
# The status code carried through in the text on purpose: it is what
# `notices.classify` reads to tell "your key is wrong" (401) apart from
# "you are being rate limited" (429), which want different reactions
# from you and which the message body alone does not always separate.
status = getattr(exc, "status_code", "")
raise AgentError(
f"the API call failed: {f'{status} ' if status else ''}{exc}") from exc
text = next((b.text for b in response.content if b.type == "text"), "")
return Reply(
data=_parse(text),
backend="api",
model=response.model or model,
usage=response.usage.model_dump() if response.usage else {},
)
def _ask_claude_code(settings: AgentConfig, system: str, question: dict[str, Any],
schema: dict[str, Any], model: str, thinking: str = "off",
on_partial: Partial | None = None) -> Reply:
try:
from claude_agent_sdk import (
ClaudeAgentOptions, ClaudeSDKError, ResultMessage, StreamEvent, query,
)
except ImportError as exc:
raise AgentError(
"the Claude Agent SDK isn't installed — "
"`uv pip install claude-agent-sdk`"
) from exc
options = ClaudeAgentOptions(
system_prompt=system,
model=model or None,
# A classification call, not an agent: nothing to run, nothing to
# read, and no project settings — so no CLAUDE.md, no skills and no
# MCP servers get in front of the model on the way to a tag list.
tools=[],
allowed_tools=[],
permission_mode="dontAsk",
setting_sources=None,
# Always stated, never left to the CLI's default, which is on. With
# it on the progress bar also stops meaning anything: thinking emits
# nothing `_Answered` can count, so a chunk streams no `"uid"` until
# the thinking is done, and the bar sits on the previous chunk's
# number until the answer arrives in a rush.
thinking=_thinking(thinking),
output_format={"type": "json_schema", "schema": schema},
# Only when someone is listening. The partials are the same bytes as
# the final answer sent again in pieces, and asking the CLI for them
# when nothing reads them is noise on the pipe for no one's benefit.
include_partial_messages=on_partial is not None,
cli_path=settings.cli_path or None,
# `max_tokens` has to mean the same thing on both backends or it means
# nothing. The Messages API takes it as a parameter and the CLI takes
# it only as an environment variable, so without this the setting was
# quietly API-only — which is how a call configured for 8,192 tokens
# answered with 11,706 and nothing objected. The SDK merges this into
# the inherited environment rather than replacing it, so naming one
# variable does not cost the CLI its PATH.
env={"CLAUDE_CODE_MAX_OUTPUT_TOKENS": str(settings.max_tokens),
**({"CLAUDE_CODE_OAUTH_TOKEN": oauth} if (oauth := credentials.token(settings)) else {})},
stderr=lambda line: log.debug("claude: %s", line.rstrip()),
)
async def ask() -> Reply:
answer: Reply | None = None
async for message in query(prompt=_dumps(question), options=options):
if isinstance(message, StreamEvent) and on_partial is not None:
piece = _delta(message.event)
if piece:
on_partial(piece)
continue
if not isinstance(message, ResultMessage):
continue
if message.is_error:
raise AgentError(_explain(
"; ".join(message.errors or []) or message.result
or f"the agent ended as {message.subtype}"))
data = message.structured_output
if not isinstance(data, dict):
# A run that ended without structured output — a limit, an
# interruption. `result` is the raw text if there is any, and
# that is worth one attempt at parsing before giving up.
data = _parse(message.result or "")
answer = Reply(
data=data,
backend="claude_code",
model=model,
cost_usd=message.total_cost_usd or 0.0,
usage=message.usage or {},
)
if answer is None:
raise AgentError("the agent returned no result")
return answer
try:
return _sync(ask(), settings.timeout)
except ClaudeSDKError as exc:
# The SDK has its own exception hierarchy — a CLI that exited, a line
# it could not parse, an answer that overran `max_tokens` — and none of
# it is an `AgentError`. Left alone those escape the per-chunk `except
# AgentError` in `with_model`, so one bad chunk takes down a run that
# is contractually supposed to leave its rows for next time. Every
# failure of this backend is an `AgentError` or none of them are.
#
# The text is passed through intact rather than summarised: `notices`
# matches on prose, by its own deliberate design, so the wording *is*
# the thing that decides which sentence you get told.
raise AgentError(_explain(str(exc))) from exc
# --------------------------------------------------------------- plumbing
def _field(event: Any, name: str) -> Any:
"""One field of a stream event, whichever shape it arrived in.
The two backends carry the same events in different clothes: the API SDK
parses them into objects, and the CLI passes the wire format straight
through as dicts. Reading both through here is what lets `_delta` be
written once.
"""
return event.get(name) if isinstance(event, dict) else getattr(event, name, None)
def _delta(event: Any) -> str:
"""The answer text one stream event carries, or "" for the many with none.
The events worth anything here are `content_block_delta`, whose delta is
`text` for a plain answer and `partial_json` when the schema is enforced
through a tool. Everything else — message start and stop, block
boundaries, usage — is dropped.
Deliberately reads defensively rather than by type. An event shape nobody
here recognises must be nothing, not an exception: this drives a progress
bar, and it does not get to fail a run the model is answering correctly.
"""
if _field(event, "type") != "content_block_delta":
return ""
delta = _field(event, "delta")
if delta is None:
return ""
piece = _field(delta, "text") or _field(delta, "partial_json") or ""
return piece if isinstance(piece, str) else ""
def _sync(coro: Any, timeout: float) -> Reply:
"""Run one coroutine to completion from synchronous code.
Jobs are synchronous by design and the worker runs them on a thread, so
there is normally no loop here and `asyncio.run` is exactly right. The
thread pool is for the other caller — anything asking from inside a
running loop, where `asyncio.run` would raise.
"""
import asyncio
async def bounded() -> Reply:
try:
return await asyncio.wait_for(coro, timeout)
except TimeoutError as exc:
raise AgentError(f"no answer within {timeout:g}s") from exc
try:
asyncio.get_running_loop()
except RuntimeError:
return asyncio.run(bounded())
with ThreadPoolExecutor(max_workers=1) as pool:
return pool.submit(asyncio.run, bounded()).result()
def _explain(error: str) -> str:
"""Keep the raw failure text intact for the layer that classifies it.
This used to append advice ("the `claude` login looks expired, run…"),
which was the right instinct in the wrong place: the message was built here
and then swallowed three frames up by a `except AgentError: continue`, so
the advice never reached anyone. Recognising the failure and deciding what
to say about it now lives in `desk/notices.py`, which has somewhere to
put the answer.
What stays here is the part only this module knows: the CLI can end a run
with an empty error and a subtype, and "" is not something to classify.
"""
return error.strip() or "the agent ended without saying why"
def _dumps(payload: dict[str, Any]) -> str:
return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
def _parse(text: str) -> dict[str, Any]:
"""The answer as a dict. Tolerant of a fenced or chatty response.
Both backends are asked for JSON through a schema, so this should be a
plain `json.loads`. It isn't, because "should be" is not a guarantee worth
losing a batch over — a model that wrapped its answer in a code fence has
still done the work.
"""
body = text.strip()
if body.startswith("```"):
body = body.split("\n", 1)[-1].rsplit("```", 1)[0]
start, end = body.find("{"), body.rfind("}")
if start == -1 or end <= start:
raise AgentError(f"no JSON object in the response: {text[:200]!r}")
try:
data = json.loads(body[start:end + 1])
except ValueError as exc:
raise AgentError(f"the response wasn't valid JSON: {exc}") from exc
if not isinstance(data, dict):
raise AgentError("the response wasn't a JSON object")
return data
+212
View File
@@ -0,0 +1,212 @@
"""Structured article extraction, independent of fetching and persistence.
The tree is data, never HTML. Code is kept verbatim; plain text is a derived
view of the same tree. Custom extractors can return ArticleContent; callers
own persistence and renderers consume the closed vocabulary below.
"""
from __future__ import annotations
import hashlib
from datetime import UTC, datetime
from html.parser import HTMLParser
from typing import Literal, Self
from urllib.parse import urljoin, urlsplit
from xml.etree import ElementTree as ET
from pydantic import Field, model_validator
from desk.models import Model
class ContentNode(Model):
kind: Literal[
"document", "text", "paragraph", "heading", "link", "code", "list",
"item", "table", "row", "cell", "strong", "emphasis", "quote", "break",
"span", "inline_code",
]
text: str = ""
children: list[ContentNode] = Field(default_factory=list)
level: int = Field(default=1, ge=1, le=6)
ordered: bool = False
header: bool = False
url: str = ""
@model_validator(mode="after")
def consistent_node(self) -> Self:
if self.kind in {"text", "code", "inline_code", "break"}:
if self.children:
raise ValueError("leaf nodes cannot have children")
elif self.text:
raise ValueError("container text belongs in text children")
if self.url and (self.kind != "link" or not _link(self.url, "")):
raise ValueError("only links can carry absolute HTTP(S) URLs")
return self
def plain_text(self) -> str:
if self.kind == "break":
return "\n"
if self.kind in {"code", "inline_code", "text"}:
return self.text
separator = {"document": "\n\n", "list": "\n", "table": "\n", "row": "\t"}.get(self.kind, "")
return separator.join(child.plain_text() for child in self.children)
class ArticleContent(Model):
format: Literal["plain", "structured-v1"]
content: str | ContentNode
source_url: str
method: str
acquired_at: str = Field(default_factory=lambda: datetime.now(UTC).isoformat())
# Finding a readable article does not prove that it was not truncated by
# the publisher. The caller must supply evidence before declaring it full.
completeness: Literal["unknown", "complete", "partial"] = "unknown"
removed_gutters: int = Field(default=0, ge=0)
@model_validator(mode="after")
def consistent_format(self) -> Self:
if self.format == "plain":
if not isinstance(self.content, str):
raise ValueError("plain content must be a string")
elif not isinstance(self.content, ContentNode) or self.content.kind != "document":
raise ValueError("structured-v1 content must be a document node")
return self
@property
def plain_text(self) -> str:
return self.content if isinstance(self.content, str) else self.content.plain_text()
@property
def revision(self) -> str:
value = self.content if isinstance(self.content, str) else self.content.model_dump_json()
return hashlib.sha256(f"{self.format}\n{value}".encode()).hexdigest()
class _Gutters(HTMLParser):
"""Remove only explicit code-gutter nodes, retaining their following text.
No numeric-line heuristics. Unknown publisher markup is preserved. The
void-element handling matters: a br inside a gutter must not eat the rest
of the article by leaving the suppression depth unbalanced.
"""
VOID = frozenset({"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"})
MARKERS = frozenset({"line-number", "line-numbers-rows", "lineno", "linenos"})
def __init__(self) -> None:
super().__init__(convert_charrefs=False)
self.parts: list[str] = []
self.stack: list[tuple[str, bool, bool]] = []
self.removed = 0
def handle_starttag(self, tag, attrs):
classes = set((dict(attrs).get("class") or "").split())
context = tag in {"pre", "code"} or bool(classes & {"highlight", "highlighttable"}) or any(c for _, c, _ in self.stack)
suppressed = any(s for _, _, s in self.stack)
gutter = context and bool(classes & self.MARKERS)
if gutter and not suppressed:
self.removed += 1
if not (gutter or suppressed):
self.parts.append(self.get_starttag_text() or "")
if tag not in self.VOID:
self.stack.append((tag, context, gutter or suppressed))
def handle_startendtag(self, tag, attrs):
self.handle_starttag(tag, attrs)
if tag not in self.VOID:
self.handle_endtag(tag)
def handle_endtag(self, tag):
suppressed = any(s for _, _, s in self.stack)
for index in range(len(self.stack) - 1, -1, -1):
if self.stack[index][0] == tag:
del self.stack[index:]
break
if not suppressed:
self.parts.append(f"</{tag}>")
def handle_data(self, data):
if not any(s for _, _, s in self.stack):
self.parts.append(data)
def handle_entityref(self, name):
self.handle_data(f"&{name};")
def handle_charref(self, name):
self.handle_data(f"&#{name};")
def _link(value: str, source_url: str) -> str:
try:
resolved = urljoin(source_url, value)
return resolved if urlsplit(resolved).scheme in {"http", "https"} else ""
except ValueError:
return ""
def _node(element: ET.Element, source_url: str) -> ContentNode:
children: list[ContentNode] = []
if element.text:
children.append(ContentNode(kind="text", text=element.text))
for child in element:
children.append(_node(child, source_url))
if child.tail:
children.append(ContentNode(kind="text", text=child.tail))
match element.tag:
case "code":
return ContentNode(kind="code", text="".join(element.itertext()))
case "head":
level = element.get("rend", "h1").removeprefix("h")
return ContentNode(kind="heading", level=int(level) if level in "123456" and len(level) == 1 else 1, children=children)
case "ref":
return ContentNode(kind="link", url=_link(element.get("target", ""), source_url), children=children)
case "list":
return ContentNode(kind="list", ordered=element.get("rend") == "ol", children=children)
case "cell":
return ContentNode(kind="cell", header=element.get("role") == "head", children=children)
case "hi":
if element.get("rend") == "#t":
return ContentNode(kind="inline_code", text="".join(element.itertext()))
kind = {"#b": "strong", "#i": "emphasis"}.get(element.get("rend", ""), "span")
return ContentNode.model_validate({"kind": kind, "children": children})
case "lb":
return ContentNode(kind="break")
case _:
kinds = {"main": "document", "p": "paragraph", "item": "item", "table": "table", "row": "row", "quote": "quote"}
return ContentNode.model_validate({"kind": kinds.get(element.tag, "paragraph"), "children": children})
def extract_article(html: str, source_url: str) -> ArticleContent | None:
"""Extract supplied HTML without I/O; return None when no article is found.
XML is an intermediate representation only. Its tags and attributes never
escape into the content contract, and no raw HTML is passed to a renderer.
"""
import trafilatura
cleaned = _Gutters()
cleaned.feed(html)
cleaned.close()
extracted = trafilatura.extract(
"".join(cleaned.parts), url=source_url, output_format="xml",
include_comments=False, include_tables=True, include_links=True,
include_formatting=True,
)
if not extracted:
return None
main = ET.fromstring(extracted).find("main")
if main is None:
return None
# XML pretty-printing inserts whitespace between blocks. It is not source
# content; inside code, however, every byte of whitespace belongs to it.
for element in main.iter():
if element.tag in {"main", "table", "row", "list"}:
if not (element.text or "").strip():
element.text = None
for child in element:
if not (child.tail or "").strip():
child.tail = None
return ArticleContent(
format="structured-v1", content=_node(main, source_url), source_url=source_url,
method="trafilatura-xml", removed_gutters=cleaned.removed,
)
+254
View File
@@ -0,0 +1,254 @@
"""Render a page in headless Chromium, accepting a tracking-consent gate on the way.
The getter a check sends a page to when plain HTTP brought back a placeholder:
an app shell whose article is drawn by a script, or a consent gate standing in
front of every article on a host. Golem is the second kind. Every article URL
redirects to `/sonstiges/zustimmung/auswahl.html`, where Sourcepoint's CMP
draws its dialog in an iframe; accepting it sets the cookies and redirects
back to the article.
from desk.tools import browser
got = browser.render("https://www.golem.de/news/...")
got.status, got.html, got.url, got.consent, got.cookies
browser.remember(cache, got.url, got.cookies)
browser.cookie_header(cache, url) # "golem_consent20=cmp|250101; ..."
**Consent is accepted once and then reused.** Chromium costs a few seconds
and a few hundred megabytes per page. The cookies it earns open the gate for
a plain request too, so the caller stores them per host with `remember` and
sends them with `cookie_header` on the next HTTP fetch. The browser runs again
only when HTTP meets the gate again, which is when the cookies expired or the
outlet changed what it accepts. Golem's last three months.
**A CMP is one entry in `CONSENTS`**: the selector of the iframe it draws in
and a pattern for the accept button's name. A page with no such frame is
rendered without clicking anything, after waiting `gate_wait` seconds for one
to appear. Only accept buttons are listed. A bot check that wants a human is
not something this clicks; it stays `js_required` or `denied`.
Needs the `browser` extra and a Chromium build: `playwright install chromium`.
Imported only inside `render`, so the rest of `desk.tools` does not pay for it.
"""
from __future__ import annotations
import contextlib
import importlib.util
import json
import re
import threading
import time
from collections.abc import Mapping, MutableMapping, Sequence
from dataclasses import dataclass, field
from typing import Any
from urllib.parse import urlsplit
from desk.tools.framing import host_of
#: Sits alongside `embed:` and `body:` in the same table.
CACHE_PREFIX = "cookies:"
#: Seconds for navigation and for the click. A gate that has not redirected
#: back by then is not going to.
TIMEOUT = 30.0
#: Seconds to wait for a consent frame on a page that may not have one. The
#: CMP injects its iframe from a script; on golem.de it is there within a
#: second.
GATE_WAIT = 5.0
#: Seconds to let a script-drawn page settle after `domcontentloaded`. Not
#: `networkidle` as a condition: an ad-carrying page never goes idle, so this is
#: a ceiling.
SETTLE = 3.0
#: One Chromium at a time in this process. Body batches run side by side, and
#: the first articles of a gated host would otherwise each start a browser
#: before the first one has earned the cookies.
_ONE_AT_A_TIME = threading.Lock()
def installed() -> bool:
"""Whether the `browser` extra is. Chromium itself is checked by launching."""
return importlib.util.find_spec("playwright") is not None
@dataclass(frozen=True)
class Consent:
"""How to accept one CMP's dialog."""
name: str
#: CSS selector of the iframe the dialog is drawn in.
frame: str
#: Case-insensitive regex, matched against the accept button's
#: accessible name. Anchored, so "Accept" does not match "Accept only
#: necessary" and a reject button is never the one clicked.
accept: str
#: A page this was found on.
seen: str = ""
def __post_init__(self) -> None:
re.compile(self.accept)
CONSENTS: tuple[Consent, ...] = (
Consent("sourcepoint", "iframe[id^='sp_message_iframe']",
r"^(Zustimmen und weiter|Alle akzeptieren|Akzeptieren|Zustimmen"
r"|Accept all|Accept|I agree|Agree)$",
seen="https://www.golem.de/sonstiges/zustimmung/auswahl.html"),
)
@dataclass(frozen=True)
class Cookie:
name: str
value: str
domain: str
path: str = "/"
#: Unix seconds; -1 for a session cookie, which is kept until the gate
#: comes back rather than until a browser that no longer exists closes.
expires: float = -1
@dataclass(frozen=True)
class Rendered:
"""What Chromium ended up showing. `url` is where it ended up."""
status: int
html: str
url: str
headers: Mapping[str, str] = field(default_factory=dict)
#: Every cookie the host of `url` would be sent.
cookies: tuple[Cookie, ...] = ()
#: The `Consent.name` that was accepted, or "" when no gate was met.
consent: str = ""
def render(url: str, *, consents: Sequence[Consent] = CONSENTS,
timeout: float = TIMEOUT, gate_wait: float = GATE_WAIT,
settle: float = SETTLE) -> Rendered:
"""Load `url`, accept a consent dialog if one appears, return the page.
Raises what Playwright raises: its `TimeoutError` names itself a timeout
to `failures.of_exception`, and a missing Chromium build is `unknown`.
One browser per call. A pull renders a page only where HTTP failed, so a
browser kept alive between calls would sit idle in a worker.
"""
from playwright.sync_api import TimeoutError as PlaywrightTimeout
from playwright.sync_api import sync_playwright
ms = int(timeout * 1000)
with _ONE_AT_A_TIME, sync_playwright() as playwright:
# /dev/shm is 64 MB in a container, which Chromium outgrows and
# crashes on; this has it use /tmp instead.
chromium = playwright.chromium.launch(
headless=True, args=["--disable-dev-shm-usage"])
try:
context = chromium.new_context()
page = context.new_page()
# The last document response of the main frame, so the status is
# the article's after a gate's redirect rather than the gate's.
documents: list[Any] = []
page.on("response", lambda response: documents.append(response)
if response.request.is_navigation_request()
and response.frame == page.main_frame else None)
# `domcontentloaded`, not `load`: an ad-carrying page such as a
# golem.de article may not fire `load` within any useful timeout.
page.goto(url, wait_until="domcontentloaded", timeout=ms)
accepted = _accept(page, consents, ms, gate_wait, PlaywrightTimeout)
with contextlib.suppress(PlaywrightTimeout):
page.wait_for_load_state("networkidle", timeout=int(settle * 1000))
last = documents[-1] if documents else None
final = page.url
return Rendered(
status=last.status if last else 0,
html=page.content(),
url=final,
headers=dict(last.headers) if last else {},
cookies=_for_host(_hostname(final), context.cookies()),
consent=accepted)
finally:
chromium.close()
def _accept(page: Any, consents: Sequence[Consent], ms: int, gate_wait: float,
timeout_error: type[Exception]) -> str:
"""Click the first consent dialog's accept button. The CMP's name, or ""."""
deadline = time.monotonic() + gate_wait
for consent in consents:
left = max(0, int((deadline - time.monotonic()) * 1000))
frame = page.locator(consent.frame).first
try:
frame.wait_for(state="attached", timeout=max(left, 1))
except timeout_error:
continue
button = page.frame_locator(consent.frame).first.get_by_role(
"button", name=re.compile(consent.accept, re.IGNORECASE))
button.first.click(timeout=ms)
# Gone either way it is dismissed: an overlay removes its frame, a
# gate page is navigated away from.
frame.wait_for(state="detached", timeout=ms)
page.wait_for_load_state("domcontentloaded", timeout=ms)
return consent.name
return ""
def _for_host(host: str, cookies: Sequence[Mapping[str, Any]]) -> tuple[Cookie, ...]:
return tuple(
Cookie(str(c["name"]), str(c["value"]), str(c["domain"]),
str(c.get("path", "/")), float(c.get("expires", -1)))
for c in cookies if _matches(host, str(c.get("domain", ""))))
def _hostname(url: str) -> str:
"""The host without a port, which is what a cookie's domain is matched to."""
try:
return (urlsplit(url).hostname or "").lower()
except ValueError:
return ""
def _matches(host: str, domain: str) -> bool:
domain = domain.lstrip(".").lower()
return bool(host) and (host == domain or host.endswith("." + domain))
# ------------------------------------------------------------- the cookie jar
def remember(cache: MutableMapping[str, str], url: str,
cookies: Sequence[Cookie]) -> None:
"""Keep the cookies a render earned, for the next plain fetch from this host.
Replaces whatever the host had: a render that met the gate again earned
the current set, and the old one is what stopped working.
"""
kept = [c for c in cookies if _matches(_hostname(url), c.domain)]
if kept:
cache[CACHE_PREFIX + host_of(url)] = json.dumps(
[[c.name, c.value, c.domain, c.path, c.expires] for c in kept])
def cookie_header(cache: Mapping[str, str], url: str, *,
now: float | None = None) -> str:
"""The `Cookie` header for a plain request to `url`, or "" for none.
Expired cookies are left out rather than deleted. A host whose cookies
have all expired sends none, meets the gate, and is rendered again.
"""
stored = cache.get(CACHE_PREFIX + host_of(url), "")
if not stored:
return ""
now = time.time() if now is None else now
try:
rows = json.loads(stored)
except ValueError:
return ""
return "; ".join(f"{name}={value}" for name, value, _domain, _path, expires in rows
if expires < 0 or expires > now)
def forget(cache: MutableMapping[str, str], url: str) -> None:
"""Drop a host's cookies, so its next fetch goes without them."""
cache.pop(CACHE_PREFIX + host_of(url), None)
+227
View File
@@ -0,0 +1,227 @@
"""Did the getter bring back a page worth converting, and if not, what next?
`scrape` fetches, this judges, `article` converts. A 200 is not an article
because it answered 200: an app shell serves a `<noscript>` notice asking for
JavaScript, a bot check serves a few words and a script. Converting either
stores the notice as the article.
A check answers with a `Verdict`: the failure category from
`desk.tools.failures`, the name of what matched, and which getters are worth
trying next. That last part is where what we learn goes. A Mastodon post is
a shell over HTTP and a status over Mastodon's API, so its signature names
`mastodon_api`; a bot check names `browser`. A getter the app does not have is
skipped, so a signature can name one before it is built.
Checks run in order and the first match decides:
1. the status code
2. no HTML at all
3. `SIGNATURES`, the table below, most specific first
4. the structural rule, for a shell nobody has written a signature for yet
**Adding a signature** is one entry in `SIGNATURES` and one test page in
`tests/test_checks.py`. The entry names the page it was found on, so whoever
reads it later can fetch that page and see whether it still says this.
Signature(
name="mastodon_shell", # logged, and on the attempt
where="noscript", # title | text | noscript | html | url
pattern=r"To use the Mastodon web application", # regex, case-insensitive
category="js_required", # from desk.tools.failures
next=("mastodon_api", "browser"), # getters to try, in order
seen="https://grapheneos.social/@GrapheneOS/117225539756835649",
)
`where` is what the pattern is matched against: `title` the `<title>`, `text`
the words a reader without JavaScript sees outside `<noscript>`, `noscript`
the words inside it, `html` the raw markup (script sources, class names),
`url` the address the getter ended up at after redirects. Prefer the narrowest
one that works: a phrase in `text` can appear in an article about the thing
it describes, a class name in `html` usually cannot.
A pattern that does not compile, a category the failure vocabulary does not
have, or two signatures with one name is refused at import.
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from dataclasses import dataclass, field
from html.parser import HTMLParser
from typing import Literal
from desk.tools import failures
Where = Literal["title", "text", "noscript", "html", "url"]
@dataclass(frozen=True)
class Page:
"""What one getter brought back. `url` is where it ended up."""
status: int
html: str
url: str = ""
headers: Mapping[str, str] = field(default_factory=dict)
@dataclass(frozen=True)
class Verdict:
"""Why this page is not worth converting, and what might get the real one."""
category: str
#: The signature or rule that matched. What a person reads in the log to
#: decide whether the table needs a new entry or a narrower one.
name: str
next: tuple[str, ...] = ()
@dataclass(frozen=True)
class Signature:
name: str
where: Where
pattern: str
category: str
next: tuple[str, ...] = ()
#: A page this was found on.
seen: str = ""
def __post_init__(self) -> None:
re.compile(self.pattern)
if self.category not in failures.TRANSIENT | failures.SETTLED:
raise ValueError(f"signature {self.name!r} names category "
f"{self.category!r}, which desk.tools.failures has not")
#: Most specific first. See the module docstring for the format.
SIGNATURES: tuple[Signature, ...] = (
Signature("mastodon_shell", "noscript", r"To use the Mastodon web application",
"js_required", ("mastodon_api", "browser"),
seen="https://grapheneos.social/@GrapheneOS/117225539756835649"),
Signature("bluesky_shell", "noscript", r"This is a heavily interactive web application, and JavaScript is required",
"js_required", ("bluesky_api", "browser"),
seen="https://bsky.app/profile/bsky.app/post/3mv3shqdfuc2e"),
Signature("create_react_app_shell", "noscript", r"You need to enable JavaScript to run this app",
"js_required", ("browser",),
seen="https://rune.build/blog/rune-is-now-open-source"),
Signature("cloudflare_turnstile", "html", r"challenges\.cloudflare\.com/turnstile",
"js_required", ("browser",),
seen="https://openreview.net/forum?id=pc7fqaOcAH"),
Signature("anubis_check", "text", r"automated check to get rid of most bots|Making sure you're not a bot",
"js_required", ("browser",),
seen="https://codeberg.org/forgejo/forgejo/src/branch/forgejo/release-notes-published/16.0.4.md"),
# Every article redirects here without the consent cookie. The browser
# accepts Sourcepoint's dialog, and the cookies it earns open the gate for
# HTTP until they expire. See `desk.tools.browser`.
Signature("golem_consent_gate", "html", r"GolemConsent\.init",
"consent_wall", ("browser",),
seen="https://www.golem.de/sonstiges/zustimmung/auswahl.html"),
)
#: Below this many words outside `<noscript>`, a page carrying a script is a
#: placeholder. OpenReview's check is 36 words, Anubis 42; a real article is
#: several hundred.
SCRIPT_PAGE_WORDS = 80
#: The longest `<noscript>` read as a notice. Mastodon's is 21 words,
#: Bluesky's 76. A longer one is a site serving its content without scripts.
NOTICE_WORDS = 150
#: Where the structural rule sends a page nobody has a signature for.
UNRECOGNISED_SHELL_NEXT = ("browser",)
def _unique_names(signatures: tuple[Signature, ...]) -> None:
names = [s.name for s in signatures]
if len(names) != len(set(names)):
raise ValueError(f"two signatures share a name: "
f"{sorted({n for n in names if names.count(n) > 1})}")
_unique_names(SIGNATURES)
def check(page: Page | None, signatures: tuple[Signature, ...] = SIGNATURES) -> Verdict | None:
"""None when the page is worth converting, else why not."""
if page is None:
# The downloader logs which way it failed and returns nothing: DNS, a
# refused connection, a reset and TLS all arrive here as one.
return Verdict("unreachable", "no_response")
if why := failures.of_status(page.status):
return Verdict(why, f"status_{page.status}")
if not page.html.strip():
return Verdict("empty", "no_html")
parts = _Parts.of(page)
for signature in signatures:
if re.search(signature.pattern, parts[signature.where], re.IGNORECASE):
return Verdict(signature.category, signature.name, signature.next)
if (parts.scripts and len(_words(parts["text"])) < SCRIPT_PAGE_WORDS
and len(_words(parts["noscript"])) <= NOTICE_WORDS):
return Verdict("js_required", "script_page", UNRECOGNISED_SHELL_NEXT)
return None
def _flat(pieces: list[str]) -> str:
return " ".join(" ".join(pieces).split())
def _words(text: str) -> list[str]:
return re.findall(r"\w+", text)
class _Parts(dict[str, str]):
"""The five things a signature can match against, read in one pass."""
scripts: int
@classmethod
def of(cls, page: Page) -> _Parts:
reader = _Reader()
try:
reader.feed(page.html)
reader.close()
except Exception: # malformed markup: match what was read so far
pass
# Whitespace collapsed in the three text parts, so a signature is
# written with single spaces however the page wrapped its lines.
parts = cls(title=_flat(reader.title), text=_flat(reader.text),
noscript=_flat(reader.noscript), html=page.html, url=page.url)
parts.scripts = reader.scripts
return parts
class _Reader(HTMLParser):
HIDDEN = frozenset({"script", "style", "template", "svg"})
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self.title: list[str] = []
self.text: list[str] = []
self.noscript: list[str] = []
self.scripts = 0
self._hidden = 0
self._in: dict[str, int] = {"title": 0, "noscript": 0}
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
self.scripts += tag == "script"
if tag in self.HIDDEN:
self._hidden += 1
elif tag in self._in:
self._in[tag] += 1
def handle_endtag(self, tag: str) -> None:
if tag in self.HIDDEN:
self._hidden = max(0, self._hidden - 1)
elif tag in self._in:
self._in[tag] = max(0, self._in[tag] - 1)
def handle_data(self, data: str) -> None:
if self._hidden:
return
if self._in["title"]:
self.title.append(data)
elif self._in["noscript"]:
self.noscript.append(data)
else:
self.text.append(data)
+32
View File
@@ -0,0 +1,32 @@
"""Local credential settings. Read per call so running workers see replacements."""
import os
import sqlite3
from contextlib import closing
from desk.config import AgentConfig
def saved_token(settings: AgentConfig) -> str:
if not settings.credentials_database:
return ""
with closing(sqlite3.connect(settings.credentials_database)) as conn:
if not conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='credentials'"
).fetchone():
return ""
row = conn.execute("SELECT value FROM credentials WHERE name='claude_oauth'").fetchone()
return row[0] if row else ""
def token(settings: AgentConfig) -> str:
return saved_token(settings) or os.environ.get("CLAUDE_CODE_OAUTH_TOKEN", "")
def save_token(settings: AgentConfig, value: str) -> None:
with closing(sqlite3.connect(settings.credentials_database)) as conn, conn:
conn.execute("CREATE TABLE IF NOT EXISTS credentials (name TEXT PRIMARY KEY, value TEXT NOT NULL)")
if value:
conn.execute("INSERT OR REPLACE INTO credentials VALUES ('claude_oauth', ?)", (value,))
else:
conn.execute("DELETE FROM credentials WHERE name='claude_oauth'")
+115
View File
@@ -0,0 +1,115 @@
"""Turning an article into a vector.
A local sentence-transformer turns each article into ~384 floats. What it
buys, and why it is worth an optional 100MB download:
**It generalises across phrasings, which TF-IDF cannot.** The bag-of-words
model learns the token "semiconductor". It learns nothing at all from that
about an article on "chip fabrication", because the two share no words. The
embedding puts them next to each other, so a vote on one moves the other.
**It works from about five votes.** The taste vector is
`centroid(upvoted) - centroid(downvoted)` — a direction in the space, scored
by cosine. There is no training step, no fitting, and no threshold below which
it does nothing useful; it just gets sharper as votes arrive.
This module is only the encoder. Where the vectors go, and the arithmetic over
them, is `newsdesk/storage/vectors/`; which articles get encoded, and when, is
`newsdesk/jobs/ingest.py`.
`ranking.embedding_model` is recorded on every stored row, because a vector is
only comparable to another from the same model. Change the setting and the
old vectors are not wrong, they are *meaningless* — a different geometry
entirely — so they are dropped and re-computed rather than silently compared.
Not to be confused with `desk/tools/framing.py`, which is about whether
an outlet lets its pages be put in an iframe — that used to be called
`embed.py`, and the collision with `ranking.embedding_model` is why both were
renamed.
"""
from __future__ import annotations
import logging
from collections.abc import Sequence
from typing import Protocol
import numpy as np
log = logging.getLogger(__name__)
#: How much of the article body the encoder sees. These models truncate at a
#: few hundred tokens anyway, and the headline plus the opening paragraphs is
#: what actually says what a story is about.
BODY_CHARS = 1500
class Embedder(Protocol):
"""Anything that can turn strings into vectors.
A protocol rather than the concrete class, so the tests can hand the rest
of the system a deterministic encoder and never download a model. That is
not a testing hack — it is also what a future backend (an API embedder, a
different local library) would implement.
"""
model_id: str
def encode(self, texts: Sequence[str]) -> np.ndarray: ...
class SentenceTransformer:
"""The real one. Loads the model on first use, not on import.
A pull with `backend: tfidf` must not pay for a torch import, so
nothing here is imported at module level and this class is only constructed
by `load()`, which the embeddings path calls and the TF-IDF path does not.
"""
def __init__(self, model_id: str) -> None:
try:
from sentence_transformers import SentenceTransformer as ST
except ImportError as exc:
raise ImportError(
"ranking.backend is 'embeddings', which needs sentence-transformers.\n"
" uv sync --extra embeddings\n"
"or set ranking.backend back to 'tfidf' in config.yaml."
) from exc
self.model_id = model_id
log.info("loading embedding model %s (first run downloads it)", model_id)
self._model = ST(model_id)
def encode(self, texts: Sequence[str]) -> np.ndarray:
# Normalised on the way out, so cosine similarity is a dot product and
# every consumer can stop worrying about magnitudes.
return np.asarray(
self._model.encode(list(texts), normalize_embeddings=True,
show_progress_bar=False),
dtype=np.float32,
)
def load(model_id: str) -> Embedder:
return SentenceTransformer(model_id)
class Embeddable(Protocol):
"""What `document` reads: a title and a body, however the app stores them."""
title: str
excerpt: str
body: str
def document(row: Embeddable) -> str:
"""What gets embedded.
Title and body only — deliberately no `__source_` tokens. Those exist in
`rank._document` because TF-IDF has no other way to learn "I like this
outlet", and they would be noise here: the encoder would try to find
meaning in a made-up word. Outlet preference belongs in the linear model of
Stage 2, as its own one-hot feature, not smuggled into the text.
"""
body = row.body or row.excerpt
return f"{row.title}\n\n{' '.join((body or '').split())[:BODY_CHARS]}"
+185
View File
@@ -0,0 +1,185 @@
"""What went wrong, in one word a retry policy can be written against.
Acquisition fails in ways that mean opposite things. A host that took too long
will very likely answer in a minute; a page that is gone will be gone in a
minute, in an hour, and tomorrow. Until this existed the difference had nowhere
to live: `fetch_bodies` caught every exception in one `except`, wrote an empty
body, and the row said only that something had been tried — so the one policy
the archive could have was "never ask again", and it was the policy for the
timeout as much as for the 404.
A category is an answer to exactly one question: **would asking again get a
different answer?** `TRANSIENT` is yes, `SETTLED` is no, and `unknown` is "we
do not know", which is why nothing retries it unless a plugin names it.
from desk.tools import failures
failures.of_status(503) # 'server_error'
failures.of_exception(exc) # 'timeout'
failures.hint(response.headers) # 120.0, when the server named its own wait
Kept away from any HTTP library on purpose. `httpx` and `urllib3` are both
optional here and either may be the one that raised, so an exception is named
by walking the class names in its ancestry rather than by importing the
packages that define them. It costs the precision of an `isinstance` and buys a
tool that can name a failure raised by a library it has never imported — which
is what a plugin's own fetch will be.
The names are the vocabulary `RetryPolicy.on` is validated against, so adding
one here is adding one to the plugin contract: do it deliberately, and only
where the new name would lead somewhere different from an existing one.
"""
from __future__ import annotations
import logging
from collections.abc import Mapping
from email.utils import parsedate_to_datetime
from datetime import UTC, datetime
log = logging.getLogger(__name__)
#: Failures worth another attempt: the world was busy, not unwilling.
TRANSIENT = frozenset({
# The request did not finish in the time it was given.
"timeout",
# It never got an answer at all: DNS, a refused connection, a reset, TLS.
# One name rather than four because nothing downstream would do anything
# different about them, and because the downloader that reports most of
# them does not say which it was.
"unreachable",
# 429. The one failure that usually comes with the server's own answer to
# "when, then" — see `hint`.
"rate_limited",
# 5xx. The publisher's own machine is having a bad minute.
"server_error",
# Asked for before it existed. Not an HTTP failure at all: auto-captions
# on a two-hour upload take hours to appear, so a recording that has none
# yet is a wait rather than a refusal — and the same word covers anything
# else a source publishes late. See `segment_profile` in
# `desk/sources/podcast.py`, which is what declares a bound on that wait.
"not_ready",
})
#: Failures that will say the same thing next time. Retrying one of these is
#: how a source ends up fetched four times an evening for nothing.
SETTLED = frozenset({
# 404, 410. The address is not an article and will not become one.
"not_found",
# 401, 403, 451. Refused rather than broken — a login, a bot check, a
# jurisdiction. Another identical request is refused identically.
"denied",
# The page came back and it was the consent gate the host shows instead of
# the article. Deliberately here and never in a default retry list: it is
# the same answer however many times it is asked for. See `desk.tools.walls`.
"consent_wall",
# The page was an app shell or a bot challenge: a placeholder for what a
# script would have rendered. The same request gets the same placeholder,
# so it is settled for a retry; a different way of fetching, a browser, is
# what can answer it. See `desk.tools.checks`.
"js_required",
# Fetched, extracted, and there was no article in it.
"empty",
})
#: Neither, and on purpose. A failure nothing here recognises is not declared
#: transient on its behalf — a policy that wants it retried has to say so.
UNKNOWN = "unknown"
#: Everything a plugin may name in `RetryPolicy.on`.
CATEGORIES = TRANSIENT | SETTLED | frozenset({UNKNOWN})
#: Substrings of an exception class name, in the order they are tried, and the
#: category each means. Ordered because `ReadTimeoutError` is both a timeout
#: and a connection error by name, and it is a timeout.
_BY_NAME: tuple[tuple[str, str], ...] = (
("timeout", "timeout"),
("nameresolution", "unreachable"),
("ssl", "unreachable"),
("connect", "unreachable"),
("connection", "unreachable"),
("protocol", "unreachable"),
("transport", "unreachable"),
("network", "unreachable"),
)
#: The longest server hint worth waiting on. A `Retry-After` of a day is not a
#: wait, it is a refusal with a number on it, and honouring it would leave a
#: job row holding this item's place until tomorrow. Beyond this the job's own
#: backoff is used instead, which ends in a bounded number of attempts.
MAX_HINT = 6 * 60 * 60
def of_status(status: int) -> str:
"""The category of an HTTP status that is not a success.
A 2xx or 3xx is not a failure and has no category — it is `""`, so a
caller can pass any status and ask.
"""
if 200 <= status < 400:
return ""
if status == 429:
return "rate_limited"
if status in (401, 403, 451):
return "denied"
if status in (404, 410):
return "not_found"
if 500 <= status < 600:
return "server_error"
return UNKNOWN
def of_exception(exc: BaseException) -> str:
"""The category of a raised failure, named from its ancestry.
`TimeoutError` is the builtin every socket timeout already is; the rest are
matched on the class names in the exception's MRO, so `httpx.ConnectError`
and `urllib3.exceptions.NewConnectionError` both come out `unreachable`
without this module importing either package. A cause is followed, because
a wrapper around the real failure is the usual shape: `ProtocolError` with
a `TimeoutError` underneath it is a timeout.
"""
if isinstance(exc, TimeoutError):
return "timeout"
names = [cls.__name__.lower() for cls in type(exc).__mro__]
for fragment, category in _BY_NAME:
if any(fragment in name for name in names):
return category
cause = exc.__cause__ or exc.__context__
if cause is not None and cause is not exc:
return of_exception(cause)
return UNKNOWN
def hint(headers: Mapping[str, str] | None) -> float | None:
"""The wait a server named for itself, in seconds, or None for no answer.
`Retry-After` comes in two forms and both are in use: a number of seconds,
and an HTTP date. Honouring it is the point — guessing costs either a
minute of doing nothing when the service was ready again, or another
refusal charged against a bound that is ours rather than the server's.
Unreadable, negative, or longer than `MAX_HINT` is no answer: this returns
None and the job's own backoff decides. A header nobody can parse is not
worth failing over.
"""
if not headers:
return None
raw = next((v for k, v in headers.items()
if k.lower() == "retry-after"), "").strip()
if not raw:
return None
try:
seconds = float(raw)
except ValueError:
try:
when = parsedate_to_datetime(raw)
except (TypeError, ValueError):
log.debug("unreadable Retry-After: %r", raw)
return None
if when.tzinfo is None:
when = when.replace(tzinfo=UTC)
seconds = (when - datetime.now(UTC)).total_seconds()
if seconds <= 0 or seconds > MAX_HINT:
return None
return seconds
+132
View File
@@ -0,0 +1,132 @@
"""Will this page load inside an iframe?
Publishers say no with one of two headers:
X-Frame-Options: DENY | SAMEORIGIN
Content-Security-Policy: ...; frame-ancestors <list>
Both are enforced by the browser, silently: a blocked frame is a blank box with
no event you can catch from the parent page. Cross-origin means you cannot read
the frame's document to see what happened either. So the check has to happen
server-side, before the frame is ever created — which is what this module is.
Both headers are set per host, not per article, so one HTTP request per host is
enough and the answer is cached in the `state` table. In practice this resolves
to "no" for most newspapers: at the time of writing the Guardian, BBC, Ars
Technica and Al Jazeera all send SAMEORIGIN, and the NYT sends DENY.
"""
from __future__ import annotations
from collections.abc import MutableMapping
import logging
import re
from typing import Any
from urllib.parse import urlsplit
log = logging.getLogger(__name__)
#: Kept as `embed:` — the rows are already in every database.
CACHE_PREFIX = "embed:"
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/131.0 Safari/537.36")
_FRAME_ANCESTORS = re.compile(r"frame-ancestors\s+([^;]*)", re.IGNORECASE)
def host_of(url: str) -> str:
try:
return urlsplit(url).netloc.lower()
except ValueError:
return ""
def _verdict(headers: dict[str, str]) -> tuple[bool, str]:
"""Read the two headers. Returns (embeddable, human reason)."""
lower = {k.lower(): v for k, v in headers.items()}
xfo = (lower.get("x-frame-options") or "").strip().lower()
if xfo.startswith("deny"):
return False, "sends X-Frame-Options: DENY"
if xfo.startswith("sameorigin"):
return False, "sends X-Frame-Options: SAMEORIGIN"
match = _FRAME_ANCESTORS.search(lower.get("content-security-policy", ""))
if match:
sources = match.group(1).split()
# 'self' and 'none' both exclude us; a wildcard or an explicit list
# might not, but we're never on anyone's list, so only * lets us in.
if "*" not in sources:
return False, f"CSP frame-ancestors {' '.join(sources) or 'none'}"
return True, "no framing restriction"
def cached(cache: MutableMapping[str, str], url: str) -> dict[str, Any] | None:
"""The answer already known for this host, or None if nobody has asked.
The difference between this and `check` is a network request, and it is
the whole reason both exist: a page being rendered must not wait on an
outlet to find out whether it may be framed, but it does need to know the
difference between "no" and "not asked yet". None is the second one, and
the caller shows it as pending rather than as a refusal.
"""
host = host_of(url)
if not host or not url.lower().startswith(("http://", "https://")):
return {"embeddable": False, "reason": "not a web page", "host": host,
"certain": True}
stored = cache.get(CACHE_PREFIX + host, "")
if not stored:
return None
ok, _, reason = stored.partition(":")
return {"embeddable": ok == "1", "reason": reason, "host": host,
"certain": True}
def check(cache: MutableMapping[str, str], url: str, refresh: bool = False) -> dict[str, Any]:
"""Ask once per host, then remember.
`cache` is any mapping of strings — the app's state table, or a dict in a
test — because a tool has no store of its own to remember things in.
An unreachable host is reported as embeddable with `certain: False` — better
to let the browser try and possibly succeed than to refuse on our own
guess. Only a header we actually read counts as a no.
"""
host = host_of(url)
if not host or not url.lower().startswith(("http://", "https://")):
return {"embeddable": False, "reason": "not a web page", "host": host,
"certain": True}
key = CACHE_PREFIX + host
if not refresh:
cached = cache.get(key, "")
if cached:
ok, _, reason = cached.partition(":")
return {"embeddable": ok == "1", "reason": reason, "host": host,
"certain": True}
try:
import httpx
except ImportError:
return {"embeddable": True, "reason": "httpx not installed, not checked",
"host": host, "certain": False}
try:
with httpx.Client(follow_redirects=True, timeout=10.0,
headers={"User-Agent": UA}) as client:
resp = client.get(url)
# A 403 still carries the headers — the NYT refuses the request *and*
# refuses framing — so don't gate on the status code.
embeddable, reason = _verdict(dict(resp.headers))
# Redirects can land on a different host; cache the answer under both.
final = host_of(str(resp.url))
except Exception as exc:
log.debug("embed check failed for %s: %s", host, exc)
return {"embeddable": True, "reason": f"could not check ({type(exc).__name__})",
"host": host, "certain": False}
payload = f"{'1' if embeddable else '0'}:{reason}"
cache[key] = payload
if final and final != host:
cache[CACHE_PREFIX + final] = payload
return {"embeddable": embeddable, "reason": reason, "host": host, "certain": True}
+39
View File
@@ -0,0 +1,39 @@
"""One shared HTTP client per call chain, without making anyone pass one.
Reading an episode is three or four requests to two hosts, and doing them on
one client is the difference between one connection and four. But the public
functions here are also meant to be usable one at a time — `transcript(video)`
should not require the caller to have stood up a client first.
So every entry point takes an optional client and wraps it in this: given one,
it is borrowed and left open for whoever owns it; given `None`, a client is
made and closed on the way out. The `with` statement at the call site reads the
same either way.
"""
from __future__ import annotations
from contextlib import AbstractContextManager
import httpx
class _Borrowed(AbstractContextManager["httpx.Client"]):
"""Someone else's client. Enters and exits without owning anything."""
def __init__(self, client: httpx.Client) -> None:
self._client = client
def __enter__(self) -> httpx.Client:
return self._client
def __exit__(self, *exc: object) -> None:
return None
def session(client: httpx.Client | None,
timeout: float = 30) -> AbstractContextManager[httpx.Client]:
"""A context manager yielding a client, owning it only if it made it."""
if client is not None:
return _Borrowed(client)
return httpx.Client(timeout=timeout, follow_redirects=True)
+123
View File
@@ -0,0 +1,123 @@
"""What kind of thing is this, when it isn't an ordinary article?
An outlet's feed carries more than its journalism. Golem's carries adverts
("Anzeige:") and pieces behind its paywall ("(g+)"); Hacker News marks paid
links "[$]"; other feeds mark neither. There is no convention across outlets to
detect, so each plugin declares how its publisher says it — as `MarkRule`s in
its spec, and nowhere else. Typing a token in against one feed was how this
worked before every outlet was a plugin; step 19 retired that editor, and
`classify` below is the only way a mark is now found.
What is general is everything after that. The outlet's token is input: it is
matched, cut out of the title, and never shown. One glyph stands for a mark
everywhere, so "what is this" reads the same whoever published it, instead of
being a private code you learn one publisher at a time. The token is noise in
a headline as well — "Anzeige: Schon 10.000-mal bestellt" is worse than the
line underneath it.
The vocabulary below is closed on purpose. Marks are structural — they say what
a thing *is*, and the reader filters and styles them — which is a different
question from what a thing is *about*; that one has tags, an open vocabulary the
model can extend. Adding a mark means adding a row here and a token to the feeds
that use it.
"""
from __future__ import annotations
from dataclasses import dataclass
from collections.abc import Sequence
from typing import Any, Protocol
from desk.models import Item
@dataclass(frozen=True, slots=True)
class Mark:
slug: str
glyph: str
label: str
#: What the reader is told the glyph means, on hover and to a screen reader.
note: str
#: Whether it is worth fetching the page for a body. False for both marks we
#: have: a paywalled page hands back its gate, and an advert's body is the
#: advert. Declared here rather than assumed by the backfill, so a mark that
#: *does* want its body only has to say so.
fetch_body: bool = False
#: What an embedded frame of such an item would actually show, for the
#: reader offered a control they cannot use. Here for the same reason
#: `fetch_body` is: framing is a shared behaviour, and the mark is what
#: knows whether a frame of it would be worth anything. See
#: `OriginalView.blocked_by`.
framed: str = ""
MARKS: tuple[Mark, ...] = (
Mark("paywalled", "\U0001F512", "Paywalled",
"Behind this outlet's paywall", framed="the outlet's paywall"),
Mark("sponsored", "\U0001F4E3", "Sponsored",
"An advertisement, not journalism", framed="the advert"),
)
BY_SLUG = {m.slug: m for m in MARKS}
#: The marks whose items the body backfill should not spend a request on.
NO_BODY = frozenset(m.slug for m in MARKS if not m.fetch_body)
class Rule(Protocol):
@property
def field(self) -> str: ...
@property
def method(self) -> str: ...
@property
def value(self) -> str: ...
@property
def mark(self) -> str: ...
@property
def strip(self) -> bool: ...
def classify(item: Item, rules: Sequence[Rule]) -> Item:
"""Classify original inputs, retaining independent acquisition marks.
Evidence lives in raw alongside the publisher categories until those
acquisition inputs gain typed storage. Repeating this replaces only marks
owned by the previous rule pass, never a custom acquisition hook's marks.
"""
previous = item.raw.get("mark_evidence", {})
original = item.source_title or item.title
independent = previous.get("independent", item.marks)
found: list[str] = []
evidence: list[dict[str, Any]] = []
spans: list[tuple[int, int]] = []
for rule in rules:
if rule.field == "inputs.source_categories":
values = item.raw.get("categories", [])
matched = next((v for v in values if isinstance(v, str)
and v.lower() == rule.value.lower()), None)
at = 0 if matched is not None else -1
value = matched or ""
else:
value = item.excerpt if rule.field == "inputs.feed_text" else original
lower, token = value.lower(), rule.value.lower()
at = lower.find(token)
if rule.method == "prefix" and not lower.startswith(token):
at = -1
elif rule.method == "suffix":
at = len(value) - len(rule.value) if lower.endswith(token) else -1
if at < 0:
continue
found.append(rule.mark)
evidence.append({"field": rule.field, "method": rule.method,
"value": rule.value, "mark": rule.mark,
"strip": rule.strip,
"matched": value[at:at + len(rule.value)]})
if rule.strip:
spans.append((at, at + len(rule.value)))
item.source_title = original
item.title = ("".join(c for i, c in enumerate(original)
if not any(start <= i < end for start, end in spans)).strip() or original
) if spans else original
item.marks = sorted(set(independent) | set(found))
item.raw["mark_evidence"] = {"version": 1, "independent": list(independent),
"matches": evidence}
return item
+193
View File
@@ -0,0 +1,193 @@
"""A Mastodon or Bluesky post read through its network's public API.
Both web apps serve a script shell to a plain request, so `checks` names these
getters before `browser`: the API answers in a fraction of a second, needs no
Chromium and no login, and returns the post rather than a rendering of it.
Each getter builds a small HTML page from the answer, so `scrape` checks and
converts it exactly as it would a publisher's page.
from desk.tools import posts
posts.is_mastodon("https://hachyderm.io/@simontatham/117201594980991062")
page = posts.mastodon(url) # checks.Page
page = posts.bluesky("https://bsky.app/profile/bsky.app/post/3mv3shqdfuc2e")
The Mastodon and Bluesky plugins register them with `scrape` as the
`mastodon_api` and `bluesky_api` getters for their URLs; see `Source.getters`.
**A thread is read as far as its author continued it.** A post a link points
to is often the first of several by the same person; the replies by anyone
else are not part of it.
X is not here: its API bills every read and needs a token, and x.com shows a
browser a login wall.
GET https://{host}/api/v1/statuses/{id} Mastodon, any instance
GET https://{host}/api/v1/statuses/{id}/context
GET https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread
"""
from __future__ import annotations
import html
import re
from typing import Any
from urllib.parse import urlsplit
import httpx
from desk.tools.checks import Page
TIMEOUT = 20.0
#: Says what is asking, the way `desk.sources.base.USER_AGENT` does for the
#: plugins; a tool sits below `sources` and cannot import it.
USER_AGENT = "newsdesk (+https://github.com/; reading a linked post)"
BLUESKY_API = "https://public.api.bsky.app/xrpc"
#: `/@user/123` is what an instance links; `/users/user/statuses/123` is the
#: ActivityPub id some links carry instead.
_MASTODON_PATH = re.compile(r"^/(?:@[^/]+|users/[^/]+/statuses)/(\d+)/?$")
_BLUESKY_PATH = re.compile(r"^/profile/([^/]+)/post/([a-z0-9]+)/?$")
#: How deep a Bluesky thread is asked for. A self-thread longer than this is
#: cut, which is rare and costs the tail rather than the post.
THREAD_DEPTH = 20
def _get(url: str, params: dict[str, Any] | None = None) -> httpx.Response:
# Module-level, so a test replaces this one function.
return httpx.get(url, params=params, timeout=TIMEOUT,
headers={"User-Agent": USER_AGENT}, follow_redirects=True)
# ---------------------------------------------------------------- Mastodon
def _mastodon_id(url: str) -> tuple[str, str] | None:
parts = urlsplit(url)
found = _MASTODON_PATH.match(parts.path)
return (parts.netloc, found.group(1)) if parts.scheme == "https" and found else None
def is_mastodon(url: str) -> bool:
"""Whether `url` has a Mastodon post's shape. Any host: an instance is
recognised by the check that found its web app's shell, not by name."""
return _mastodon_id(url) is not None
def mastodon(url: str) -> Page | None:
"""The post at `url`, and its author's continuation, from the instance's API."""
host, status_id = _mastodon_id(url) or ("", "")
answer = _get(f"https://{host}/api/v1/statuses/{status_id}")
if not _json(answer):
# Not an instance: `/@user/123` is a Mastodon address's shape, and
# another site can have it. Nothing to say about the page, so the
# ordinary path runs.
return None
if answer.status_code != 200:
return Page(answer.status_code, "", url, dict(answer.headers))
status = answer.json()
author = status.get("account") or {}
thread = [status]
context = _get(f"https://{host}/api/v1/statuses/{status_id}/context")
if context.status_code == 200 and _json(context):
# Descendants come in thread order; the author's continuation is the
# chain of their replies, each to the one before.
for reply in context.json().get("descendants") or []:
if ((reply.get("account") or {}).get("id") == author.get("id")
and reply.get("in_reply_to_id") == thread[-1].get("id")):
thread.append(reply)
name = author.get("display_name") or author.get("acct") or host
card = status.get("card") or {}
return Page(200, _page(
title=f"{name}: {_first_words(_text(status.get('content', '')))}",
byline=f"{name} (@{author.get('acct', '')})",
published=status.get("created_at", ""),
paragraphs=[s.get("content", "") for s in thread],
link=(card.get("url", ""), card.get("title", "")),
alts=[m.get("description") or "" for s in thread
for m in s.get("media_attachments") or []],
escaped=False), url)
# ---------------------------------------------------------------- Bluesky
def _bluesky_uri(url: str) -> str:
parts = urlsplit(url)
found = _BLUESKY_PATH.match(parts.path)
if parts.netloc not in {"bsky.app", "www.bsky.app"} or not found:
return ""
return f"at://{found.group(1)}/app.bsky.feed.post/{found.group(2)}"
def is_bluesky(url: str) -> bool:
return bool(_bluesky_uri(url))
def bluesky(url: str) -> Page | None:
"""The post at `url`, and its author's continuation, from Bluesky's AppView."""
answer = _get(f"{BLUESKY_API}/app.bsky.feed.getPostThread",
{"uri": _bluesky_uri(url), "depth": THREAD_DEPTH, "parentHeight": 0})
if answer.status_code != 200:
# A post that is gone is a 400 `NotFound` here, not a 404.
gone = answer.status_code == 400 and "NotFound" in answer.text
return Page(404 if gone else answer.status_code, "", url, dict(answer.headers))
thread = answer.json().get("thread") or {}
post = thread.get("post") or {}
author = post.get("author") or {}
posts = [post]
node = thread
while True:
# The author's own reply, if there is one, is the next part of the thread.
nxt = next((r for r in node.get("replies") or []
if ((r.get("post") or {}).get("author") or {}).get("did") == author.get("did")),
None)
if nxt is None:
break
posts.append(nxt["post"])
node = nxt
record = post.get("record") or {}
external = ((post.get("embed") or {}).get("external") or {})
name = author.get("displayName") or author.get("handle", "")
return Page(200, _page(
title=f"{name}: {_first_words(record.get('text', ''))}",
byline=f"{name} (@{author.get('handle', '')})",
published=record.get("createdAt", ""),
paragraphs=[(p.get("record") or {}).get("text", "") for p in posts],
link=(external.get("uri", ""), external.get("title", "")),
alts=[image.get("alt") or "" for p in posts
for image in (p.get("embed") or {}).get("images") or []],
escaped=True), url)
# ---------------------------------------------------------------- the page
def _page(*, title: str, byline: str, published: str, paragraphs: list[str],
link: tuple[str, str], alts: list[str], escaped: bool) -> str:
"""An article page for the converter. Mastodon's content is HTML already;
Bluesky's is text, escaped here and split into paragraphs on blank lines."""
if escaped:
paragraphs = ["".join(f"<p>{html.escape(block).replace(chr(10), '<br>')}</p>"
for block in re.split(r"\n\s*\n", text) if block.strip())
for text in paragraphs]
body = "\n".join(paragraphs)
url, label = link
if url:
body += f'\n<p><a href="{html.escape(url)}">{html.escape(label or url)}</a></p>'
body += "".join(f"\n<p>[Image: {html.escape(alt)}]</p>" for alt in alts if alt.strip())
return (f"<!DOCTYPE html><html><head><title>{html.escape(title)}</title></head><body>"
f"<article><header><p>{html.escape(byline)}</p>"
f"<time datetime=\"{html.escape(published)}\">{html.escape(published)}</time></header>"
f"\n{body}\n</article></body></html>")
def _json(answer: httpx.Response) -> bool:
return "json" in answer.headers.get("content-type", "")
def _text(markup: str) -> str:
return html.unescape(re.sub(r"<[^>]+>", " ", markup))
def _first_words(text: str, count: int = 8) -> str:
words = text.split()
return " ".join(words[:count]) + ("…" if len(words) > count else "")
+240
View File
@@ -0,0 +1,240 @@
"""Get an article out of a URL: fetch, check, convert, and fetch another way
when the check says so.
Three parts, each replaceable on its own:
- a **getter** fetches a page: `HTTP`, `BROWSER` (Playwright, with the
`browser` extra), or one a plugin registered for its own URLs, such as a
post through its network's API
- `desk.tools.checks` **checks** the page and answers with a `Verdict`: the
failure category, what matched, and which getters to try next
- `desk.tools.article` **converts** HTML that passed into the structured
article, preserving code blocks and tables
got = scrape(url) # standard: HTTP, check, convert
got = scrape(url, getters=ADVANCED) # a failed check may try another getter
got = scrape(url, getters=advanced(cache, plugin_getters))
# and a consent the browser accepted is sent by
# HTTP next time, and a plugin's URLs go to it
**A getter that claims a URL goes first.** A plugin that knows how to read its
own network's addresses registers a getter with `claims=True`: when `handles`
matches, it runs before `http`. What it brings back is converted like any
page. A status it answers with ends the attempt, because the network's own API
is the authority on its posts; no answer at all, or a page that fails the
check, and the ordinary path runs as if it had not been asked.
Standard and advanced run the same loop. The only difference is which getters
are available to a verdict: with `http` alone a bot check ends as
`js_required` on the row, with `browser` available the verdict's `next` sends
the same URL there, and a page that passes is converted exactly as an HTTP one
would have been. A getter named by a verdict and not available is skipped, so
the checks can say where a page should go before that getter exists.
Nothing but a verdict moves to another getter. A page that passed the check
and converted to nothing is `empty` and ends there, because an extractor that
finds no article in real HTML finds none in the same HTML from a browser.
"""
from __future__ import annotations
import copy
import logging
from collections.abc import Callable, Iterable, Mapping, MutableMapping
from configparser import ConfigParser
from dataclasses import dataclass
from desk.tools import article, browser, failures
from desk.tools.article import ArticleContent
from desk.tools.checks import Page, Verdict, check
log = logging.getLogger(__name__)
@dataclass(frozen=True)
class Getter:
"""One way of fetching a page. `fetch` returns None for a request that
never completed and raises for anything it can name. `handles` is for a
getter that only knows some hosts, such as a platform's API; one that does
not handle a URL is skipped without an attempt."""
name: str
fetch: Callable[[str], Page | None]
handles: Callable[[str], bool] = lambda url: True
#: Run before `http` for every URL `handles` matches, rather than only
#: when a check names it. For a getter that owns a kind of address.
claims: bool = False
@dataclass(frozen=True)
class Attempt:
getter: str
failure: str = ""
#: The signature or rule that failed the check, empty on success.
matched: str = ""
status: int = 0
@dataclass(frozen=True)
class Scraped:
"""The article, or the failure of the last getter tried, and every try."""
content: ArticleContent | None
failure: str
getter: str
attempts: tuple[Attempt, ...]
#: The server's own Retry-After, in seconds, when one was given.
retry_after: float | None = None
def _http(url: str, cookie: str = "") -> Page | None:
# Looked up per call, so a test that stubs the downloader stubs this.
import trafilatura
if cookie:
answer = trafilatura.fetch_response(url, decode=True, with_headers=True,
config=_with_cookie(cookie))
else:
answer = trafilatura.fetch_response(url, decode=True, with_headers=True)
if answer is None:
return None
html = answer.html or ""
data = getattr(answer, "data", None)
if isinstance(data, bytes) and data[:4] == ZSTD_MAGIC:
html = _unzstd(data)
return Page(status=answer.status, html=html,
url=getattr(answer, "url", "") or url,
headers=answer.headers or {})
#: The first bytes of a zstd frame.
ZSTD_MAGIC = b"\x28\xb5\x2f\xfd"
def _unzstd(data: bytes) -> str:
"""A zstd body as the HTML trafilatura would have decoded.
Python 3.14 has `compression.zstd`, so urllib3 offers zstd, and Cloudflare
hosts such as opendatascience.com answer with it. Streaming the body,
urllib3 hands trafilatura the compressed bytes; trafilatura decompresses
zstd only with the `zstandard` package, which refuses a frame that does not
state its size, and Cloudflare's frames do not. Undecoded, the page is
binary that passes the check and converts to `empty`.
"""
from compression import zstd
from trafilatura.utils import decode_file
return decode_file(zstd.decompress(data))
def _with_cookie(cookie: str) -> ConfigParser:
"""trafilatura's own config with a `Cookie` header in it.
Its downloader takes no headers, only a config. A config that is not the
default replaces the default headers, so the user agent is set again to
the one it would have sent. `%` is escaped for `ConfigParser`'s
interpolation; consent strings are full of it.
"""
from trafilatura.downloads import USER_AGENT
from trafilatura.settings import DEFAULT_CONFIG
config = copy.deepcopy(DEFAULT_CONFIG)
config.set("DEFAULT", "USER_AGENTS", USER_AGENT)
config.set("DEFAULT", "COOKIE", cookie.replace("%", "%%"))
return config
def _browser(url: str, cache: MutableMapping[str, str] | None = None) -> Page | None:
got = browser.render(url)
if cache is not None and got.consent:
browser.remember(cache, got.url, got.cookies)
return Page(status=got.status, html=got.html, url=got.url, headers=got.headers)
#: One plain request through trafilatura's downloader: its headers, its size
#: cap, and the status and headers handed back rather than thrown away.
HTTP = Getter("http", _http)
#: Headless Chromium. Accepts a consent dialog it recognises; see
#: `desk.tools.browser`.
BROWSER = Getter("browser", _browser)
STANDARD: Mapping[str, Getter] = {"http": HTTP}
#: The getters this module has itself. `browser` only with the `browser`
#: extra installed; a verdict naming a getter that is not here skips it. The
#: rest come from plugins, through `advanced`.
ADVANCED: Mapping[str, Getter] = {
"http": HTTP, **({"browser": BROWSER} if browser.installed() else {})}
def advanced(cache: MutableMapping[str, str],
registered: Iterable[Getter] = ()) -> Mapping[str, Getter]:
"""`ADVANCED` and the getters plugins `registered`, sharing one cookie jar.
The browser keeps the cookies of a consent it accepted in `cache`, per
host, and HTTP sends them. So a gate costs one render per host until its
cookies expire, and every other article from that host is a plain request.
A registered getter cannot replace `http` or `browser`.
"""
getters = {getter.name: getter for getter in registered}
getters.update(ADVANCED)
getters["http"] = Getter("http", lambda url: _http(url, browser.cookie_header(cache, url)))
if "browser" in getters:
getters["browser"] = Getter("browser", lambda url: _browser(url, cache))
return getters
def scrape(url: str, getters: Mapping[str, Getter] = STANDARD,
first: str = "http") -> Scraped:
"""A claiming getter, then `first`, then whatever each failed check names."""
attempts: list[Attempt] = []
hint: float | None = None
for claimant in getters.values():
if not (claimant.claims and claimant.handles(url)):
continue
content, verdict, page = _one(claimant, url)
attempts.append(Attempt(claimant.name, "" if content else verdict.category,
"" if content else verdict.name,
page.status if page else 0))
if content is not None:
return Scraped(content, "", claimant.name, tuple(attempts), hint)
if page is not None and page.status >= 400:
return Scraped(None, verdict.category, claimant.name, tuple(attempts),
failures.hint(page.headers))
log.debug("%s: %s via %s (%s); the ordinary path runs", url,
verdict.category, claimant.name, verdict.name)
plan = [first]
while plan:
getter = getters.get(plan.pop(0))
if getter is None or not getter.handles(url) or any(
a.getter == getter.name for a in attempts):
continue
content, verdict, page = _one(getter, url)
if content is not None:
attempts.append(Attempt(getter.name, status=page.status if page else 0))
return Scraped(content, "", getter.name, tuple(attempts), hint)
attempts.append(Attempt(getter.name, verdict.category, verdict.name,
page.status if page else 0))
if page is not None and page.status >= 400:
hint = hint or failures.hint(page.headers)
log.debug("%s: %s via %s (%s)", url, verdict.category, getter.name, verdict.name)
plan = list(verdict.next)
# The row gets the last answer a page gave. A fallback that brought back
# nothing (a browser that would not start, an API that did not answer)
# says nothing about the page, and must not replace what the check found.
answered = [a for a in attempts if a.status] or attempts
last = answered[-1] if answered else Attempt("", "unknown")
return Scraped(None, last.failure, last.getter, tuple(attempts), hint)
def _one(getter: Getter, url: str) -> tuple[ArticleContent | None, Verdict, Page | None]:
try:
page = getter.fetch(url)
except Exception as exc:
return None, Verdict(failures.of_exception(exc), type(exc).__name__), None
if verdict := check(page):
return None, verdict, page
assert page is not None # check() refuses a missing page
content = article.extract_article(page.html, url)
if content is None or not content.plain_text.strip():
return None, Verdict("empty", "no_article"), page
return content, Verdict("", ""), page
+100
View File
@@ -0,0 +1,100 @@
# `desk.tools` — the machinery
## What it is
Things a pipeline stage picks up and puts down: plain inputs, plain outputs, no idea what the app is for. The test for belonging here is *would this make sense in a different application*.
| Tool | What it does | Needs |
|---|---|---|
| `agent` | ask Claude, get JSON back | `anthropic` **or** `claude-agent-sdk` (both optional) |
| `article` | supplied HTML to typed structured content, preserving tables and code; derived plain text and acquisition provenance | `trafilatura` |
| `browser` | a page rendered in headless Chromium, past a consent dialog it knows; the cookies it earned, per host | `playwright` (optional, plus `playwright install chromium`) |
| `embedding` | text in, vector out (~384 floats) | `sentence-transformers` (optional, ~100MB+) |
| `failures` | what went wrong, in one word a retry policy can be written against | stdlib |
| `framing` | will this page load in an iframe? | `httpx` |
| `http` | one shared HTTP client per call chain | `httpx` |
| `marks` | what kind of thing is this, when it isn't an ordinary article | stdlib |
| `posts` | a Mastodon or Bluesky post and its author's thread, through the public API, as a page to convert; registered by those plugins | `httpx` |
| `checks` | is a fetched page worth converting, and if not, which getter to try next | stdlib |
| `scrape` | fetch, check, convert, and fetch another way when the check says so | `trafilatura` |
| `walls` | did we fetch the article, or the consent page in front of it | stdlib |
| `video/` | a recording, read as text with timings | `httpx`, `youtube-transcript-api` |
**Nothing is imported in `desk/tools/__init__.py` on purpose** — each import is paid for by the caller that wants it. Import the module, not the package: `from desk.tools import agent`.
## `agent` — the model
Two interchangeable backends behind one call. `agent.backend(cfg)` says which will answer, or `""` for nobody — which is a failure, not a third backend.
- `api` — one Claude API call. Wants `ANTHROPIC_API_KEY`. The unattended path.
- `claude_code` — the Agent SDK drives the `claude` CLI on your subscription. No key, nothing billed. The development path. (**Spawns a process.**)
- `off` / nothing available — `backend()` answers `""`, and the caller fails the batch with a notice naming the reason. Nothing waits to be answered by hand.
```python
from desk.tools import agent
reply = agent.ask_json(
config.agent, # desk.config.AgentConfig
system="You tag articles.", # kept byte-identical across a run — it is what caches
question={"articles": [...]},
schema={"type": "object", ...}, # both backends take the same JSON schema
model="claude-haiku-4-5-20251001",
thinking="off", # or "adaptive"
on_partial=lambda piece: ..., # streamed text, for per-item progress
)
reply.data, reply.backend, reply.model, reply.cost_usd, reply.usage
```
Failures of every kind raise `AgentError` — treat it as "no answer this time". The `claude_code` path deliberately gets **no tools, no CLAUDE.md, no skills, no MCP, no session**: one text-in, JSON-out turn, the same shape as the API call.
## The others, in one line each
`article.extract_article(html, source_url)` returns `ArticleContent` or `None` when extraction finds nothing. `structured-v1` is a typed tree, not trusted HTML; `plain` supports legacy strings. `plain_text` and `revision` are derived from content. Explicit gutter nodes are removed before extraction, while meaningful numeric lines and code whitespace remain. Completeness defaults to unknown. Newsdesk's body-fetch job now uses this tool and atomically persists its content and derived plain projection; the reader renders its closed node vocabulary. Fetching, persistence, wall detection and rendering remain the caller's responsibilities. The extractor requires trafilatura 2.2 or newer, verified against the representative fixture.
```python
from desk.tools import embedding, failures, framing, marks, video, walls
from desk.tools.http import session
vecs = embedding.load("all-MiniLM-L6-v2").encode(["some text"]) # (n, dim) ndarray
verdict = framing.check(cache, "https://x.com/a") # {embeddable, reason, host, certain}; per host
item = marks.mark(item, {"paywalled": "(g+)"}) # cuts the token out, records the mark
reason = walls.note(cache, url, body) # "" or why this body is the same page every time
why = failures.of_status(503) # 'server_error'; of_exception(exc) for a raise
wait = failures.hint(resp.headers) # the server's own Retry-After, in seconds
vid, segments = video.read("https://youtu.be/GUYEofBebDc") # Video + Segment[]
with session(client) as http: ... # borrows one if given, makes and closes one if not
```
`failures` is the third of those, and the newest. A category answers one question — *would asking again get a different answer?* — so `TRANSIENT` (`timeout`, `unreachable`, `rate_limited`, `server_error`, `not_ready`) and `SETTLED` (`not_found`, `denied`, `consent_wall`, `js_required`, `empty`) are the whole of the vocabulary, plus `unknown` for a failure nothing recognises, which is deliberately in neither. Those names are what a plugin's `RetryPolicy.on` is validated against, so adding one here adds one to the plugin contract. An exception is named by walking the class names in its ancestry rather than by `isinstance`, because `httpx` and `urllib3` are both optional and either may be the one that raised — it costs precision and buys a tool that can name a failure raised by a library it has never imported. A consent wall is a category and is in no default retry list: it is the same answer however many times it is asked for.
`marks` and `walls` are the two that read what came back rather than fetching anything, which is why neither needs a dependency. `marks` turns an outlet's own notation — Golem's `(g+)`, Hacker News's `[$]` — into one glyph that means the same thing whoever published it, and cuts the token out of the title. `walls` catches the opposite problem: a site that answers every article URL with its consent gate, which trafilatura extracts faithfully and which is worse than storing nothing. The tell is that two different articles cannot have the same body and two walls always do, so the detector is a fingerprint and a cache, with no per-site rules to keep up to date.
`python -m desk.tools.video <url>` prints a video's segments — a dry run with no app around it.
## `scrape` and `checks` — getting the article out of a URL
`scrape.scrape(url, getters)` fetches with `http`, runs `checks.check` on the page, converts a page that passed with `article.extract_article`, and on a failed check tries the getters the `Verdict` names. It returns `Scraped`: the content or the last failure, the getter that answered, every `Attempt` with the signature that failed it, and the server's `Retry-After`.
```python
from desk.tools import checks, scrape
got = scrape.scrape(url) # STANDARD: http only
got = scrape.scrape(url, scrape.ADVANCED) # a verdict may send it to another getter
got = scrape.scrape(url, scrape.advanced(cache, getters)) # what the body fetch uses: plugins' URL getters first,
# and HTTP sends the cookies a browser earned
verdict = checks.check(checks.Page(200, html)) # None, or Verdict(category, name, next)
```
`browser.render(url)` launches Chromium, clicks the accept button of a dialog in `browser.CONSENTS` if one appears within `GATE_WAIT` seconds, and returns `Rendered`: status, HTML, final URL, headers, the host's cookies and which consent it accepted. `remember(cache, url, cookies)`, `cookie_header(cache, url)` and `forget(cache, url)` are the jar, in the same kind of mapping `walls` uses. One browser per call and one at a time per process; `browser.installed()` says whether the extra is there, and `ADVANCED` has `browser` only if it is.
How the loop runs, the signature format and how to add one: `docs/architecture.md` §2.1.
## Limits
- **All synchronous and blocking.** Meant to be called from a worker thread.
- **`ask_json` is one turn, no tools, no conversation.** No streaming API for the caller beyond `on_partial` (raw text pieces, not parsed JSON); chunking, retries and applying the answer are `desk.modeltask`'s job.
- **Optional deps really are optional, but "optional" is not "interchangeable".** A missing model SDK is an `AgentError`, and there are two backends, so the other one may well answer. A missing `sentence-transformers` is an `ImportError` and there is nothing behind it: `ranking.backend: embeddings` does **not** fall back to tfidf. The embed step logs the failure and returns zero, articles keep no vectors, and everything scores a neutral 0.5 for taste and ranks on its recency alone. Which reads a lot like a ranker that is working and is not.
- **`cost_usd` comes from the `claude_code` path, not the API one.** Backwards from what you would guess: the CLI's `ResultMessage` carries what those tokens *would* have cost, so it is reported even though a subscription bills nothing for them, and the API path — the one that really is billed — leaves the field at zero because the Messages response does not price itself. Read it as an estimate of what a run would cost on the API, and never as a bill.
- **`video` is YouTube-only today.** No local-file transcription yet (yt-dlp + whisper is the planned third backend); a video with no captions still yields chapters, just no words.
- **`framing` answers per host, not per article**, and caches; most newspapers answer "no".
- **No tool knows about the store, the queue, or config beyond its own section** — and `tools` must not import `pipeline` or `sources`.
+103
View File
@@ -0,0 +1,103 @@
"""Video, read as text with timings on it.
The generic half of watching things so you don't have to. Give it a video and
it gives back a list of `Segment`s — each one a stretch of the recording with a
title, the words spoken in it, and a link that plays it from that second.
read() a video -> Video + Segment[] the whole thing, one call
youtube captions, metadata, comments where both inputs come from
marks an outline -> chapter marks description, notes, comments
media a MediaRef -> a playable link YouTube today, a file later
segment cues + marks -> Segment[] the cut, and transcript text
**This package knows nothing about newsdesk.** No store, no queue, no config,
no `Item`. That is what makes it reusable, and it is the seam the design is
built on: `desk/sources/podcast.py` sits on top and adds the things that
*are* specific to a show — its feed, its weekly cadence, the article links in
its notes, and turning segments into rows.
Two decisions worth knowing before reading further.
**A video is a `MediaRef`, not a URL.** Today every ref is a YouTube id; the
reading pane's link is computed from it on the way out. Point one at a file on
disk in config and every link to it changes, with nothing rewritten and no
item's identity touched. See `media.py`.
**Transcripts come from whoever can supply one.** `youtube-transcript-api`
first, and a built-in fetcher against YouTube's own player endpoint when the
library can't get the track. Transcribing audio locally (yt-dlp plus
faster-whisper) is the third backend and the one that makes local video work;
it is not written yet, and `read` is shaped so that adding it changes nothing
above.
Try it:
python -m desk.tools.video https://www.youtube.com/watch?v=GUYEofBebDc
"""
from __future__ import annotations
import logging
import httpx
from desk.tools.video import marks, media, youtube
from desk.tools.http import session
from desk.tools.video.marks import Chapter, Link
from desk.tools.video.media import MediaRef
from desk.tools.video.segment import Segment, Video, cut, opening, paragraphs
from desk.tools.video.youtube import Blocked, Cue, TranscriptUnavailable
log = logging.getLogger(__name__)
__all__ = [
"Blocked", "Chapter", "Cue", "Link", "MediaRef", "Segment",
"TranscriptUnavailable", "Video", "cut", "describe", "marks", "media",
"opening", "paragraphs", "read", "youtube",
]
def read(url: str, *, comments: bool = True,
client: httpx.Client | None = None) -> tuple[Video, list[Segment]]:
"""One video, cut into its chapters. The whole package in one call.
`comments` is the only cost worth thinking about: it is an extra request to
the watch page and is only consulted when the description has no outline in
it, so for a show that publishes chapter marks it never happens.
"""
with session(client, timeout=40) as http:
video = describe(url, client=http)
try:
cues = youtube.transcript(video.key, client=http)
except TranscriptUnavailable as exc:
# Not fatal on its own. The outline still gives every chapter, its
# links and its timecode; only the quoted words are missing, and
# auto-captions usually appear within a few hours of an upload.
log.warning("%s", exc)
cues = []
chapters = marks.best(video.notes)
if not chapters and comments:
chapters = marks.in_comments(youtube.comments(video.key, client=http))
return video, cut(video, cues, chapters)
def describe(url: str, *, client: httpx.Client | None = None) -> Video:
"""What a video says about itself: title, author, length, description.
YouTube only, for now. A local file would come back with its stem as the
title and no notes, which is why `Video.notes` is allowed to be empty and
why `read` has a comments fallback at all.
"""
key = youtube.video_id(url)
with session(client) as http:
meta = youtube.details(key, client=http)
return Video(
ref=media.youtube(key),
title=meta["title"],
url=media.youtube(key).at(0),
author=meta["author"],
duration=meta["duration"],
notes=meta["description"],
)
+120
View File
@@ -0,0 +1,120 @@
"""Cut a video up and print what came out.
python -m desk.tools.video <youtube url>
python -m desk.tools.video <youtube url> --json > segments.json
Videos only. A podcast episode — where the chapter marks and the article links
live on the show's own page rather than on YouTube — goes through the layer
above:
python -m desk.sources.podcast <episode url | feed url>
"""
from __future__ import annotations
import argparse
import dataclasses
import json
import logging
import sys
import desk.tools.video as video_pkg
from desk.tools.video import marks
from desk.tools.video.segment import Segment, Video
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m desk.tools.video",
description="Cut a video into one item per chapter mark.",
)
parser.add_argument("url", help="a YouTube link or bare video id")
parser.add_argument("--json", action="store_true",
help="dump everything, transcripts included, as JSON")
parser.add_argument("--chars", type=int, default=160,
help="how much transcript to preview per segment (default 160)")
parser.add_argument("--no-comments", action="store_true",
help="don't fall back to chapter marks typed in the comments")
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(
level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)-7s %(message)s",
)
# Windows still opens a console in the system codepage, and a video title
# with a "ñ" in it — or the arrows this prints — is an unhandled
# UnicodeEncodeError half way through the report rather than a bad glyph.
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
try:
video, segments = video_pkg.read(args.url, comments=not args.no_comments)
except Exception as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
if args.json:
json.dump(dump(video, segments), sys.stdout, indent=2, default=str)
print()
else:
report(video, segments, args.chars)
return 0
def report(video: Video, segments: list[Segment], chars: int) -> None:
"""The human-readable view. Also used by the podcast module."""
words = sum(len(s.transcript.split()) for s in segments)
length = video.duration or (segments[-1].end if segments else 0)
print(f"{video.title}")
print(f" {video.author or '—'} {video.ref.kind}:{video.ref.key}"
f" {marks.hhmmss(length)}")
print(f" {len(segments)} segments, {words:,} words of transcript")
print()
heading = None
for segment in segments:
if segment.heading != heading:
heading = segment.heading
print(f" {heading or '(unfiled)'}")
preview = segment.transcript[:chars]
if len(segment.transcript) > chars:
preview += "…"
print(f" {segment.timecode:>8} {segment.title[:78]}")
print(f" {marks.hhmmss(segment.duration)} long · {segment.url()}")
for link in segment.links:
print(f" → {link.url}")
print(f" {preview or '(no transcript under this mark)'}")
print()
def dump(video: Video, segments: list[Segment]) -> dict[str, object]:
"""The shape a stored item is built from, plus the fields a dry run wants
to eyeball. The stored half is `Segment.record()` and is defined there;
desk/sources/podcast.py turns the rest into a row."""
return {
"video": {
"ref": dataclasses.asdict(video.ref),
"title": video.title,
"url": video.url,
"author": video.author,
"published_at": video.published_at,
"duration": video.duration,
},
"segments": [
{
**s.record(),
"external_id": s.external_id,
"title": s.title,
"heading": s.heading,
"timecode": s.timecode,
"url": s.url(),
"transcript": s.transcript,
}
for s in segments
],
}
if __name__ == "__main__":
raise SystemExit(main())
+265
View File
@@ -0,0 +1,265 @@
"""Chapter marks: a video's own table of contents, wherever it was written.
The important discovery about this kind of video is that **the segmentation
already exists and somebody else did it**. Last Week in AI ships every episode
with a timestamped outline — a rubric, then one line per story, each with the
time it starts and a link to the article being discussed:
Tools & Apps
(00:03:32) Google announces Gemini 3.7 Flash ... - Ars Technica
(00:13:11) SpaceXAI Releases Grok 4.6 ... - MarkTechPost
That is a better set of boundaries than any topic-segmentation model would
produce, for nothing, with the source links attached. So this module parses it
rather than inferring it. An LLM is still wanted downstream — to summarise what
was *said* in each stretch — but not to work out where one stretch ends.
**Three places to find that outline**, in descending order of usefulness, all
parsed by the same function because they are the same convention written in
different markup:
1. **The show's own page or feed body** — HTML, so the links are real
`<a href>` and each chapter arrives with the article it is about. Only a
show with a website has this, which is why the podcast source hands it in
rather than this module going to look.
2. **The video description** — plain text. Usually the same outline with the
links flattened away.
3. **The comments** — see `in_comments`. When a show publishes no marks at
all, a listener has often typed some out; YouTube itself reads those to
draw its chapter markers.
`best` takes them in that order and returns the first that is real. Nothing
here is specific to one show beyond the shape "timestamp, then a title,
optionally under a heading", which is the near-universal convention.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Iterable
from dataclasses import dataclass, field
from html.parser import HTMLParser
log = logging.getLogger(__name__)
#: `(00:03:32)`, `00:03:32`, `3:32` — at the start of the line, which is what
#: makes it a chapter mark rather than a time mentioned in a sentence.
TIMESTAMP = re.compile(
# The character class ends with hyphen, en dash and em dash on purpose:
# a show separates the time from the title with whichever of the three its
# editor's keyboard produced, and means the same thing by all of them.
r"^\(?\s*(\d{1,2}):([0-5]\d)(?::([0-5]\d))?\s*\)?[\s\-–—:]*" # noqa: RUF001
)
_URL = re.compile(r"https?://[^\s<>\"')\]]+")
#: Tags that end a line of the outline. Everything else is inline and stays
#: on the line it was found in, which is what keeps `<span>(00:03:32) </span>`
#: attached to the `<a>` that follows it.
_BLOCKS = frozenset({
"li", "p", "br", "div", "tr", "ul", "ol", "h1", "h2", "h3", "h4", "blockquote",
})
@dataclass(frozen=True, slots=True)
class Link:
"""One article the hosts were talking about."""
url: str
title: str = ""
@dataclass(slots=True)
class Chapter:
"""One entry in the outline: where a story starts and what it is about."""
start: int # seconds into the episode
title: str
heading: str = "" # the rubric it sits under
links: list[Link] = field(default_factory=list)
@property
def url(self) -> str:
"""The article being discussed, if the notes linked one."""
return self.links[0].url if self.links else ""
def parse(body: str) -> list[Chapter]:
"""Every chapter mark in a set of show notes, in the order they play.
Takes HTML or plain text and works out which it was given. Returns `[]`
for notes with no outline in them at all, which is a real answer — plenty
of shows don't write one, and the caller falls back to treating the episode
as a single undivided item rather than guessing at boundaries.
"""
lines = _html_lines(body) if "<" in body and ">" in body else _text_lines(body)
chapters: list[Chapter] = []
heading = ""
for text, links in lines:
match = TIMESTAMP.match(text)
if match is None:
# A heading is only a heading once the outline has started;
# before that every line is preamble — the blurb, the sponsors,
# the "Timestamps:" label itself.
if chapters and _is_heading(text, links):
heading = text.rstrip(":")
continue
title = text[match.end():].strip(" -–—|") # noqa: RUF001 — as above
# Prefer the anchor's own text: a line that links two follow-ups reads
# "First story | The Verge + Second story", and the first link's text
# is the story, where the whole line is the story plus its footnote.
if links and links[0].title:
title = links[0].title
chapters.append(Chapter(
start=_seconds(match), title=title, heading=heading, links=links,
))
if not chapters:
log.debug("no chapter marks found in %d chars of notes", len(body))
return chapters
def best(*candidates: str, minimum: int = 3) -> list[Chapter]:
"""The first candidate that yields a real outline.
Chapter marks come from up to three places and they are not equally good,
so the order they are passed in is the order of preference — for a podcast
that is: the show's own notes as HTML (links intact), then the YouTube
description (same outline, links flattened away), then the comments.
`minimum` is what stops a stray "at 3:20 he says" in a blurb becoming a
one-chapter outline that suppresses the good source behind it. Three marks
is the point where a line of text stops being a coincidence.
"""
for text in candidates:
found = parse(text)
if len(found) >= minimum:
return found
return []
def in_comments(comments: Iterable[str], minimum: int = 3) -> list[Chapter]:
"""The outline somebody typed into the comments, if anybody did.
Read as the fallback for shows that publish no chapter marks of their own,
where it is common for a listener to write one out — YouTube itself treats
such a comment as good enough to draw chapter markers from.
Each comment is parsed on its own rather than the lot being concatenated.
Joined together, an outline comment and thirty unrelated replies produce
one bogus outline interleaved with chatter; separately, the outline is the
comment with the most marks in it and the chatter scores zero.
"""
outlines = [parse(comment) for comment in comments]
outlines = [found for found in outlines if len(found) >= minimum]
if not outlines:
return []
richest = max(outlines, key=len)
log.info("took %d chapter marks from a comment", len(richest))
return richest
def hhmmss(seconds: float) -> str:
"""3932 -> '1:05:32'. What a timecode looks like to a reader."""
total = int(seconds)
hours, rest = divmod(total, 3600)
minutes, secs = divmod(rest, 60)
return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}"
# ------------------------------------------------------------------ internals
def _seconds(match: re.Match[str]) -> int:
a, b, c = match.group(1), match.group(2), match.group(3)
if c is None: # mm:ss
return int(a) * 60 + int(b)
return int(a) * 3600 + int(b) * 60 + int(c)
def _is_heading(text: str, links: list[Link]) -> bool:
"""A rubric like 'Policy & Safety', not a sentence and not a story.
Deliberately strict. Anything that gets through becomes the heading for
every chapter after it, so a false positive mislabels the rest of the
episode — where a false negative just leaves a few items unfiled.
"""
return (
not links
and len(text) <= 60
and len(text.split()) <= 6
and not text.endswith((".", "!", "?"))
)
def _text_lines(body: str) -> list[tuple[str, list[Link]]]:
"""Plain-text notes — a YouTube description. Bare URLs become links."""
lines = []
for raw in body.splitlines():
line = raw.strip()
if not line:
continue
links = [Link(url=u) for u in _URL.findall(line)]
lines.append((_URL.sub("", line).strip(), links))
return lines
def _html_lines(body: str) -> list[tuple[str, list[Link]]]:
parser = _Outline()
parser.feed(body)
parser.close()
return parser.lines
class _Outline(HTMLParser):
"""HTML to (line, links) pairs, without pulling in a parser library.
A full HTML tree would be nicer and is not worth a dependency for this:
the only structure that matters is where the line breaks fall and which
anchors landed on which line. `convert_charrefs` (on by default) means
entities arrive already decoded, so `Tools &amp; Apps` is `Tools & Apps`
by the time it reaches `handle_data`.
"""
def __init__(self) -> None:
super().__init__()
self.lines: list[tuple[str, list[Link]]] = []
self._text: list[str] = []
self._links: list[Link] = []
self._href = ""
self._anchor: list[str] = []
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if tag in _BLOCKS:
self._flush()
elif tag == "a":
self._href = dict(attrs).get("href") or ""
self._anchor = []
def handle_endtag(self, tag: str) -> None:
if tag in _BLOCKS:
self._flush()
elif tag == "a":
if self._href:
self._links.append(
Link(url=self._href, title=_squash("".join(self._anchor)))
)
self._href, self._anchor = "", []
def handle_data(self, data: str) -> None:
self._text.append(data)
if self._href:
self._anchor.append(data)
def _flush(self) -> None:
text = _squash("".join(self._text))
if text or self._links:
self.lines.append((text, self._links))
self._text, self._links = [], []
def close(self) -> None:
super().close()
self._flush()
def _squash(text: str) -> str:
return re.sub(r"\s+", " ", text or "").strip()
+113
View File
@@ -0,0 +1,113 @@
"""Where a video actually lives, and how to link into the middle of it.
Every segment newsdesk stores has to answer one question: *press play at
1:08:27 — where?* Today the answer is a YouTube URL. It will not always be.
The same episode may sit on disk, ripped or recorded; a show may take its
uploads down; a video may only ever have existed as a local file.
So nothing in this codebase stores a YouTube URL as a fact. It stores a
`MediaRef` — a kind and a key — and asks this module for a link when it needs
one. Swapping YouTube for a copy on disk is then a line of config rather than
an UPDATE across every item that mentioned it.
video:
local_media:
GUYEofBebDc: D:/media/lwiai/255.mp4
`resolve` applies that map. Everything downstream keeps working because it was
never holding a URL in the first place.
The one thing this deliberately does not do is *find* local copies. Matching
"is this file the same episode as that video" by filename or duration is a
guess, and a wrong guess points a citation at the wrong recording. The mapping
is written by a person, or it does not exist.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path, PurePath
#: A local file, playable in the reading pane's <video> element. Anything else
#: on disk (mkv, wav) still resolves; the pane just may not play it.
PLAYABLE = frozenset({".mp4", ".m4v", ".webm", ".ogg", ".mp3", ".m4a", ".wav"})
@dataclass(frozen=True, slots=True)
class MediaRef:
"""A video, identified by where it is rather than by a URL.
`kind` is what the key means: `youtube` is an 11-character video id,
`spotify` is an episode id, `file` is a path on this machine, `url` is
anything else somebody handed us. `key` is stable — it is what
`Segment.external_id` is built from — so re-pointing a video at a local
copy must *not* change it. That is why `resolve` returns a separate
playback ref instead of rewriting this one.
"""
kind: str
key: str
@property
def playable_locally(self) -> bool:
return self.kind == "file" and PurePath(self.key).suffix.lower() in PLAYABLE
def at(self, seconds: float = 0) -> str:
"""A link that opens this video at that moment.
The media fragment (`#t=`) is the same idea in every case: it is what
YouTube's `&t=` and Spotify's `?t=` do, what a `<video>` element
honours natively, and what most desktop players understand from a
`file://` URL.
"""
offset = max(0, int(seconds))
if self.kind == "youtube":
return f"https://www.youtube.com/watch?v={self.key}&t={offset}s"
if self.kind == "spotify":
return f"https://open.spotify.com/episode/{self.key}?t={offset}"
if self.kind == "file":
return Path(self.key).absolute().as_uri() + (f"#t={offset}" if offset else "")
return self.key + (f"#t={offset}" if offset else "")
def embed(self, seconds: float = 0) -> str:
"""The player a reading pane can frame at that moment, or "".
Only the hosts that publish a player built for framing have one: a
YouTube watch page and a Spotify episode page both refuse to be
framed, and each has an `/embed/` form that does not. A local file
needs none — the pane plays it in a `<video>` element — and an
arbitrary URL is not known to allow it.
"""
offset = max(0, int(seconds))
if self.kind == "youtube":
return f"https://www.youtube.com/embed/{self.key}?start={offset}"
if self.kind == "spotify":
return f"https://open.spotify.com/embed/episode/{self.key}?t={offset}"
return ""
def youtube(video_id: str) -> MediaRef:
return MediaRef(kind="youtube", key=video_id)
def local(path: str | Path) -> MediaRef:
return MediaRef(kind="file", key=str(path))
def resolve(ref: MediaRef, overrides: dict[str, str] | None = None) -> MediaRef:
"""The ref to *play*, after applying the local-media map.
Keyed on the original key, so a mapping written once keeps working however
the item was ingested. Returns the ref unchanged when there is no override
or the file has since gone — a stale mapping should fall back to the
original rather than hand out a link to a file that isn't there.
"""
if not overrides:
return ref
replacement = overrides.get(ref.key)
if not replacement:
return ref
path = Path(replacement).expanduser()
if not path.exists():
return ref
return local(path)
+261
View File
@@ -0,0 +1,261 @@
"""A video, and the pieces it breaks into.
This is the generic half and it knows nothing about podcasts, feeds or
newsdesk's tables. A `Video` is something with a transcript and a duration; a
`Segment` is a span of one, with a title, a start, an end and the words spoken
inside it. Cutting one into the other needs two inputs and no model:
Video --+-- transcript -> timed cues -----+
| +-> cut() -> Segment[]
+-- chapter marks (marks.best) ----+
What supplies those inputs is somebody else's problem, and deliberately so.
YouTube supplies both today. A show's own web page supplies better marks, with
the article links attached — that is what `desk/sources/podcast.py` does,
by passing its own marks in. A local recording plus a Whisper transcript would
supply both without touching this file.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass, field
from datetime import UTC, datetime
from typing import Any
from desk.models import ItemLink
from desk.tools.video import marks as marks_mod
from desk.tools.video import youtube
from desk.tools.video.marks import Chapter, Link
from desk.tools.video.media import MediaRef
log = logging.getLogger(__name__)
@dataclass(slots=True)
class Video:
"""One video, before anything has been done to it."""
ref: MediaRef
title: str = ""
#: The page it was published on, when that is a different thing from the
#: video itself — a Substack post, a show's episode page. Empty for a video
#: that is only a video.
url: str = ""
author: str = ""
published_at: datetime = field(default_factory=lambda: datetime.now(UTC))
duration: float = 0.0
#: Whatever text came with it and might carry chapter marks: a YouTube
#: description, a feed body, show notes as HTML.
notes: str = ""
@property
def key(self) -> str:
"""What identifies this video for the life of the archive.
The media *ref's* key, not a URL, and never the resolved playback
location — pointing an item at a local copy must not change its
identity or every segment of that video re-ingests as a new item.
"""
return self.ref.key
@dataclass(slots=True)
class Segment:
"""One span of a video: the unit that becomes a newsdesk item."""
video: Video
index: int
start: int
end: int
title: str
heading: str = "" # the rubric it was filed under
links: list[Link] = field(default_factory=list)
transcript: str = "" # verbatim, this span only
@property
def external_id(self) -> str:
"""Stable across re-ingests of the same video.
Keyed on the video and the start time rather than the position in the
list, so a show that inserts a chapter it forgot doesn't renumber —
and therefore doesn't duplicate — everything after it. Re-cutting
against corrected timestamps *does* produce new ids, which is the
honest outcome: they are different spans of the recording.
"""
return f"{self.video.key}#t={self.start}"
def url(self, overrides: dict[str, str] | None = None) -> str:
"""A link that plays this segment, wherever the video currently lives.
Resolved on the way out rather than stored, which is the whole point of
`media.MediaRef`: mapping a YouTube id to a file on disk changes every
link in the archive and rewrites nothing.
"""
from desk.tools.video import media
return media.resolve(self.video.ref, overrides).at(self.start)
@property
def duration(self) -> int:
return max(0, self.end - self.start)
@property
def timecode(self) -> str:
return marks_mod.hhmmss(self.start)
def record(self) -> dict[str, Any]:
"""The segment as a JSON-able dict: what newsdesk stores in `items.raw`.
One definition of the layout, so the two writers (a podcast segment,
a video added by hand) and the readers (the reading pane, the
summariser's query) agree on the keys. `media` is the reference and
never a URL — see `media.py`; `duration` is denormalised so a queue row
can show a length chip without arithmetic across two JSON extractions.
The articles are not in here: they are a typed collection of their own
now — see `reading`. `raw` is what no shared field has a name for, and
a link the whole reader understands is not that.
"""
return {
"media": {"kind": self.video.ref.kind, "key": self.video.ref.key},
"start": self.start,
"end": self.end,
"duration": self.duration,
"index": self.index,
}
def reading(self) -> list[ItemLink]:
"""What the hosts were reading, as links the whole app understands.
The outline's links with a role on them, which is the difference
between "a URL this show happened to print" and "the article this
stretch is about": the reader lists them under the player, the
summariser names them in the prompt, and neither has to know a podcast
produced them.
A line of show notes is whatever an editor typed, so a relative href
or a `mailto:` is an ordinary outcome rather than a fault — dropped
with a note, because one malformed link in the outline must not cost
the segment it sits on.
"""
out = []
for link in self.links:
try:
out.append(ItemLink(url=link.url, title=link.title, role="reading"))
except ValueError:
log.debug("%s: dropping unusable outline link %r",
self.external_id, link.url[:120])
return out
def cut(video: Video, cues: list[youtube.Cue],
chapters: list[Chapter] | None = None) -> list[Segment]:
"""Cut a video into one segment per chapter mark.
A chapter runs until the next one starts; the last one runs to the end of
the recording. That is the only boundary rule, and it is exact, because the
marks were written by the people who recorded it.
With no chapter marks at all the video comes back as a single segment
holding the whole transcript. That is deliberately not a failure: an
unsegmented video is still worth having, and inventing boundaries where
nobody published any is a job for a model and a later stage, not for a
slicing function.
"""
chapters = chapters if chapters is not None else marks_mod.parse(video.notes)
end_of_video = int(video.duration or (cues[-1].end if cues else 0))
if not chapters:
return [Segment(
video=video, index=0, start=0, end=end_of_video,
title=video.title, transcript=youtube.text_between(cues, 0),
)]
ordered = sorted(chapters, key=lambda c: c.start)
out = []
for index, chapter in enumerate(ordered):
following = (ordered[index + 1].start if index + 1 < len(ordered)
else end_of_video)
# A show that mistypes a timestamp can put one mark after the next.
# Clamping keeps the slice empty rather than reversed, and leaves the
# neighbouring segments intact.
end = max(chapter.start, following)
out.append(Segment(
video=video,
index=index,
start=chapter.start,
end=end,
title=chapter.title,
heading=chapter.heading,
links=chapter.links,
transcript=youtube.text_between(cues, chapter.start, end or None),
))
empty = sum(1 for s in out if not s.transcript)
if empty:
log.warning("%s: %d of %d segments have no transcript under them",
video.key, empty, len(out))
return out
# ------------------------------------------------------ transcript as text
_SENTENCE_END = re.compile(r"(?<=[.!?])\s+")
def opening(transcript: str, sentences: int = 2, cap: int = 400) -> str:
"""The first sentence or two, as a stand-in for a summary that isn't
written yet. An empty `text` is an empty row in the queue."""
if not transcript:
return ""
parts = _SENTENCE_END.split(transcript.strip())
return " ".join(parts[:sentences])[:cap]
def paragraphs(text: str, target: int = 700) -> str:
"""Break a captions stream into paragraphs a reading pane can render.
Captions arrive as one unbroken run, and the reader splits a body on blank
lines — so without this a forty-minute talk is one paragraph. Sentences
are grouped until roughly `target` characters, then a break: arbitrary,
but readable, and the alternative is a wall.
Auto-generated captions often have no punctuation at all, so one
"sentence" can be the whole talk. Those are broken on a word boundary
instead — worse than a real sentence end, better than the wall.
"""
text = " ".join((text or "").split())
if not text:
return ""
out: list[str] = []
current: list[str] = []
length = 0
for sentence in _SENTENCE_END.split(text):
for piece in _words(sentence, target * 2):
current.append(piece)
length += len(piece) + 1
if length >= target:
out.append(" ".join(current))
current, length = [], 0
if current:
out.append(" ".join(current))
return "\n\n".join(out)
def _words(run: str, limit: int) -> list[str]:
"""Split an over-long run at spaces, keeping every piece under `limit`."""
if len(run) <= limit:
return [run]
pieces: list[str] = []
current: list[str] = []
length = 0
for word in run.split(" "):
if current and length + len(word) + 1 > limit:
pieces.append(" ".join(current))
current, length = [], 0
current.append(word)
length += len(word) + 1
if current:
pieces.append(" ".join(current))
return pieces
+600
View File
@@ -0,0 +1,600 @@
"""Captions, with their timings, for one YouTube video.
YouTube has always published a machine-readable transcript for anything with
captions — including the auto-generated ones, which is most podcasts. It is the
cheapest transcript in existence: somebody else already paid for the ASR.
Getting at it is the fiddly part, and it is fiddly for a reason worth writing
down. The caption URLs embedded in a `watch` page are signed for a *browser*
session, and fetching one from anywhere else now returns `200 OK` with an empty
body — no error, no redirect, nothing to catch. So the obvious implementation
(scrape the watch page, follow `baseUrl`) fails silently, which is the worst
way for an ingest step to fail.
What works is asking the same endpoint the mobile app asks. `/youtubei/v1/player`
with a mobile client context hands back a player response whose caption URLs are
not tied to a browser session, and those fetch fine. That is the whole trick,
and it is why this module talks to an undocumented endpoint instead of scraping
a page: the scrape is the fragile one.
Two clients are tried in order, because they fail differently — `IOS` returns
JSON3 when asked and `ANDROID` returns SRV3 whatever you ask for, so both
formats are parsed. If Google breaks both, the fallbacks are `yt-dlp`, which
tracks this arms race full-time, and transcribing the audio yourself. Neither
is in here — `__init__.py` says where they would go. Until one of them is
written, a break in both clients raises `TranscriptUnavailable` rather than
storing an empty transcript: loudly, which is the whole point of the paragraph
above.
**A refusal is not an absence.** YouTube gates by address: from a VPN exit or
a datacentre range a client is answered `LOGIN_REQUIRED` ("Sign in to confirm
you're not a bot") for every video, captioned or not. Measured on 2026-09-21,
one machine, one video, seven clients, each asked bare and again with the
`visitorData` a watch page hands out:
VPN on `IOS`, `ANDROID`, `WEB`, `MWEB`, `TVHTML5` and `ANDROID_VR` all
get the bot check, and `WEB_EMBEDDED_PLAYER` an error. No tracks
from any of them, and `visitorData` changes none of it.
`youtube-transcript-api` raises `RequestBlocked`.
VPN off `IOS` and `ANDROID` return tracks that fetch. The rest are no
use even here: `WEB`, `MWEB` and `WEB_EMBEDDED_PLAYER` are
unplayable without a PO token, and `TVHTML5` and `ANDROID_VR`
are answered with the bot check from a clean address too.
So there is no anonymous client that gets past the gate and none to add: the
fix is on the network. That case raises `Blocked`, a `TranscriptUnavailable` a
caller can tell apart, because "wait for the captions" is the wrong answer to
it and would be given for ever.
"""
from __future__ import annotations
import html
import json
import logging
import re
import xml.etree.ElementTree as ET
from collections.abc import Iterator
from dataclasses import dataclass
from typing import Any
from urllib.parse import parse_qs, urlsplit
import httpx
from desk.tools.http import session
log = logging.getLogger(__name__)
INNERTUBE = "https://www.youtube.com/youtubei/v1/player?prettyPrint=false"
#: Only for the watch page, which is read for comments. A caption fetch
#: deliberately identifies as a mobile app instead — see CLIENTS.
BROWSER_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
#: Tried in order. Each is (client context, the User-Agent that goes with it) —
#: they have to match, or the response comes back playable but captionless.
CLIENTS = (
(
{"clientName": "IOS", "clientVersion": "20.10.4",
"deviceModel": "iPhone16,2", "hl": "en"},
"com.google.ios.youtube/20.10.4 (iPhone16,2; U; CPU iOS 18_3_2 like Mac OS X)",
),
(
{"clientName": "ANDROID", "clientVersion": "20.10.38",
"androidSdkVersion": 30, "hl": "en"},
"com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip",
),
)
_VIDEO_ID = re.compile(r"^[\w-]{11}$")
#: Every shape a YouTube link comes in, including the one Substack embeds
#: (`<div class="youtube-wrap" id="youtube2-VIDEOID">`).
_IN_TEXT = re.compile(
r"(?:youtu\.be/|youtube\.com/(?:watch\?(?:[^\"'\s]*&)?v=|embed/|live/|shorts/)"
r"|youtube2-)([\w-]{11})"
)
class TranscriptUnavailable(Exception):
"""No captions to be had — yet, or at all.
Raised rather than returning `[]` because the two cases an ingest has to
tell apart both look like an empty list: a video with no captions, and a
video whose captions are still being generated. The message says which,
and the caller decides whether to retry on the next pull.
"""
class Blocked(TranscriptUnavailable):
"""YouTube refused this machine, whatever the video has.
A subclass, so a caller that only wants "no captions, carry on" — a link
added by hand, a repair — is unchanged. A caller that waits for captions
has to catch this first: a refusal is not lifted by waiting, it answers
every video the same way, and retrying it quietly is how a show stores
nothing for a fortnight with nobody told. See `_refused`.
"""
@dataclass(frozen=True, slots=True)
class Cue:
"""One caption line: when it starts, how long it lasts, what was said."""
start: float
duration: float
text: str
@property
def end(self) -> float:
return self.start + self.duration
def video_id(value: str) -> str:
"""The 11-character id, from a bare id, any YouTube URL, or a blob of HTML.
Deliberately tolerant about its input: the id arrives from three different
places in this pipeline — typed by hand, in a feed entry's link, and buried
in the HTML of a Substack post — and each of them would otherwise need its
own extraction.
"""
value = (value or "").strip()
if _VIDEO_ID.match(value):
return value
if match := _IN_TEXT.search(value):
return match.group(1)
# A URL shape the pattern above doesn't cover, but with ?v= in the query.
found = parse_qs(urlsplit(value).query).get("v")
if found and _VIDEO_ID.match(found[0]):
return found[0]
raise ValueError(f"no YouTube video id in {value[:120]!r}")
def transcript(video: str, *, language: str = "en",
client: httpx.Client | None = None) -> list[Cue]:
"""Every caption cue for one video, in order.
`language` is matched as a prefix, so "en" accepts "en-US" — which matters,
because a channel that uploads a real transcript files it under a regional
code while the auto-generated one is plain "en".
Two backends, tried in that order. `youtube-transcript-api` first — it is
a dependency, and it is somebody's full-time job to keep up with YouTube's
changes. The built-in fetcher below when the library can't get the track,
so one broken path doesn't cost the transcript.
"""
cues = _via_library(video, language)
if cues is not None:
return cues
return _via_innertube(video, language, client)
#: Seconds one `requests` call inside `youtube-transcript-api` gets. Roughly
#: what `desk/tools/http.py` gives an httpx call, for the same reason.
LIBRARY_TIMEOUT = 30
def _timed_client() -> Any:
"""A `requests` session with a deadline on it, for a library that owns its HTTP.
`youtube-transcript-api` does its own fetching, on `requests`, and a
`requests` call with no `timeout=` waits on a socket read for ever. What
that costs is not a slow transcript. The work runs in a worker thread;
cancellation is cooperative, so a thread parked in a read is one the Kill
button cannot reach, and a restart re-queues the job to park on the same
video. The library takes a session of ours through `http_client`, which is
its own supported way of being told this; a `requests.Session` has no
timeout of its own to set, so the default goes on one method down.
**This is why there is no `socket.setdefaulttimeout` anywhere.** A
process-wide default is the tempting one-liner for library code that hides
its HTTP, and it is the wrong trade here: it would be inherited by every
socket the process opens, uvicorn's listener and each websocket included,
to buy a deadline for call sites that can each be told one directly. There
were two. The other was feedparser, and it is handled the same way — by
fetching where a timeout can be set; see `desk/sources/rss.py`.
Typed as `Any` because `requests` is only imported inside the guard below.
It is a `requests.Session`.
"""
import requests
class Timed(requests.Session):
def request(self, *args: Any, **kwargs: Any) -> Any:
kwargs.setdefault("timeout", LIBRARY_TIMEOUT)
return super().request(*args, **kwargs)
return Timed()
def _via_library(video: str, language: str) -> list[Cue] | None:
"""`youtube-transcript-api`. None means "not this way".
Every failure is None rather than an exception, including a video with no
captions at all: the fallback below raises the error that gets reported, so
there is one message for "no transcript" instead of two that depend on
which packages happen to be present.
"""
try:
from youtube_transcript_api import YouTubeTranscriptApi
client = _timed_client()
except ImportError:
return None
try:
tracks = YouTubeTranscriptApi(http_client=client).list(video)
wanted = [language, f"{language}-US", f"{language}-GB"]
try:
track = tracks.find_manually_created_transcript(wanted)
except Exception:
track = tracks.find_generated_transcript(wanted)
fetched = track.fetch()
except Exception as exc:
log.debug("youtube-transcript-api could not fetch %s: %s", video, exc)
return None
cues = [
Cue(start=c.start, duration=c.duration, text=clean)
for c in fetched
if (clean := _clean(c.text))
]
log.info("%s: %d cues via youtube-transcript-api", video, len(cues))
return cues or None
def _via_innertube(video: str, language: str,
client: httpx.Client | None) -> list[Cue]:
with session(client) as http:
player = _player(video, http)
track = _pick(_tracks(player), language)
if track is None:
raise TranscriptUnavailable(
f"{video}: no {language} captions. "
"Auto-captions can take a few hours to appear on a long upload."
)
# `&fmt=json3` is a request, not a promise: the ANDROID client answers
# it with SRV3 anyway. Both are parsed, so it doesn't matter which
# arrives — see `_cues`.
response = http.get(track["baseUrl"] + "&fmt=json3",
headers={"User-Agent": player.get("_caption_agent", BROWSER_UA)})
if _throttled(response):
raise Blocked(_refusal(video, [f"captions: HTTP {response.status_code}"]))
response.raise_for_status()
body = response.text
if not body.strip():
raise TranscriptUnavailable(
f"{video}: caption track fetched empty. YouTube does this when "
"the request doesn't look like the client that asked for it."
)
cues = _cues(body)
log.info("%s: %d cues, %.0f minutes", video, len(cues),
(cues[-1].end / 60) if cues else 0)
return cues
def details(video: str, *, client: httpx.Client | None = None) -> dict[str, Any]:
"""Title, author, length and description for a video.
The description is here because for a show that publishes no newsletter
it is the only place the chapter marks live — `marks.parse` reads it as
plain text. Everything else is the metadata an episode needs when YouTube
is the only source there is.
"""
with session(client) as http:
player = _player(video, http, require_captions=False)
video_details = player.get("videoDetails", {})
return {
"title": video_details.get("title", ""),
"author": video_details.get("author", ""),
"duration": float(video_details.get("lengthSeconds") or 0),
"description": video_details.get("shortDescription", ""),
}
def comments(video: str, *, limit: int = 100,
client: httpx.Client | None = None) -> list[str]:
"""Top-level comments, newest-and-most-liked first, as plain text.
Here for one reason: when a show does not publish an outline, somebody in
the comments usually has. A viewer typing out "0:00 intro / 3:32 Gemini /
13:11 Grok" is doing the same work the show notes would have done, and
YouTube itself treats such a comment as authoritative enough to draw
chapter markers from it. `marks.parse` reads these exactly as it reads a
description — see `marks.best`.
Returns `[]` rather than raising when comments are off, or when YouTube
changes the shape again. A missing fallback is not worth failing an ingest
that already has a transcript.
"""
try:
with session(client) as http:
page = http.get(f"https://www.youtube.com/watch?v={video}",
headers={"User-Agent": BROWSER_UA,
"Accept-Language": "en-US,en"}).text
context = _json_after(page, '"INNERTUBE_CONTEXT":')
initial = _json_after(page, "ytInitialData = ")
if not context or not initial:
return []
out: list[str] = []
# Several continuations live in the page (the related rail, the
# description); only one of them is the comment feed, and which
# index that is moves around. Trying them in order and keeping the
# first that yields comments is cheaper than pinning a path
# through a structure Google reshuffles.
for token in _continuations(initial)[:4]:
response = http.post(
"https://www.youtube.com/youtubei/v1/next?prettyPrint=false",
json={"context": context, "continuation": token},
headers={"Content-Type": "application/json",
"User-Agent": BROWSER_UA},
)
out = _comment_texts(response.json())
if out:
break
except Exception as exc:
log.debug("could not read comments for %s: %s", video, exc)
return []
log.info("%s: %d comments read", video, len(out))
return out[:limit]
def text_between(cues: list[Cue], start: float, end: float | None = None) -> str:
"""The words spoken in a span, as one paragraph.
A cue counts as inside the span when it *starts* inside it. Overlap would
be the other defensible rule, but it double-counts the sentence that
straddles a boundary — and these boundaries come from chapter marks a human
typed, so a sentence of drift at the seam is already there and doesn't need
doubling.
"""
return " ".join(
cue.text for cue in cues
if cue.start >= start and (end is None or cue.start < end)
).strip()
# ------------------------------------------------- internals: the watch page
def _json_after(page: str, marker: str) -> dict[str, Any] | None:
"""The JSON object that follows `marker` in a watch page.
Brace-matched rather than matched with a regex. The two blobs this reads —
`ytInitialData` and `INNERTUBE_CONTEXT` — are hundreds of kilobytes of
nested JSON containing every bracket character inside string values, and a
non-greedy regex stops at the first `}` inside a caption or a title. This
walks the string properly, which is about fifteen lines and always right.
"""
start = page.find(marker)
if start < 0:
return None
start = page.find("{", start + len(marker))
if start < 0:
return None
depth, in_string, escaped = 0, False, False
for index in range(start, len(page)):
char = page[index]
if in_string:
if escaped:
escaped = False
elif char == "\\":
escaped = True
elif char == '"':
in_string = False
continue
if char == '"':
in_string = True
elif char == "{":
depth += 1
elif char == "}":
depth -= 1
if depth == 0:
try:
return json.loads(page[start:index + 1])
except ValueError:
return None
return None
def _walk(node: Any, key: str) -> Iterator[Any]:
"""Every value stored under `key`, anywhere in a nested structure."""
if isinstance(node, dict):
if key in node:
yield node[key]
for value in node.values():
yield from _walk(value, key)
elif isinstance(node, list):
for value in node:
yield from _walk(value, key)
def _continuations(initial: dict[str, Any]) -> list[str]:
return [
token for renderer in _walk(initial, "continuationItemRenderer")
if (token := (renderer.get("continuationEndpoint", {})
.get("continuationCommand", {}) or {}).get("token"))
]
def _comment_texts(response: dict[str, Any]) -> list[str]:
"""Comment bodies out of a `next` response.
Comments arrive as `commentEntityPayload` entities in a flat batch beside
the renderers that position them, so reading them needs no understanding of
the layout at all — which is the part that keeps changing.
"""
out = []
for payload in _walk(response, "commentEntityPayload"):
text = (payload.get("properties", {}).get("content", {}) or {}).get("content", "")
if text:
out.append(text)
return out
# ------------------------------------------------------------------ internals
def _player(video: str, http: httpx.Client,
*, require_captions: bool = True) -> dict[str, Any]:
"""The player response, from whichever client answers with captions.
`require_captions` is what separates the two callers. A transcript wants
the first client that actually offers caption tracks; metadata wants the
first that answers at all, and would otherwise fail on a video that plays
fine and simply has no captions yet.
Every client's answer is kept for the error, not only the last one's: the
two fail differently, and which of them said what is the first thing
anyone asks when this breaks.
"""
said: list[str] = []
refused = False
for context, agent in CLIENTS:
name = context["clientName"]
payload = {"context": {"client": context}, "videoId": video,
"contentCheckOk": True, "racyCheckOk": True}
try:
response = http.post(
INNERTUBE, json=payload,
headers={"Content-Type": "application/json", "User-Agent": agent},
)
if _throttled(response):
refused = True
said.append(f"{name}: HTTP {response.status_code}")
continue
data = response.json()
except (httpx.HTTPError, ValueError) as exc:
said.append(f"{name}: {exc}")
continue
playability = data.get("playabilityStatus", {})
status = playability.get("status", "?")
if _tracks(data) or (not require_captions and status == "OK"):
# Caption URLs are bound to the client that obtained the player.
data["_caption_agent"] = agent
return data
refused = refused or _refused(playability)
reason = playability.get("reason") or ""
said.append(f"{name}: {status}" + (f" ({reason})" if reason else "")
+ ", no caption tracks")
log.debug("innertube %s", said[-1])
if refused:
raise Blocked(_refusal(video, said))
raise TranscriptUnavailable(
f"{video}: {'; '.join(said) or 'no client returned captions'}")
def _refused(playability: dict[str, Any]) -> bool:
"""Whether a player response is YouTube declining to serve this machine.
The status alone, not the reason beside it. The bot check reads "Sign in
to confirm you're not a bot" today, and a match on that sentence would go
back to reporting a refusal as captions not being ready on the day it is
reworded — silently, which is the failure this exists to end. The status
is the stable half, and it is sufficient: nothing asked anonymously is
ever let past `LOGIN_REQUIRED` by being asked again later.
That also takes in a private or age-gated video, where the refusal is
about the video rather than the address. It is still not a wait, and the
reason is quoted in the error for whoever reads it.
"""
return playability.get("status") == "LOGIN_REQUIRED"
def _throttled(response: httpx.Response) -> bool:
"""The same refusal, made in HTTP: a 403, a 429, or Google's `/sorry/`
interstitial, which arrives as a redirect to a captcha page."""
return response.status_code in (403, 429) or "/sorry/" in response.url.path
def _refusal(video: str, said: list[str]) -> str:
"""`Blocked`'s sentence. "YouTube refused" is what `desk/notices.py` reads
to tell this from a feed that failed, so the wording is load-bearing."""
return (f"{video}: YouTube refused this address and wants a sign-in — "
+ "; ".join(said))
def _tracks(player: dict[str, Any]) -> list[dict[str, Any]]:
return (player.get("captions", {})
.get("playerCaptionsTracklistRenderer", {})
.get("captionTracks", []) or [])
def _pick(tracks: list[dict[str, Any]], language: str) -> dict[str, Any] | None:
"""Best track for a language: a real transcript over an auto-generated one.
A channel that uploads its own captions has punctuation, speaker turns and
correctly spelled names; ASR has none of those and spells "Qwen" six ways.
Worth preferring, and never worth requiring — most shows only have ASR.
"""
wanted = [t for t in tracks
if (t.get("languageCode") or "").lower().startswith(language.lower())]
wanted.sort(key=lambda t: t.get("kind") == "asr")
return wanted[0] if wanted else None
def _cues(body: str) -> list[Cue]:
"""Parse whichever of YouTube's three caption formats turned up."""
body = body.strip()
if body.startswith("{"):
return _from_json3(json.loads(body))
root = ET.fromstring(body)
# SRV1 is `transcript > text[start,dur]`; SRV3 is `timedtext > body > p[t,d]`.
if root.tag == "transcript":
return _from_srv1(root)
return _from_srv3(root)
def _from_json3(data: dict[str, Any]) -> list[Cue]:
cues = []
for event in data.get("events", []):
text = "".join(seg.get("utf8", "") for seg in event.get("segs", []))
clean = _clean(text)
if not clean:
continue
cues.append(Cue(
start=event.get("tStartMs", 0) / 1000,
duration=event.get("dDurationMs", 0) / 1000,
text=clean,
))
return cues
def _from_srv1(root: ET.Element) -> list[Cue]:
cues = []
for node in root.iter("text"):
clean = _clean(node.text or "")
if not clean:
continue
cues.append(Cue(
start=float(node.get("start", 0)),
duration=float(node.get("dur", 0)),
text=clean,
))
return cues
def _from_srv3(root: ET.Element) -> list[Cue]:
cues = []
for node in root.iter("p"):
clean = _clean("".join(node.itertext()))
if not clean:
continue
cues.append(Cue(
start=int(node.get("t", 0)) / 1000,
duration=int(node.get("d", 0)) / 1000,
text=clean,
))
return cues
def _clean(text: str) -> str:
"""Entities out, whitespace collapsed, sound cues dropped.
Unescaped twice on purpose: SRV1 sends `&amp;#39;` — an entity for an
entity — because the caption text was escaped once when it was stored and
once again on the way into the XML.
"""
text = html.unescape(html.unescape(text or ""))
text = re.sub(r"\[(?:Music|Applause|Laughter)\]", " ", text)
return re.sub(r"\s+", " ", text).strip()
+128
View File
@@ -0,0 +1,128 @@
"""Did we fetch the article, or the page the outlet shows instead of it?
Some sites answer every article URL with the same thing: a consent gate, an
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
faithfully — it is real text on a real page — and we store it as the body. That
is worse than storing nothing, because the reader prefers the body to the feed's
summary, so a wall doesn't merely fail to add anything: it *hides* the two
readable sentences the feed already gave us.
The tell is repetition. Different articles cannot have the same body; a wall
always does. That is the whole detector — no keyword list, no per-site rules,
nothing to keep up to date. It learns golem.de's German consent page and a
Cloudflare challenge with the same line of code, and it costs no extra
requests, because the evidence is a by-product of a fetch we were making
anyway.
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
requests to one host earns a throttle or challenge page, that page is served to
whichever two articles asked next, and the two fingerprint the same. Refetching
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
was about our request rate and not about the outlet. A wall repeats on every
article, so a third body costs one extra fetch per walled host, once; a throttle
page that repeats three consecutive times is the site answering nothing else.
The verdict is per host, because the wall is: the site serves it, it isn't
chosen per article. Once a host is known to answer with one, we stop fetching
from it altogether — which makes pulls faster, not just more correct. That is
also why the count matters: `known` short-circuits the request, so a wall
verdict is the last evidence a host ever gets to give. Nothing can disprove it
except `forget`.
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
in a test), because a tool has no store of its own to remember things in. The
one way this is wrong is an outlet that really does publish the same body
`STRIKES` times running; the cost is that its articles fall back to their feed
summaries until someone calls `forget`, which is what the re-check button in
the Sources tab is for.
"""
from __future__ import annotations
import hashlib
import logging
from collections.abc import Mapping, MutableMapping
from desk.tools.framing import host_of
log = logging.getLogger(__name__)
#: Sits alongside `embed:` in the same table. The two verdicts answer different
#: questions about a host — will it let us frame it, will it give up its text —
#: and neither implies the other.
CACHE_PREFIX = "body:"
_WALL = "wall:"
REASON = "every article comes back as the same page"
#: How many consecutive identical bodies settle it. See the module docstring:
#: two was one throttle page away from retiring a working outlet for good.
STRIKES = 3
def _fingerprint(body: str) -> str:
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
def _seen(value: str) -> tuple[str, int]:
"""`(fingerprint, how many in a row)` from a stored value.
A bare fingerprint is one sighting, which is what every row written before
`STRIKES` existed holds — so an upgraded database is read correctly and
needs no migration of its own.
"""
mark, _, run = value.partition("*")
return mark, (int(run) if run.isdigit() else 1)
def known(cache: Mapping[str, str], url: str) -> str:
"""Why this host's bodies aren't worth fetching, or "" if they are.
Asked before the request, so a settled verdict costs nothing at all. Takes
a read-only mapping, because this one only ever reads — which lets the
Sources tab hand over the snapshot it already loaded rather than the live
table, and says in the signature that opening a panel changes nothing.
"""
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
"""Record a body we just extracted. Returns a reason if it proves a wall.
Only the previous body from the host is kept and how many articles running
have handed it over — not every body ever seen. A wall repeats on the
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
and one key per host beats a growing set per host.
One distinct body resets the run to one. That is what makes the count mean
"in a row" rather than "ever", and it is the whole difference between a
site that answers with a gate and a site that answered twice from a cache.
"""
host = host_of(url)
if not host or not body.strip():
return ""
key = CACHE_PREFIX + host
stored = cache.get(key, "")
if stored.startswith(_WALL):
return stored[len(_WALL):]
mark = _fingerprint(body)
seen, run = _seen(stored)
run = run + 1 if seen == mark else 1
if run >= STRIKES:
cache[key] = _WALL + REASON
log.info("%s serves a wall, not articles: %s (%d in a row)",
host, REASON, run)
return REASON
cache[key] = mark if run == 1 else f"{mark}*{run}"
return ""
def forget(cache: MutableMapping[str, str], url: str) -> None:
"""Drop what we learned about a host, so the next fetch decides again.
Outlets change their minds — a wall goes up for a quarter and comes down
again — and a verdict with no way back would be a permanent decision made
on two HTTP responses.
"""
cache.pop(CACHE_PREFIX + host_of(url), None)