Bring in desk from news

The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-23 12:29:17 +02:00
1 parent 9d666d0937
commit 248ea1636d
36 files changed
+8393

No files matched your search

+33
View File
@@ -0,0 +1,33 @@
"""The shared core: what any desk-shaped app is built from.
An app here is something that pulls things in from sources, runs work over
them as jobs, asks a model for the parts a model is good at, and shows the
result. newsdesk is the first one. A library of things added by hand and a
board of job postings are the next two, and they share everything in this
package and nothing outside it.
tools/ machinery with no opinion about the app: a model, an
encoder, an HTTP session, a video read as text
sources/ the plugin contract, the registry that finds plugins on
disk, and the built-in sources
models.py the shapes an app hands round — an item, a link — plus the
queue's own, re-exported from `pipeline.models`
modeltask.py a job that needs a model: chunk the batch, ask, apply the
answers, and give up on a row no model will settle rather
than paying for it every run
notices.py which failures newsdesk will interrupt you about, and the
sentence it uses. The queue's half of this is
`pipeline/notices.py`
config.py the settings sections an app composes into its own config
The job queue used to be `desk/pipeline/`. It is `pipeline/` at the top of
the repository now, one tree below this one, because it turned out to be the
part worth copying into other projects and a folder cannot be copied if it
imports the tree above it. `docs/pipeline-package.md` is why and how.
Nothing in here imports from an app. `tests/test_architecture.py` holds it
to that, and to one door per package: from outside, `desk.sources` is
imported by that name and nothing behind it.
"""
__version__ = "0.1.0"
+161
View File
@@ -0,0 +1,161 @@
"""The settings sections an app composes into its own config.
`Section` is the base: a block of YAML validated into a model, with an
unrecognised key being an error rather than a silent no-op. `AgentConfig`
is how a model is reached and `ModelTaskConfig` is how a task for one is
paced — both read by the pipeline, so both live here rather than in any one
app's config. An app declares its own `Config(Section)` with these as fields
and whatever else it needs next to them.
"""
from __future__ import annotations
from typing import Any, Literal, Protocol, get_args, get_origin
from pydantic import BaseModel, ConfigDict, Field, model_validator
#: The bare words YAML 1.1 resolves to each boolean, in the order a field gets
#: asked about them. See `Section._spell_yaml_booleans_back`.
_YAML_WORDS = {False: ("off", "no", "false"), True: ("on", "yes", "true")}
def _literal_strings(annotation: Any) -> set[str]:
"""Every string a `Literal` annotation accepts, through any wrapping."""
if get_origin(annotation) is Literal:
return {a for a in get_args(annotation) if isinstance(a, str)}
return {s for arg in get_args(annotation) for s in _literal_strings(arg)}
class Section(BaseModel):
"""A block of settings.
`extra="forbid"` is the whole point: an unrecognised key is a mistake — a
typo, or a setting that was renamed and never updated — and saying so is
more useful than ignoring it. `protected_namespaces=()` because pydantic
reserves the `model_` prefix for its own methods and two settings here are
honestly named `model` and `model_weight`.
"""
model_config = ConfigDict(extra="forbid", protected_namespaces=())
@model_validator(mode="before")
@classmethod
def _spell_yaml_booleans_back(cls, data: Any) -> Any:
"""Let a setting whose value is spelled `off` be written as `off`.
YAML 1.1 reads the bare words `off no false on yes true` as booleans,
so `backend: off` arrives here as `False` and fails against a `str`
field — with an error that helpfully lists `off` as one of the values
it will accept. The documented spelling being the one spelling that
doesn't work is not a thing to answer with a note in the example file;
`agent.backend: off` is what the example and the module that reads it
both tell you to write.
So a boolean landing on a field whose `Literal` lists a word for that
boolean is read as the word. Nothing else moves: `off` on a field that
has no `off` to offer is still an error, and a real `bool` field is
untouched, because `True` and `False` are what it asked for.
"""
if not isinstance(data, dict):
return data
spelled = data
for name, value in data.items():
if not isinstance(value, bool):
continue
field = cls.model_fields.get(name)
if field is None:
continue
choices = _literal_strings(field.annotation)
word = next((w for w in _YAML_WORDS[value] if w in choices), None)
if word is not None:
# A copy, because this dict is the caller's — `load` hands the
# parsed YAML straight in, and reading a file shouldn't edit it.
spelled = dict(spelled) if spelled is data else spelled
spelled[name] = word
return spelled
class AgentConfig(Section):
"""Who answers when a job needs a model. See desk/tools/agent.py."""
#: `auto` prefers the API when there are credentials and otherwise drives
#: the `claude` CLI on your subscription through the Claude Agent SDK.
#: Naming one skips the detection; `off` means no model is reachable, so
#: every batch that needs one fails with a notice saying which switch did
#: it. A stage whose batches all fail still reports and the cascade
#: carries on — turning the model off stops the model, not the pull.
backend: Literal["auto", "api", "claude_code", "off"] = "auto"
#: Overrides the per-task model everywhere. Empty means each job picks its
#: own — tagging asks for Haiku, because it is Haiku work.
model: str = ""
#: Seconds for one call, either backend. The Claude Code path spawns a
#: process and is the slower of the two by a few seconds.
timeout: float = Field(300.0, gt=0)
max_tokens: int = Field(8192, ge=1)
#: Where the `claude` binary is, if it isn't on PATH.
cli_path: str = ""
credentials_database: str = Field("", exclude=True, repr=False)
class ModelTaskConfig(Section):
"""What every task for the model is governed by. See desk/modeltask.py."""
#: Off is off everywhere: the pull, the button and `run("tag")` alike all
#: find nothing to do and say so. Not a refusal — a task that is off has
#: an empty backlog, so a cascade carries straight on past it.
enabled: bool = True
#: Whether a *pull* runs it, and nothing else — the button never reads
#: this. `auto` runs it during a pull when `agent.backend` resolves to
#: something and skips it quietly when it does not, so a machine with no
#: model pulls in silence rather than filing a failure per batch about
#: being set up the way it was meant to be. `always` runs it either way,
#: and a missing backend fails each batch saying so. `never` keeps it out
#: of the pull altogether, leaving the button.
on_ingest: Literal["auto", "always", "never"] = "auto"
model: str = ""
#: Whether the model reasons before it answers. Per *task* rather than per
#: backend on purpose: both backends take the same shapes, so how hard to
#: think is a property of the work, not of how the model is reached.
#:
#: `off` by default because neither task here is thinking work and the cost
#: is not marginal — tagging twenty-five headlines measured 128s with
#: thinking and 17s without, for tags that came back the same. Turn it on
#: for a task whose answer is a judgement rather than a lookup.
#:
#: `effort` is deliberately not offered. The CLI has it and the Messages
#: API does not, and a setting that silently means nothing on one backend
#: is exactly the asymmetry this seam exists to avoid.
thinking: Literal["off", "adaptive"] = "off"
#: Rows per model call — how the task slices its own backlog, and the one
#: number here that is genuinely the task's. Twenty-five is what one call
#: and one schema hold; it is prompt economics, and no generic queue could
#: know it.
#:
#: How many run at once is not here and should not be: that is how many
#: workers there are, and two tasks that both want a model share them
#: rather than each helping itself. See `workers` on the app's config.
batch_size: int = Field(25, ge=1)
class WorkersConfig(Section):
"""How many workers of each kind the queue has.
A property of the machine and the subscription, not of any one task, which
is why it is here and not next to `batch_size`. `llm` is what one Claude
subscription answers without queueing behind itself; `http` is wider
because a fetch is mostly waiting; `cpu` is one because the ranker and the
embedder want the machine. A resource nobody names gets one worker.
"""
llm: int = Field(4, ge=1)
http: int = Field(8, ge=1)
cpu: int = Field(1, ge=1)
class HasAgent(Protocol):
"""What the pipeline asks of an app's config: that it has an `agent`
section. `modeltask` reads nothing else from it — the rest is handed to
the task's own `settings` callable, which knows the app."""
@property
def agent(self) -> AgentConfig: ...
+241
View File
@@ -0,0 +1,241 @@
"""The shapes newsdesk hands round: an item, a link, a result from the model.
One set of models, not two — there is no ORM here, so the row the table
returns and the object the API serves are the same declaration.
The queue's own shapes — `Job`, `Run`, `JobsSnapshot`, `Notice` and the
`Model` base they share — live in `pipeline/models.py`, because the pipeline
is a folder that gets copied into other projects and cannot depend on this
one. They are re-exported here so that `desk.models` stays the one place an
app's code looks: the API serves a `Job` next to an `Item`, and which folder
each was declared in is not something a route should have to know.
"""
from __future__ import annotations
import hashlib
from dataclasses import dataclass, field
from datetime import UTC, datetime
from typing import Any, Literal
from urllib.parse import urlsplit
from pydantic import field_validator
from pipeline.models import (
TERMINAL, CascadeProgress, Job, JobsSnapshot, JobState, JsonDict, JsonIds,
Model, Notice, NoticeKind, NoticeLevel, Run, TaskProgress,
)
# ------------------------------------------------------------ model tasks
class ModelTaskResult(Model):
"""What one run of a task for the model did. Never an error: when it can't
run it says why. See desk/modeltask.py.
Subclasses add the counters that mean something for their task and say
which of them is `done`. Everything the runner adds up is here.
"""
skipped: bool = False
pending: int = 0
reason: str | None = None
batches: int = 0
#: Batches that asked the model and got nothing back. Kept apart from
#: `batches` because a run with 0 done and 0 failed had nothing to do,
#: while 0 done and 4 failed is a broken backend — and reporting both as
#: "0 done" is exactly the silence this is here to end.
failed: int = 0
#: What the model answered that couldn't be used: a slug not in the
#: vocabulary, a uid not in the batch, an abstract too short to be one.
rejected: list[str] = []
#: Rows this run stopped offering: handed to the model often enough,
#: without an answer, that asking again is only spending. Reported rather
#: than left to be inferred from a backlog that quietly got smaller — see
#: `ModelTask.missed`.
given_up: int = 0
#: Which of desk/tools/agent.py's backends answered — `api` or
#: `claude_code`, and empty when nothing did. Worth keeping: it is the
#: difference between a run that cost money and one that didn't.
backend: str = ""
@property
def done(self) -> int:
"""How many rows this run finished. The task's own counter."""
raise NotImplementedError
def absorb(self, other: ModelTaskResult) -> None:
"""Add one chunk's result into a running total.
Counts add, lists extend; `skipped`, `pending`, `reason` and
`backend` are about the run, not the chunk, and are left to the
runner. Written once here so a new task's counters count without it
saying how.
"""
for name in type(other).model_fields:
if name in ("skipped", "pending", "reason", "backend"):
continue
mine, theirs = getattr(self, name), getattr(other, name)
if isinstance(mine, list):
mine.extend(theirs)
elif isinstance(mine, int) and not isinstance(mine, bool):
setattr(self, name, mine + theirs)
class ItemLink(Model):
"""A source-supplied link with a shared meaning, separate from the destination."""
url: str
title: str = ""
role: Literal["reading", "discussion", "source"] = "source"
@field_validator("url")
@classmethod
def http_url(cls, value: str) -> str:
parsed = urlsplit(value)
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
raise ValueError("item links must be absolute HTTP(S) URLs")
return value
@dataclass(slots=True)
class Item:
"""One thing that showed up: an article, a post, a link."""
#: The source's own id for this thing. With `source`, it makes the `uid`,
#: so refetching the same thing recognises it rather than duplicating it.
source_id: str
#: Where the original lives. Any URI — `https://` for a feed entry,
#: `file:///…` for something added by hand.
location: str
title: str
#: The short version the source handed over: a feed summary, a post.
excerpt: str = ""
#: The whole thing, when the source already has it. Left empty by every
#: source that doesn't: `ingest.fetch_bodies` looks for an empty body
#: and fetches the page with trafilatura, so filling this in is also how a
#: source says "don't go and scrape this" — which is what a podcast
#: segment wants, its body being a transcript rather than a web page.
body: str = ""
author: str = ""
#: Always set and always UTC once `__post_init__` has run — a plugin may
#: omit it, and gets `now`; a plugin that supplies one must make it aware,
#: and it is converted. Declared as a plain `datetime` rather than
#: `datetime | None` because that is what every reader is handed; the
#: `None` only ever exists between the two.
#:
#: UTC and not merely aware, because this is the one field that is written
#: to storage as its own text and then *compared as text* — ordered,
#: `MAX`ed, ranged over. A stamp keeping the offset its API sent sorts by
#: the digits it happens to start with, so 10:00+02:00 comes out newer than
#: 09:00+00:00 when it is an hour older. Converting here rather than at the
#: `INSERT` is what makes every reader of the field agree with the column.
published_at: datetime = field(default_factory=lambda: datetime.now(UTC))
#: Which feed or handle you follow that produced it. The only attribution
#: an item carries: who publishes it — the outlet, and the section of it —
#: is a fact about *this subscription*, held once on the subscription row
#: and joined when something needs it. It used to be copied onto every
#: item, which meant a rename left four hundred rows disagreeing with the
#: feed they came from until a repair pass walked the table.
subscription: str = ""
source: str = "" # plugin name; filled in by the runner
#: What this source made, in the app's own words. A free string, and
#: deliberately: which shapes exist is the app's business, not the plugin
#: contract's. newsdesk reads `article`, `post` and `segment` and lays each
#: out its own way; a jobs board on this core will say `role` and `desk`
#: has no opinion about either list.
#:
#: What was *made*, not what carried it — an RSS feed of a podcast, once
#: the audio is fetched and cut, yields segments and should say so. This
#: used to read `article | video | audio`, which named the medium; the
#: medium is not the shape, and every row in the table said `article`
#: anyway because no plugin ever set it.
kind: str = "article"
#: Which of its plugin's declared profiles made this, by name. A plugin
#: that declares one uses `default` and never sets this; a show that
#: publishes both whole episodes and cut segments names the profile each
#: item belongs to, and that recorded name is what decides how the item is
#: processed and laid out for the rest of its life. Never worked out later
#: from what the item happens to carry — see `desk/sources/spec.py`.
profile: str = "default"
#: What this is, when it is not an ordinary article: `paywalled`,
#: `sponsored`. Usually left alone — `ingest` reads them off the title
#: using the tokens the subscription declares (see desk/tools/marks.py).
#: A plugin talking to an API that says so outright can set them directly.
marks: list[str] = field(default_factory=list)
#: The title as it was published. `marks.classify` writes it onto every
#: item it looks at, whether or not it cut anything, because a title with
#: a token already cut off cannot be reclassified later — the evidence
#: goes with it, and "we looked and there was nothing to cut" is a fact
#: worth having rather than an absence to re-derive.
#:
#: Empty therefore means *unknown*: a row stored before originals were
#: kept, or one no rule pass has seen. `Store` reads it that way — see the
#: unknown-original count `reclassify_marks` reports.
source_title: str = ""
#: Whatever else the source returned, kept as JSON on the row. One key is
#: read by name: `categories`, the source's own labels for this item, which
#: the tagger takes as a weak hint. Deliberately not merged into `tags` —
#: every outlet's scheme is its own ("Television & radio", "Putin, Vladimir
#: V"), and a filter built from the union of all of them is unusable. A
#: plugin that sets it caps its own list; a dozen is plenty.
raw: dict[str, Any] = field(default_factory=dict)
links: list[ItemLink] = field(default_factory=list)
def __post_init__(self) -> None:
# A plugin passing `published_at=None` explicitly bypasses the default
# factory. Unreachable as far as the annotation is concerned, and the
# annotation is what every *reader* should go by — but the plugin
# contract is a published interface and this used to be allowed, so a
# drop-in source under `plugin_paths` must not start raising.
if self.published_at is None:
self.published_at = datetime.now(UTC) # type: ignore[unreachable]
# A naive stamp is a source bug, and the only one of these three that
# cannot be repaired here: which hour it means is a fact about the API
# that sent it, and this end has nothing to read it off. It used to be
# taken as UTC, which is a guess wearing a fixed offset — wrong by up
# to a day for anything that sends local time, and wrong silently,
# since the row that comes back looks like every other row. A source
# that knows its API's convention says so where it parses; see
# `desk/sources/base.py`.
elif self.published_at.utcoffset() is None:
raise ValueError(
f"{self.source or 'a source'} gave a naive published_at "
f"({self.published_at.isoformat()}) for {self.source_id!r}. "
"A source must hand over an aware datetime: attach the offset "
"the API sent, or UTC when it documents UTC and says nothing."
)
else:
self.published_at = self.published_at.astimezone(UTC)
self.title = (self.title or "").strip()
self.excerpt = (self.excerpt or "").strip()
if not self.title and self.excerpt:
# Social posts have no title. Use the first line, capped.
first = self.excerpt.split("\n", 1)[0]
self.title = first[:200] + ("…" if len(first) > 200 else "")
@property
def uid(self) -> str:
return hashlib.sha1(
f"{self.source}\x00{self.source_id}".encode()
).hexdigest()
__all__ = [
"TERMINAL",
"CascadeProgress",
"Item",
"ItemLink",
"Job",
"JobState",
"JobsSnapshot",
"JsonDict",
"JsonIds",
"Model",
"ModelTaskResult",
"Notice",
"NoticeKind",
"NoticeLevel",
"Run",
"TaskProgress",
]
+505
View File
@@ -0,0 +1,505 @@
"""A task for the model: rows that are waiting, one call per chunk, answers
written back. The machine tagging and summarising both run in.
`agent.py` answers *how do I reach a model* — the API, or the local `claude`.
This module answers the layer above: *what do I send it, and what do I do with
what comes back*. Everything between those two lives here, once.
ModelTask the parts: waiting, build_batch, system, schema, apply,
missed
one one batch: one model call, one apply. What a job is.
of that as a `Task`: the batches, and how to find them
run every batch here and now, for a caller with no queue
Tagging and summarising used to be near-twins — the same loop, the same
failure handling, the same job body and the same resume, spelled out twice
and drifting by a line each merge. What actually differs between them is
small and is all *content*: which rows are waiting, what goes in the batch,
what the model is told, what shape it must answer in, what to do with the
answer. That is a `ModelTask`; this module is everything that is not one.
The third one — clustering, or whatever comes next — is a prompt, a schema
and two functions over the store, and no loop.
"""
from __future__ import annotations
import logging
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import Any
from desk.config import HasAgent, ModelTaskConfig
from pipeline.context import NULL, RunContext
from pipeline.task import Batch, Tally, Task, resolve
from desk.models import ModelTaskResult, Notice
from desk.notices import classify, stops_the_run
from desk.tools import agent
log = logging.getLogger(__name__)
#: `notices.py`'s code for "the model's safeguards would not take this text".
#: Named here because `one` has to recognise it by more than its wording: it
#: is the only failure that says something about a particular row, and so the
#: only one worth asking about again with fewer rows.
REFUSED = "agent.refused"
@dataclass(frozen=True, slots=True)
class ModelTask[R: ModelTaskResult]:
"""What one task for the model consists of. See tagging.TASK for one."""
#: The job kind.
kind: str
label: str
#: For progress lines and the idle result: "tagging" / "tags" /
#: "nothing untagged".
verb: str
noun: str
idle: str
#: The key under which `build_batch` lists its rows. Both of newsdesk's
#: say "items", and they used to say "articles" and "segments": a batch is
#: a mixed archive and naming it after one shape was a claim about the
#: rows in it that stopped being true.
items: str
#: Which config section governs it.
settings: Callable[[Any], ModelTaskConfig]
#: How many rows are waiting: `(config, store)`.
pending: Callable[[Any, Any], int]
#: Which rows are waiting, by uid, most recent first: `(config, store, limit)`.
#: What the task splits over — one batch per model call — so it asks for
#: identifiers rather than the payload, which each batch builds for itself
#: when it runs.
waiting: Callable[[Any, Any, int], list[str]]
#: The payload the model works from, its prompt included:
#: `(config, store, limit, uids)`. Given uids it builds that batch, and
#: asks the store again rather than trusting the list — a row may have been
#: done in the seconds since the task shared the work out.
build_batch: Callable[..., dict[str, Any]]
# All three take the config because *which rows are waiting* is no longer a
# question the store can answer on its own: summarising is governed by what
# each installed plugin declares about its own items, and the plugins are
# found through the config. A task that does not care ignores the argument.
#: The system prompt, given the store rather than fixed, because tagging's
#: carries the vocabulary and that grows between one run and the next.
system: Callable[[Any], str]
schema: dict[str, Any]
#: Write an answer back: `(store, answer, batch, config)`. Tolerant of a
#: sloppy response, and the only place the answer's shape is trusted.
apply: Callable[[Any, dict[str, Any], dict[str, Any] | None, Any], R]
#: `(store, uids, why="")`: the model was handed these rows and said
#: nothing usable about them. Count that against each one still waiting,
#: and answer how many of them that has now given up on. `why` is
#: `refused` when the safeguards turned the row down, so the Given up list
#: can say so.
#:
#: Every task needs one and there is no default, because the default is the
#: bug. A row the model cannot satisfy — a segment whose only honest
#: abstract is shorter than the guard, a uid it drops from every reply — is
#: otherwise found by the next `waiting`, sent again, and paid for again,
#: on every pull, for as long as the row exists. Where the count is kept is
#: the store's business; that there is a bound at all is this module's.
missed: Callable[..., int]
#: The result type, so totals start as the right kind of zero.
result: type[R]
#: `module:function` run once when every batch of the run is over, given
#: `(config, store, tally)` and the run context. "" for a task with
#: nothing to settle, which summarising is: an abstract is written for one
#: segment by the batch that read it, and no other batch has an opinion.
#:
#: For the decision a batch is in no position to make. Tagging's new tags
#: are the case: the cap is *per run*, and a batch that can see neither
#: the other thirty-nine nor what they proposed can only enforce it
#: forty times over. So a batch reports what it wants, and this is where
#: the run's own answer is worked out — once, over the whole tally.
#:
#: It returns the fields of `result` that settling added, and both paths
#: take it from there: queued, the final job reports it; direct, `_fold`
#: absorbs it into the total. Same function either way.
settle: str = ""
#: The field every answer in the reply names its row by. Both tasks say
#: `uid`, which is not a coincidence — an answer that cannot say which row
#: it is about is not applicable — and counting it in the streamed reply is
#: what moves the progress bar per item rather than per batch. A task whose
#: answers are not one-per-row sets it to "" and gets the old behaviour.
answer_key: str = "uid"
class _Answered:
"""Counts finished answers in a reply that is still arriving.
The model writes its answers one after another, so the `"uid"` that opens
each one is a reliable "another row is done" marker tens of seconds before
the request that carries it completes. Counting a substring is a blunt
instrument and deliberately so: the alternative is incremental JSON
parsing, which is a great deal of machinery to be *approximately as* right
about a number that only drives a bar. It cannot overcount — the key
appears once per answer, and the payload it is counted in is the model's
own output, not the batch we sent — and if it undercounts, the bar catches
up when the chunk lands and the real total is known.
`tail` is why this is a class rather than a closure: a key can be split
across two deltas, so the last few characters of each are carried into the
next. Carrying `len(needle) - 1` is exactly enough to catch one that spans
the join, and too few for anything already counted to be seen twice.
"""
def __init__(self, key: str, report: Callable[[int], None]) -> None:
self._needle = f'"{key}"'
self._report = report
self._tail = ""
self.count = 0
def __call__(self, piece: str) -> None:
window = self._tail + piece
found = window.count(self._needle)
self._tail = window[1 - len(self._needle):] if len(self._needle) > 1 else ""
if found:
self.count += found
self._report(self.count)
def one[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
uids: Sequence[str], *, context: RunContext = NULL) -> R:
"""One batch: one model call, one apply. What a job is.
The unit the queue admits. It used to be a whole backlog inside a single
job, which opened four threads of its own — so the queue counted one thing
running where four requests were in the air, and four model workers was a
number nobody was keeping. Now the job and the request are
the same thing and the count is true.
The prompt is built here rather than when the batch was handed out: for
tagging that carries the vocabulary as it stands when this batch starts,
which is what somebody adding a tag by hand mid-run should reach. What it
does *not* carry is what a sibling batch proposed a second ago — nothing
of this run's proposals exists yet, because they are settled once, when
every batch is over. See `settle`.
Raises `agent.AgentError` when no model can be reached, or when the one
that could gave nothing. The job fails, saying which — and the notice says
what to do about it, because both of those are things you fix rather than
things to work around.
A row the model answers is done with. One it leaves out, or answers in a
way `apply` will not take, is charged a turn against `task.missed` — so it
comes round again a bounded number of times rather than on every pull for
as long as it exists.
A batch the model's safeguards refuse outright is split and asked again in
halves rather than failed, because a refusal is about the text of one row
and the other rows in it are owed their answers. Down to a single row it
is charged like any other unanswered one and the run carries on. See
`_narrow`.
It counts its own rows and nothing else. As a job that is all there is; run
through `Task.run` instead, the caller shifts the progress into the whole
backlog, which is the only place that knows how big it is.
"""
context.check()
cfg = task.settings(config)
batch = task.build_batch(config, store, len(uids) or cfg.batch_size,
list(uids))
rows = batch[task.items]
if not rows:
# Done by someone else between the task sharing the work out and
# now, which is ordinary:
# every task rediscovers its backlog and the batches overlap at the
# edges. Nothing to ask, nothing to write.
return task.result(reason=task.idle)
name = agent.backend(config.agent)
if not name:
# The notice goes out from here rather than being left to the failure,
# because nothing was asked and so the `except` below never runs. Left
# to itself this reached the screen only as "Tagging a batch failed",
# filed as a bug, about a machine that is working correctly and is one
# install or one config line from working here too. The configured
# backend is what is passed, not the one that ran: there wasn't one,
# and `off` is the difference between a missing install and a switch.
unreachable = agent.AgentError("no model is reachable from here")
_report(context, classify(unreachable, backend=config.agent.backend))
raise unreachable
total = len(rows)
def report(done: int) -> None:
done = min(done, total)
context.progress(done, total, f"{task.verb} {task.items}, {done} of {total}")
context.progress(0, total, f"asking {name} about {total} {task.items}")
counted = (_Answered(task.answer_key, report) if task.answer_key else None)
try:
context.check()
reply = agent.ask_json(
config.agent,
system=task.system(store),
question={task.items: rows},
schema=task.schema,
model=config.agent.model or cfg.model,
thinking=cfg.thinking,
using=name,
on_partial=counted,
)
except agent.AgentError as exc:
notice = classify(exc, backend=name)
# A refusal is the one failure that is about the rows rather than
# about the call, and so the one that answers "which of them?" by
# being asked again with fewer. Everything else falls through to the
# raise below unchanged.
if notice.code == REFUSED and len(rows) > 1:
return _narrow(task, config, store, _uids(task, rows),
context=context)
log.exception("%s batch of %d failed on %s", task.verb, len(rows), name)
# The notice carries the sentence and the fix — "Claude Code is signed
# out" and the command that mends it, rather than the token that
# expired — to where you will see it. Then the batch fails: its rows
# keep their watermark and come round on the next run — uncharged,
# because nothing was asked of a model.
#
# Its siblings are unaffected only while the failure was this batch's.
# When it is the account's, `_report` stops the ones still queued: see
# `notices.stops_the_run`.
if notice.code == REFUSED:
# Down to one row, so the notice can name it — and has to. "The
# safeguards wouldn't take this text" is not a sentence anyone can
# act on without knowing which of the day's articles it is about,
# and the row is about to leave the backlog quietly. `key` is the
# title too, so two refusals in one run are two lines rather than
# one line with a count of two.
about = rows[0].get("title") or _uids(task, rows)[0]
notice = notice.model_copy(update={
"text": f"{notice.text} — {about}", "key": str(about)})
_report(context, notice)
if notice.code == REFUSED:
# Down to one row, and that row is the one. Charged and not
# raised: the rule about never charging a failure is a rule about
# failures that prove nothing, and this one has just proved
# exactly which row it is about. Left uncharged it is found by
# the next `waiting`, sent again, refused again, and poisons
# whichever batch it lands in for as long as the row exists —
# which is how two articles held eighteen others' abstracts.
totals = task.result(reason=notice.text)
totals.failed = 1
totals.given_up = task.missed(store, _uids(task, rows), why="refused")
totals.pending = task.pending(config, store)
return totals
raise
# Apply even if killed during the call: the spend has happened. The worker
# records cancellation at return while preserving the paid-for answer.
totals = task.apply(store, reply.data, batch, config)
# Whatever is still waiting now is what the model was asked about and did
# not settle, so it is charged one turn — before `pending` is read, so a
# row this call gave up on has already left the number the run reports.
# Charged on a reply and never on a failure: a batch that never reached a
# model cost nothing and proved nothing about its rows.
totals.given_up = task.missed(store, uids)
totals.batches = 1
totals.backend = reply.backend
totals.pending = task.pending(config, store)
report(total)
return totals
def _uids(task: ModelTask[Any], rows: Sequence[dict[str, Any]]) -> list[str]:
"""Which rows a built batch is about, from the batch itself.
`one` is given uids, but what it sent is `rows` — `build_batch` asks the
store again and drops whatever was finished in between. On the path this
serves, that difference is the whole point: a row nobody sent is a row
nobody may charge.
"""
return [str(row[task.answer_key]) for row in rows if row.get(task.answer_key)]
def _narrow[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
uids: Sequence[str], *,
context: RunContext) -> R:
"""Ask about these rows again in two halves, and add up what comes back.
A refusal names no row. It is one answer about the whole request, so the
only way to learn which row it is about is to send fewer — and the reason
to bother is that the rest of the batch is fine and is owed its abstracts.
The split costs a handful of extra calls, once, against one call that
summarises nobody and is made again on every pull.
Halves rather than one row at a time, because a clean half lands whole:
the cost is set by how many rows are flagged rather than by how many are
in the batch, and two flagged rows in one batch is ordinary — they arrive
from the same day's news.
Each half is a whole `one`, so a half that meets an unrelated failure
raises it exactly as it would have unsplit, and the batch fails the way it
always did.
The bad case, if the classifier ever took against the *prompt* rather than
any row: every single-row call is refused, every row is charged, and three
pulls empty the backlog into given-up. Bounded and visible — the status
line counts them and `reopen_given_up` asks again — but it is why the
notice names the text rather than the row.
"""
middle = len(uids) // 2
totals = task.result()
for half in (uids[:middle], uids[middle:]):
if not half:
continue
landed = one(task, config, store, half, context=context)
totals.absorb(landed)
# `absorb` leaves the fields that are about the run rather than the
# chunk, and which backend answered is one of them — the same reading
# `_fold` takes of the same shape.
totals.backend = landed.backend or totals.backend
totals.pending = task.pending(config, store)
return totals
def _report(context: RunContext, notice: Notice) -> None:
"""Put one failure on screen, and stop the run if it will meet the same one.
The two halves belong together because they are one judgement made once. A
batch that cannot reach a model, or is told the spend limit is reached, has
learned something about the *account* and not about its own rows — and the
twenty-two batches queued behind it would each pay a call to be told it
again, and each put the same line on screen. One batch fails, one notice
goes out, and the rest are cancelled rather than spent.
Which failures those are is the catalogue's to say and not this module's: a
timeout or a reply that was not JSON is this batch's bad luck, and the next
batch deserves its turn. See `notices.stops_the_run`.
"""
context.notice(notice)
if stops_the_run(notice):
context.stop(notice.text)
# ----------------------------------------------------------------- the task
def batches(task: ModelTask[Any], limit: int | None = None) -> Any:
"""The finder: what is waiting, cut into one payload per model call.
How many to a call is the task's own business — twenty-five articles is
what one call and one schema hold, and no generic queue could know it. How
many run at once is the queue's, because that is how many workers it has.
It asks for identifiers rather than the rows, and each batch asks the store
again when it runs: a row may have been done in the seconds since the work
was shared out.
`enabled: false` is answered here, and answered as nothing to do rather
than as a refusal. Here, because this is the one place every route to the
work passes through: the stage of a cascade, the button, and `run("tag")`
are three callers of this same finder, and a switch only one of them read
would be a switch that turned the task off in one place. As nothing to do,
because a task that finds nothing still queues its final job — so a
cascade whose tagging is off carries on to whatever follows rather than
ending on it.
"""
def find(config: HasAgent, store: Any) -> list[Batch]:
cfg = task.settings(config)
if not cfg.enabled:
return []
uids = task.waiting(config, store,
limit if limit is not None else 100_000)
size = max(cfg.batch_size, 1)
cut = [uids[at:at + size] for at in range(0, len(uids), size)]
return [Batch({"uids": chunk}, label=_label(task, len(chunk), n, len(cut)))
for n, chunk in enumerate(cut, start=1)]
return find
def _label(task: ModelTask[Any], rows: int, n: int, of: int) -> str:
"""What one batch is called in the queue: "Tagging 25 items, 2 of 9".
The kind's own label is "Tagging a batch" on all nine of them, which is
true and useless — the queue lists nine identical rows and nothing says
which is which or how much is left. The slice is the only thing that tells
them apart, and this is the one place it is known: the finder cut them, and
the batch that runs half an hour later knows only its own uids.
Its shape is the progress line's, capitalised because it is a name rather
than a sentence about what is happening. The tail is dropped for a task of
one batch, where "1 of 1" says nothing.
"""
said = f"{task.verb} {rows} {task.items}".capitalize()
return said if of < 2 else f"{said}, {n} of {of}"
def _fold(task: ModelTask[Any]) -> Any:
"""Add the batches up, for the direct path only. Under the queue each batch
is its own job with its own row, and there is nothing to add.
The settle runs here too, off the same tally shape the final job builds,
because "what may this run add" is a question about the run and not about
the queue. Otherwise `run("tag")` in a notebook would be the one caller
with no cap at all.
"""
def fold(config: HasAgent, store: Any, results: list[Any]) -> dict[str, Any]:
if not results:
return task.result(reason=task.idle).model_dump()
totals = task.result()
for result in results:
if isinstance(result, Exception):
# One batch that could not be answered is not a failed run: as
# eight jobs it would be one failing and seven landing. Its rows
# keep their watermark and come round next time.
totals.failed += 1
totals.reason = str(result)
continue
landed = task.result.model_validate(result)
totals.absorb(landed)
# `absorb` adds up counts and leaves the fields that are about the
# run rather than the batch. Which backend answered is one of
# those, and the run wants the last one to have said anything.
totals.backend = landed.backend or totals.backend
if task.settle:
answered = tuple(r for r in results if isinstance(r, dict))
settled = resolve(task.settle)(
config, store, Tally(done=len(answered), results=answered),
context=NULL) or {}
totals.absorb(task.result.model_validate(settled))
# After the settle, so a tag it added — and the rows it went back and
# tagged — are counted in what the run reports it did.
totals.pending = task.pending(config, store)
return totals.model_dump()
return fold
def of(task: ModelTask[Any], *, limit: int | None = None, name: str = "",
callback: str = "", carries: dict[str, Any] | None = None) -> Task:
"""The `Task` for one model task: its batches, and how to find them.
A `ModelTask` is the *content* — which rows are waiting, what the model is
told, what to do with the answer. This is that content as a step of the
pipeline, which is the same shape as every other step and is queued and
killed the same way.
"""
return Task(
kind=f"{task.kind}_batch", name=name or task.kind,
finder=batches(task, limit), callback=callback, settle=task.settle,
carries=dict(carries or {}),
size=lambda payload: len(payload.get("uids") or ()),
fold=_fold(task),
)
def run[R: ModelTaskResult](task: ModelTask[R], config: HasAgent, store: Any,
limit: int | None = None, *, context: RunContext = NULL) -> dict[str, Any]:
"""Every batch, here and now, with no queue. The direct path and nothing else.
What a Claude Code session, a notebook and a test get: one call that blocks
and hands back the total. Under the queue this same task is `of(...)`
started against the queue, its batches are independent jobs, and there is
no total until the final job reads how they went — which is the whole point
of splitting it.
Same `one` either way, and the same batches from the same finder, so there
is no second implementation to drift.
"""
return of(task, limit=limit).run(config, store, context=context)
+382
View File
@@ -0,0 +1,382 @@
"""What newsdesk will interrupt you about, and the sentence it uses.
The generic half of this — the notice a failed job gets, the check that stops
it being said twice, the rendering a terminal wants — is `pipeline/notices.py`,
and the pipeline uses only that. This module is the half that could only be
newsdesk's: a catalogue of the ways *its* model backends, *its* feeds and *its*
config keys fail, and what to tell you to do about each.
`classify` is the interface. Everything else is the table it reads.
Deliberately string-matching rather than exception-type-matching. These errors
arrive as text through two SDKs and a subprocess, and half of them are
`ResultMessage.errors` from a CLI that owes us no stable exception hierarchy.
Matching prose is the honest reflection of that. The fallback is a notice that
still says something and is marked `bug` — because a failure nobody wrote a
rule for is one to go and read about, and the property that matters most is
that nothing fails silently.
`job_failed`, `already_said` and `render` are re-exported so that a caller
reaching for newsdesk's notices gets the whole vocabulary from one import
rather than having to know which half a given name lives in.
"""
from __future__ import annotations
import logging
from collections.abc import Iterable
from pipeline.models import Notice, NoticeKind, NoticeLevel
from pipeline.notices import (
already_said, job_failed, render, whose_fault as _whose_fault,
)
log = logging.getLogger(__name__)
class Rule:
"""One row of the catalogue: how to recognise a failure and what to say.
`per_backend` exists for the one case where the same failure wants
genuinely different advice depending on who hit it. A 401 means "run
`claude auth login`" on the subscription path and "check your
ANTHROPIC_API_KEY" on the API one, and telling you to re-run a CLI login
when the problem is an environment variable is worse than saying nothing.
Its keys are usually the backend that ran. `agent.unreachable` is the
exception, because there nobody ran: the caller passes the configured
`agent.backend` instead, and `off` is a real answer that wants its own
sentence — "you turned this off" is not "you have nothing installed".
`standing` says the cause is the account or the machine rather than this
batch, so every sibling batch of the same run will be told exactly the
same thing. A spend limit does not lift between one call and the next and
a signed-out CLI does not sign itself back in; a timeout or a reply that
was not JSON says nothing at all about the next batch. It is what a run
reads to stop before it spends the other twenty-two calls — see
`stops_the_run`.
"""
__slots__ = ("code", "fix", "kind", "level", "patterns", "per_backend",
"standing", "text")
def __init__(self, code: str, level: NoticeLevel, kind: NoticeKind,
text: str, fix: str, patterns: Iterable[str],
per_backend: dict[str, tuple[str, str]] | None = None,
*, standing: bool = False) -> None:
self.code = code
self.level = level
self.kind = kind
self.text = text
self.fix = fix
self.patterns = tuple(patterns)
self.per_backend = per_backend or {}
self.standing = standing
def matches(self, error: str) -> bool:
return any(p in error for p in self.patterns)
def say(self, backend: str) -> tuple[str, str]:
return self.per_backend.get(backend, (self.text, self.fix))
#: The catalogue, most specific first — `classify` takes the first match, so
#: "out of credit" must be tried before the generic "the API refused".
#:
#: Two backends produce these and they word things differently, so most rules
#: carry patterns from both: `credit balance is too low` is the API's phrasing,
#: `usage limit reached` is the subscription CLI's, and they are the same
#: problem from where you are sitting.
#:
#: Every one of these is `expected`. That is not an accident — between them
#: they are the ways a model call fails when nothing is wrong with newsdesk,
#: and the point of enumerating them is precisely so that what falls through
#: can be called a bug with a straight face.
RULES: tuple[Rule, ...] = (
Rule(
"agent.auth", "error", "expected",
"Claude Code is signed out, so nothing that needs a model can run.",
# Two places, two fixes. At a keyboard the login is interactive; in the
# container there is nobody to prompt, so the CLI is authenticated by a
# token in the server's env file and that is the thing to replace. One
# line naming only the login sends whoever is holding the server to a
# command that cannot work where they are running it.
"Run `claude auth login` here, or paste a fresh token from "
"`claude setup-token` into Settings. New calls use it without a restart.",
("oauth", "unauthor", "authentication", "not logged in", "sign in",
"log in", "login", "expired", "401", "invalid_api_key",
"invalid api key", "could not resolve authentication"),
{"api": ("The Claude API rejected the key.",
"Check ANTHROPIC_API_KEY, or set agent.backend: claude_code "
"in config.yaml to use your subscription instead.")},
standing=True,
),
Rule(
"agent.credit", "error", "expected",
"The Claude API account is out of credit.",
"Top up at console.anthropic.com, or set agent.backend: claude_code "
"in config.yaml to use your subscription instead.",
("credit balance", "insufficient_quota", "insufficient quota",
"billing", "payment"),
standing=True,
),
# The subscription CLI has more than one way of saying "not until it
# resets", and the patterns have to carry all of them, because what falls
# through here is filed as a defect of ours. "You've hit your monthly
# spend limit · raise it at claude.ai/settings/usage?from=cc_cli_limit_
# message · your weekly limit resets 6am" matched none of the original
# words — `usage` occurs in it only inside the URL — so twenty-three
# batches of one run each reported a working machine as a bug. The text
# arrives verbatim from the CLI (`desk/tools/agent.py:_explain`), so
# matching its own words is the only thing there is to match.
Rule(
"agent.limit", "warning", "expected",
"Claude's usage limit is reached — tagging will pick up where it left "
"off once it resets.",
"Nothing to do; it resets on its own. A spend limit can be raised at "
"claude.ai/settings/usage.",
("usage limit", "spend limit", "spend cap", "limit reached",
"limit resets", "weekly limit", "monthly limit", "hourly limit",
"hit your limit", "rate_limit", "rate limit", "429",
"too many requests", "overloaded", "quota"),
standing=True,
),
# A refusal about the *rows*, not about us and not about the account. The
# platform runs a classifier over the request, and a news archive walks
# into it regularly: reporting on a worm or a supply-chain compromise is
# written in the vocabulary the classifier watches for, and an outlet that
# covers security covers it weekly.
#
# Deliberately not `standing`. The account is fine and the next batch is
# about different rows, so stopping the run here would throw away every
# abstract the rest of the backlog was about to get. What it costs instead
# is one split — see `modeltask.one`, which re-asks the batch in halves so
# the flagged row is found and charged rather than taking its batch-mates
# down with it.
Rule(
"agent.refused", "warning", "expected",
# Short, because the title is appended to it and the two together are
# what the band shows — on a phone, over the article being read. What
# it means and what to do are the fix's, which is behind `more` there
# and open in the log. It says "item" and not "abstract": tagging
# meets the same refusal.
"The model's safeguards refused this item",
# The URL spelled out rather than left to the raw error. The one
# decision this notice exists to inform — whether the verification is
# worth applying for — cannot be made from a line that says a link
# exists.
"Usually security reporting, and the item itself is still here and "
"readable. Nothing to do: after three tries it is left alone rather "
"than sent again on every pull, it is said on screen only the first "
"time, and Jobs & history lists it under Given up with a Try again. "
"These stop for an account that has applied to "
"Anthropic's Cyber Verification Program — "
"https://support.claude.com/en/articles/"
"14604842-real-time-cyber-safeguards-on-claude",
("[cyber]", "safeguards flagged", "cyber verification program",
"flagged by safeguards"),
),
Rule(
"agent.output", "warning", "expected",
"The model's answer was longer than agent.max_tokens allows, so the "
"batch came back with nothing.",
"Raise agent.max_tokens in config.yaml, or lower the task's batch_size "
"so there is less to answer in one call.",
("output token maximum", "max_output_tokens", "max output tokens",
"claude_code_max_output_tokens", "exceed your max_tokens"),
),
Rule(
"agent.key", "error", "expected",
"No Claude API key, and no local `claude` to fall back on.",
"Set ANTHROPIC_API_KEY, or install Claude Code and run "
"`claude auth login`.",
("no model backend", "anthropic sdk isn't installed",
"claude agent sdk isn't installed"),
standing=True,
),
# The two halves of what a job that cannot reach a model has to be able to
# say — "none is reachable" and "the one there is would not answer".
# (`docs/pipeline-levels.md` used to promise both; the file is gone and
# the promise is not.) Both used to fall past every rule into the
# fallback, so a scratch box with `backend: off` and a machine with no
# `claude` on PATH each reported every batch as a defect of ours and sent
# you to a traceback about a program that is working correctly.
Rule(
"agent.unreachable", "error", "expected",
"No model is reachable from here, so nothing that needs one can run.",
"Install Claude Code and run `claude auth login`, or set "
"ANTHROPIC_API_KEY.",
("no model is reachable",),
{"off": ("agent.backend is `off`, so nothing that needs a model runs.",
"Set agent.backend to `auto` in config.yaml, or name `api` "
"or `claude_code` outright.")},
standing=True,
),
Rule(
"agent.cli", "error", "expected",
"Claude Code is the backend, but there is no `claude` here to run.",
"Install the CLI — `irm https://claude.ai/install.ps1 | iex` on "
"Windows, `npm install -g @anthropic-ai/claude-code` elsewhere — or "
"point agent.cli_path at one in config.yaml.",
("claude code not found", "cli path not resolved"),
standing=True,
),
Rule(
"agent.timeout", "warning", "expected",
"The model didn't answer in time.",
"Usually a slow moment; it retries on the next run. Raise "
"agent.timeout in config.yaml if it keeps happening.",
("no answer within", "timeout", "timed out"),
),
Rule(
"agent.offline", "warning", "expected",
"Couldn't reach Claude — the network looks down.",
"",
("connection", "getaddrinfo", "name or service not known",
"temporary failure in name resolution", "network is unreachable",
"ssl", "certificate"),
),
Rule(
"agent.response", "warning", "bug",
"The model answered with something that wasn't the JSON asked for.",
"",
("no json object", "wasn't valid json", "wasn't a json object",
"returned no result"),
),
)
#: Where an unrecognised failure lands, and marked `bug` on purpose: the rules
#: above are the ways this fails when nothing is wrong, so falling past all of
#: them is itself the signal. Better to send you to the log for something that
#: turns out to be mundane than to file a real defect under "expected".
FALLBACK = Rule(
"agent.failed", "warning", "bug",
"A model call failed for a reason newsdesk doesn't recognise.",
"", (),
)
#: Exception names that mean the *remote end* misbehaved rather than that we
#: mishandled it. A feed that 403s or times out is news about the feed; a
#: KeyError parsing its XML is news about us. Everything not listed is treated
#: as ours, which is the safer default — a bug filed as expected is a bug that
#: never gets looked at.
def classify(error: BaseException | str, *, backend: str = "",
key: str = "") -> Notice:
"""The notice for a model-backend failure.
`backend` is folded into the sentence when it is known, because "Claude
Code is signed out" and "the API rejected the key" want different hands on
different keyboards, and the caller is the only one that knows which ran.
When nothing ran at all, the caller passes the configured `agent.backend`
instead — see `Rule.per_backend`.
"""
detail = str(error).strip()
rule = next((r for r in RULES if r.matches(detail.lower())), FALLBACK)
text, fix = rule.say(backend)
if rule is FALLBACK:
text = _unrecognised(backend, detail)
return Notice(
code=rule.code, level=rule.level, kind=rule.kind, text=text, fix=fix,
detail=detail, key=key, where="the tagging lines in the server log",
)
def stops_the_run(notice: Notice) -> bool:
"""Whether the rest of this run should be abandoned rather than attempted.
True for the failures a `Rule` marks `standing`: the ones caused by the
account or the machine, which will answer every sibling batch of the run
identically. The batch that met it has already failed; what this decides
is whether the twenty-two behind it are worth the calls.
It is a question about the notice rather than about the exception because
the notice is where the catalogue's judgement already is. A caller that
classified a failure to say something useful about it has, by then, been
told everything it needs to know about whether to carry on.
"""
return any(rule.code == notice.code and rule.standing for rule in RULES)
def source_failed(source: str, error: BaseException | str) -> Notice:
"""A subscription that blew up mid-fetch.
Keyed by source, so a feed that has been failing all week collapses into
one line rather than one per pull — and, more to the point, is *visible*,
which a source quietly returning nothing has never been. That is the one
failure you cannot spot by reading the feed: a dead source and a quiet news
day look identical.
One failure gets its own sentence, because the general one sends you the
wrong way. When YouTube refuses the machine the show's feed is fine, so
the Check job would report every subscription healthy; what is wrong is
the address the request left from. See `source_blocked`.
"""
if any(said in str(error).lower() for said in BLOCKED):
return source_blocked(source, error)
return Notice(
code="source.failed", level="warning", kind=_whose_fault(error),
key=source,
text=f"{source} failed to fetch — its items are missing from this pull.",
fix="Run the Check job to see which subscriptions still resolve.",
detail=str(error).strip(),
where=f"the `source {source} blew up mid-fetch` traceback in the log",
)
#: How `desk.tools.video.youtube.Blocked` reads once it is text. Matched as
#: prose like everything else here, and for one more reason: `notices` sits
#: below `tools` in the layering, so the type is not this module's to import.
BLOCKED = ("youtube refused",)
def source_blocked(source: str, error: BaseException | str) -> Notice:
"""A source YouTube would not serve, because of where the request came from.
Always `expected`: nothing in newsdesk is wrong, and the episodes are not
lost. They were deliberately not marked done, so the first pull from an
address YouTube accepts reads them. Until then this is said on every pull,
which is the point — it used to be an INFO line that retried for ever.
"""
return Notice(
code="source.blocked", level="warning", kind="expected", key=source,
text=f"YouTube refused {source} — it takes this address for a bot, "
"so no episodes were read.",
fix="Usually a VPN or a datacentre address: pull again without it. "
"Nothing is lost; unread episodes are tried on every pull.",
detail=str(error).strip(),
where=f"the `source {source} blew up mid-fetch` traceback in the log",
)
def _unrecognised(backend: str, detail: str) -> str:
"""The fallback's sentence, with the failure's own words in it.
It used to say only "failed for a reason newsdesk doesn't recognise",
which is true and useless. This is precisely the case where nobody wrote a
rule, so the raw text is the only thing anyone has to go on — and it was
reachable only by pressing `why` on a notice that fades in half a minute,
or by going to the log.
Trimmed to the first line and a screenful, because a stack trace does not
belong in a strip. `Notice.detail` still carries every character of it.
"""
who = _name(backend) if backend else "A model call"
said = detail.splitlines()[0].strip() if detail else ""
if len(said) > 160:
said = said[:159].rstrip() + "…"
if not said:
return f"{who} failed for a reason newsdesk doesn't recognise."
return f"{who} failed: {said}"
def _name(backend: str) -> str:
return {"api": "The Claude API",
"claude_code": "Claude Code"}.get(backend, backend or "The model")
__all__ = [
"BLOCKED", "FALLBACK", "RULES", "Rule", "already_said", "classify",
"job_failed", "render", "source_blocked", "source_failed", "stops_the_run",
]
+175
View File
@@ -0,0 +1,175 @@
"""The plugins: where items come from. One door, used from both sides.
from desk.sources import Source, Item # writing one
from desk.sources import build, available # running them
A plugin is a file with a `Source` subclass in it — `base.py` is the contract
— dropped into this package or into a directory under `plugin_paths:` in
config.yaml. It yields `Item`s and touches nothing else: no store, no queue,
no config beyond its own `options`. What it needs to remember between pulls
goes in `self.state`, a mapping the runner backs with the database. That is
what makes a new source one file, and what lets everything above treat every
source the same way.
base.py `Source`, `Item`, `Suggestion`, `CheckResult`, and the
helpers every plugin ends up wanting
spec.py `PluginSpec`: what a plugin *declares* — its profiles, the
shared tools that acquire its content, what the shared
summariser and tagger should do with it, how it is laid out
registry.py finds the plugins on disk and builds the configured ones
bluesky.py, mastodon.py, x.py, podcast.py the built-in ones
rss.py feed fetching, which a publisher's plugin subclasses
Declaring a `spec` is optional in the contract and every plugin installed here
declares one: `spec_of` adapts a plugin that declares none, and such a plugin
runs exactly as it always has, but nothing about it is a claim its author made.
`PodcastSource` and `RssSource` are exported next to `Source` because that is
how a publisher's plugin is written: subclass the class that already does the
fetching, give it a name and a `spec`, and add nothing else.
`plugins/lastweekin_ai.py` and `plugins/guardian.py` are the two models.
`RssSource` is `shared` and is therefore not a plugin itself — there is no
generic "Any RSS feed" source to paste a masthead into.
"""
from __future__ import annotations
import logging
from typing import Any, Protocol
from desk.models import Item, ItemLink
from desk.sources.base import (
MULTIWORD_OUTLETS, USER_AGENT, CheckResult, Getter, Source, Suggestion,
entry_time, http_failure, outlet_name, parse_timestamp, split_label, strip_html,
)
from desk.sources.podcast import PodcastSource, segment_profile
from desk.sources.registry import available
from desk.sources.rss import RssSource, check_feed
from desk.sources.spec import (
API_VERSION, ARTICLE_READER, FEED_READER, LINK_READER, POST_READER,
RECORDING_READER, RSS, Discovery,
ExternalLink, MarkRule, OriginalView, PluginSpec, Presentation, Profile,
QueueLayout, RetryPolicy, Section, Step, SummaryPolicy, TaggingPolicy,
Target, TargetSpec, Tasks, View, Vocabulary, Wall, problems, spec_of,
)
__all__ = [
"API_VERSION",
"ARTICLE_READER",
"FEED_READER",
"LINK_READER",
"MULTIWORD_OUTLETS",
"POST_READER",
"RECORDING_READER",
"RSS",
"USER_AGENT",
"CheckResult",
"Discovery",
"ExternalLink",
"Getter",
"Item",
"ItemLink",
"MarkRule",
"OriginalView",
"PluginSpec",
"PodcastSource",
"Presentation",
"Profile",
"QueueLayout",
"RetryPolicy",
"RssSource",
"Section",
"Source",
"SourcesConfig",
"Step",
"Suggestion",
"SummaryPolicy",
"TaggingPolicy",
"Target",
"TargetSpec",
"Tasks",
"View",
"Vocabulary",
"Wall",
"available",
"build",
"check_feed",
"entry_time",
"faults",
"http_failure",
"outlet_name",
"parse_timestamp",
"problems",
"segment_profile",
"spec_of",
"split_label",
"strip_html",
]
log = logging.getLogger(__name__)
class SourcesConfig(Protocol):
"""What `build` needs from an app's config: where plugins are found, and
the section each configured one gets. Anything with these two attributes
will do; newsdesk's `Config` is one."""
@property
def plugin_paths(self) -> list[str]: ...
@property
def sources(self) -> dict[str, Any]: ...
def build(config: SourcesConfig,
vocabulary: Vocabulary = Vocabulary()) -> list[Source]:
"""Instantiate every source that is both installed and configured.
`vocabulary` is what this application supports, and an empty one — the
default — restricts nothing, which is every caller today. When an app
passes its own, a plugin whose declared spec names a tool, kind or
component the app does not have is left out with its diagnostics in the
log, and the others are built. One outlet's mistake is not an evening
without news.
A plugin that declares no spec is never left out on these grounds: there
is nothing in an adapted spec that anyone wrote. See `spec.spec_of`.
"""
classes = available(config.plugin_paths)
instances = []
for name, options in config.sources.items():
cls = classes.get(name)
if cls is None:
log.warning("config has `sources.%s` but no plugin defines it (have: %s)",
name, ", ".join(sorted(classes)) or "none")
continue
if broken := faults(cls, vocabulary):
log.error("source %s is not loadable here and is disabled:\n %s",
name, "\n ".join(broken))
continue
try:
instance = cls(options or {})
except Exception:
log.exception("source %s could not be constructed; continuing", name)
continue
if not instance.enabled:
log.info("source %s is disabled in config", name)
continue
instances.append(instance)
return instances
def faults(cls: type[Source], vocabulary: Vocabulary) -> list[str]:
"""Why this application cannot run this plugin, or nothing.
A spec that will not even build is caught here rather than at import,
because a plugin is free to construct its spec in `__init__`; either way
the plugin is disabled and the reason is the message.
Exported because a plugin left out of a pull is also a row in a list of
installed plugins, saying why it is unavailable — and asking that
question twice, in two places, is how the two answers start to differ.
"""
try:
return problems(spec_of(cls), vocabulary)
except Exception as exc:
return [f"its spec could not be read: {exc}"]
+325
View File
@@ -0,0 +1,325 @@
"""The plugin contract.
Every source lives in its own file. Drop a .py file into `desk/sources/`
or into any directory listed under `plugin_paths:` in config.yaml, define a
subclass of `Source`, and it will be discovered automatically.
Minimum viable plugin:
from desk.sources import Source, Item
class MySource(Source):
name = "mysource"
description = "One line shown in the Sources tab."
def fetch(self):
yield Item(
source_id="unique-per-source",
location="https://example.com/story",
title="Headline",
excerpt="Summary or body",
author="someone",
published_at=datetime.now(timezone.utc), # aware; stored as UTC
)
`self.options` is whatever you put under `sources.mysource:` in the config —
by the time a pull runs, its subscription list has been replaced with the
fetch-enabled rows from the Sources tab (see newsdesk/jobs/subscriptions.py).
Two optional methods round out the contract, both with do-nothing defaults:
`discover` proposes accounts to follow, and `check` says whether each
subscription still resolves. Neither is required for a source to be useful.
"""
from __future__ import annotations
import re
from abc import ABC, abstractmethod
from collections.abc import Iterator, MutableMapping
from dataclasses import dataclass
from datetime import UTC, datetime
from typing import Any, ClassVar
from desk import __version__
from desk.models import Item
from desk.tools.scrape import Getter
#: Sent by every plugin that talks HTTP, so an outlet reading its logs can see
#: who this is. (`desk/tools/framing.py` deliberately sends a browser string
#: instead — it is asking what a browser would be told, so it has to ask as
#: one.)
USER_AGENT = f"desk/{__version__} (personal aggregator)"
_HTML_TAGS = re.compile(r"<[^>]+>")
_WHITESPACE = re.compile(r"\s+")
#: Tidying up after the line above — see `strip_html`. Deliberately only the
#: punctuation that always closes: an apostrophe or a dash can legitimately
#: have a space in front of it.
_SPACE_BEFORE_PUNCT = re.compile(r" +([.,;:!?%)\]}])")
_SPACE_AFTER_OPENING = re.compile(r"([(\[{]) +")
def parse_timestamp(value: str) -> datetime:
"""An ISO-8601 stamp from a social API, as an aware datetime, or now.
Every one of these APIs writes UTC as a trailing `Z`, which
`datetime.fromisoformat` did not accept before 3.11 and which is cheaper to
translate than to special-case. A missing or malformed stamp is not worth
dropping a post over: `now` sorts it to the top of the queue, where you
will see it and can judge it yourself.
A stamp with no offset at all is read as UTC, and that is a decision this
helper is entitled to make where `Item` is not. Here we know whose stamp it
is — an API that documents UTC and then omits the `Z` on some of its rows —
whereas the model is handed a bare datetime by anything at all, and a wrong
guess there is an item filed hours out with nothing on the row to show it.
A plugin whose API sends local time should not be reaching for this.
"""
try:
parsed = datetime.fromisoformat((value or "").replace("Z", "+00:00"))
except (ValueError, AttributeError):
return datetime.now(UTC)
return parsed if parsed.utcoffset() is not None else parsed.replace(tzinfo=UTC)
def entry_time(entry: Any) -> datetime:
"""When a feedparser entry was published, or now if it doesn't say.
feedparser hands back a 9-tuple in UTC under `published_parsed`, falling
back to `updated_parsed`. Three modules used to do this slice on their
own; the tuple is sliced to six here, once.
"""
for key in ("published_parsed", "updated_parsed"):
parsed = entry.get(key)
if parsed:
return datetime(*parsed[:6], tzinfo=UTC) # type: ignore[misc] # 9-tuple, sliced to 6
return datetime.now(UTC)
def http_failure(exc: Exception) -> str:
"""One line for a failed request, for a check result: the status when the
server answered, the exception's name when it didn't."""
response = getattr(exc, "response", None)
if response is not None:
return f"HTTP {response.status_code}"
return f"unreachable: {type(exc).__name__}"
def strip_html(html: str) -> str:
"""Tags out, whitespace collapsed. Feeds and Mastodon both hand us HTML.
A tag becomes a *space* rather than nothing, because `</p><p>` has to stop
the last word of one paragraph running into the first word of the next.
That is right for block tags and wrong for inline ones — `a <b>link</b>.`
would come out as "a link ." — so the space is taken back off where it
landed against punctuation. It reads better, and the ranker and the tagger
are both looking at these strings.
"""
text = _WHITESPACE.sub(" ", _HTML_TAGS.sub(" ", html or "")).strip()
return _SPACE_BEFORE_PUNCT.sub(r"\1", _SPACE_AFTER_OPENING.sub(r"\1", text))
#: Outlets whose name is more than one word, so "Al Jazeera Sport" doesn't
#: become outlet "Al". Anything not listed splits on the first space, which is
#: right for "Guardian Tech" and "BBC World". Set `outlet:`/`section:` on the
#: feed in config.yaml when you want neither guess.
MULTIWORD_OUTLETS = (
"Al Jazeera", "Hacker News", "Ars Technica", "Washington Post",
"Financial Times", "New York Times", "Wall Street Journal",
"Der Spiegel", "Le Monde", "Foreign Policy", "MIT Technology Review",
)
def split_label(label: str) -> tuple[str, str]:
"""'Guardian Tech' -> ('Guardian', 'Tech'). Fallback when config is silent.
A reading of a feed's *label*, which is why it is part of the source
contract rather than of any store: the RSS plugin applies it to a feed it
was given with no outlet, and an app applies it to a subscription seeded
the same way. One rule, and the plugin side owns it.
"""
label = (label or "").strip()
if not label:
return "", ""
for outlet in MULTIWORD_OUTLETS:
if label == outlet:
return outlet, ""
if label.startswith(outlet + " "):
return outlet, label[len(outlet):].strip()
head, _, rest = label.partition(" ")
return head, rest.strip()
#: What sits between a site's name and its tagline, for `outlet_name`. Spelled
#: with escapes because a literal en dash next to a hyphen inside a character
#: class is unreadable, and ruff is right to say so.
_TAGLINE = re.compile("\\s+[-\u2013\u2014|:\u00b7]\\s+")
def outlet_name(raw: str) -> str:
"""A name a site gave for itself, cut back to something a sidebar can hold.
Both places that ask a server what it is called get the SEO title back,
tagline and all: WordPress keeps ODSC's as "Open Data Science - Your News
Source for AI, Machine Learning & more", and a feed's `<title>` is often
"Ars Technica - All content". Everything after the first separator is the
tagline, and the name is the part in front of it.
A reading of what a *server* said, which is why it is part of the source
contract next to `split_label` — that one reads a label you wrote, this
one reads a string you never saw.
"""
return _TAGLINE.split((raw or "").strip(), 1)[0].strip() or (raw or "").strip()
@dataclass(slots=True)
class Suggestion:
"""An account you don't follow yet, that the graph thinks you should."""
handle: str
display_name: str = ""
reason: str = ""
score: float = 0.0
url: str = ""
@dataclass(slots=True)
class CheckResult:
"""Whether one subscription still resolves, and what the server said it is.
`target` is the feed URL or the handle — what the subscription row holds —
and `label` is what *you* call it, so the Jobs view can name a failing row.
`outlet` and `section` are the other half: what the server calls itself.
They are how a plugin that learns a publisher's real name from the network
hands it over — WordPress reads it off `/wp-json`, a feed off its own
`<title>` — and they are written onto the subscription, once, by
`name_grouping`. Empty from every plugin that has nothing to add, which is
most of them: an outlet is normally known from the label before anything
is fetched.
Deliberately *not* `label`. That field already carries the subscription's
own name in every plugin and in the Jobs view's failing-row line, and a
publisher's name is a different fact about a different thing.
"""
kind: str
target: str
ok: bool
detail: str
label: str = ""
outlet: str = ""
section: str = ""
class Source(ABC):
"""Base class for every ingest plugin."""
#: config key under `sources:` in config.yaml. Required, must be unique.
name: str = ""
#: True for a class that exists to be subclassed rather than to be run:
#: `RssSource` is the fetching every publisher's plugin inherits, not a
#: plugin of its own. The registry passes over one, so there is no
#: "Any RSS feed" row to paste a masthead into — see `desk.sources.rss`.
#:
#: Read off the class's own `__dict__` and never inherited, or declaring
#: it once would take every plugin that subclasses that class with it.
shared: ClassVar[bool] = False
#: shown in the Sources tab, under Plugins
description: str = ""
#: Which key under `sources.<name>:` holds this plugin's subscriptions, and
#: therefore whether one is a URL or a handle. Declared here rather than
#: inferred, because everything downstream needs the answer: what
#: `subscriptions.apply` puts back into the config for a pull, what the Add
#: box in the Sources tab asks for, and what the UI validates.
#:
#: This used to be `name == "rss"` spelled out at each of those sites,
#: which meant the second URL-shaped plugin silently had its feed list
#: replaced by an empty account list and fetched nothing.
subscription_field: ClassVar[str] = "accounts" # "accounts" or "feeds"
#: The outlet every subscription of this plugin sits under, when the plugin
#: is a network rather than a publisher — a Bluesky handle is a channel of
#: Bluesky, not a masthead of its own. Written onto a subscription when it
#: is added, and only over a name nobody chose; `check` overrides it for a
#: plugin that can ask the network what a subscription is really called.
#: Empty means "read it off the label", which is right for a feed.
default_outlet: str = ""
#: The addresses this plugin knows how to read better than a web page, and
#: how: a `Getter` whose `handles` matches them and whose `claims` puts it
#: ahead of the plain request. The body fetch collects these from every
#: active plugin and hands them to `desk.tools.scrape`, so a Hacker News
#: link to a Bluesky post is read through Bluesky's API while the Bluesky
#: plugin is on, and as a web page while it is off. A getter only gets the
#: page: which link an item follows is decided when the plugin makes it.
getters: ClassVar[tuple[Getter, ...]] = ()
def __init__(self, options: dict[str, Any] | None = None) -> None:
self.options = options or {}
#: What this plugin remembers between pulls — a spend counter, which
#: episodes are already done — as a plain mapping of strings. A pull
#: hands in one backed by the database; driven directly (a test, the
#: dry-run `main` in a plugin file) it is this dict and lives as long
#: as the instance. It is the only thing a plugin is given by the
#: application, which is what keeps a plugin one file.
self.state: MutableMapping[str, str] = {}
@property
def enabled(self) -> bool:
return bool(self.options.get("enabled", True))
@abstractmethod
def fetch(self) -> Iterator[Item]:
"""Yield everything new. Runner handles dedup, so over-fetching is fine."""
raise NotImplementedError
def discover(self) -> list[Suggestion]:
"""Optional: propose accounts to follow. Return [] if unsupported."""
return []
def check(self) -> list[CheckResult]:
"""Optional: does each subscription still resolve, and what is it called?
One result per feed or account, and never a paid request — the `check`
job runs this for every source at once. Return [] if there is nothing
to ask.
This is also the naming channel. A plugin that can only learn a
publisher's real name by asking the network says so here, in the
result's `outlet`/`section`, and the app writes it onto the
subscription. It is asked of one target at a time when you add a
subscription — build the source with that one entry in its options —
so a URL you paste comes back named.
"""
return []
def params_for(self, target: str) -> dict[str, Any]:
"""What this subscription asks for, when it is a query and not an address.
A feed URL and a handle need nothing; a search takes keywords, and
maybe a location or a floor. Those live on the subscription, not in the
plugin's own settings, because they are what distinguishes one of your
saved searches from the next — and so that adding one is the same act
as adding a feed, with the same switches on it afterwards.
"""
for entry in self.options.get(self.subscription_field) or []:
if isinstance(entry, dict):
this = str(entry.get("url") or entry.get("handle") or "")
if this == target:
got = entry.get("params")
return dict(got) if isinstance(got, dict) else {}
return {}
def targets(self) -> list[tuple[str, str]]:
"""The subscriptions this instance was configured with, as
`(target, label)` pairs — a URL or a handle, and what it is called.
A bare string in the list is its own label."""
out = []
for entry in self.options.get(self.subscription_field) or []:
if isinstance(entry, dict):
target = str(entry.get("url") or entry.get("handle") or "")
label = str(entry.get("label") or target)
else:
target = label = str(entry)
if target:
out.append((target, label))
return out
+242
View File
@@ -0,0 +1,242 @@
"""Bluesky via the public AppView. No auth, no key, no bill.
Public app.bsky.* endpoints can be called directly against
https://public.api.bsky.app, which is what this plugin does. Endpoints used:
app.bsky.feed.getAuthorFeed - posts by one account
app.bsky.graph.getFollows - who an account follows (for discovery)
app.bsky.graph.getSuggestedFollowsByActor - accounts similar to a given one
"""
from __future__ import annotations
import logging
from collections import defaultdict
from collections.abc import Iterator
from typing import Any
import httpx
from desk.sources.base import (
USER_AGENT, CheckResult, Getter, Item, Source, Suggestion, http_failure,
parse_timestamp,
)
from desk.sources.spec import (
API_VERSION, POST_READER, PluginSpec, Profile, Step, SummaryPolicy,
TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools import posts
log = logging.getLogger(__name__)
API = "https://public.api.bsky.app/xrpc"
class BlueskySource(Source):
name = "bluesky"
description = "Posts from Bluesky accounts (free public API)."
default_outlet = "Bluesky"
#: A `bsky.app/profile/.../post/...` link from any outlet is read through
#: the same public API, not as the web app's script shell.
getters = (Getter("bluesky_api", posts.bluesky, posts.is_bluesky, claims=True),)
#: A network declares no accounts, because which ones you follow are
#: yours. `mode="editable"` is that sentence in the contract, and it is
#: what puts an Add box under this plugin and nothing but Fetch and Show
#: under a publisher's — see §8 of docs/plugin-rework.md.
#:
#: Both policies are declared as these items already behave rather than
#: as anyone would design them fresh, which is step 11's rule that
#: silence is not a decision. Posts are tagged today and have been for as
#: long as there has been a tagger; a post is already the length an
#: abstract would be, so summarising is off and says so.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Bluesky",
targets=TargetSpec(mode="editable", kind="account", defaults=[],
validator="bluesky_handle"),
profiles={"default": Profile(
kind="post",
# The fetching a link post has always had: `links_only` keeps the
# ones that point somewhere, and where they point is extracted
# into the field the reader shows under the post itself. A post
# linking to nothing carries its own permalink as `location` and
# as `raw.post_url`, and the body fetch skips it.
acquisition=[Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"})],
tasks=Tasks(
summary=SummaryPolicy(enabled=False),
tagging=TaggingPolicy(
enabled=True, input_fields=["inputs.feed_text"],
context_fields=["author"]),
),
presentation=POST_READER,
)},
)
def __init__(self, options: dict[str, Any] | None = None) -> None:
super().__init__(options)
self.client = httpx.Client(
timeout=20,
headers={"User-Agent": USER_AGENT},
)
def _get(self, method: str, **params) -> dict[str, Any]:
r = self.client.get(f"{API}/{method}", params=params)
r.raise_for_status()
return r.json()
# ------------------------------------------------------------- ingest
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("posts_per_account", 40))
skip_replies = self.options.get("skip_replies", True)
links_only = self.options.get("links_only", False)
for handle, label in self.targets():
try:
data = self._get(
"app.bsky.feed.getAuthorFeed",
actor=handle,
limit=min(limit, 100),
filter="posts_no_replies" if skip_replies else "posts_with_replies",
)
except httpx.HTTPError as exc:
log.warning("bluesky: %s failed (%s)", handle, exc)
continue
for entry in data.get("feed", []):
post = entry.get("post") or {}
record = post.get("record") or {}
author = post.get("author") or {}
text = record.get("text", "")
link = self._external_link(post)
if links_only and not link:
continue
rkey = (post.get("uri") or "").rsplit("/", 1)[-1]
yield Item(
source_id=post.get("uri", ""),
location=link
or f"https://bsky.app/profile/{author.get('handle', handle)}/post/{rkey}",
title="", # base class derives one from the text
excerpt=text,
author=author.get("displayName") or author.get("handle", handle),
published_at=parse_timestamp(record.get("createdAt", "")),
subscription=label,
kind="post",
raw={
"likes": post.get("likeCount", 0),
"reposts": post.get("repostCount", 0),
"post_url": f"https://bsky.app/profile/"
f"{author.get('handle', handle)}/post/{rkey}",
},
)
@staticmethod
def _external_link(post: dict[str, Any]) -> str:
"""Prefer the article a post links to over the post itself."""
embed = post.get("embed") or {}
external = embed.get("external") or {}
if external.get("uri"):
return external["uri"]
for facet in (post.get("record") or {}).get("facets", []) or []:
for feature in facet.get("features", []):
if feature.get("$type", "").endswith("#link") and feature.get("uri"):
return feature["uri"]
return ""
# --------------------------------------------------------------- check
def check(self) -> list[CheckResult]:
return [self._check_one(handle) for handle, _ in self.targets()]
def _check_one(self, handle: str) -> CheckResult:
try:
data = self._get("app.bsky.actor.getProfile", actor=handle)
except httpx.HTTPError as exc:
return CheckResult("bluesky", handle, False, http_failure(exc))
return CheckResult(
"bluesky", handle, True,
f"{data.get('postsCount', '?')} posts, "
f"{data.get('followersCount', '?')} followers",
)
# ----------------------------------------------------------- discovery
def discover(self) -> list[Suggestion]:
settings = self.options.get("discover") or {}
if not settings.get("enabled", True):
return []
seeds = [target for target, _ in self.targets()][: settings.get("max_seeds", 25)]
if not seeds:
return []
following = {s.lower() for s in seeds}
# Fractional, because signal 2 is weighted above signal 1 — so not a
# Counter, whose values typeshed (correctly) says are ints.
counts: defaultdict[str, float] = defaultdict(float)
meta: dict[str, dict[str, str]] = {}
# Signal 1: accounts that several of your accounts follow.
for seed in seeds:
for account in self._paged_follows(seed):
handle = account.get("handle", "").lower()
if not handle or handle in following:
continue
counts[handle] += 1
meta.setdefault(handle, {
"display_name": account.get("displayName", ""),
"description": (account.get("description") or "")[:160],
})
# Signal 2: Bluesky's own "similar to this account" endpoint.
for seed in seeds[:10]:
try:
data = self._get("app.bsky.graph.getSuggestedFollowsByActor", actor=seed)
except httpx.HTTPError:
continue
for account in data.get("suggestions", []):
handle = account.get("handle", "").lower()
if not handle or handle in following:
continue
counts[handle] += 1.5 # weight it slightly above a raw co-follow
meta.setdefault(handle, {
"display_name": account.get("displayName", ""),
"description": (account.get("description") or "")[:160],
})
threshold = settings.get("min_overlap", 3)
out = []
ranked = sorted(counts.items(), key=lambda kv: -kv[1])[:200]
for handle, score in ranked:
if score < threshold:
break
info = meta.get(handle, {})
out.append(Suggestion(
handle=handle,
display_name=info.get("display_name", ""),
reason=f"followed by {int(score)} of the accounts you follow",
score=float(score),
url=f"https://bsky.app/profile/{handle}",
))
return out
def _paged_follows(self, actor: str, cap: int = 400) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
cursor: str | None = None
while len(out) < cap:
params = {"actor": actor, "limit": 100}
if cursor:
params["cursor"] = cursor
try:
data = self._get("app.bsky.graph.getFollows", **params)
except httpx.HTTPError as exc:
log.warning("bluesky: follows for %s failed (%s)", actor, exc)
break
out.extend(data.get("follows", []))
cursor = data.get("cursor")
if not cursor:
break
return out
+196
View File
@@ -0,0 +1,196 @@
"""Mastodon via each instance's public REST API.
Uses /api/v1/accounts/lookup, /api/v1/accounts/{id}/statuses and
/api/v1/accounts/{id}/following. Public statuses are readable without a token
on Mastodon 2.7.0 and later, so no app registration is needed for public
accounts. Instances that run in whitelist mode will 401; those get skipped.
Accounts are written as user@instance.tld.
"""
from __future__ import annotations
import logging
import time
from collections import Counter
from collections.abc import Iterator
from typing import Any
import httpx
from desk.sources.base import (
USER_AGENT, CheckResult, Getter, Item, Source, Suggestion, http_failure,
parse_timestamp, strip_html,
)
from desk.sources.spec import (
API_VERSION, POST_READER, PluginSpec, Profile, Step, SummaryPolicy,
TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools import posts
log = logging.getLogger(__name__)
class MastodonSource(Source):
name = "mastodon"
description = "Public posts from Mastodon accounts (no token needed)."
default_outlet = "Mastodon"
#: A post link on any instance (`/@user/123`) is read through that
#: instance's API. The shape is not proof of Mastodon; a host that does not
#: answer the API in JSON gets the ordinary path.
getters = (Getter("mastodon_api", posts.mastodon, posts.is_mastodon, claims=True),)
#: The same declaration Bluesky makes, for the same reasons — read that
#: one. What differs is the handle: `user@instance.tld` names the server
#: as well as the person, so the check is its own.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Mastodon",
targets=TargetSpec(mode="editable", kind="account", defaults=[],
validator="mastodon_handle"),
profiles={"default": Profile(
kind="post",
acquisition=[Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"})],
tasks=Tasks(
summary=SummaryPolicy(enabled=False),
tagging=TaggingPolicy(
enabled=True, input_fields=["inputs.feed_text"],
context_fields=["author"]),
),
presentation=POST_READER,
)},
)
def __init__(self, options: dict[str, Any] | None = None) -> None:
super().__init__(options)
self.client = httpx.Client(
timeout=20,
follow_redirects=True,
headers={"User-Agent": USER_AGENT},
)
self._delay = float(self.options.get("request_delay", 0.3))
@staticmethod
def _split(acct: str) -> tuple[str, str]:
acct = acct.lstrip("@")
user, _, host = acct.partition("@")
return user, host
def _get(self, host: str, path: str, **params) -> Any:
time.sleep(self._delay)
r = self.client.get(f"https://{host}/api/v1/{path}", params=params)
r.raise_for_status()
return r.json()
def _lookup(self, acct: str) -> dict[str, Any] | None:
user, host = self._split(acct)
if not host:
log.warning("mastodon: %s is missing an instance (use user@instance)", acct)
return None
try:
return self._get(host, "accounts/lookup", acct=user)
except httpx.HTTPError as exc:
log.warning("mastodon: lookup %s failed (%s)", acct, exc)
return None
# ------------------------------------------------------------- ingest
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("posts_per_account", 40))
for acct, label in self.targets():
account = self._lookup(acct)
if not account:
continue
_, host = self._split(acct)
try:
statuses = self._get(
host,
f"accounts/{account['id']}/statuses",
limit=min(limit, 40),
exclude_replies=str(self.options.get("skip_replies", True)).lower(),
exclude_reblogs=str(self.options.get("skip_boosts", True)).lower(),
)
except httpx.HTTPError as exc:
log.warning("mastodon: statuses for %s failed (%s)", acct, exc)
continue
for status in statuses:
card = status.get("card") or {}
text = strip_html(status.get("content", ""))
yield Item(
source_id=status.get("uri") or status.get("url", ""),
location=card.get("url") or status.get("url", ""),
title=card.get("title", ""),
excerpt=text,
author=(status.get("account") or {}).get("display_name") or acct,
published_at=parse_timestamp(status.get("created_at", "")),
subscription=label,
kind="post",
raw={
"boosts": status.get("reblogs_count", 0),
"favourites": status.get("favourites_count", 0),
"post_url": status.get("url", ""),
},
)
# --------------------------------------------------------------- check
def check(self) -> list[CheckResult]:
return [self._check_one(acct) for acct, _ in self.targets()]
def _check_one(self, acct: str) -> CheckResult:
user, host = self._split(acct)
if not host:
return CheckResult("mastodon", acct, False,
"missing instance (use user@instance)")
try:
data = self._get(host, "accounts/lookup", acct=user)
except httpx.HTTPError as exc:
return CheckResult("mastodon", acct, False, http_failure(exc))
return CheckResult("mastodon", acct, True,
f"{data.get('statuses_count', '?')} posts")
# ----------------------------------------------------------- discovery
def discover(self) -> list[Suggestion]:
settings = self.options.get("discover") or {}
if not settings.get("enabled", True):
return []
seeds = [target for target, _ in self.targets()]
following = {s.lstrip("@").lower() for s in seeds}
counts: Counter[str] = Counter()
meta: dict[str, str] = {}
for acct in seeds:
account = self._lookup(acct)
if not account:
continue
_, host = self._split(acct)
try:
# Many accounts hide this list; a 401/403 just means skip.
follows = self._get(host, f"accounts/{account['id']}/following", limit=80)
except httpx.HTTPError:
continue
for other in follows:
handle = other.get("acct", "")
if "@" not in handle:
handle = f"{handle}@{host}"
if handle.lower() in following:
continue
counts[handle] += 1
meta.setdefault(handle, other.get("display_name", ""))
threshold = settings.get("min_overlap", 2)
return [
Suggestion(
handle=handle,
display_name=meta.get(handle, ""),
reason=f"followed by {n} of the accounts you follow",
score=float(n),
url=f"https://{handle.split('@')[-1]}/@{handle.split('@')[0]}",
)
for handle, n in counts.most_common(200)
if n >= threshold
]
+546
View File
@@ -0,0 +1,546 @@
"""A news podcast, read as one item per story rather than one per episode.
The general half. `desk/tools/video/` below knows how to cut a video into
chapters; this knows the things that are true of a *show*: it has a feed, it
publishes on a cycle, its episodes are somewhere on YouTube, and its stories
belong in your queue next to the RSS copies of those same articles.
`plugins/lastweekin_ai.py` above knows one show.
That three-layer split is not decoration. A show that publishes a timestamped
outline works here with nothing show-specific involved — Dwarkesh and Latent
Space both do. What varies between shows is smaller than it looks, and is
declared as two methods rather than guessed at:
video_key(entry) where this show puts its video. The default reads
the feed body, which is the common case and is
wrong for several of the biggest shows.
episode_number(title) how this show writes an episode number.
Where each piece comes from, for one episode:
the feed title, date, the audio enclosure, the video link,
and the show notes as HTML — one fetch, everything
the notes chapter marks, sometimes with the article links attached,
which is what makes a segment a story rather than a topic
YouTube the transcript, with timings
video.cut() one Segment per chapter
this file one Item per Segment
**The subscription is just a feed.** `feeds:` is the same config key `rss`
uses, so a show imports into the `subscriptions` table like any other
subscription and gets add/remove/fetch/show, the Sources tab and the sidebar
for free — none of which had to know podcasts exist.
Config, for any show that is not one somebody has written a plugin for:
sources:
podcast:
feeds:
- url: https://api.substack.com/feed/podcast/69345.rss
label: Dwarkesh
episodes_per_feed: 2
Two things this deliberately does not do. It does not summarise: `text` starts
as the opening of the transcript and is replaced when a summary lands. What the
hosts made of a story is the one thing here that no feed already gives you, and
so the only part of this worth a model — which makes it a separate job, whose
failure costs you the abstracts and not the pull. See
newsdesk/jobs/summarise.py.
And it does not store the episode as an item, or as anything else: an episode
is a date and a video id, both of which live on every segment it produced, so
"every segment of episode 255, in order" is already a query and needs no
entity behind it.
**A show with no outline gets nothing stored, on purpose.** That claim above
used to hold only by luck: with no marks anywhere, `cut` hands back the whole
episode as one segment — right for a link you added by hand, wrong here — and
this stored it, so a two-hour recording arrived as a single eleven-thousand-word
item whose standfirst was the first two sentences of the intro and which then
queued itself for a summariser expecting one story. Now `_one_episode` stops
before the transcript is even fetched and says so in the log. Only shows that
publish a timestamped outline work here, which was always true and is now also
enforced.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from typing import Any
import feedparser
import httpx
from desk.sources.base import USER_AGENT, CheckResult, Item, Source, entry_time
from desk.sources.rss import check_feed
from desk.sources.spec import (
API_VERSION, RECORDING_READER, RSS, PluginSpec, Profile, RetryPolicy, Step,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks,
)
from desk.tools.http import session
from desk.tools.video import Segment, Video, cut, marks, media, opening, youtube
log = logging.getLogger(__name__)
#: "LWiAI Podcast #255 - Gemini 3.7, ..." -> 255. Every show numbers its
#: episodes somewhere; the title is where they all agree to put it.
_NUMBER = re.compile(r"(?:#|episode\s+|ep\.?\s*)(\d{1,5})\b", re.IGNORECASE)
def segment_profile(instructions: str = "", tags: str = "",
target_words: int = 70) -> Profile:
"""What a show that publishes a timestamped outline produces, declared.
A reusable typed preset, so that a plugin for one show is its feed, its
numbering and its editorial line — and not a second copy of this
machinery's description. `plugins/lastweekin_ai.py` is the model.
The two steps are what `_one_episode` below does, in the contract's words:
retrieve the recording's captions *once*, then cut them at the marks the
show published. Nothing executes them yet — the runner is a later step of
`docs/plugin-rework.md` — and they are written now because they are the
part a future reader has to be able to check this file against.
`instructions` is the one thing a show is expected to fill in: what its
own segments want said about them. What is general about summarising
speech — write what they made of it, keep the disagreement, do not quote a
mangled caption back — belongs to the shared task and is written once in
`newsdesk/jobs/summarise.py`, not per show.
"""
return Profile(
# One profile, because one thing is stored: the stories, not the
# episodes they came out of. An episode is a date and a video id, both
# of which are on every segment it produced.
kind="segment",
acquisition=[
Step(id="captions", tool="retrieve_captions",
inputs={"media": "inputs.media"},
outputs={"transcript": "inputs.transcript"},
# Auto-captions on a two-hour upload can take hours to
# appear, so absence is a wait and not a failure. Today that
# wait is "the next pull, for ever", because the episode is
# simply not marked done; a bound is what this declares.
retry=RetryPolicy(max_attempts=4, delay_seconds=3600,
backoff=2.0, on=["not_ready"])),
Step(id="segments", tool="cut_segments", needs=["captions"],
# The marks come from the show's own notes, which arrive in
# the feed entry — HTML, so each story keeps the article link
# beside its timestamp.
inputs={"transcript": "inputs.transcript",
"marks": "inputs.feed_text"},
outputs={"transcript": "inputs.transcript",
"links": "inputs.links"}),
],
tasks=Tasks(
summary=SummaryPolicy(
enabled=True, input_fields=["inputs.transcript"],
require_complete=True,
# No floor: a two-sentence housekeeping segment gets one flat
# sentence saying so, which is worth having and is what the
# shared prompt already asks for.
min_words=0, target_words=target_words,
instructions=instructions),
tagging=TaggingPolicy(
# The transcript, not the abstract: tagging a segment must not
# quietly wait on a summariser that failed. Naming it decides
# *which* text and not how much of it — the tagger reads the
# opening few hundred characters of whatever a profile names,
# because it is choosing from a closed list of labels and a
# two-hour transcript would be thirty times the tokens for
# the same answer. See `newsdesk/jobs/tagging.py`.
enabled=True, input_fields=["inputs.transcript"],
context_fields=["title", "inputs.links"], instructions=tags),
),
# The player first, then what the hosts made of it; the articles
# they were reading stay above it in every view. See the preset.
presentation=RECORDING_READER,
)
class PodcastSource(Source):
name = "podcast"
description = "A show's feed, cut into one item per story discussed."
subscription_field = "feeds"
default_outlet = "Podcast"
#: The generic source declares the same profile it hands its subclasses,
#: with no editorial line of its own: it is any show, and it cannot know
#: what one of them wants said. Its targets are editable because they are
#: whatever you put under `podcast:` in config.yaml — a show with a plugin
#: of its own declares a catalogue instead.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="Podcast",
discovery=RSS(),
targets=TargetSpec(mode="editable", kind="rss_feed", defaults=[]),
profiles={"default": segment_profile()},
)
# -------------------------------------------------------- what a show is
#
# The two things that differ between shows, as methods rather than as
# config knobs — because neither has a value you could write down. Finding
# the video means knowing where that show puts it, and numbering an episode
# means knowing how that show writes a number. A subclass that knows one
# show answers both; everything else in this file is the same for all of
# them.
def video_key(self, entry: Any) -> str:
"""The YouTube id for this episode, or "" if there isn't one to find.
**The one place a show's publishing habits really bite.** The default
reads the feed body, which works when the show embeds or links its own
video there — Last Week in AI, Dwarkesh, Latent Space all do.
It is not general, and it fails in a way worth naming: The Vergecast,
Hard Fork, TWiT, All-In and Lex Fridman all publish full video to
YouTube and none of them mentions it in the feed, so all five come back
empty here. Fixing that means matching a feed entry against a channel's
uploads — which title, which date, whose channel — and that is a
per-show judgement, not a parser. This is where it goes when someone
wants one of those shows: override this, return the id, and every other
line in this file already works.
"""
try:
return youtube.video_id(_body(entry))
except ValueError:
return ""
def episode_number(self, title: str) -> int | None:
"""Which episode this is, from its title, or None if it doesn't say.
Every show numbers its episodes and the title is where they all agree
to put it — but not in one format. `#255` and `Episode 42` and `Ep. 7`
are the common family and are what the default reads; `TWiT 1100` and
`SN 1094` are not, and an interview show may not number at all.
Nothing downstream depends on the answer: it is stored on the item for
display and for "every segment of episode 255, in order". So the honest
default is a regex that catches most of them and returns None rather
than guessing when it doesn't, and a subclass overrides it when its
show writes numbers its own way.
"""
found = _NUMBER.search(title or "")
return int(found.group(1)) if found else None
# ------------------------------------------------------------------ fetch
def fetch(self) -> Iterator[Item]:
limit = int(self.options.get("episodes_per_feed", 2))
with session(None, timeout=40) as http:
for entry in self.options.get("feeds", []):
url = entry["url"] if isinstance(entry, dict) else entry
label = _label(entry, url)
try:
yield from self._one_feed(http, url, label, limit)
except youtube.Blocked:
# The one failure that is not this show's. YouTube refused
# the machine, so every other show here would be refused
# too, and each attempt is one more request from an address
# already taken for a bot. Out of `fetch` altogether: the
# runner keeps what was yielded and turns this into a
# notice, which is what the log line below never was.
raise
except Exception:
# One dead show must not cost the others their episodes.
log.exception("podcast feed %s failed", url)
def _one_feed(self, http: httpx.Client, url: str, label: str,
limit: int) -> Iterator[Item]:
response = http.get(url, headers={"User-Agent": USER_AGENT})
response.raise_for_status()
# The bytes, not `.text`. A feed declares its own encoding in its XML
# declaration and plenty of them are not UTF-8; handing the parser a
# string that httpx has already decoded by its own guess throws that
# declaration away, and what comes back is an episode title with a
# mojibake apostrophe in it. `check_feed` parses the bytes too.
feed = feedparser.parse(response.content)
if not feed.entries:
log.warning("no episodes in %s", url)
return
for entry in feed.entries[:limit]:
episode = self._episode(entry, label)
if episode is None:
continue
if self._already_done(episode.key):
log.debug("%s: already ingested", episode.title)
continue
try:
yield from self._one_episode(http, episode, entry, label)
except youtube.Blocked:
raise # not this episode's failure — see `fetch`
except Exception:
log.exception("episode %s failed; continuing the feed", episode.key)
def _one_episode(self, http: httpx.Client | None, episode: Video, entry: Any,
label: str) -> Iterator[Item]:
# The show's own notes first: same outline as the video description,
# but the links survived. Then the description, then the comments.
chapters = marks.best(episode.notes)
if not chapters:
described = youtube.details(episode.key, client=http)
chapters = marks.best(described["description"])
episode.duration = episode.duration or described["duration"]
if not chapters and self.options.get("read_comments", True):
chapters = marks.in_comments(youtube.comments(episode.key, client=http))
if not chapters:
# Nothing published an outline: not the notes, not the description,
# not a listener in the comments. `cut` would hand back the whole
# episode as one segment — correct of it, and what a hand-added
# link wants — but this source's contract is one item per *story*,
# and quietly substituting one item per episode is a different
# product wearing its name.
#
# It is also unusable downstream in a way that is easy to miss. The
# item's `text` would be `opening(transcript)`, the first two
# sentences of a two-hour recording, which is what the ranker and
# the tag filters would then judge it on; and eleven thousand words
# would go to a summariser whose prompt promises it a single story
# with the article the hosts were reading. Better to store nothing and say
# why than to store one wall and three wrong things about it.
#
# Marked done regardless. Reaching here means the episode was
# fetched and read successfully; there was simply nothing to cut
# along. A show that never publishes an outline would otherwise
# re-download a full transcript on every pull, for ever, to throw
# it away again — which is the waste `_already_done` exists to stop.
log.warning(
"%s: no timestamped outline in the notes, the description or "
"the comments, so there are no stories to cut it into — "
"storing nothing for this episode", episode.title)
self._mark_done(episode.key)
return
try:
cues = youtube.transcript(episode.key, client=http)
except youtube.Blocked:
# A `TranscriptUnavailable` too, and the one that must not be
# treated as one here. Waiting is the answer to captions that are
# still being made; to a refusal it is the same nothing on every
# pull, at INFO, with the episode never marked done and nobody
# told. Still not marked done, so the pull after the address is
# clean reads it — but it leaves as a failure, and reaches the
# screen. `details` above raises the same thing the same way.
raise
except youtube.TranscriptUnavailable as exc:
# Not done, so no marker is written and the next pull tries again.
# Auto-captions on a two-hour upload can take hours to appear, and
# an episode without them is worth waiting for rather than
# ingesting as twenty titles with nothing under them.
log.info("%s: %s", episode.title, exc)
return
segments = cut(episode, cues, chapters)
log.info("%s: %d segments", episode.title, len(segments))
for segment in segments:
yield _item(segment, entry, label,
number=self.episode_number(episode.title))
self._mark_done(episode.key)
def check(self) -> list[CheckResult]:
"""A show is a feed, so the feed check is the right one."""
return [check_feed(url, label) for url, label in self.targets()]
def _episode(self, entry: Any, show: str) -> Video | None:
"""A feed entry as a `Video`. None when there is no video to read.
A show whose video this cannot find is not a failure — see
`video_key`, which is the part a subclass replaces. Saying so once,
quietly, beats failing per episode.
"""
key = self.video_key(entry)
if not key:
# Deliberately not "audio-only shows need local transcription",
# which is what this used to say and which was wrong for the five
# biggest shows it fires on: they all post video, they just do not
# say so in the feed. Naming the actual condition is what stops the
# next person going and building a transcriber they did not need.
log.info("no video id in the feed entry for %r — the show may still "
"publish one; see PodcastSource.video_key",
(entry.get("title") or "")[:60])
return None
return Video(
ref=media.youtube(key),
title=(entry.get("title") or "").strip(),
url=entry.get("link") or "",
author=entry.get("author") or show,
published_at=entry_time(entry),
notes=_body(entry),
)
# ------------------------------------------------------------ bookkeeping
def _already_done(self, key: str) -> bool:
"""Whether this episode has been through here successfully.
In the `state` table rather than in a table of its own, because that is
genuinely all the state an episode has: it is done or it is not. An
episode is not a thing you read, so it needs no row of its own, and one
key in a table that already exists needs no migration.
The key is written only on success, so an episode whose captions
YouTube had not finished generating is left unmarked and comes round
again on the next pull.
Without this, a weekly pull would re-fetch and re-cut every transcript
in the window on every run and then discard the lot at the dedup check
— correct, and several wasted seconds per pull.
"""
return bool(self.state.get(self._done_key(key)))
def _mark_done(self, key: str) -> None:
self.state[self._done_key(key)] = datetime.now(UTC).isoformat()
def _done_key(self, key: str) -> str:
"""Namespaced by the plugin, not hard-coded to `podcast`.
Two shows can publish the same video id — a guest episode cross-posted,
a network re-run — and with one shared prefix the second would find the
first's marker and skip an episode it had never read. One subclass per
show means one namespace per show for free.
"""
return f"{self.name}:done:{key}"
# ------------------------------------------------------------------ internals
def _label(entry: Any, url: str) -> str:
"""What one configured show is called. Which outlet that stands for is the
subscription's business — see `Item.subscription`."""
if not isinstance(entry, dict):
return url
return entry.get("label") or url
def _item(segment: Segment, entry: Any, label: str,
number: int | None = None) -> Item:
"""One segment, as the row newsdesk stores.
Three of the field choices are not obvious:
`body` is the transcript, which is both where it belongs and how this
tells `fetch_bodies` not to go and scrape a YouTube page for one. It
is not the `excerpt`, because the excerpt is what the queue row renders,
what the ranker's document is built from and what search indexes, and
sixteen hundred words of speech would swamp all three.
`excerpt` is a placeholder until a summary is written over it. It is the
opening of the transcript rather than empty, because an item with no
excerpt is an empty row in the queue, and the ranker and the tagger both
read it. It used to be accompanied by a `summarised: False` flag, which
was this source telling the summariser that the row was waiting for it.
Nothing needs telling now: the profile declares that these items are
summarised, and an empty `summary` column is the whole of "not yet".
`published_at` is the episode's stamp plus the segment's offset, so twenty
items published in the same instant still sort in the order they were said.
`links` is the reason a segment is a story rather than a topic: the
articles the outline attached to this stretch, typed and roled, so the
reading pane and the summariser read them from the same field an outlet's
links live in rather than from this plugin's own corner of `raw`.
"""
episode = segment.video
return Item(
source_id=segment.external_id,
location=segment.url(),
title=segment.title,
excerpt=opening(segment.transcript),
body=segment.transcript,
author=episode.author,
published_at=episode.published_at + timedelta(seconds=segment.start),
subscription=label,
links=segment.reading(),
# Not `article`, which is what every one of these used to be stored as
# — twenty-three transcripts filed as prose. What the pipeline made,
# not what carried it: this arrived over RSS and left as segments.
kind="segment",
raw={
# The chapter this segment sits under. Per *item* — it changes
# down the episode — so it is here rather than on the
# subscription's `section`, which names a branch of an outlet and
# is the same for everything that subscription hands over.
"heading": segment.heading,
# Where it plays and what it linked to: the record every stored
# segment carries, defined once on `Segment`.
**segment.record(),
"show": label,
"episode": number,
"episode_title": episode.title,
"episode_url": episode.url,
"episode_published": episode.published_at.isoformat(),
"audio_url": _enclosure(entry),
},
)
def _body(entry: Any) -> str:
"""The richest form of the notes the feed offers."""
for content in entry.get("content") or []:
if value := content.get("value"):
return value
return entry.get("summary") or ""
def _enclosure(entry: Any) -> str:
return next(
(e.get("href", "") for e in entry.get("enclosures") or []
if (e.get("type") or "").startswith("audio")),
"",
)
# ----------------------------------------------------------------- dry run
def main(argv: list[str] | None = None) -> int:
"""`python -m desk.sources.podcast <feed url>` — what a pull would store.
Runs the plugin with no store behind it and prints the items instead of
writing them. This is the thing to run when a show changes the shape of
its notes: everything else in the pipeline is downstream of these fields.
"""
import argparse
import sys
parser = argparse.ArgumentParser(prog="python -m desk.sources.podcast")
parser.add_argument("feed", help="the show's RSS feed")
parser.add_argument("--episodes", type=int, default=1)
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)-7s %(message)s")
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
source = PodcastSource({
"feeds": [args.feed], "episodes_per_feed": args.episodes,
})
items = list(source.fetch())
if not items:
print("no items — run with -v to see why", file=sys.stderr)
return 1
print(f"{len(items)} items from {args.feed}\n")
heading = None
for item in items:
if item.raw["heading"] != heading:
heading = item.raw["heading"]
print(f" {heading or '(unfiled)'}")
stamp = marks.hhmmss(item.raw["start"])
print(f" {stamp:>8} {item.title[:80]}")
print(f" {item.location}")
for link in item.links:
print(f" → {link.url}")
print(f" {len(item.body.split()):,} words · {item.excerpt[:90]}")
print()
return 0
if __name__ == "__main__":
raise SystemExit(main())
+103
View File
@@ -0,0 +1,103 @@
"""Finds Source subclasses on disk. `desk.sources.build` instantiates them."""
from __future__ import annotations
import importlib.util
import inspect
import logging
import pkgutil
import sys
import hashlib
import threading
from functools import lru_cache
from pathlib import Path
from desk.sources.base import Source
log = logging.getLogger(__name__)
_load_lock = threading.RLock()
def _classes_in_module(module) -> list[type[Source]]:
found = []
for _, obj in inspect.getmembers(module, inspect.isclass):
if not issubclass(obj, Source) or obj is Source:
continue
if inspect.isabstract(obj):
continue
# A class written to be subclassed is not a plugin. `RssSource` is
# the fetching a publisher's plugin inherits; it has no feeds, no
# archive and nothing to switch on.
if obj.__dict__.get("shared", False):
continue
if not getattr(obj, "name", ""):
log.warning("%s.%s has no `name`, skipping", module.__name__, obj.__name__)
continue
# Only take classes defined in this module, not imported into it.
if obj.__module__ != module.__name__:
continue
found.append(obj)
return found
def _load_file(path: Path) -> list[type[Source]]:
path = path.resolve()
try:
stamp = path.stat()
# A concurrent status request must not execute the same plugin twice.
with _load_lock:
return _load_version(path, stamp.st_mtime_ns, stamp.st_size)
except OSError:
log.exception("cannot read plugin %s", path)
return []
@lru_cache(maxsize=128)
def _load_version(path: Path, modified: int, size: int) -> list[type[Source]]:
"""Reuse unchanged plugins while allowing edits to take effect on next use."""
identity = hashlib.sha256(str(path).encode()).hexdigest()[:16]
mod_name = f"newsdesk_plugin_{path.stem}_{identity}"
spec = importlib.util.spec_from_file_location(mod_name, path)
if spec is None or spec.loader is None:
return []
module = importlib.util.module_from_spec(spec)
sys.modules[mod_name] = module
try:
spec.loader.exec_module(module)
except Exception:
sys.modules.pop(mod_name, None)
log.exception("failed to load plugin %s", path)
return []
return _classes_in_module(module)
def available(plugin_paths: list[str] | None = None) -> dict[str, type[Source]]:
"""Map of name -> Source class. Later definitions win, so a user plugin
named `rss` overrides the built-in one, with a warning naming both classes."""
classes: dict[str, type[Source]] = {}
import desk.sources as builtin_pkg
for info in pkgutil.iter_modules(builtin_pkg.__path__):
if info.name == "base":
continue
module = importlib.import_module(f"desk.sources.{info.name}")
for cls in _classes_in_module(module):
classes[cls.name] = cls
for raw in plugin_paths or []:
directory = Path(raw).expanduser()
if not directory.is_dir():
log.warning("plugin_path %s is not a directory", directory)
continue
for path in sorted(directory.glob("*.py")):
if path.name.startswith("_"):
continue
for cls in _load_file(path):
if (previous := classes.get(cls.name)) and previous is not cls:
log.warning("plugin name %s: %s.%s overrides %s.%s",
cls.name, cls.__module__, cls.__name__,
previous.__module__, previous.__name__)
classes[cls.name] = cls
return classes
+236
View File
@@ -0,0 +1,236 @@
"""RSS and Atom feed fetching, as a base class for a publisher's plugin.
Not itself a source: see `RssSource`, which is what a masthead subclasses.
`check_feed` is the same fetching asked whether one address still answers.
"""
from __future__ import annotations
import logging
from collections.abc import Iterator
from datetime import UTC, datetime
from typing import Any
import feedparser
import httpx
from desk.sources.base import (
USER_AGENT, CheckResult, Item, Source, entry_time, http_failure, outlet_name,
strip_html,
)
from desk.tools.http import session
log = logging.getLogger(__name__)
#: Seconds one feed gets, per operation. Which number it is matters far less
#: than there being one at all — see `_read`.
FETCH_TIMEOUT = 30
class FeedsFailed(Exception):
"""Some of a pull's feeds could not be fetched, and the rest were.
Raised at the end of `fetch` rather than where it happened, and that is the
whole of the design: everything the live feeds gave has already been
yielded and stored by the time this lands, so one outlet returning 403
costs you that outlet and not the other nineteen. The runner turns it into
`notices.source_failed`, which is what puts a feed that has been dead all
week on screen — the one failure a reader cannot spot by eye, because a
dead source and a quiet news day look identical.
Worded through `http_failure`, so every line in it reads `HTTP 403` or
`unreachable: ReadTimeout`. That is also what has the notice file it as the
outlet's business rather than as a defect of ours; see
`desk/notices.py`.
"""
def _categories(entry) -> list[str]:
"""Whatever the outlet put in <category>.
The Guardian, Ars Technica, the NYT and Al Jazeera all populate these;
the BBC and Hacker News don't. They're inconsistent enough between outlets
to be useless as a filter — the NYT files people ('Putin, Vladimir V'),
the Guardian files desks ('Television & radio') — so they're carried as a
hint for the tagger rather than shown to you.
"""
seen, out = set(), []
for tag in entry.get("tags") or []:
term = strip_html(tag.get("term", ""))
if term and term.lower() not in seen:
seen.add(term.lower())
out.append(term)
return out
def check_feed(url: str, label: str = "", kind: str = "rss") -> CheckResult:
"""Fetch one feed and say what is in it. Shared with the podcast source,
whose subscription is a feed too."""
try:
r = httpx.get(url, headers={"User-Agent": USER_AGENT}, timeout=20,
follow_redirects=True)
except httpx.HTTPError as exc:
return CheckResult(kind, url, False, f"unreachable: {type(exc).__name__}", label)
if r.status_code != 200:
return CheckResult(kind, url, False, f"HTTP {r.status_code}", label)
parsed = feedparser.parse(r.content)
n = len(parsed.entries)
if n == 0:
reason = parsed.get("bozo_exception")
return CheckResult(kind, url, False, f"0 entries ({reason or 'not a feed?'})", label)
return CheckResult(kind, url, True, f"{n} entries, {_age(parsed.entries)}", label,
outlet=_proposed_outlet(parsed, url, label))
def _read(http: httpx.Client, url: str) -> Any:
"""One feed, fetched here and parsed from the bytes it answered with.
Deliberately not `feedparser.parse(url)`, which is the same call with the
fetching left to somebody else and which goes through urllib with **no
timeout**. A
host that accepts the connection and then says nothing holds the thread for
ever, and the thread it holds is a worker's: the pull is `solo`, so nothing
else pulls behind it; cancellation is cooperative, so Kill cannot reach a
thread parked in a socket read; and a restart releases the lease only to
re-queue the job and stall it on the same feed. One bad feed stopped the
application from pulling at all until somebody edited the subscription.
feedparser offers no way to pass urllib a deadline, so the fetch comes
here, where the client has one.
Two things fall out of doing it here, both of which `check_feed` has had
all along. The request goes out as `USER_AGENT`, which `base.py` promises
every plugin sends and which feedparser was quietly replacing with its own.
And the status is read, so a 403, 410 or 429 — each of which arrives with
an empty body that parses perfectly well as a feed with nothing in it — is
a failure of that feed rather than a shrug about a slow news day.
"""
response = http.get(url, headers={"User-Agent": USER_AGENT})
response.raise_for_status()
# The bytes rather than `.text`, so the document's own XML declaration
# decides its encoding instead of a guess made from a header the outlet
# may never have set.
return feedparser.parse(response.content)
def _proposed_outlet(parsed: Any, url: str, label: str) -> str:
"""What this feed calls itself — but only when you didn't say.
A label you wrote is the better name and the check must never talk you out
of it: "Guardian Tech" splits into an outlet and a section that a feed's
own `<title>` ("Technology | The Guardian") cannot improve on. A label is
only ever *this* when you pasted a URL and named nothing, and then the
alternative is filing your reading under `https://…`, which the feed's
title beats outright.
"""
if label and label != url:
return ""
return outlet_name(strip_html(parsed.feed.get("title", "")))
def _age(entries) -> str:
stamps = [entry_time(e) for e in entries
if e.get("published_parsed") or e.get("updated_parsed")]
if not stamps:
return "no dates"
hours = (datetime.now(UTC) - max(stamps)).total_seconds() / 3600
return f"newest {hours:.0f}h ago" if hours < 48 else f"newest {hours / 24:.0f}d ago"
class RssSource(Source):
"""Feed fetching, for a publisher's plugin to inherit. Not a plugin.
`shared` is what says so, and retiring the generic "Any RSS feed" source
it used to be is §9 of the rework: an outlet is a plugin, so a masthead
has a file that declares its own feeds, marks and policies rather than a
row somebody pasted a URL into. Everything below is unchanged — this is
the same fetching the Guardian, the BBC, Ars Technica, Golem and Hacker
News have always run on, reached now by subclassing rather than by
configuring.
A subclass states its own `name`; the empty one here is deliberate, so
that forgetting it is a plugin the registry skips loudly rather than a
second class claiming the archive of the first.
"""
name = ""
shared = True
#: Inherited by a subclass that states none of its own.
description = "An RSS or Atom feed."
subscription_field = "feeds"
def fetch(self) -> Iterator[Item]:
feeds = self.options.get("feeds", [])
failed: list[str] = []
# One client for the whole pull, so twenty feeds off one host are one
# connection rather than twenty. Closed on the way out however this
# generator ends.
with session(None, timeout=FETCH_TIMEOUT) as http:
for entry in feeds:
# The label is the whole of what an item says about where it came
# from. Which outlet and which section of it that label stands for
# is the subscription's business, settled once when you added it —
# see `CheckResult.outlet`, which is how this plugin has its say.
if isinstance(entry, dict):
url = entry["url"]
label = entry.get("label") or url
else:
url = label = entry
try:
parsed = _read(http, url)
except (httpx.HTTPError, httpx.InvalidURL) as exc:
# Everything this feed's own end can do to us: a status, a
# host that never answers, an address that was mistyped
# when it was pasted. Noted and carried past, because the
# next feed in the list has nothing to do with any of it.
#
# Nothing broader is caught. What else could land here is a
# wrong assumption of ours about the shape of a response,
# and notices.py is explicit that those have to reach you
# as bugs rather than be folded in with the weather.
log.warning("could not fetch %s: %s", url, http_failure(exc))
failed.append(f"{label} ({http_failure(exc)})")
continue
if parsed.bozo and not parsed.entries:
# It answered, and what came back is not a feed. Empty
# rather than failed: the server did its part, and there is
# nothing here for anyone to go and fix.
log.warning("no entries from %s (%s)", url, parsed.get("bozo_exception"))
continue
yield from self._items(parsed, url, label)
if failed:
raise FeedsFailed(
f"{len(failed)} of {len(feeds)} feeds failed: " + "; ".join(failed))
def _items(self, parsed: Any, url: str, label: str) -> Iterator[Item]:
"""One feed's entries, as rows. Split out of `fetch` only so that the
loop over your subscriptions stays readable beside what it catches."""
for e in parsed.entries[: self.options.get("max_items", 60)]:
yield self.item_from_entry(e, url, label, parsed.feed.get("title", ""))
def item_from_entry(self, entry: Any, url: str, label: str, author: str = "") -> Item:
"""Map one parsed entry; outlets may refine fields without owning HTTP."""
link = entry.get("link") or entry.get("id") or ""
body = ""
if entry.get("content"):
body = strip_html(entry.content[0].get("value", ""))
body = body or strip_html(entry.get("summary", ""))
raw: dict[str, Any] = {"feed": url}
if cats := _categories(entry):
raw["categories"] = cats[:12]
return Item(
source_id=entry.get("id") or link,
location=link,
title=strip_html(entry.get("title", "")),
excerpt=body,
author=entry.get("author", "") or author,
published_at=entry_time(entry),
subscription=label,
kind="article",
raw=raw,
)
def check(self) -> list[CheckResult]:
return [check_feed(url, label) for url, label in self.targets()]
+110
View File
@@ -0,0 +1,110 @@
# `desk.sources` — the plugins
## What it is
**"Source" and "plugin" are the same thing.** One `.py` file with a `Source` subclass in it. It answers one question: *where do items come from* — fetch, prepare, yield. Nothing else: no store, no queue, no config beyond its own options.
| | |
|---|---|
| Contract | `base.py` — `Source`, `Item`, `Suggestion`, `CheckResult` |
| Discovery | `registry.available()` — every module in `desk/sources/`, plus every `*.py` in the dirs under `plugin_paths:` in config.yaml (leading `_` skipped) |
| Built-ins | `bluesky`, `mastodon`, `x`, `podcast`. `RssSource` is here too and is **not** a plugin: it is `shared`, which the registry passes over, and a publisher subclasses it |
| Packages | `httpx` fetches, `feedparser` parses what a feed answered with. No plugin framework, no entry points |
| Door | `from desk.sources import Source, Item, build, available, PodcastSource` |
| Override | later definition wins — a user plugin named `bluesky` replaces the built-in, with a warning naming both |
## The three steps
1. **Fetch** — `fetch()` is a generator; hit the network, yield as you go. Put a timeout on every request you make — see Limits.
2. **Prepare** — normalise into `Item`. Over-fetching is fine: the runner dedups on `uid = sha1(source + source_id)`.
3. **Output** — `Item`s. The runner fills `source`, applies the default outlet, and stores them.
`Item(source_id, location, title, excerpt="", body="", author="", published_at=now, subscription="", kind="article", marks=[], raw={}, links=[])`. `published_at` must be an aware datetime; it is stored as UTC. A non-empty `body` means "don't scrape this page" — that's how a podcast segment keeps its transcript. A post whose `location` equals `raw["post_url"]` links nothing and is not fetched either. The source's own labels go in `raw` under `categories`, which the tagger reads as a weak hint. `ItemLink`, exported through the same door, carries an absolute HTTP(S) URL, title and `reading`/`discussion`/`source` role separately from the item's primary destination.
**A plugin may register how to read its own URLs.** `getters = (Getter("bluesky_api", posts.bluesky, posts.is_bluesky, claims=True),)` on the class says that a `bsky.app` post link, from whichever outlet, is read through that function rather than as a web page, while this plugin is active. It gets the page only; which link an item points to is decided in `fetch`. See `docs/architecture.md` §1.1.
**A plugin declares what should be done with its items, and the app does it.** `spec.py` is that declaration — `PluginSpec`, its profiles, and a `SummaryPolicy` and `TaggingPolicy` per profile. Both shared jobs now run off one: a profile says whether its items are summarised and tagged, which stored text each task reads in order of preference, how short is too short for an abstract and how long one should be, and what this outlet wants said or emphasised about its own items. Those instructions travel with each item in a mixed batch and never apply to another outlet's. See `newsdesk/jobs/summarise.py` and `newsdesk/jobs/tagging.py`.
**An outlet is a plugin.** There is no generic "Any RSS feed" source to paste a masthead into — step 19 retired it, because a feed belongs to a publisher and a publisher has extraction, marks, retries and a summary floor of its own. `RssSource` is the fetching those plugins share: subclass it, give it a `name` and a `spec`, declare the feeds. `plugins/ars_technica.py` is the smallest one, a catalogue of a single feed. A feed nobody has written a plugin for keeps its archive and its Show switch and is simply not fetched; the v28 migration names each one in the log.
**Declaring nothing is not declaring no.** A plugin with no `spec` gets an adapted one with both tasks off, and the two jobs read that silence differently on purpose. Nothing undeclared has ever been summarised, so the summariser takes it as no and loses nothing. Tagging has run over every row in the archive since long before this contract, so it takes it as *today's behaviour* — the acquired text, then the feed's own, then an abstract as a last resort — and only a profile that says `enabled=False` out loud is left untagged. A converted outlet must therefore say `enabled=True` where its items were already tagged; Guardian and Hacker News say so. Which field a tagging policy names decides *which* text and not how much of it: the tagger reads the opening few hundred characters whatever is named, because it is choosing from a closed list of ~35 labels and a whole transcript would be thirty times the tokens for the same answer.
**A profile also declares how its items are read.** The drawer is fixed: Summary, Text, Original and a new-tab action, in that order, for every plugin. The content is the plugin's: which slots it fills, what to call them (a segment's text is its Transcript, its original is Watch), and an ordered list of sections per view, each `prose`, `links` or `media` bound to a named field. It cannot add a fourth view, emit markup, or overrule an outlet that refuses framing. `header` holds links shown above every view: the host of the page the item is, the discussion thread of an aggregator's link, the articles a podcast segment discusses. `QueueLayout.destination` adds that host to the queue row too, and is for a plugin whose outlet is not the publisher. Name a preset rather than restating one: `ARTICLE_READER` for a publisher's own article, `LINK_READER` for a link to someone else's page, `FEED_READER` for a walled outlet, `POST_READER` for a social post, `RECORDING_READER` for a recording's segment. The table in `docs/architecture.md` §1.4 says which flow each is. A view may declare what to show `instead` when its own field is empty, with its own label: `ARTICLE_READER`'s Summary control falls through to the feed's own text and calls itself RSS feed, which is what the queue row beside it has always done. The rename is half of it: a publisher's teaser under a control marked Summary reads as our abstract of the piece, and as a poor one.
**A mark and a wall are different facts, and only one of them changes the flow.** `framing` asks whether the browser will render a page in a frame, and the page behind a paywalled item renders perfectly — it is the paywall. `OriginalView.blocked_by` names the marks that refuse the embed for the items carrying one (`paywalled` by default); the layout does not change, because the piece beside it in the same feed is an ordinary article, and the control greys out with the mark's own words (`Mark.framed`). A `Wall` on a profile is the other thing: there is no article behind any URL this outlet publishes, no abstract will ever be written from one, and the destination frames the gate — so the flow differs rather than the ordinary one standing there with two permanently dead controls. `FEED_READER` is that flow, the feed's own text and the way out, and a walled profile declaring an Original view of its destination is refused where it is written. Its reason rides on `ReaderModel.note` as the one line in the bar, since there is no greyed button left to hover. A plugin that learns to get past its wall deletes the declaration and is an ordinary article profile again. The server resolves the declaration against one item and serves the answer; the browser renders it and decides nothing. See `newsdesk/web/reader.py`.
A show subclasses `PodcastSource`, which finds the episode, retrieves its captions once and cuts them at the marks the show published. `plugins/lastweekin_ai.py` is the model: a catalog of one feed, `segment_profile()` — the machinery's own declaration of what it produces, exported through this door for a show to reuse — and nothing of its own beyond how this show numbers an episode and what it wants said about its segments. Each segment carries the articles its outline linked as `links` with the `reading` role, so the reading pane and the summariser read them from the same field an outlet's links live in.
A show whose site already publishes the cut needs none of that. `plugins/ai_daily_brief.py` subclasses `Source` directly: the site lists its editions in `/agent.json`, each as JSON nuggets and a minute-marked transcript, so the plugin makes one segment per nugget from two requests, with no feed, no captions and no summariser.
RSS outlets subclass `RssSource`. Its `item_from_entry(entry, url, label, author)` hook refines one parsed entry while shared code owns HTTP, feed parsing, limits and failure reporting. `plugins/hacker_news.py` uses it to preserve publisher destinations and HN discussion links while keeping points/comment metadata out of preview text. Guardian needs no override. Both declare fixed catalogs; unknown custom feeds remain supported through generic RSS.
**There is no `outlet` and no `section`.** Who publishes a thing is a fact about the subscription it came from, held once on the subscription row and joined when something needs it — so `subscription` is the only attribution an item carries. They used to be fields, copied onto every item, and a rename left four hundred rows disagreeing with the feed they came from until somebody walked the table.
## What a plugin declares about failing
`spec.py` is the other half of the contract: not where items come from but what should be done with them. `Step.retry` is the part of it that runs today.
```python
Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"},
retry=RetryPolicy(max_attempts=2, delay_seconds=300, backoff=1.0,
on=["timeout", "unreachable", "server_error", "rate_limited"],
manual=True))
```
**Per step and per plugin, which is the whole reason it is declared.** A transcript that was not ready an hour ago and a host that timed out are different waits, and retrying the whole outlet to get one of them is how a source ends up fetched four times an evening. `max_attempts` counts the first attempt, so the default of 1 is "no retry" and is what every plugin that declares nothing gets — behaviour unchanged from before any of this existed.
**`on` names failure categories from `desk.tools.failures`**, and a name that is not one of them is refused where the plugin is written rather than becoming a policy that silently matches nothing. Name the ones that could answer differently next time. A consent wall is a category and belongs in nobody's list: it is the same answer however many times it is asked for. Where the server named its own wait — a `Retry-After` — that is honoured in place of `delay_seconds`, but it buys no extra attempt: the bound is the plugin's and the wait is the world's.
**`manual` is whether the reader offers a person the retry** the policy would not take on its own. The app still checks that the offer means anything: the plugin must be active, and the item must actually have been tried and come back empty. A target's Fetch switch is *not* checked for a manual retry — Fetch governs the evening's automatic acquisition, not a repair somebody asked for.
## Minimum example
```python
# plugins/mysource.py (or desk/sources/mysource.py)
from desk.sources import Source, Item
class MySource(Source):
name = "mysource" # the key under `sources:` in config.yaml
description = "One line shown in the Sources tab."
subscription_field = "accounts" # or "feeds" for URL-shaped subscriptions
def fetch(self):
for target, label in self.targets(): # subscriptions from config/UI
yield Item(source_id="abc123", location="https://example.com/x",
title="Headline", excerpt="Summary", subscription=target)
```
```yaml
# config.yaml
plugin_paths: ["plugins"]
sources:
mysource:
enabled: true
accounts: ["someone"]
```
Optional, both defaulting to nothing: `discover() -> [Suggestion]` (accounts worth following) and `check() -> [CheckResult]` (does each subscription still resolve — free requests only, it runs for every source at once).
## What a plugin is given
- `self.options` — its own config block. By pull time the subscription list has been replaced with the fetch-enabled rows from the Sources tab.
- `self.state` — a `MutableMapping[str, str]` backed by the database, for what it must remember between pulls (a spend counter, which episodes are done). Driven directly it's a plain dict.
**`self.state` is one namespace shared by every plugin**, not one per source. Nothing prefixes your keys for you, so prefix them yourself: `x.py` writes `x_spend_2026_09`, and `PodcastSource` derives its done-marker key from the subclass name. A bare key like `last_seen` is a collision waiting for the second plugin that wants one, and the symptom is one source silently reading another's bookkeeping.
Helpers worth knowing: `strip_html`, `parse_timestamp`, `entry_time`, `split_label`, `outlet_name`, `MULTIWORD_OUTLETS`, `http_failure`, `USER_AGENT`, `check_feed`.
Two of those read names, and which one you want depends on who wrote the string. `split_label` reads a label *you* wrote and cuts it into outlet and section — "Guardian Tech" into the two — with `MULTIWORD_OUTLETS` as the list of names that contain a space themselves, so "Hacker News Best" does not come out as an outlet called Hacker. `outlet_name` reads a name a *server* gave for itself and cuts the tagline off it, because both a feed's `<title>` and a site's own title tend to arrive as the SEO version.
## Limits
- **One file, no state but `self.state`.** No store, no queue, no cross-source access. That's the deal that keeps a new source one file.
- **A failed target and an empty one are different answers.** Yielding nothing means there was nothing new; raising means something is wrong and you are owed a notice about it — `desk/notices.py` decides from the exception whether it reads as the world misbehaving or as a bug of ours. A source with several subscriptions gets both at once by yielding everything that worked and raising once at the end for what didn't, which is what stops one dead feed costing you the other nineteen. `rss.FeedsFailed` is that exception.
- **`fetch` is synchronous and blocking.** Runs in the worker's thread. Async, rate-limiting and *timeouts* are yours. A slow source holds the pull; a source that fetches without a deadline holds it for ever, because cancellation is cooperative and Kill cannot reach a thread parked in a socket read. The case to watch for is a library that does its own fetching: `feedparser.parse(url)` and `requests` with no `timeout=` both wait indefinitely by default, so `rss.py` fetches the bytes itself and hands the parser a document.
- **No sandbox, no versioning.** A plugin file is imported and trusted; a plugin that raises on import is logged and skipped, one that raises inside `fetch` fails that pull's source.
- **`name` must be unique and set** — no `name`, not discovered.
- **Discovery is flat**: `*.py` in a directory, not packages, not installed distributions.
- **A source can't create jobs.** It yields items; what happens next (body, vectors, tags) is the app's ingest job.
+1012
View File
File diff suppressed because it is too large. Load diff
+203
View File
@@ -0,0 +1,203 @@
"""X (Twitter) via the official API v2. Off by default, because it costs money.
X moved to pay-per-usage billing. As published on the official pricing page,
post reads are $0.005 per resource and user reads are $0.010 per resource,
with pay-per-usage plans capped at 3 million post reads per billing cycle.
Reads of your *own* account's data ("Owned Reads") are $0.001 per resource.
So a config with 30 accounts x 20 posts x 4 polls a day is roughly
30*20*4*30 = 72,000 reads/month = about $360/month. Check the estimate this
plugin prints before you enable it.
This plugin therefore:
- is disabled unless you set `enabled: true`
- refuses to run without `budget_usd_per_month`
- tracks spend in the `state` table and stops when the budget is gone
- caches user IDs so it never pays for a lookup twice
Source for rates: https://docs.x.com/x-api/getting-started/pricing
"""
from __future__ import annotations
from urllib.parse import urlsplit
import logging
import os
from collections.abc import Iterator
from datetime import UTC, datetime
from typing import Any
import httpx
from desk.sources.base import USER_AGENT, Item, Source, Suggestion, parse_timestamp
from desk.sources.spec import (
API_VERSION, POST_READER, PluginSpec, Profile, Step, SummaryPolicy,
TaggingPolicy, TargetSpec, Tasks,
)
log = logging.getLogger(__name__)
API = "https://api.x.com/2"
POST_READ_USD = 0.005
USER_READ_USD = 0.010
def _is_internal(url: str) -> bool:
"""Match the host boundary so linux.com remains an external article."""
try:
host = (urlsplit(url).hostname or "").lower().rstrip(".")
except ValueError:
return False
return any(host == domain or host.endswith("." + domain)
for domain in ("x.com", "twitter.com"))
class XSource(Source):
name = "x"
default_outlet = "X"
description = "X/Twitter timelines (paid API — off by default)."
#: Declared like the other two networks. The budget, the token and the
#: refusal to run without either stay in the code above: they are what
#: this plugin *does*, and none of them is a thing the shared jobs or the
#: reader need to be told about.
spec = PluginSpec(
api_version=API_VERSION, id=name, version="1", label="X",
targets=TargetSpec(mode="editable", kind="account", defaults=[],
validator="x_handle"),
profiles={"default": Profile(
kind="post",
acquisition=[Step(id="article", tool="extract_article",
inputs={"url": "inputs.original_url"},
outputs={"content": "inputs.article_text"})],
tasks=Tasks(
summary=SummaryPolicy(enabled=False),
tagging=TaggingPolicy(
enabled=True, input_fields=["inputs.feed_text"],
context_fields=["author"]),
),
presentation=POST_READER,
)},
)
def __init__(self, options: dict[str, Any] | None = None) -> None:
super().__init__(options)
token_env = self.options.get("bearer_token_env", "X_BEARER_TOKEN")
self.token = os.environ.get(token_env, "")
self.client = httpx.Client(
timeout=25,
headers={
"Authorization": f"Bearer {self.token}",
"User-Agent": USER_AGENT,
},
)
@property
def enabled(self) -> bool:
return bool(self.options.get("enabled", False))
def estimate_monthly_usd(self, polls_per_day: int = 4) -> float:
n = len(self.options.get("accounts", []))
per_poll = n * int(self.options.get("posts_per_account", 10))
return per_poll * polls_per_day * 30 * POST_READ_USD
# ------------------------------------------------------------ budget
def _spent(self) -> float:
month = datetime.now(UTC).strftime("%Y-%m")
return float(self.state.get(f"x_spend_{month}", "0") or 0)
def _charge(self, usd: float) -> None:
month = datetime.now(UTC).strftime("%Y-%m")
self.state[f"x_spend_{month}"] = str(self._spent() + usd)
def _user_id(self, handle: str) -> str | None:
cache_key = f"x_userid_{handle.lower()}"
cached = self.state.get(cache_key)
if cached:
return cached
r = self.client.get(f"{API}/users/by/username/{handle}")
if r.status_code != 200:
log.warning("x: lookup %s returned %s", handle, r.status_code)
return None
self._charge(USER_READ_USD)
uid = (r.json().get("data") or {}).get("id")
if uid:
self.state[cache_key] = uid
return uid
# ------------------------------------------------------------- ingest
def fetch(self) -> Iterator[Item]:
if not self.token:
log.error("x: no bearer token in $%s",
self.options.get("bearer_token_env", "X_BEARER_TOKEN"))
return
budget = self.options.get("budget_usd_per_month")
if budget is None:
log.error("x: refusing to run without `budget_usd_per_month` in config")
return
per_account = int(self.options.get("posts_per_account", 10))
exclude = ",".join(self.options.get("exclude", ["retweets", "replies"]))
for handle, label in self.targets():
spent = self._spent()
if spent >= float(budget):
log.warning("x: monthly budget of $%.2f exhausted ($%.2f spent), stopping",
float(budget), spent)
return
uid = self._user_id(handle)
if not uid:
continue
r = self.client.get(
f"{API}/users/{uid}/tweets",
params={
"max_results": max(5, min(per_account, 100)),
"exclude": exclude,
"tweet.fields": "created_at,entities,public_metrics",
},
)
if r.status_code != 200:
log.warning("x: timeline %s returned %s: %s",
handle, r.status_code, r.text[:200])
continue
payload = r.json()
posts = payload.get("data", []) or []
self._charge(len(posts) * POST_READ_USD)
for post in posts:
urls = ((post.get("entities") or {}).get("urls") or [])
external = next(
(u.get("expanded_url") for u in urls
if u.get("expanded_url")
and not _is_internal(u["expanded_url"])),
"",
)
yield Item(
source_id=post["id"],
location=external or f"https://x.com/{handle}/status/{post['id']}",
title="",
excerpt=post.get("text", ""),
author=handle,
published_at=parse_timestamp(post.get("created_at", "")),
subscription=label,
kind="post",
raw={
"metrics": post.get("public_metrics", {}),
"post_url": f"https://x.com/{handle}/status/{post['id']}",
},
)
def discover(self) -> list[Suggestion]:
# The follow-graph endpoints are billed per user read, which makes
# graph-walking on X expensive enough that it is not worth doing
# automatically. Use Bluesky/Mastodon discovery instead.
return []
# `check` is left at the default too: a user lookup is a billed read, and
# a check job must never cost money.
+20
View File
@@ -0,0 +1,20 @@
"""Reusable machinery with no opinion about news.
A tool is something a pipeline stage picks up and puts down: it takes plain
inputs, returns plain outputs, and knows nothing about subscriptions, feeds,
or what a job is for. That is the test for whether a module belongs here —
if it would make sense in a different application, it is a tool.
agent.py who answers when something needs a model: the Claude API
with a key, the `claude` CLI on a subscription, or nobody
embedding.py text in, vector out
framing.py will this page load inside an iframe?
http.py one shared HTTP client per call chain
marks.py what kind of thing is this, if not an article?
video/ a recording, read as text with timings on it
walls.py was that the article, or the page shown instead of it?
Nothing is imported here on purpose. `video` pulls in httpx and the caption
parsers, `embedding` a sentence-transformer, `agent` an SDK; each is paid for
by the caller that wants it and by nobody else.
"""
+461
View File
@@ -0,0 +1,461 @@
"""Who answers when a job needs a model.
There is one question this module exists to answer — *ask Claude this, get JSON
back* — and two ways to answer it. They are interchangeable, so a handler
asks without knowing which one ran:
api One call to the Claude API. Wants ANTHROPIC_API_KEY (or an
`ant auth login` profile). No local install, no subprocess,
per-token billing.
claude_code The Claude Agent SDK drives the `claude` CLI already on this
machine, on the subscription it is already signed in with.
No key, nothing to bill. This is what the server runs on
today, authenticated by CLAUDE_CODE_OAUTH_TOKEN.
`backend()` answers `""` when neither can run, and that is a failure rather
than a third way: the caller says so with a notice naming the reason — no
key, no CLI, `backend: off` — and the batch fails without asking anybody.
Nothing waits for a person to answer it by hand.
The seam is deliberately narrow: `ask_json` takes a system prompt, a payload
and a JSON schema, and returns parsed JSON. Both backends take the *same*
schema — the Agent SDK accepts the Messages API's `json_schema` output format
and hands the parsed object back on `ResultMessage.structured_output` — so
neither path is the degraded one, and moving between them is a config line
rather than a second implementation.
Both also *stream*, and hand the text out through `on_partial` as it arrives.
Nothing here does anything with it — this module still knows only "JSON in,
JSON out" — but a batch of twenty-five articles is one request that takes the
best part of a minute, and the answers inside it are finished one at a time
long before the request is. Passing the pieces up is what lets the layer that
knows what an answer looks like count them, and a progress bar move per
article instead of per batch.
What the `claude_code` backend deliberately does not get: no tools
(`tools=[]`), no filesystem settings (`setting_sources=None`, so no CLAUDE.md,
no skills, no MCP servers leak into a tagging call), no session to resume. One
text-in, JSON-out turn — the same shape as the API call it stands in for.
`thinking` is the one knob a caller turns, and it is stated to both backends
rather than left to either. It has to be: the CLI is built for agent work and
thinks unprompted, the Messages API thinks only when asked, so "say nothing"
means opposite things on the two sides. Saying `disabled` out loud is what
makes them the interchangeable pair this module claims — and it is worth
saying, because on a batch of twenty-five headlines the CLI's default cost
128s against 17s for the same tags. What to think about is the task's business
(`ModelTaskConfig.thinking`); making both backends agree on it is this one's.
"""
from __future__ import annotations
import importlib.util
import json
import logging
import os
from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
from typing import Any
from desk.config import AgentConfig
from desk.tools import credentials
log = logging.getLogger(__name__)
#: The backends `backend()` can name, best first. `auto` walks this list.
BACKENDS = ("api", "claude_code")
#: Where the answer goes as it arrives, a piece at a time. Called with raw
#: text, not parsed anything — half a JSON object is not a JSON object, and
#: what to make of a partial answer is the caller's business, not this
#: module's.
Partial = Callable[[str], None]
class AgentError(Exception):
"""The backend was there and the call still didn't produce an answer.
Raised for every failure mode of both paths — no SDK installed, no CLI on
PATH, an API error, a timeout, a response that isn't the shape asked for.
Callers treat it as "no answer this time", which for a job means either
moving on to the next batch or handing off to a human-driven agent.
"""
@dataclass(slots=True)
class Reply:
"""One answer, and enough about where it came from to say so in the UI."""
data: dict[str, Any]
backend: str = ""
model: str = ""
#: Filled by the `claude_code` path only, which is the reverse of what you
#: would expect: the SDK's `ResultMessage` prices the turn, so a
#: subscription that is billed nothing for it still reports a figure, while
#: the API path — the one actually billed — leaves this at zero because the
#: Messages response does not price itself. An estimate of what the run
#: would have cost on the API, then, and never a bill. `usage` is the field
#: to trust if you want to price it yourself.
cost_usd: float = 0.0
usage: dict[str, Any] = field(default_factory=dict)
# ------------------------------------------------------------ which backend
def has_api_credentials() -> bool:
"""Whether the API path can run unattended.
An unset ANTHROPIC_API_KEY isn't proof there are no credentials — the SDK
also reads ANTHROPIC_AUTH_TOKEN and an `ant auth login` profile — but env
vars are the only part we can check without importing the SDK.
"""
return any(os.environ.get(name) for name in (
"ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_PROFILE",
))
def has_claude_code() -> bool:
"""Whether the Agent SDK is here, and so whether there is a CLI to drive.
The SDK bundles its own `claude`, so having the package is enough — a
binary on PATH is a bonus, not a requirement, and checking for one would
wrongly rule out a machine where the SDK is the only install.
Not proof the call will work — the CLI may be signed out — but it is the
check that can be made in microseconds, which is what a job asking "can I
do this myself?" needs. A signed-out CLI surfaces as an `AgentError` on
the call, and the caller falls back exactly as if this had returned False.
"""
return importlib.util.find_spec("claude_agent_sdk") is not None
def backend(settings: AgentConfig) -> str:
"""Which backend will answer, or `""` for nobody.
`agent.backend: auto` prefers the API when there are credentials, because
that is the unattended path and the one production will use; it falls back
to the local Claude Code subscription, which is the one that costs nothing
while you are building. Naming a backend skips the detection — including
`off`, which is how you make everything that wants a model fail.
"""
want = settings.backend
if want == "off":
return ""
if want in BACKENDS:
return want
if has_api_credentials():
return "api"
if has_claude_code():
return "claude_code"
return ""
# ------------------------------------------------------------------ the ask
def _thinking(setting: str) -> Any:
"""One `thinking` setting in the form both SDKs take.
Returns `Any` on purpose. Each SDK declares its own TypedDict for the same
two-word object — `ThinkingConfigDisabledParam` on one side,
`ThinkingConfigDisabled` on the other — and neither can be named here
without importing an SDK this module is careful to keep optional. One
honest `Any` at the seam beats a hard dependency at the top of the file.
Anything that is not a mode we offer is off, rather than an error. This is
reached from config pydantic has already constrained to the same two words,
so a third one arriving here would be a bug in our own plumbing — and the
honest answer to that is the same one the default gives, not a failed run
the model would happily have answered.
"""
return {"type": "adaptive"} if setting == "adaptive" else {"type": "disabled"}
def ask_json(
settings: AgentConfig,
*,
system: str,
question: dict[str, Any],
schema: dict[str, Any],
model: str = "",
using: str = "",
thinking: str = "off",
on_partial: Partial | None = None,
) -> Reply:
"""Ask Claude `question`, constrained to `schema`, and parse the answer.
`system` is expected to be byte-identical across the calls in a run — it is
what the API path caches. `question` is the part that changes. `using`
forces a backend for one call; otherwise `backend()` decides. `thinking` is
`off` or `adaptive`, and defaults to off because the caller who wants it is
rarer than the caller who would be surprised to be waiting for it.
`on_partial` is called with each piece of the answer as it arrives, in
order, on the thread this was called on. It is a courtesy, not a contract:
a backend that cannot stream simply never calls it, and the return value
is the same either way.
"""
name = using or backend(settings)
model = model or settings.model
if thinking == "adaptive" and model.startswith(
("claude-haiku-4-5", "claude-sonnet-4-5", "claude-opus-4-5")):
# These models support manual extended thinking only; fail before spend.
# https://platform.claude.com/docs/en/build-with-claude/extended-thinking
raise AgentError(f"{model} does not support adaptive thinking; set thinking: off "
"or choose a model supporting adaptive thinking")
if not name:
raise AgentError("no model backend is available")
if name == "api":
return _ask_api(settings, system, question, schema, model, thinking, on_partial)
if name == "claude_code":
return _ask_claude_code(settings, system, question, schema, model, thinking,
on_partial)
raise AgentError(f"unknown backend {name!r}")
def _ask_api(settings: AgentConfig, system: str, question: dict[str, Any],
schema: dict[str, Any], model: str, thinking: str = "off",
on_partial: Partial | None = None) -> Reply:
try:
import anthropic
except ImportError as exc:
raise AgentError(
"the anthropic SDK isn't installed — `uv pip install anthropic`"
) from exc
try:
client = anthropic.Anthropic(timeout=settings.timeout)
# Streamed rather than awaited whole, always — not only when someone
# passed `on_partial`. One code path is worth more than the few lines a
# non-streaming branch would save, and a minute-long request held open
# by a single response is the shape that trips idle timeouts in
# whatever sits between here and the API.
with client.messages.stream(
model=model,
max_tokens=settings.max_tokens,
system=[{
"type": "text",
"text": system,
# The instructions are byte-identical across every batch and
# every run; only the payload below changes.
"cache_control": {"type": "ephemeral"},
}],
messages=[{"role": "user", "content": _dumps(question)}],
thinking=_thinking(thinking),
output_config={"format": {"type": "json_schema", "schema": schema}},
) as stream:
for event in stream:
if on_partial is not None and (piece := _delta(event)):
on_partial(piece)
response = stream.get_final_message()
except (anthropic.APIError, TypeError) as exc:
# The status code carried through in the text on purpose: it is what
# `notices.classify` reads to tell "your key is wrong" (401) apart from
# "you are being rate limited" (429), which want different reactions
# from you and which the message body alone does not always separate.
status = getattr(exc, "status_code", "")
raise AgentError(
f"the API call failed: {f'{status} ' if status else ''}{exc}") from exc
text = next((b.text for b in response.content if b.type == "text"), "")
return Reply(
data=_parse(text),
backend="api",
model=response.model or model,
usage=response.usage.model_dump() if response.usage else {},
)
def _ask_claude_code(settings: AgentConfig, system: str, question: dict[str, Any],
schema: dict[str, Any], model: str, thinking: str = "off",
on_partial: Partial | None = None) -> Reply:
try:
from claude_agent_sdk import (
ClaudeAgentOptions, ClaudeSDKError, ResultMessage, StreamEvent, query,
)
except ImportError as exc:
raise AgentError(
"the Claude Agent SDK isn't installed — "
"`uv pip install claude-agent-sdk`"
) from exc
options = ClaudeAgentOptions(
system_prompt=system,
model=model or None,
# A classification call, not an agent: nothing to run, nothing to
# read, and no project settings — so no CLAUDE.md, no skills and no
# MCP servers get in front of the model on the way to a tag list.
tools=[],
allowed_tools=[],
permission_mode="dontAsk",
setting_sources=None,
# Always stated, never left to the CLI's default, which is on. With
# it on the progress bar also stops meaning anything: thinking emits
# nothing `_Answered` can count, so a chunk streams no `"uid"` until
# the thinking is done, and the bar sits on the previous chunk's
# number until the answer arrives in a rush.
thinking=_thinking(thinking),
output_format={"type": "json_schema", "schema": schema},
# Only when someone is listening. The partials are the same bytes as
# the final answer sent again in pieces, and asking the CLI for them
# when nothing reads them is noise on the pipe for no one's benefit.
include_partial_messages=on_partial is not None,
cli_path=settings.cli_path or None,
# `max_tokens` has to mean the same thing on both backends or it means
# nothing. The Messages API takes it as a parameter and the CLI takes
# it only as an environment variable, so without this the setting was
# quietly API-only — which is how a call configured for 8,192 tokens
# answered with 11,706 and nothing objected. The SDK merges this into
# the inherited environment rather than replacing it, so naming one
# variable does not cost the CLI its PATH.
env={"CLAUDE_CODE_MAX_OUTPUT_TOKENS": str(settings.max_tokens),
**({"CLAUDE_CODE_OAUTH_TOKEN": oauth} if (oauth := credentials.token(settings)) else {})},
stderr=lambda line: log.debug("claude: %s", line.rstrip()),
)
async def ask() -> Reply:
answer: Reply | None = None
async for message in query(prompt=_dumps(question), options=options):
if isinstance(message, StreamEvent) and on_partial is not None:
piece = _delta(message.event)
if piece:
on_partial(piece)
continue
if not isinstance(message, ResultMessage):
continue
if message.is_error:
raise AgentError(_explain(
"; ".join(message.errors or []) or message.result
or f"the agent ended as {message.subtype}"))
data = message.structured_output
if not isinstance(data, dict):
# A run that ended without structured output — a limit, an
# interruption. `result` is the raw text if there is any, and
# that is worth one attempt at parsing before giving up.
data = _parse(message.result or "")
answer = Reply(
data=data,
backend="claude_code",
model=model,
cost_usd=message.total_cost_usd or 0.0,
usage=message.usage or {},
)
if answer is None:
raise AgentError("the agent returned no result")
return answer
try:
return _sync(ask(), settings.timeout)
except ClaudeSDKError as exc:
# The SDK has its own exception hierarchy — a CLI that exited, a line
# it could not parse, an answer that overran `max_tokens` — and none of
# it is an `AgentError`. Left alone those escape the per-chunk `except
# AgentError` in `with_model`, so one bad chunk takes down a run that
# is contractually supposed to leave its rows for next time. Every
# failure of this backend is an `AgentError` or none of them are.
#
# The text is passed through intact rather than summarised: `notices`
# matches on prose, by its own deliberate design, so the wording *is*
# the thing that decides which sentence you get told.
raise AgentError(_explain(str(exc))) from exc
# --------------------------------------------------------------- plumbing
def _field(event: Any, name: str) -> Any:
"""One field of a stream event, whichever shape it arrived in.
The two backends carry the same events in different clothes: the API SDK
parses them into objects, and the CLI passes the wire format straight
through as dicts. Reading both through here is what lets `_delta` be
written once.
"""
return event.get(name) if isinstance(event, dict) else getattr(event, name, None)
def _delta(event: Any) -> str:
"""The answer text one stream event carries, or "" for the many with none.
The events worth anything here are `content_block_delta`, whose delta is
`text` for a plain answer and `partial_json` when the schema is enforced
through a tool. Everything else — message start and stop, block
boundaries, usage — is dropped.
Deliberately reads defensively rather than by type. An event shape nobody
here recognises must be nothing, not an exception: this drives a progress
bar, and it does not get to fail a run the model is answering correctly.
"""
if _field(event, "type") != "content_block_delta":
return ""
delta = _field(event, "delta")
if delta is None:
return ""
piece = _field(delta, "text") or _field(delta, "partial_json") or ""
return piece if isinstance(piece, str) else ""
def _sync(coro: Any, timeout: float) -> Reply:
"""Run one coroutine to completion from synchronous code.
Jobs are synchronous by design and the worker runs them on a thread, so
there is normally no loop here and `asyncio.run` is exactly right. The
thread pool is for the other caller — anything asking from inside a
running loop, where `asyncio.run` would raise.
"""
import asyncio
async def bounded() -> Reply:
try:
return await asyncio.wait_for(coro, timeout)
except TimeoutError as exc:
raise AgentError(f"no answer within {timeout:g}s") from exc
try:
asyncio.get_running_loop()
except RuntimeError:
return asyncio.run(bounded())
with ThreadPoolExecutor(max_workers=1) as pool:
return pool.submit(asyncio.run, bounded()).result()
def _explain(error: str) -> str:
"""Keep the raw failure text intact for the layer that classifies it.
This used to append advice ("the `claude` login looks expired, run…"),
which was the right instinct in the wrong place: the message was built here
and then swallowed three frames up by a `except AgentError: continue`, so
the advice never reached anyone. Recognising the failure and deciding what
to say about it now lives in `desk/notices.py`, which has somewhere to
put the answer.
What stays here is the part only this module knows: the CLI can end a run
with an empty error and a subtype, and "" is not something to classify.
"""
return error.strip() or "the agent ended without saying why"
def _dumps(payload: dict[str, Any]) -> str:
return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
def _parse(text: str) -> dict[str, Any]:
"""The answer as a dict. Tolerant of a fenced or chatty response.
Both backends are asked for JSON through a schema, so this should be a
plain `json.loads`. It isn't, because "should be" is not a guarantee worth
losing a batch over — a model that wrapped its answer in a code fence has
still done the work.
"""
body = text.strip()
if body.startswith("```"):
body = body.split("\n", 1)[-1].rsplit("```", 1)[0]
start, end = body.find("{"), body.rfind("}")
if start == -1 or end <= start:
raise AgentError(f"no JSON object in the response: {text[:200]!r}")
try:
data = json.loads(body[start:end + 1])
except ValueError as exc:
raise AgentError(f"the response wasn't valid JSON: {exc}") from exc
if not isinstance(data, dict):
raise AgentError("the response wasn't a JSON object")
return data
+212
View File
@@ -0,0 +1,212 @@
"""Structured article extraction, independent of fetching and persistence.
The tree is data, never HTML. Code is kept verbatim; plain text is a derived
view of the same tree. Custom extractors can return ArticleContent; callers
own persistence and renderers consume the closed vocabulary below.
"""
from __future__ import annotations
import hashlib
from datetime import UTC, datetime
from html.parser import HTMLParser
from typing import Literal, Self
from urllib.parse import urljoin, urlsplit
from xml.etree import ElementTree as ET
from pydantic import Field, model_validator
from desk.models import Model
class ContentNode(Model):
kind: Literal[
"document", "text", "paragraph", "heading", "link", "code", "list",
"item", "table", "row", "cell", "strong", "emphasis", "quote", "break",
"span", "inline_code",
]
text: str = ""
children: list[ContentNode] = Field(default_factory=list)
level: int = Field(default=1, ge=1, le=6)
ordered: bool = False
header: bool = False
url: str = ""
@model_validator(mode="after")
def consistent_node(self) -> Self:
if self.kind in {"text", "code", "inline_code", "break"}:
if self.children:
raise ValueError("leaf nodes cannot have children")
elif self.text:
raise ValueError("container text belongs in text children")
if self.url and (self.kind != "link" or not _link(self.url, "")):
raise ValueError("only links can carry absolute HTTP(S) URLs")
return self
def plain_text(self) -> str:
if self.kind == "break":
return "\n"
if self.kind in {"code", "inline_code", "text"}:
return self.text
separator = {"document": "\n\n", "list": "\n", "table": "\n", "row": "\t"}.get(self.kind, "")
return separator.join(child.plain_text() for child in self.children)
class ArticleContent(Model):
format: Literal["plain", "structured-v1"]
content: str | ContentNode
source_url: str
method: str
acquired_at: str = Field(default_factory=lambda: datetime.now(UTC).isoformat())
# Finding a readable article does not prove that it was not truncated by
# the publisher. The caller must supply evidence before declaring it full.
completeness: Literal["unknown", "complete", "partial"] = "unknown"
removed_gutters: int = Field(default=0, ge=0)
@model_validator(mode="after")
def consistent_format(self) -> Self:
if self.format == "plain":
if not isinstance(self.content, str):
raise ValueError("plain content must be a string")
elif not isinstance(self.content, ContentNode) or self.content.kind != "document":
raise ValueError("structured-v1 content must be a document node")
return self
@property
def plain_text(self) -> str:
return self.content if isinstance(self.content, str) else self.content.plain_text()
@property
def revision(self) -> str:
value = self.content if isinstance(self.content, str) else self.content.model_dump_json()
return hashlib.sha256(f"{self.format}\n{value}".encode()).hexdigest()
class _Gutters(HTMLParser):
"""Remove only explicit code-gutter nodes, retaining their following text.
No numeric-line heuristics. Unknown publisher markup is preserved. The
void-element handling matters: a br inside a gutter must not eat the rest
of the article by leaving the suppression depth unbalanced.
"""
VOID = frozenset({"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"})
MARKERS = frozenset({"line-number", "line-numbers-rows", "lineno", "linenos"})
def __init__(self) -> None:
super().__init__(convert_charrefs=False)
self.parts: list[str] = []
self.stack: list[tuple[str, bool, bool]] = []
self.removed = 0
def handle_starttag(self, tag, attrs):
classes = set((dict(attrs).get("class") or "").split())
context = tag in {"pre", "code"} or bool(classes & {"highlight", "highlighttable"}) or any(c for _, c, _ in self.stack)
suppressed = any(s for _, _, s in self.stack)
gutter = context and bool(classes & self.MARKERS)
if gutter and not suppressed:
self.removed += 1
if not (gutter or suppressed):
self.parts.append(self.get_starttag_text() or "")
if tag not in self.VOID:
self.stack.append((tag, context, gutter or suppressed))
def handle_startendtag(self, tag, attrs):
self.handle_starttag(tag, attrs)
if tag not in self.VOID:
self.handle_endtag(tag)
def handle_endtag(self, tag):
suppressed = any(s for _, _, s in self.stack)
for index in range(len(self.stack) - 1, -1, -1):
if self.stack[index][0] == tag:
del self.stack[index:]
break
if not suppressed:
self.parts.append(f"</{tag}>")
def handle_data(self, data):
if not any(s for _, _, s in self.stack):
self.parts.append(data)
def handle_entityref(self, name):
self.handle_data(f"&{name};")
def handle_charref(self, name):
self.handle_data(f"&#{name};")
def _link(value: str, source_url: str) -> str:
try:
resolved = urljoin(source_url, value)
return resolved if urlsplit(resolved).scheme in {"http", "https"} else ""
except ValueError:
return ""
def _node(element: ET.Element, source_url: str) -> ContentNode:
children: list[ContentNode] = []
if element.text:
children.append(ContentNode(kind="text", text=element.text))
for child in element:
children.append(_node(child, source_url))
if child.tail:
children.append(ContentNode(kind="text", text=child.tail))
match element.tag:
case "code":
return ContentNode(kind="code", text="".join(element.itertext()))
case "head":
level = element.get("rend", "h1").removeprefix("h")
return ContentNode(kind="heading", level=int(level) if level in "123456" and len(level) == 1 else 1, children=children)
case "ref":
return ContentNode(kind="link", url=_link(element.get("target", ""), source_url), children=children)
case "list":
return ContentNode(kind="list", ordered=element.get("rend") == "ol", children=children)
case "cell":
return ContentNode(kind="cell", header=element.get("role") == "head", children=children)
case "hi":
if element.get("rend") == "#t":
return ContentNode(kind="inline_code", text="".join(element.itertext()))
kind = {"#b": "strong", "#i": "emphasis"}.get(element.get("rend", ""), "span")
return ContentNode.model_validate({"kind": kind, "children": children})
case "lb":
return ContentNode(kind="break")
case _:
kinds = {"main": "document", "p": "paragraph", "item": "item", "table": "table", "row": "row", "quote": "quote"}
return ContentNode.model_validate({"kind": kinds.get(element.tag, "paragraph"), "children": children})
def extract_article(html: str, source_url: str) -> ArticleContent | None:
"""Extract supplied HTML without I/O; return None when no article is found.
XML is an intermediate representation only. Its tags and attributes never
escape into the content contract, and no raw HTML is passed to a renderer.
"""
import trafilatura
cleaned = _Gutters()
cleaned.feed(html)
cleaned.close()
extracted = trafilatura.extract(
"".join(cleaned.parts), url=source_url, output_format="xml",
include_comments=False, include_tables=True, include_links=True,
include_formatting=True,
)
if not extracted:
return None
main = ET.fromstring(extracted).find("main")
if main is None:
return None
# XML pretty-printing inserts whitespace between blocks. It is not source
# content; inside code, however, every byte of whitespace belongs to it.
for element in main.iter():
if element.tag in {"main", "table", "row", "list"}:
if not (element.text or "").strip():
element.text = None
for child in element:
if not (child.tail or "").strip():
child.tail = None
return ArticleContent(
format="structured-v1", content=_node(main, source_url), source_url=source_url,
method="trafilatura-xml", removed_gutters=cleaned.removed,
)
+254
View File
@@ -0,0 +1,254 @@
"""Render a page in headless Chromium, accepting a tracking-consent gate on the way.
The getter a check sends a page to when plain HTTP brought back a placeholder:
an app shell whose article is drawn by a script, or a consent gate standing in
front of every article on a host. Golem is the second kind. Every article URL
redirects to `/sonstiges/zustimmung/auswahl.html`, where Sourcepoint's CMP
draws its dialog in an iframe; accepting it sets the cookies and redirects
back to the article.
from desk.tools import browser
got = browser.render("https://www.golem.de/news/...")
got.status, got.html, got.url, got.consent, got.cookies
browser.remember(cache, got.url, got.cookies)
browser.cookie_header(cache, url) # "golem_consent20=cmp|250101; ..."
**Consent is accepted once and then reused.** Chromium costs a few seconds
and a few hundred megabytes per page. The cookies it earns open the gate for
a plain request too, so the caller stores them per host with `remember` and
sends them with `cookie_header` on the next HTTP fetch. The browser runs again
only when HTTP meets the gate again, which is when the cookies expired or the
outlet changed what it accepts. Golem's last three months.
**A CMP is one entry in `CONSENTS`**: the selector of the iframe it draws in
and a pattern for the accept button's name. A page with no such frame is
rendered without clicking anything, after waiting `gate_wait` seconds for one
to appear. Only accept buttons are listed. A bot check that wants a human is
not something this clicks; it stays `js_required` or `denied`.
Needs the `browser` extra and a Chromium build: `playwright install chromium`.
Imported only inside `render`, so the rest of `desk.tools` does not pay for it.
"""
from __future__ import annotations
import contextlib
import importlib.util
import json
import re
import threading
import time
from collections.abc import Mapping, MutableMapping, Sequence
from dataclasses import dataclass, field
from typing import Any
from urllib.parse import urlsplit
from desk.tools.framing import host_of
#: Sits alongside `embed:` and `body:` in the same table.
CACHE_PREFIX = "cookies:"
#: Seconds for navigation and for the click. A gate that has not redirected
#: back by then is not going to.
TIMEOUT = 30.0
#: Seconds to wait for a consent frame on a page that may not have one. The
#: CMP injects its iframe from a script; on golem.de it is there within a
#: second.
GATE_WAIT = 5.0
#: Seconds to let a script-drawn page settle after `domcontentloaded`. Not
#: `networkidle` as a condition: an ad-carrying page never goes idle, so this is
#: a ceiling.
SETTLE = 3.0
#: One Chromium at a time in this process. Body batches run side by side, and
#: the first articles of a gated host would otherwise each start a browser
#: before the first one has earned the cookies.
_ONE_AT_A_TIME = threading.Lock()
def installed() -> bool:
"""Whether the `browser` extra is. Chromium itself is checked by launching."""
return importlib.util.find_spec("playwright") is not None
@dataclass(frozen=True)
class Consent:
"""How to accept one CMP's dialog."""
name: str
#: CSS selector of the iframe the dialog is drawn in.
frame: str
#: Case-insensitive regex, matched against the accept button's
#: accessible name. Anchored, so "Accept" does not match "Accept only
#: necessary" and a reject button is never the one clicked.
accept: str
#: A page this was found on.
seen: str = ""
def __post_init__(self) -> None:
re.compile(self.accept)
CONSENTS: tuple[Consent, ...] = (
Consent("sourcepoint", "iframe[id^='sp_message_iframe']",
r"^(Zustimmen und weiter|Alle akzeptieren|Akzeptieren|Zustimmen"
r"|Accept all|Accept|I agree|Agree)$",
seen="https://www.golem.de/sonstiges/zustimmung/auswahl.html"),
)
@dataclass(frozen=True)
class Cookie:
name: str
value: str
domain: str
path: str = "/"
#: Unix seconds; -1 for a session cookie, which is kept until the gate
#: comes back rather than until a browser that no longer exists closes.
expires: float = -1
@dataclass(frozen=True)
class Rendered:
"""What Chromium ended up showing. `url` is where it ended up."""
status: int
html: str
url: str
headers: Mapping[str, str] = field(default_factory=dict)
#: Every cookie the host of `url` would be sent.
cookies: tuple[Cookie, ...] = ()
#: The `Consent.name` that was accepted, or "" when no gate was met.
consent: str = ""
def render(url: str, *, consents: Sequence[Consent] = CONSENTS,
timeout: float = TIMEOUT, gate_wait: float = GATE_WAIT,
settle: float = SETTLE) -> Rendered:
"""Load `url`, accept a consent dialog if one appears, return the page.
Raises what Playwright raises: its `TimeoutError` names itself a timeout
to `failures.of_exception`, and a missing Chromium build is `unknown`.
One browser per call. A pull renders a page only where HTTP failed, so a
browser kept alive between calls would sit idle in a worker.
"""
from playwright.sync_api import TimeoutError as PlaywrightTimeout
from playwright.sync_api import sync_playwright
ms = int(timeout * 1000)
with _ONE_AT_A_TIME, sync_playwright() as playwright:
# /dev/shm is 64 MB in a container, which Chromium outgrows and
# crashes on; this has it use /tmp instead.
chromium = playwright.chromium.launch(
headless=True, args=["--disable-dev-shm-usage"])
try:
context = chromium.new_context()
page = context.new_page()
# The last document response of the main frame, so the status is
# the article's after a gate's redirect rather than the gate's.
documents: list[Any] = []
page.on("response", lambda response: documents.append(response)
if response.request.is_navigation_request()
and response.frame == page.main_frame else None)
# `domcontentloaded`, not `load`: an ad-carrying page such as a
# golem.de article may not fire `load` within any useful timeout.
page.goto(url, wait_until="domcontentloaded", timeout=ms)
accepted = _accept(page, consents, ms, gate_wait, PlaywrightTimeout)
with contextlib.suppress(PlaywrightTimeout):
page.wait_for_load_state("networkidle", timeout=int(settle * 1000))
last = documents[-1] if documents else None
final = page.url
return Rendered(
status=last.status if last else 0,
html=page.content(),
url=final,
headers=dict(last.headers) if last else {},
cookies=_for_host(_hostname(final), context.cookies()),
consent=accepted)
finally:
chromium.close()
def _accept(page: Any, consents: Sequence[Consent], ms: int, gate_wait: float,
timeout_error: type[Exception]) -> str:
"""Click the first consent dialog's accept button. The CMP's name, or ""."""
deadline = time.monotonic() + gate_wait
for consent in consents:
left = max(0, int((deadline - time.monotonic()) * 1000))
frame = page.locator(consent.frame).first
try:
frame.wait_for(state="attached", timeout=max(left, 1))
except timeout_error:
continue
button = page.frame_locator(consent.frame).first.get_by_role(
"button", name=re.compile(consent.accept, re.IGNORECASE))
button.first.click(timeout=ms)
# Gone either way it is dismissed: an overlay removes its frame, a
# gate page is navigated away from.
frame.wait_for(state="detached", timeout=ms)
page.wait_for_load_state("domcontentloaded", timeout=ms)
return consent.name
return ""
def _for_host(host: str, cookies: Sequence[Mapping[str, Any]]) -> tuple[Cookie, ...]:
return tuple(
Cookie(str(c["name"]), str(c["value"]), str(c["domain"]),
str(c.get("path", "/")), float(c.get("expires", -1)))
for c in cookies if _matches(host, str(c.get("domain", ""))))
def _hostname(url: str) -> str:
"""The host without a port, which is what a cookie's domain is matched to."""
try:
return (urlsplit(url).hostname or "").lower()
except ValueError:
return ""
def _matches(host: str, domain: str) -> bool:
domain = domain.lstrip(".").lower()
return bool(host) and (host == domain or host.endswith("." + domain))
# ------------------------------------------------------------- the cookie jar
def remember(cache: MutableMapping[str, str], url: str,
cookies: Sequence[Cookie]) -> None:
"""Keep the cookies a render earned, for the next plain fetch from this host.
Replaces whatever the host had: a render that met the gate again earned
the current set, and the old one is what stopped working.
"""
kept = [c for c in cookies if _matches(_hostname(url), c.domain)]
if kept:
cache[CACHE_PREFIX + host_of(url)] = json.dumps(
[[c.name, c.value, c.domain, c.path, c.expires] for c in kept])
def cookie_header(cache: Mapping[str, str], url: str, *,
now: float | None = None) -> str:
"""The `Cookie` header for a plain request to `url`, or "" for none.
Expired cookies are left out rather than deleted. A host whose cookies
have all expired sends none, meets the gate, and is rendered again.
"""
stored = cache.get(CACHE_PREFIX + host_of(url), "")
if not stored:
return ""
now = time.time() if now is None else now
try:
rows = json.loads(stored)
except ValueError:
return ""
return "; ".join(f"{name}={value}" for name, value, _domain, _path, expires in rows
if expires < 0 or expires > now)
def forget(cache: MutableMapping[str, str], url: str) -> None:
"""Drop a host's cookies, so its next fetch goes without them."""
cache.pop(CACHE_PREFIX + host_of(url), None)
+227
View File
@@ -0,0 +1,227 @@
"""Did the getter bring back a page worth converting, and if not, what next?
`scrape` fetches, this judges, `article` converts. A 200 is not an article
because it answered 200: an app shell serves a `<noscript>` notice asking for
JavaScript, a bot check serves a few words and a script. Converting either
stores the notice as the article.
A check answers with a `Verdict`: the failure category from
`desk.tools.failures`, the name of what matched, and which getters are worth
trying next. That last part is where what we learn goes. A Mastodon post is
a shell over HTTP and a status over Mastodon's API, so its signature names
`mastodon_api`; a bot check names `browser`. A getter the app does not have is
skipped, so a signature can name one before it is built.
Checks run in order and the first match decides:
1. the status code
2. no HTML at all
3. `SIGNATURES`, the table below, most specific first
4. the structural rule, for a shell nobody has written a signature for yet
**Adding a signature** is one entry in `SIGNATURES` and one test page in
`tests/test_checks.py`. The entry names the page it was found on, so whoever
reads it later can fetch that page and see whether it still says this.
Signature(
name="mastodon_shell", # logged, and on the attempt
where="noscript", # title | text | noscript | html | url
pattern=r"To use the Mastodon web application", # regex, case-insensitive
category="js_required", # from desk.tools.failures
next=("mastodon_api", "browser"), # getters to try, in order
seen="https://grapheneos.social/@GrapheneOS/117225539756835649",
)
`where` is what the pattern is matched against: `title` the `<title>`, `text`
the words a reader without JavaScript sees outside `<noscript>`, `noscript`
the words inside it, `html` the raw markup (script sources, class names),
`url` the address the getter ended up at after redirects. Prefer the narrowest
one that works: a phrase in `text` can appear in an article about the thing
it describes, a class name in `html` usually cannot.
A pattern that does not compile, a category the failure vocabulary does not
have, or two signatures with one name is refused at import.
"""
from __future__ import annotations
import re
from collections.abc import Mapping
from dataclasses import dataclass, field
from html.parser import HTMLParser
from typing import Literal
from desk.tools import failures
Where = Literal["title", "text", "noscript", "html", "url"]
@dataclass(frozen=True)
class Page:
"""What one getter brought back. `url` is where it ended up."""
status: int
html: str
url: str = ""
headers: Mapping[str, str] = field(default_factory=dict)
@dataclass(frozen=True)
class Verdict:
"""Why this page is not worth converting, and what might get the real one."""
category: str
#: The signature or rule that matched. What a person reads in the log to
#: decide whether the table needs a new entry or a narrower one.
name: str
next: tuple[str, ...] = ()
@dataclass(frozen=True)
class Signature:
name: str
where: Where
pattern: str
category: str
next: tuple[str, ...] = ()
#: A page this was found on.
seen: str = ""
def __post_init__(self) -> None:
re.compile(self.pattern)
if self.category not in failures.TRANSIENT | failures.SETTLED:
raise ValueError(f"signature {self.name!r} names category "
f"{self.category!r}, which desk.tools.failures has not")
#: Most specific first. See the module docstring for the format.
SIGNATURES: tuple[Signature, ...] = (
Signature("mastodon_shell", "noscript", r"To use the Mastodon web application",
"js_required", ("mastodon_api", "browser"),
seen="https://grapheneos.social/@GrapheneOS/117225539756835649"),
Signature("bluesky_shell", "noscript", r"This is a heavily interactive web application, and JavaScript is required",
"js_required", ("bluesky_api", "browser"),
seen="https://bsky.app/profile/bsky.app/post/3mv3shqdfuc2e"),
Signature("create_react_app_shell", "noscript", r"You need to enable JavaScript to run this app",
"js_required", ("browser",),
seen="https://rune.build/blog/rune-is-now-open-source"),
Signature("cloudflare_turnstile", "html", r"challenges\.cloudflare\.com/turnstile",
"js_required", ("browser",),
seen="https://openreview.net/forum?id=pc7fqaOcAH"),
Signature("anubis_check", "text", r"automated check to get rid of most bots|Making sure you're not a bot",
"js_required", ("browser",),
seen="https://codeberg.org/forgejo/forgejo/src/branch/forgejo/release-notes-published/16.0.4.md"),
# Every article redirects here without the consent cookie. The browser
# accepts Sourcepoint's dialog, and the cookies it earns open the gate for
# HTTP until they expire. See `desk.tools.browser`.
Signature("golem_consent_gate", "html", r"GolemConsent\.init",
"consent_wall", ("browser",),
seen="https://www.golem.de/sonstiges/zustimmung/auswahl.html"),
)
#: Below this many words outside `<noscript>`, a page carrying a script is a
#: placeholder. OpenReview's check is 36 words, Anubis 42; a real article is
#: several hundred.
SCRIPT_PAGE_WORDS = 80
#: The longest `<noscript>` read as a notice. Mastodon's is 21 words,
#: Bluesky's 76. A longer one is a site serving its content without scripts.
NOTICE_WORDS = 150
#: Where the structural rule sends a page nobody has a signature for.
UNRECOGNISED_SHELL_NEXT = ("browser",)
def _unique_names(signatures: tuple[Signature, ...]) -> None:
names = [s.name for s in signatures]
if len(names) != len(set(names)):
raise ValueError(f"two signatures share a name: "
f"{sorted({n for n in names if names.count(n) > 1})}")
_unique_names(SIGNATURES)
def check(page: Page | None, signatures: tuple[Signature, ...] = SIGNATURES) -> Verdict | None:
"""None when the page is worth converting, else why not."""
if page is None:
# The downloader logs which way it failed and returns nothing: DNS, a
# refused connection, a reset and TLS all arrive here as one.
return Verdict("unreachable", "no_response")
if why := failures.of_status(page.status):
return Verdict(why, f"status_{page.status}")
if not page.html.strip():
return Verdict("empty", "no_html")
parts = _Parts.of(page)
for signature in signatures:
if re.search(signature.pattern, parts[signature.where], re.IGNORECASE):
return Verdict(signature.category, signature.name, signature.next)
if (parts.scripts and len(_words(parts["text"])) < SCRIPT_PAGE_WORDS
and len(_words(parts["noscript"])) <= NOTICE_WORDS):
return Verdict("js_required", "script_page", UNRECOGNISED_SHELL_NEXT)
return None
def _flat(pieces: list[str]) -> str:
return " ".join(" ".join(pieces).split())
def _words(text: str) -> list[str]:
return re.findall(r"\w+", text)
class _Parts(dict[str, str]):
"""The five things a signature can match against, read in one pass."""
scripts: int
@classmethod
def of(cls, page: Page) -> _Parts:
reader = _Reader()
try:
reader.feed(page.html)
reader.close()
except Exception: # malformed markup: match what was read so far
pass
# Whitespace collapsed in the three text parts, so a signature is
# written with single spaces however the page wrapped its lines.
parts = cls(title=_flat(reader.title), text=_flat(reader.text),
noscript=_flat(reader.noscript), html=page.html, url=page.url)
parts.scripts = reader.scripts
return parts
class _Reader(HTMLParser):
HIDDEN = frozenset({"script", "style", "template", "svg"})
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self.title: list[str] = []
self.text: list[str] = []
self.noscript: list[str] = []
self.scripts = 0
self._hidden = 0
self._in: dict[str, int] = {"title": 0, "noscript": 0}
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
self.scripts += tag == "script"
if tag in self.HIDDEN:
self._hidden += 1
elif tag in self._in:
self._in[tag] += 1
def handle_endtag(self, tag: str) -> None:
if tag in self.HIDDEN:
self._hidden = max(0, self._hidden - 1)
elif tag in self._in:
self._in[tag] = max(0, self._in[tag] - 1)
def handle_data(self, data: str) -> None:
if self._hidden:
return
if self._in["title"]:
self.title.append(data)
elif self._in["noscript"]:
self.noscript.append(data)
else:
self.text.append(data)
+32
View File
@@ -0,0 +1,32 @@
"""Local credential settings. Read per call so running workers see replacements."""
import os
import sqlite3
from contextlib import closing
from desk.config import AgentConfig
def saved_token(settings: AgentConfig) -> str:
if not settings.credentials_database:
return ""
with closing(sqlite3.connect(settings.credentials_database)) as conn:
if not conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='credentials'"
).fetchone():
return ""
row = conn.execute("SELECT value FROM credentials WHERE name='claude_oauth'").fetchone()
return row[0] if row else ""
def token(settings: AgentConfig) -> str:
return saved_token(settings) or os.environ.get("CLAUDE_CODE_OAUTH_TOKEN", "")
def save_token(settings: AgentConfig, value: str) -> None:
with closing(sqlite3.connect(settings.credentials_database)) as conn, conn:
conn.execute("CREATE TABLE IF NOT EXISTS credentials (name TEXT PRIMARY KEY, value TEXT NOT NULL)")
if value:
conn.execute("INSERT OR REPLACE INTO credentials VALUES ('claude_oauth', ?)", (value,))
else:
conn.execute("DELETE FROM credentials WHERE name='claude_oauth'")
+115
View File
@@ -0,0 +1,115 @@
"""Turning an article into a vector.
A local sentence-transformer turns each article into ~384 floats. What it
buys, and why it is worth an optional 100MB download:
**It generalises across phrasings, which TF-IDF cannot.** The bag-of-words
model learns the token "semiconductor". It learns nothing at all from that
about an article on "chip fabrication", because the two share no words. The
embedding puts them next to each other, so a vote on one moves the other.
**It works from about five votes.** The taste vector is
`centroid(upvoted) - centroid(downvoted)` — a direction in the space, scored
by cosine. There is no training step, no fitting, and no threshold below which
it does nothing useful; it just gets sharper as votes arrive.
This module is only the encoder. Where the vectors go, and the arithmetic over
them, is `newsdesk/storage/vectors/`; which articles get encoded, and when, is
`newsdesk/jobs/ingest.py`.
`ranking.embedding_model` is recorded on every stored row, because a vector is
only comparable to another from the same model. Change the setting and the
old vectors are not wrong, they are *meaningless* — a different geometry
entirely — so they are dropped and re-computed rather than silently compared.
Not to be confused with `desk/tools/framing.py`, which is about whether
an outlet lets its pages be put in an iframe — that used to be called
`embed.py`, and the collision with `ranking.embedding_model` is why both were
renamed.
"""
from __future__ import annotations
import logging
from collections.abc import Sequence
from typing import Protocol
import numpy as np
log = logging.getLogger(__name__)
#: How much of the article body the encoder sees. These models truncate at a
#: few hundred tokens anyway, and the headline plus the opening paragraphs is
#: what actually says what a story is about.
BODY_CHARS = 1500
class Embedder(Protocol):
"""Anything that can turn strings into vectors.
A protocol rather than the concrete class, so the tests can hand the rest
of the system a deterministic encoder and never download a model. That is
not a testing hack — it is also what a future backend (an API embedder, a
different local library) would implement.
"""
model_id: str
def encode(self, texts: Sequence[str]) -> np.ndarray: ...
class SentenceTransformer:
"""The real one. Loads the model on first use, not on import.
A pull with `backend: tfidf` must not pay for a torch import, so
nothing here is imported at module level and this class is only constructed
by `load()`, which the embeddings path calls and the TF-IDF path does not.
"""
def __init__(self, model_id: str) -> None:
try:
from sentence_transformers import SentenceTransformer as ST
except ImportError as exc:
raise ImportError(
"ranking.backend is 'embeddings', which needs sentence-transformers.\n"
" uv sync --extra embeddings\n"
"or set ranking.backend back to 'tfidf' in config.yaml."
) from exc
self.model_id = model_id
log.info("loading embedding model %s (first run downloads it)", model_id)
self._model = ST(model_id)
def encode(self, texts: Sequence[str]) -> np.ndarray:
# Normalised on the way out, so cosine similarity is a dot product and
# every consumer can stop worrying about magnitudes.
return np.asarray(
self._model.encode(list(texts), normalize_embeddings=True,
show_progress_bar=False),
dtype=np.float32,
)
def load(model_id: str) -> Embedder:
return SentenceTransformer(model_id)
class Embeddable(Protocol):
"""What `document` reads: a title and a body, however the app stores them."""
title: str
excerpt: str
body: str
def document(row: Embeddable) -> str:
"""What gets embedded.
Title and body only — deliberately no `__source_` tokens. Those exist in
`rank._document` because TF-IDF has no other way to learn "I like this
outlet", and they would be noise here: the encoder would try to find
meaning in a made-up word. Outlet preference belongs in the linear model of
Stage 2, as its own one-hot feature, not smuggled into the text.
"""
body = row.body or row.excerpt
return f"{row.title}\n\n{' '.join((body or '').split())[:BODY_CHARS]}"
+185
View File
@@ -0,0 +1,185 @@
"""What went wrong, in one word a retry policy can be written against.
Acquisition fails in ways that mean opposite things. A host that took too long
will very likely answer in a minute; a page that is gone will be gone in a
minute, in an hour, and tomorrow. Until this existed the difference had nowhere
to live: `fetch_bodies` caught every exception in one `except`, wrote an empty
body, and the row said only that something had been tried — so the one policy
the archive could have was "never ask again", and it was the policy for the
timeout as much as for the 404.
A category is an answer to exactly one question: **would asking again get a
different answer?** `TRANSIENT` is yes, `SETTLED` is no, and `unknown` is "we
do not know", which is why nothing retries it unless a plugin names it.
from desk.tools import failures
failures.of_status(503) # 'server_error'
failures.of_exception(exc) # 'timeout'
failures.hint(response.headers) # 120.0, when the server named its own wait
Kept away from any HTTP library on purpose. `httpx` and `urllib3` are both
optional here and either may be the one that raised, so an exception is named
by walking the class names in its ancestry rather than by importing the
packages that define them. It costs the precision of an `isinstance` and buys a
tool that can name a failure raised by a library it has never imported — which
is what a plugin's own fetch will be.
The names are the vocabulary `RetryPolicy.on` is validated against, so adding
one here is adding one to the plugin contract: do it deliberately, and only
where the new name would lead somewhere different from an existing one.
"""
from __future__ import annotations
import logging
from collections.abc import Mapping
from email.utils import parsedate_to_datetime
from datetime import UTC, datetime
log = logging.getLogger(__name__)
#: Failures worth another attempt: the world was busy, not unwilling.
TRANSIENT = frozenset({
# The request did not finish in the time it was given.
"timeout",
# It never got an answer at all: DNS, a refused connection, a reset, TLS.
# One name rather than four because nothing downstream would do anything
# different about them, and because the downloader that reports most of
# them does not say which it was.
"unreachable",
# 429. The one failure that usually comes with the server's own answer to
# "when, then" — see `hint`.
"rate_limited",
# 5xx. The publisher's own machine is having a bad minute.
"server_error",
# Asked for before it existed. Not an HTTP failure at all: auto-captions
# on a two-hour upload take hours to appear, so a recording that has none
# yet is a wait rather than a refusal — and the same word covers anything
# else a source publishes late. See `segment_profile` in
# `desk/sources/podcast.py`, which is what declares a bound on that wait.
"not_ready",
})
#: Failures that will say the same thing next time. Retrying one of these is
#: how a source ends up fetched four times an evening for nothing.
SETTLED = frozenset({
# 404, 410. The address is not an article and will not become one.
"not_found",
# 401, 403, 451. Refused rather than broken — a login, a bot check, a
# jurisdiction. Another identical request is refused identically.
"denied",
# The page came back and it was the consent gate the host shows instead of
# the article. Deliberately here and never in a default retry list: it is
# the same answer however many times it is asked for. See `desk.tools.walls`.
"consent_wall",
# The page was an app shell or a bot challenge: a placeholder for what a
# script would have rendered. The same request gets the same placeholder,
# so it is settled for a retry; a different way of fetching, a browser, is
# what can answer it. See `desk.tools.checks`.
"js_required",
# Fetched, extracted, and there was no article in it.
"empty",
})
#: Neither, and on purpose. A failure nothing here recognises is not declared
#: transient on its behalf — a policy that wants it retried has to say so.
UNKNOWN = "unknown"
#: Everything a plugin may name in `RetryPolicy.on`.
CATEGORIES = TRANSIENT | SETTLED | frozenset({UNKNOWN})
#: Substrings of an exception class name, in the order they are tried, and the
#: category each means. Ordered because `ReadTimeoutError` is both a timeout
#: and a connection error by name, and it is a timeout.
_BY_NAME: tuple[tuple[str, str], ...] = (
("timeout", "timeout"),
("nameresolution", "unreachable"),
("ssl", "unreachable"),
("connect", "unreachable"),
("connection", "unreachable"),
("protocol", "unreachable"),
("transport", "unreachable"),
("network", "unreachable"),
)
#: The longest server hint worth waiting on. A `Retry-After` of a day is not a
#: wait, it is a refusal with a number on it, and honouring it would leave a
#: job row holding this item's place until tomorrow. Beyond this the job's own
#: backoff is used instead, which ends in a bounded number of attempts.
MAX_HINT = 6 * 60 * 60
def of_status(status: int) -> str:
"""The category of an HTTP status that is not a success.
A 2xx or 3xx is not a failure and has no category — it is `""`, so a
caller can pass any status and ask.
"""
if 200 <= status < 400:
return ""
if status == 429:
return "rate_limited"
if status in (401, 403, 451):
return "denied"
if status in (404, 410):
return "not_found"
if 500 <= status < 600:
return "server_error"
return UNKNOWN
def of_exception(exc: BaseException) -> str:
"""The category of a raised failure, named from its ancestry.
`TimeoutError` is the builtin every socket timeout already is; the rest are
matched on the class names in the exception's MRO, so `httpx.ConnectError`
and `urllib3.exceptions.NewConnectionError` both come out `unreachable`
without this module importing either package. A cause is followed, because
a wrapper around the real failure is the usual shape: `ProtocolError` with
a `TimeoutError` underneath it is a timeout.
"""
if isinstance(exc, TimeoutError):
return "timeout"
names = [cls.__name__.lower() for cls in type(exc).__mro__]
for fragment, category in _BY_NAME:
if any(fragment in name for name in names):
return category
cause = exc.__cause__ or exc.__context__
if cause is not None and cause is not exc:
return of_exception(cause)
return UNKNOWN
def hint(headers: Mapping[str, str] | None) -> float | None:
"""The wait a server named for itself, in seconds, or None for no answer.
`Retry-After` comes in two forms and both are in use: a number of seconds,
and an HTTP date. Honouring it is the point — guessing costs either a
minute of doing nothing when the service was ready again, or another
refusal charged against a bound that is ours rather than the server's.
Unreadable, negative, or longer than `MAX_HINT` is no answer: this returns
None and the job's own backoff decides. A header nobody can parse is not
worth failing over.
"""
if not headers:
return None
raw = next((v for k, v in headers.items()
if k.lower() == "retry-after"), "").strip()
if not raw:
return None
try:
seconds = float(raw)
except ValueError:
try:
when = parsedate_to_datetime(raw)
except (TypeError, ValueError):
log.debug("unreadable Retry-After: %r", raw)
return None
if when.tzinfo is None:
when = when.replace(tzinfo=UTC)
seconds = (when - datetime.now(UTC)).total_seconds()
if seconds <= 0 or seconds > MAX_HINT:
return None
return seconds
+132
View File
@@ -0,0 +1,132 @@
"""Will this page load inside an iframe?
Publishers say no with one of two headers:
X-Frame-Options: DENY | SAMEORIGIN
Content-Security-Policy: ...; frame-ancestors <list>
Both are enforced by the browser, silently: a blocked frame is a blank box with
no event you can catch from the parent page. Cross-origin means you cannot read
the frame's document to see what happened either. So the check has to happen
server-side, before the frame is ever created — which is what this module is.
Both headers are set per host, not per article, so one HTTP request per host is
enough and the answer is cached in the `state` table. In practice this resolves
to "no" for most newspapers: at the time of writing the Guardian, BBC, Ars
Technica and Al Jazeera all send SAMEORIGIN, and the NYT sends DENY.
"""
from __future__ import annotations
from collections.abc import MutableMapping
import logging
import re
from typing import Any
from urllib.parse import urlsplit
log = logging.getLogger(__name__)
#: Kept as `embed:` — the rows are already in every database.
CACHE_PREFIX = "embed:"
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/131.0 Safari/537.36")
_FRAME_ANCESTORS = re.compile(r"frame-ancestors\s+([^;]*)", re.IGNORECASE)
def host_of(url: str) -> str:
try:
return urlsplit(url).netloc.lower()
except ValueError:
return ""
def _verdict(headers: dict[str, str]) -> tuple[bool, str]:
"""Read the two headers. Returns (embeddable, human reason)."""
lower = {k.lower(): v for k, v in headers.items()}
xfo = (lower.get("x-frame-options") or "").strip().lower()
if xfo.startswith("deny"):
return False, "sends X-Frame-Options: DENY"
if xfo.startswith("sameorigin"):
return False, "sends X-Frame-Options: SAMEORIGIN"
match = _FRAME_ANCESTORS.search(lower.get("content-security-policy", ""))
if match:
sources = match.group(1).split()
# 'self' and 'none' both exclude us; a wildcard or an explicit list
# might not, but we're never on anyone's list, so only * lets us in.
if "*" not in sources:
return False, f"CSP frame-ancestors {' '.join(sources) or 'none'}"
return True, "no framing restriction"
def cached(cache: MutableMapping[str, str], url: str) -> dict[str, Any] | None:
"""The answer already known for this host, or None if nobody has asked.
The difference between this and `check` is a network request, and it is
the whole reason both exist: a page being rendered must not wait on an
outlet to find out whether it may be framed, but it does need to know the
difference between "no" and "not asked yet". None is the second one, and
the caller shows it as pending rather than as a refusal.
"""
host = host_of(url)
if not host or not url.lower().startswith(("http://", "https://")):
return {"embeddable": False, "reason": "not a web page", "host": host,
"certain": True}
stored = cache.get(CACHE_PREFIX + host, "")
if not stored:
return None
ok, _, reason = stored.partition(":")
return {"embeddable": ok == "1", "reason": reason, "host": host,
"certain": True}
def check(cache: MutableMapping[str, str], url: str, refresh: bool = False) -> dict[str, Any]:
"""Ask once per host, then remember.
`cache` is any mapping of strings — the app's state table, or a dict in a
test — because a tool has no store of its own to remember things in.
An unreachable host is reported as embeddable with `certain: False` — better
to let the browser try and possibly succeed than to refuse on our own
guess. Only a header we actually read counts as a no.
"""
host = host_of(url)
if not host or not url.lower().startswith(("http://", "https://")):
return {"embeddable": False, "reason": "not a web page", "host": host,
"certain": True}
key = CACHE_PREFIX + host
if not refresh:
cached = cache.get(key, "")
if cached:
ok, _, reason = cached.partition(":")
return {"embeddable": ok == "1", "reason": reason, "host": host,
"certain": True}
try:
import httpx
except ImportError:
return {"embeddable": True, "reason": "httpx not installed, not checked",
"host": host, "certain": False}
try:
with httpx.Client(follow_redirects=True, timeout=10.0,
headers={"User-Agent": UA}) as client:
resp = client.get(url)
# A 403 still carries the headers — the NYT refuses the request *and*
# refuses framing — so don't gate on the status code.
embeddable, reason = _verdict(dict(resp.headers))
# Redirects can land on a different host; cache the answer under both.
final = host_of(str(resp.url))
except Exception as exc:
log.debug("embed check failed for %s: %s", host, exc)
return {"embeddable": True, "reason": f"could not check ({type(exc).__name__})",
"host": host, "certain": False}
payload = f"{'1' if embeddable else '0'}:{reason}"
cache[key] = payload
if final and final != host:
cache[CACHE_PREFIX + final] = payload
return {"embeddable": embeddable, "reason": reason, "host": host, "certain": True}
+39
View File
@@ -0,0 +1,39 @@
"""One shared HTTP client per call chain, without making anyone pass one.
Reading an episode is three or four requests to two hosts, and doing them on
one client is the difference between one connection and four. But the public
functions here are also meant to be usable one at a time — `transcript(video)`
should not require the caller to have stood up a client first.
So every entry point takes an optional client and wraps it in this: given one,
it is borrowed and left open for whoever owns it; given `None`, a client is
made and closed on the way out. The `with` statement at the call site reads the
same either way.
"""
from __future__ import annotations
from contextlib import AbstractContextManager
import httpx
class _Borrowed(AbstractContextManager["httpx.Client"]):
"""Someone else's client. Enters and exits without owning anything."""
def __init__(self, client: httpx.Client) -> None:
self._client = client
def __enter__(self) -> httpx.Client:
return self._client
def __exit__(self, *exc: object) -> None:
return None
def session(client: httpx.Client | None,
timeout: float = 30) -> AbstractContextManager[httpx.Client]:
"""A context manager yielding a client, owning it only if it made it."""
if client is not None:
return _Borrowed(client)
return httpx.Client(timeout=timeout, follow_redirects=True)
+123
View File
@@ -0,0 +1,123 @@
"""What kind of thing is this, when it isn't an ordinary article?
An outlet's feed carries more than its journalism. Golem's carries adverts
("Anzeige:") and pieces behind its paywall ("(g+)"); Hacker News marks paid
links "[$]"; other feeds mark neither. There is no convention across outlets to
detect, so each plugin declares how its publisher says it — as `MarkRule`s in
its spec, and nowhere else. Typing a token in against one feed was how this
worked before every outlet was a plugin; step 19 retired that editor, and
`classify` below is the only way a mark is now found.
What is general is everything after that. The outlet's token is input: it is
matched, cut out of the title, and never shown. One glyph stands for a mark
everywhere, so "what is this" reads the same whoever published it, instead of
being a private code you learn one publisher at a time. The token is noise in
a headline as well — "Anzeige: Schon 10.000-mal bestellt" is worse than the
line underneath it.
The vocabulary below is closed on purpose. Marks are structural — they say what
a thing *is*, and the reader filters and styles them — which is a different
question from what a thing is *about*; that one has tags, an open vocabulary the
model can extend. Adding a mark means adding a row here and a token to the feeds
that use it.
"""
from __future__ import annotations
from dataclasses import dataclass
from collections.abc import Sequence
from typing import Any, Protocol
from desk.models import Item
@dataclass(frozen=True, slots=True)
class Mark:
slug: str
glyph: str
label: str
#: What the reader is told the glyph means, on hover and to a screen reader.
note: str
#: Whether it is worth fetching the page for a body. False for both marks we
#: have: a paywalled page hands back its gate, and an advert's body is the
#: advert. Declared here rather than assumed by the backfill, so a mark that
#: *does* want its body only has to say so.
fetch_body: bool = False
#: What an embedded frame of such an item would actually show, for the
#: reader offered a control they cannot use. Here for the same reason
#: `fetch_body` is: framing is a shared behaviour, and the mark is what
#: knows whether a frame of it would be worth anything. See
#: `OriginalView.blocked_by`.
framed: str = ""
MARKS: tuple[Mark, ...] = (
Mark("paywalled", "\U0001F512", "Paywalled",
"Behind this outlet's paywall", framed="the outlet's paywall"),
Mark("sponsored", "\U0001F4E3", "Sponsored",
"An advertisement, not journalism", framed="the advert"),
)
BY_SLUG = {m.slug: m for m in MARKS}
#: The marks whose items the body backfill should not spend a request on.
NO_BODY = frozenset(m.slug for m in MARKS if not m.fetch_body)
class Rule(Protocol):
@property
def field(self) -> str: ...
@property
def method(self) -> str: ...
@property
def value(self) -> str: ...
@property
def mark(self) -> str: ...
@property
def strip(self) -> bool: ...
def classify(item: Item, rules: Sequence[Rule]) -> Item:
"""Classify original inputs, retaining independent acquisition marks.
Evidence lives in raw alongside the publisher categories until those
acquisition inputs gain typed storage. Repeating this replaces only marks
owned by the previous rule pass, never a custom acquisition hook's marks.
"""
previous = item.raw.get("mark_evidence", {})
original = item.source_title or item.title
independent = previous.get("independent", item.marks)
found: list[str] = []
evidence: list[dict[str, Any]] = []
spans: list[tuple[int, int]] = []
for rule in rules:
if rule.field == "inputs.source_categories":
values = item.raw.get("categories", [])
matched = next((v for v in values if isinstance(v, str)
and v.lower() == rule.value.lower()), None)
at = 0 if matched is not None else -1
value = matched or ""
else:
value = item.excerpt if rule.field == "inputs.feed_text" else original
lower, token = value.lower(), rule.value.lower()
at = lower.find(token)
if rule.method == "prefix" and not lower.startswith(token):
at = -1
elif rule.method == "suffix":
at = len(value) - len(rule.value) if lower.endswith(token) else -1
if at < 0:
continue
found.append(rule.mark)
evidence.append({"field": rule.field, "method": rule.method,
"value": rule.value, "mark": rule.mark,
"strip": rule.strip,
"matched": value[at:at + len(rule.value)]})
if rule.strip:
spans.append((at, at + len(rule.value)))
item.source_title = original
item.title = ("".join(c for i, c in enumerate(original)
if not any(start <= i < end for start, end in spans)).strip() or original
) if spans else original
item.marks = sorted(set(independent) | set(found))
item.raw["mark_evidence"] = {"version": 1, "independent": list(independent),
"matches": evidence}
return item
+193
View File
@@ -0,0 +1,193 @@
"""A Mastodon or Bluesky post read through its network's public API.
Both web apps serve a script shell to a plain request, so `checks` names these
getters before `browser`: the API answers in a fraction of a second, needs no
Chromium and no login, and returns the post rather than a rendering of it.
Each getter builds a small HTML page from the answer, so `scrape` checks and
converts it exactly as it would a publisher's page.
from desk.tools import posts
posts.is_mastodon("https://hachyderm.io/@simontatham/117201594980991062")
page = posts.mastodon(url) # checks.Page
page = posts.bluesky("https://bsky.app/profile/bsky.app/post/3mv3shqdfuc2e")
The Mastodon and Bluesky plugins register them with `scrape` as the
`mastodon_api` and `bluesky_api` getters for their URLs; see `Source.getters`.
**A thread is read as far as its author continued it.** A post a link points
to is often the first of several by the same person; the replies by anyone
else are not part of it.
X is not here: its API bills every read and needs a token, and x.com shows a
browser a login wall.
GET https://{host}/api/v1/statuses/{id} Mastodon, any instance
GET https://{host}/api/v1/statuses/{id}/context
GET https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread
"""
from __future__ import annotations
import html
import re
from typing import Any
from urllib.parse import urlsplit
import httpx
from desk.tools.checks import Page
TIMEOUT = 20.0
#: Says what is asking, the way `desk.sources.base.USER_AGENT` does for the
#: plugins; a tool sits below `sources` and cannot import it.
USER_AGENT = "newsdesk (+https://github.com/; reading a linked post)"
BLUESKY_API = "https://public.api.bsky.app/xrpc"
#: `/@user/123` is what an instance links; `/users/user/statuses/123` is the
#: ActivityPub id some links carry instead.
_MASTODON_PATH = re.compile(r"^/(?:@[^/]+|users/[^/]+/statuses)/(\d+)/?$")
_BLUESKY_PATH = re.compile(r"^/profile/([^/]+)/post/([a-z0-9]+)/?$")
#: How deep a Bluesky thread is asked for. A self-thread longer than this is
#: cut, which is rare and costs the tail rather than the post.
THREAD_DEPTH = 20
def _get(url: str, params: dict[str, Any] | None = None) -> httpx.Response:
# Module-level, so a test replaces this one function.
return httpx.get(url, params=params, timeout=TIMEOUT,
headers={"User-Agent": USER_AGENT}, follow_redirects=True)
# ---------------------------------------------------------------- Mastodon
def _mastodon_id(url: str) -> tuple[str, str] | None:
parts = urlsplit(url)
found = _MASTODON_PATH.match(parts.path)
return (parts.netloc, found.group(1)) if parts.scheme == "https" and found else None
def is_mastodon(url: str) -> bool:
"""Whether `url` has a Mastodon post's shape. Any host: an instance is
recognised by the check that found its web app's shell, not by name."""
return _mastodon_id(url) is not None
def mastodon(url: str) -> Page | None:
"""The post at `url`, and its author's continuation, from the instance's API."""
host, status_id = _mastodon_id(url) or ("", "")
answer = _get(f"https://{host}/api/v1/statuses/{status_id}")
if not _json(answer):
# Not an instance: `/@user/123` is a Mastodon address's shape, and
# another site can have it. Nothing to say about the page, so the
# ordinary path runs.
return None
if answer.status_code != 200:
return Page(answer.status_code, "", url, dict(answer.headers))
status = answer.json()
author = status.get("account") or {}
thread = [status]
context = _get(f"https://{host}/api/v1/statuses/{status_id}/context")
if context.status_code == 200 and _json(context):
# Descendants come in thread order; the author's continuation is the
# chain of their replies, each to the one before.
for reply in context.json().get("descendants") or []:
if ((reply.get("account") or {}).get("id") == author.get("id")
and reply.get("in_reply_to_id") == thread[-1].get("id")):
thread.append(reply)
name = author.get("display_name") or author.get("acct") or host
card = status.get("card") or {}
return Page(200, _page(
title=f"{name}: {_first_words(_text(status.get('content', '')))}",
byline=f"{name} (@{author.get('acct', '')})",
published=status.get("created_at", ""),
paragraphs=[s.get("content", "") for s in thread],
link=(card.get("url", ""), card.get("title", "")),
alts=[m.get("description") or "" for s in thread
for m in s.get("media_attachments") or []],
escaped=False), url)
# ---------------------------------------------------------------- Bluesky
def _bluesky_uri(url: str) -> str:
parts = urlsplit(url)
found = _BLUESKY_PATH.match(parts.path)
if parts.netloc not in {"bsky.app", "www.bsky.app"} or not found:
return ""
return f"at://{found.group(1)}/app.bsky.feed.post/{found.group(2)}"
def is_bluesky(url: str) -> bool:
return bool(_bluesky_uri(url))
def bluesky(url: str) -> Page | None:
"""The post at `url`, and its author's continuation, from Bluesky's AppView."""
answer = _get(f"{BLUESKY_API}/app.bsky.feed.getPostThread",
{"uri": _bluesky_uri(url), "depth": THREAD_DEPTH, "parentHeight": 0})
if answer.status_code != 200:
# A post that is gone is a 400 `NotFound` here, not a 404.
gone = answer.status_code == 400 and "NotFound" in answer.text
return Page(404 if gone else answer.status_code, "", url, dict(answer.headers))
thread = answer.json().get("thread") or {}
post = thread.get("post") or {}
author = post.get("author") or {}
posts = [post]
node = thread
while True:
# The author's own reply, if there is one, is the next part of the thread.
nxt = next((r for r in node.get("replies") or []
if ((r.get("post") or {}).get("author") or {}).get("did") == author.get("did")),
None)
if nxt is None:
break
posts.append(nxt["post"])
node = nxt
record = post.get("record") or {}
external = ((post.get("embed") or {}).get("external") or {})
name = author.get("displayName") or author.get("handle", "")
return Page(200, _page(
title=f"{name}: {_first_words(record.get('text', ''))}",
byline=f"{name} (@{author.get('handle', '')})",
published=record.get("createdAt", ""),
paragraphs=[(p.get("record") or {}).get("text", "") for p in posts],
link=(external.get("uri", ""), external.get("title", "")),
alts=[image.get("alt") or "" for p in posts
for image in (p.get("embed") or {}).get("images") or []],
escaped=True), url)
# ---------------------------------------------------------------- the page
def _page(*, title: str, byline: str, published: str, paragraphs: list[str],
link: tuple[str, str], alts: list[str], escaped: bool) -> str:
"""An article page for the converter. Mastodon's content is HTML already;
Bluesky's is text, escaped here and split into paragraphs on blank lines."""
if escaped:
paragraphs = ["".join(f"<p>{html.escape(block).replace(chr(10), '<br>')}</p>"
for block in re.split(r"\n\s*\n", text) if block.strip())
for text in paragraphs]
body = "\n".join(paragraphs)
url, label = link
if url:
body += f'\n<p><a href="{html.escape(url)}">{html.escape(label or url)}</a></p>'
body += "".join(f"\n<p>[Image: {html.escape(alt)}]</p>" for alt in alts if alt.strip())
return (f"<!DOCTYPE html><html><head><title>{html.escape(title)}</title></head><body>"
f"<article><header><p>{html.escape(byline)}</p>"
f"<time datetime=\"{html.escape(published)}\">{html.escape(published)}</time></header>"
f"\n{body}\n</article></body></html>")
def _json(answer: httpx.Response) -> bool:
return "json" in answer.headers.get("content-type", "")
def _text(markup: str) -> str:
return html.unescape(re.sub(r"<[^>]+>", " ", markup))
def _first_words(text: str, count: int = 8) -> str:
words = text.split()
return " ".join(words[:count]) + ("…" if len(words) > count else "")
+240
View File
@@ -0,0 +1,240 @@
"""Get an article out of a URL: fetch, check, convert, and fetch another way
when the check says so.
Three parts, each replaceable on its own:
- a **getter** fetches a page: `HTTP`, `BROWSER` (Playwright, with the
`browser` extra), or one a plugin registered for its own URLs, such as a
post through its network's API
- `desk.tools.checks` **checks** the page and answers with a `Verdict`: the
failure category, what matched, and which getters to try next
- `desk.tools.article` **converts** HTML that passed into the structured
article, preserving code blocks and tables
got = scrape(url) # standard: HTTP, check, convert
got = scrape(url, getters=ADVANCED) # a failed check may try another getter
got = scrape(url, getters=advanced(cache, plugin_getters))
# and a consent the browser accepted is sent by
# HTTP next time, and a plugin's URLs go to it
**A getter that claims a URL goes first.** A plugin that knows how to read its
own network's addresses registers a getter with `claims=True`: when `handles`
matches, it runs before `http`. What it brings back is converted like any
page. A status it answers with ends the attempt, because the network's own API
is the authority on its posts; no answer at all, or a page that fails the
check, and the ordinary path runs as if it had not been asked.
Standard and advanced run the same loop. The only difference is which getters
are available to a verdict: with `http` alone a bot check ends as
`js_required` on the row, with `browser` available the verdict's `next` sends
the same URL there, and a page that passes is converted exactly as an HTTP one
would have been. A getter named by a verdict and not available is skipped, so
the checks can say where a page should go before that getter exists.
Nothing but a verdict moves to another getter. A page that passed the check
and converted to nothing is `empty` and ends there, because an extractor that
finds no article in real HTML finds none in the same HTML from a browser.
"""
from __future__ import annotations
import copy
import logging
from collections.abc import Callable, Iterable, Mapping, MutableMapping
from configparser import ConfigParser
from dataclasses import dataclass
from desk.tools import article, browser, failures
from desk.tools.article import ArticleContent
from desk.tools.checks import Page, Verdict, check
log = logging.getLogger(__name__)
@dataclass(frozen=True)
class Getter:
"""One way of fetching a page. `fetch` returns None for a request that
never completed and raises for anything it can name. `handles` is for a
getter that only knows some hosts, such as a platform's API; one that does
not handle a URL is skipped without an attempt."""
name: str
fetch: Callable[[str], Page | None]
handles: Callable[[str], bool] = lambda url: True
#: Run before `http` for every URL `handles` matches, rather than only
#: when a check names it. For a getter that owns a kind of address.
claims: bool = False
@dataclass(frozen=True)
class Attempt:
getter: str
failure: str = ""
#: The signature or rule that failed the check, empty on success.
matched: str = ""
status: int = 0
@dataclass(frozen=True)
class Scraped:
"""The article, or the failure of the last getter tried, and every try."""
content: ArticleContent | None
failure: str
getter: str
attempts: tuple[Attempt, ...]
#: The server's own Retry-After, in seconds, when one was given.
retry_after: float | None = None
def _http(url: str, cookie: str = "") -> Page | None:
# Looked up per call, so a test that stubs the downloader stubs this.
import trafilatura
if cookie:
answer = trafilatura.fetch_response(url, decode=True, with_headers=True,
config=_with_cookie(cookie))
else:
answer = trafilatura.fetch_response(url, decode=True, with_headers=True)
if answer is None:
return None
html = answer.html or ""
data = getattr(answer, "data", None)
if isinstance(data, bytes) and data[:4] == ZSTD_MAGIC:
html = _unzstd(data)
return Page(status=answer.status, html=html,
url=getattr(answer, "url", "") or url,
headers=answer.headers or {})
#: The first bytes of a zstd frame.
ZSTD_MAGIC = b"\x28\xb5\x2f\xfd"
def _unzstd(data: bytes) -> str:
"""A zstd body as the HTML trafilatura would have decoded.
Python 3.14 has `compression.zstd`, so urllib3 offers zstd, and Cloudflare
hosts such as opendatascience.com answer with it. Streaming the body,
urllib3 hands trafilatura the compressed bytes; trafilatura decompresses
zstd only with the `zstandard` package, which refuses a frame that does not
state its size, and Cloudflare's frames do not. Undecoded, the page is
binary that passes the check and converts to `empty`.
"""
from compression import zstd
from trafilatura.utils import decode_file
return decode_file(zstd.decompress(data))
def _with_cookie(cookie: str) -> ConfigParser:
"""trafilatura's own config with a `Cookie` header in it.
Its downloader takes no headers, only a config. A config that is not the
default replaces the default headers, so the user agent is set again to
the one it would have sent. `%` is escaped for `ConfigParser`'s
interpolation; consent strings are full of it.
"""
from trafilatura.downloads import USER_AGENT
from trafilatura.settings import DEFAULT_CONFIG
config = copy.deepcopy(DEFAULT_CONFIG)
config.set("DEFAULT", "USER_AGENTS", USER_AGENT)
config.set("DEFAULT", "COOKIE", cookie.replace("%", "%%"))
return config
def _browser(url: str, cache: MutableMapping[str, str] | None = None) -> Page | None:
got = browser.render(url)
if cache is not None and got.consent:
browser.remember(cache, got.url, got.cookies)
return Page(status=got.status, html=got.html, url=got.url, headers=got.headers)
#: One plain request through trafilatura's downloader: its headers, its size
#: cap, and the status and headers handed back rather than thrown away.
HTTP = Getter("http", _http)
#: Headless Chromium. Accepts a consent dialog it recognises; see
#: `desk.tools.browser`.
BROWSER = Getter("browser", _browser)
STANDARD: Mapping[str, Getter] = {"http": HTTP}
#: The getters this module has itself. `browser` only with the `browser`
#: extra installed; a verdict naming a getter that is not here skips it. The
#: rest come from plugins, through `advanced`.
ADVANCED: Mapping[str, Getter] = {
"http": HTTP, **({"browser": BROWSER} if browser.installed() else {})}
def advanced(cache: MutableMapping[str, str],
registered: Iterable[Getter] = ()) -> Mapping[str, Getter]:
"""`ADVANCED` and the getters plugins `registered`, sharing one cookie jar.
The browser keeps the cookies of a consent it accepted in `cache`, per
host, and HTTP sends them. So a gate costs one render per host until its
cookies expire, and every other article from that host is a plain request.
A registered getter cannot replace `http` or `browser`.
"""
getters = {getter.name: getter for getter in registered}
getters.update(ADVANCED)
getters["http"] = Getter("http", lambda url: _http(url, browser.cookie_header(cache, url)))
if "browser" in getters:
getters["browser"] = Getter("browser", lambda url: _browser(url, cache))
return getters
def scrape(url: str, getters: Mapping[str, Getter] = STANDARD,
first: str = "http") -> Scraped:
"""A claiming getter, then `first`, then whatever each failed check names."""
attempts: list[Attempt] = []
hint: float | None = None
for claimant in getters.values():
if not (claimant.claims and claimant.handles(url)):
continue
content, verdict, page = _one(claimant, url)
attempts.append(Attempt(claimant.name, "" if content else verdict.category,
"" if content else verdict.name,
page.status if page else 0))
if content is not None:
return Scraped(content, "", claimant.name, tuple(attempts), hint)
if page is not None and page.status >= 400:
return Scraped(None, verdict.category, claimant.name, tuple(attempts),
failures.hint(page.headers))
log.debug("%s: %s via %s (%s); the ordinary path runs", url,
verdict.category, claimant.name, verdict.name)
plan = [first]
while plan:
getter = getters.get(plan.pop(0))
if getter is None or not getter.handles(url) or any(
a.getter == getter.name for a in attempts):
continue
content, verdict, page = _one(getter, url)
if content is not None:
attempts.append(Attempt(getter.name, status=page.status if page else 0))
return Scraped(content, "", getter.name, tuple(attempts), hint)
attempts.append(Attempt(getter.name, verdict.category, verdict.name,
page.status if page else 0))
if page is not None and page.status >= 400:
hint = hint or failures.hint(page.headers)
log.debug("%s: %s via %s (%s)", url, verdict.category, getter.name, verdict.name)
plan = list(verdict.next)
# The row gets the last answer a page gave. A fallback that brought back
# nothing (a browser that would not start, an API that did not answer)
# says nothing about the page, and must not replace what the check found.
answered = [a for a in attempts if a.status] or attempts
last = answered[-1] if answered else Attempt("", "unknown")
return Scraped(None, last.failure, last.getter, tuple(attempts), hint)
def _one(getter: Getter, url: str) -> tuple[ArticleContent | None, Verdict, Page | None]:
try:
page = getter.fetch(url)
except Exception as exc:
return None, Verdict(failures.of_exception(exc), type(exc).__name__), None
if verdict := check(page):
return None, verdict, page
assert page is not None # check() refuses a missing page
content = article.extract_article(page.html, url)
if content is None or not content.plain_text.strip():
return None, Verdict("empty", "no_article"), page
return content, Verdict("", ""), page
+100
View File
@@ -0,0 +1,100 @@
# `desk.tools` — the machinery
## What it is
Things a pipeline stage picks up and puts down: plain inputs, plain outputs, no idea what the app is for. The test for belonging here is *would this make sense in a different application*.
| Tool | What it does | Needs |
|---|---|---|
| `agent` | ask Claude, get JSON back | `anthropic` **or** `claude-agent-sdk` (both optional) |
| `article` | supplied HTML to typed structured content, preserving tables and code; derived plain text and acquisition provenance | `trafilatura` |
| `browser` | a page rendered in headless Chromium, past a consent dialog it knows; the cookies it earned, per host | `playwright` (optional, plus `playwright install chromium`) |
| `embedding` | text in, vector out (~384 floats) | `sentence-transformers` (optional, ~100MB+) |
| `failures` | what went wrong, in one word a retry policy can be written against | stdlib |
| `framing` | will this page load in an iframe? | `httpx` |
| `http` | one shared HTTP client per call chain | `httpx` |
| `marks` | what kind of thing is this, when it isn't an ordinary article | stdlib |
| `posts` | a Mastodon or Bluesky post and its author's thread, through the public API, as a page to convert; registered by those plugins | `httpx` |
| `checks` | is a fetched page worth converting, and if not, which getter to try next | stdlib |
| `scrape` | fetch, check, convert, and fetch another way when the check says so | `trafilatura` |
| `walls` | did we fetch the article, or the consent page in front of it | stdlib |
| `video/` | a recording, read as text with timings | `httpx`, `youtube-transcript-api` |
**Nothing is imported in `desk/tools/__init__.py` on purpose** — each import is paid for by the caller that wants it. Import the module, not the package: `from desk.tools import agent`.
## `agent` — the model
Two interchangeable backends behind one call. `agent.backend(cfg)` says which will answer, or `""` for nobody — which is a failure, not a third backend.
- `api` — one Claude API call. Wants `ANTHROPIC_API_KEY`. The unattended path.
- `claude_code` — the Agent SDK drives the `claude` CLI on your subscription. No key, nothing billed. The development path. (**Spawns a process.**)
- `off` / nothing available — `backend()` answers `""`, and the caller fails the batch with a notice naming the reason. Nothing waits to be answered by hand.
```python
from desk.tools import agent
reply = agent.ask_json(
config.agent, # desk.config.AgentConfig
system="You tag articles.", # kept byte-identical across a run — it is what caches
question={"articles": [...]},
schema={"type": "object", ...}, # both backends take the same JSON schema
model="claude-haiku-4-5-20251001",
thinking="off", # or "adaptive"
on_partial=lambda piece: ..., # streamed text, for per-item progress
)
reply.data, reply.backend, reply.model, reply.cost_usd, reply.usage
```
Failures of every kind raise `AgentError` — treat it as "no answer this time". The `claude_code` path deliberately gets **no tools, no CLAUDE.md, no skills, no MCP, no session**: one text-in, JSON-out turn, the same shape as the API call.
## The others, in one line each
`article.extract_article(html, source_url)` returns `ArticleContent` or `None` when extraction finds nothing. `structured-v1` is a typed tree, not trusted HTML; `plain` supports legacy strings. `plain_text` and `revision` are derived from content. Explicit gutter nodes are removed before extraction, while meaningful numeric lines and code whitespace remain. Completeness defaults to unknown. Newsdesk's body-fetch job now uses this tool and atomically persists its content and derived plain projection; the reader renders its closed node vocabulary. Fetching, persistence, wall detection and rendering remain the caller's responsibilities. The extractor requires trafilatura 2.2 or newer, verified against the representative fixture.
```python
from desk.tools import embedding, failures, framing, marks, video, walls
from desk.tools.http import session
vecs = embedding.load("all-MiniLM-L6-v2").encode(["some text"]) # (n, dim) ndarray
verdict = framing.check(cache, "https://x.com/a") # {embeddable, reason, host, certain}; per host
item = marks.mark(item, {"paywalled": "(g+)"}) # cuts the token out, records the mark
reason = walls.note(cache, url, body) # "" or why this body is the same page every time
why = failures.of_status(503) # 'server_error'; of_exception(exc) for a raise
wait = failures.hint(resp.headers) # the server's own Retry-After, in seconds
vid, segments = video.read("https://youtu.be/GUYEofBebDc") # Video + Segment[]
with session(client) as http: ... # borrows one if given, makes and closes one if not
```
`failures` is the third of those, and the newest. A category answers one question — *would asking again get a different answer?* — so `TRANSIENT` (`timeout`, `unreachable`, `rate_limited`, `server_error`, `not_ready`) and `SETTLED` (`not_found`, `denied`, `consent_wall`, `js_required`, `empty`) are the whole of the vocabulary, plus `unknown` for a failure nothing recognises, which is deliberately in neither. Those names are what a plugin's `RetryPolicy.on` is validated against, so adding one here adds one to the plugin contract. An exception is named by walking the class names in its ancestry rather than by `isinstance`, because `httpx` and `urllib3` are both optional and either may be the one that raised — it costs precision and buys a tool that can name a failure raised by a library it has never imported. A consent wall is a category and is in no default retry list: it is the same answer however many times it is asked for.
`marks` and `walls` are the two that read what came back rather than fetching anything, which is why neither needs a dependency. `marks` turns an outlet's own notation — Golem's `(g+)`, Hacker News's `[$]` — into one glyph that means the same thing whoever published it, and cuts the token out of the title. `walls` catches the opposite problem: a site that answers every article URL with its consent gate, which trafilatura extracts faithfully and which is worse than storing nothing. The tell is that two different articles cannot have the same body and two walls always do, so the detector is a fingerprint and a cache, with no per-site rules to keep up to date.
`python -m desk.tools.video <url>` prints a video's segments — a dry run with no app around it.
## `scrape` and `checks` — getting the article out of a URL
`scrape.scrape(url, getters)` fetches with `http`, runs `checks.check` on the page, converts a page that passed with `article.extract_article`, and on a failed check tries the getters the `Verdict` names. It returns `Scraped`: the content or the last failure, the getter that answered, every `Attempt` with the signature that failed it, and the server's `Retry-After`.
```python
from desk.tools import checks, scrape
got = scrape.scrape(url) # STANDARD: http only
got = scrape.scrape(url, scrape.ADVANCED) # a verdict may send it to another getter
got = scrape.scrape(url, scrape.advanced(cache, getters)) # what the body fetch uses: plugins' URL getters first,
# and HTTP sends the cookies a browser earned
verdict = checks.check(checks.Page(200, html)) # None, or Verdict(category, name, next)
```
`browser.render(url)` launches Chromium, clicks the accept button of a dialog in `browser.CONSENTS` if one appears within `GATE_WAIT` seconds, and returns `Rendered`: status, HTML, final URL, headers, the host's cookies and which consent it accepted. `remember(cache, url, cookies)`, `cookie_header(cache, url)` and `forget(cache, url)` are the jar, in the same kind of mapping `walls` uses. One browser per call and one at a time per process; `browser.installed()` says whether the extra is there, and `ADVANCED` has `browser` only if it is.
How the loop runs, the signature format and how to add one: `docs/architecture.md` §2.1.
## Limits
- **All synchronous and blocking.** Meant to be called from a worker thread.
- **`ask_json` is one turn, no tools, no conversation.** No streaming API for the caller beyond `on_partial` (raw text pieces, not parsed JSON); chunking, retries and applying the answer are `desk.modeltask`'s job.
- **Optional deps really are optional, but "optional" is not "interchangeable".** A missing model SDK is an `AgentError`, and there are two backends, so the other one may well answer. A missing `sentence-transformers` is an `ImportError` and there is nothing behind it: `ranking.backend: embeddings` does **not** fall back to tfidf. The embed step logs the failure and returns zero, articles keep no vectors, and everything scores a neutral 0.5 for taste and ranks on its recency alone. Which reads a lot like a ranker that is working and is not.
- **`cost_usd` comes from the `claude_code` path, not the API one.** Backwards from what you would guess: the CLI's `ResultMessage` carries what those tokens *would* have cost, so it is reported even though a subscription bills nothing for them, and the API path — the one that really is billed — leaves the field at zero because the Messages response does not price itself. Read it as an estimate of what a run would cost on the API, and never as a bill.
- **`video` is YouTube-only today.** No local-file transcription yet (yt-dlp + whisper is the planned third backend); a video with no captions still yields chapters, just no words.
- **`framing` answers per host, not per article**, and caches; most newspapers answer "no".
- **No tool knows about the store, the queue, or config beyond its own section** — and `tools` must not import `pipeline` or `sources`.
+103
View File
@@ -0,0 +1,103 @@
"""Video, read as text with timings on it.
The generic half of watching things so you don't have to. Give it a video and
it gives back a list of `Segment`s — each one a stretch of the recording with a
title, the words spoken in it, and a link that plays it from that second.
read() a video -> Video + Segment[] the whole thing, one call
youtube captions, metadata, comments where both inputs come from
marks an outline -> chapter marks description, notes, comments
media a MediaRef -> a playable link YouTube today, a file later
segment cues + marks -> Segment[] the cut, and transcript text
**This package knows nothing about newsdesk.** No store, no queue, no config,
no `Item`. That is what makes it reusable, and it is the seam the design is
built on: `desk/sources/podcast.py` sits on top and adds the things that
*are* specific to a show — its feed, its weekly cadence, the article links in
its notes, and turning segments into rows.
Two decisions worth knowing before reading further.
**A video is a `MediaRef`, not a URL.** Today every ref is a YouTube id; the
reading pane's link is computed from it on the way out. Point one at a file on
disk in config and every link to it changes, with nothing rewritten and no
item's identity touched. See `media.py`.
**Transcripts come from whoever can supply one.** `youtube-transcript-api`
first, and a built-in fetcher against YouTube's own player endpoint when the
library can't get the track. Transcribing audio locally (yt-dlp plus
faster-whisper) is the third backend and the one that makes local video work;
it is not written yet, and `read` is shaped so that adding it changes nothing
above.
Try it:
python -m desk.tools.video https://www.youtube.com/watch?v=GUYEofBebDc
"""
from __future__ import annotations
import logging
import httpx
from desk.tools.video import marks, media, youtube
from desk.tools.http import session
from desk.tools.video.marks import Chapter, Link
from desk.tools.video.media import MediaRef
from desk.tools.video.segment import Segment, Video, cut, opening, paragraphs
from desk.tools.video.youtube import Blocked, Cue, TranscriptUnavailable
log = logging.getLogger(__name__)
__all__ = [
"Blocked", "Chapter", "Cue", "Link", "MediaRef", "Segment",
"TranscriptUnavailable", "Video", "cut", "describe", "marks", "media",
"opening", "paragraphs", "read", "youtube",
]
def read(url: str, *, comments: bool = True,
client: httpx.Client | None = None) -> tuple[Video, list[Segment]]:
"""One video, cut into its chapters. The whole package in one call.
`comments` is the only cost worth thinking about: it is an extra request to
the watch page and is only consulted when the description has no outline in
it, so for a show that publishes chapter marks it never happens.
"""
with session(client, timeout=40) as http:
video = describe(url, client=http)
try:
cues = youtube.transcript(video.key, client=http)
except TranscriptUnavailable as exc:
# Not fatal on its own. The outline still gives every chapter, its
# links and its timecode; only the quoted words are missing, and
# auto-captions usually appear within a few hours of an upload.
log.warning("%s", exc)
cues = []
chapters = marks.best(video.notes)
if not chapters and comments:
chapters = marks.in_comments(youtube.comments(video.key, client=http))
return video, cut(video, cues, chapters)
def describe(url: str, *, client: httpx.Client | None = None) -> Video:
"""What a video says about itself: title, author, length, description.
YouTube only, for now. A local file would come back with its stem as the
title and no notes, which is why `Video.notes` is allowed to be empty and
why `read` has a comments fallback at all.
"""
key = youtube.video_id(url)
with session(client) as http:
meta = youtube.details(key, client=http)
return Video(
ref=media.youtube(key),
title=meta["title"],
url=media.youtube(key).at(0),
author=meta["author"],
duration=meta["duration"],
notes=meta["description"],
)
+120
View File
@@ -0,0 +1,120 @@
"""Cut a video up and print what came out.
python -m desk.tools.video <youtube url>
python -m desk.tools.video <youtube url> --json > segments.json
Videos only. A podcast episode — where the chapter marks and the article links
live on the show's own page rather than on YouTube — goes through the layer
above:
python -m desk.sources.podcast <episode url | feed url>
"""
from __future__ import annotations
import argparse
import dataclasses
import json
import logging
import sys
import desk.tools.video as video_pkg
from desk.tools.video import marks
from desk.tools.video.segment import Segment, Video
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="python -m desk.tools.video",
description="Cut a video into one item per chapter mark.",
)
parser.add_argument("url", help="a YouTube link or bare video id")
parser.add_argument("--json", action="store_true",
help="dump everything, transcripts included, as JSON")
parser.add_argument("--chars", type=int, default=160,
help="how much transcript to preview per segment (default 160)")
parser.add_argument("--no-comments", action="store_true",
help="don't fall back to chapter marks typed in the comments")
parser.add_argument("-v", "--verbose", action="store_true")
args = parser.parse_args(argv)
logging.basicConfig(
level=logging.INFO if args.verbose else logging.WARNING,
format="%(levelname)-7s %(message)s",
)
# Windows still opens a console in the system codepage, and a video title
# with a "ñ" in it — or the arrows this prints — is an unhandled
# UnicodeEncodeError half way through the report rather than a bad glyph.
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
try:
video, segments = video_pkg.read(args.url, comments=not args.no_comments)
except Exception as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
if args.json:
json.dump(dump(video, segments), sys.stdout, indent=2, default=str)
print()
else:
report(video, segments, args.chars)
return 0
def report(video: Video, segments: list[Segment], chars: int) -> None:
"""The human-readable view. Also used by the podcast module."""
words = sum(len(s.transcript.split()) for s in segments)
length = video.duration or (segments[-1].end if segments else 0)
print(f"{video.title}")
print(f" {video.author or '—'} {video.ref.kind}:{video.ref.key}"
f" {marks.hhmmss(length)}")
print(f" {len(segments)} segments, {words:,} words of transcript")
print()
heading = None
for segment in segments:
if segment.heading != heading:
heading = segment.heading
print(f" {heading or '(unfiled)'}")
preview = segment.transcript[:chars]
if len(segment.transcript) > chars:
preview += "…"
print(f" {segment.timecode:>8} {segment.title[:78]}")
print(f" {marks.hhmmss(segment.duration)} long · {segment.url()}")
for link in segment.links:
print(f" → {link.url}")
print(f" {preview or '(no transcript under this mark)'}")
print()
def dump(video: Video, segments: list[Segment]) -> dict[str, object]:
"""The shape a stored item is built from, plus the fields a dry run wants
to eyeball. The stored half is `Segment.record()` and is defined there;
desk/sources/podcast.py turns the rest into a row."""
return {
"video": {
"ref": dataclasses.asdict(video.ref),
"title": video.title,
"url": video.url,
"author": video.author,
"published_at": video.published_at,
"duration": video.duration,
},
"segments": [
{
**s.record(),
"external_id": s.external_id,
"title": s.title,
"heading": s.heading,
"timecode": s.timecode,
"url": s.url(),
"transcript": s.transcript,
}
for s in segments
],
}
if __name__ == "__main__":
raise SystemExit(main())
+265
View File
@@ -0,0 +1,265 @@
"""Chapter marks: a video's own table of contents, wherever it was written.
The important discovery about this kind of video is that **the segmentation
already exists and somebody else did it**. Last Week in AI ships every episode
with a timestamped outline — a rubric, then one line per story, each with the
time it starts and a link to the article being discussed:
Tools & Apps
(00:03:32) Google announces Gemini 3.7 Flash ... - Ars Technica
(00:13:11) SpaceXAI Releases Grok 4.6 ... - MarkTechPost
That is a better set of boundaries than any topic-segmentation model would
produce, for nothing, with the source links attached. So this module parses it
rather than inferring it. An LLM is still wanted downstream — to summarise what
was *said* in each stretch — but not to work out where one stretch ends.
**Three places to find that outline**, in descending order of usefulness, all
parsed by the same function because they are the same convention written in
different markup:
1. **The show's own page or feed body** — HTML, so the links are real
`<a href>` and each chapter arrives with the article it is about. Only a
show with a website has this, which is why the podcast source hands it in
rather than this module going to look.
2. **The video description** — plain text. Usually the same outline with the
links flattened away.
3. **The comments** — see `in_comments`. When a show publishes no marks at
all, a listener has often typed some out; YouTube itself reads those to
draw its chapter markers.
`best` takes them in that order and returns the first that is real. Nothing
here is specific to one show beyond the shape "timestamp, then a title,
optionally under a heading", which is the near-universal convention.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Iterable
from dataclasses import dataclass, field
from html.parser import HTMLParser
log = logging.getLogger(__name__)
#: `(00:03:32)`, `00:03:32`, `3:32` — at the start of the line, which is what
#: makes it a chapter mark rather than a time mentioned in a sentence.
TIMESTAMP = re.compile(
# The character class ends with hyphen, en dash and em dash on purpose:
# a show separates the time from the title with whichever of the three its
# editor's keyboard produced, and means the same thing by all of them.
r"^\(?\s*(\d{1,2}):([0-5]\d)(?::([0-5]\d))?\s*\)?[\s\-–—:]*" # noqa: RUF001
)
_URL = re.compile(r"https?://[^\s<>\"')\]]+")
#: Tags that end a line of the outline. Everything else is inline and stays
#: on the line it was found in, which is what keeps `<span>(00:03:32) </span>`
#: attached to the `<a>` that follows it.
_BLOCKS = frozenset({
"li", "p", "br", "div", "tr", "ul", "ol", "h1", "h2", "h3", "h4", "blockquote",
})
@dataclass(frozen=True, slots=True)
class Link:
"""One article the hosts were talking about."""
url: str
title: str = ""
@dataclass(slots=True)
class Chapter:
"""One entry in the outline: where a story starts and what it is about."""
start: int # seconds into the episode
title: str
heading: str = "" # the rubric it sits under
links: list[Link] = field(default_factory=list)
@property
def url(self) -> str:
"""The article being discussed, if the notes linked one."""
return self.links[0].url if self.links else ""
def parse(body: str) -> list[Chapter]:
"""Every chapter mark in a set of show notes, in the order they play.
Takes HTML or plain text and works out which it was given. Returns `[]`
for notes with no outline in them at all, which is a real answer — plenty
of shows don't write one, and the caller falls back to treating the episode
as a single undivided item rather than guessing at boundaries.
"""
lines = _html_lines(body) if "<" in body and ">" in body else _text_lines(body)
chapters: list[Chapter] = []
heading = ""
for text, links in lines:
match = TIMESTAMP.match(text)
if match is None:
# A heading is only a heading once the outline has started;
# before that every line is preamble — the blurb, the sponsors,
# the "Timestamps:" label itself.
if chapters and _is_heading(text, links):
heading = text.rstrip(":")
continue
title = text[match.end():].strip(" -–—|") # noqa: RUF001 — as above
# Prefer the anchor's own text: a line that links two follow-ups reads
# "First story | The Verge + Second story", and the first link's text
# is the story, where the whole line is the story plus its footnote.
if links and links[0].title:
title = links[0].title
chapters.append(Chapter(
start=_seconds(match), title=title, heading=heading, links=links,
))
if not chapters:
log.debug("no chapter marks found in %d chars of notes", len(body))
return chapters
def best(*candidates: str, minimum: int = 3) -> list[Chapter]:
"""The first candidate that yields a real outline.
Chapter marks come from up to three places and they are not equally good,
so the order they are passed in is the order of preference — for a podcast
that is: the show's own notes as HTML (links intact), then the YouTube
description (same outline, links flattened away), then the comments.
`minimum` is what stops a stray "at 3:20 he says" in a blurb becoming a
one-chapter outline that suppresses the good source behind it. Three marks
is the point where a line of text stops being a coincidence.
"""
for text in candidates:
found = parse(text)
if len(found) >= minimum:
return found
return []
def in_comments(comments: Iterable[str], minimum: int = 3) -> list[Chapter]:
"""The outline somebody typed into the comments, if anybody did.
Read as the fallback for shows that publish no chapter marks of their own,
where it is common for a listener to write one out — YouTube itself treats
such a comment as good enough to draw chapter markers from.
Each comment is parsed on its own rather than the lot being concatenated.
Joined together, an outline comment and thirty unrelated replies produce
one bogus outline interleaved with chatter; separately, the outline is the
comment with the most marks in it and the chatter scores zero.
"""
outlines = [parse(comment) for comment in comments]
outlines = [found for found in outlines if len(found) >= minimum]
if not outlines:
return []
richest = max(outlines, key=len)
log.info("took %d chapter marks from a comment", len(richest))
return richest
def hhmmss(seconds: float) -> str:
"""3932 -> '1:05:32'. What a timecode looks like to a reader."""
total = int(seconds)
hours, rest = divmod(total, 3600)
minutes, secs = divmod(rest, 60)
return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}"
# ------------------------------------------------------------------ internals
def _seconds(match: re.Match[str]) -> int:
a, b, c = match.group(1), match.group(2), match.group(3)
if c is None: # mm:ss
return int(a) * 60 + int(b)
return int(a) * 3600 + int(b) * 60 + int(c)
def _is_heading(text: str, links: list[Link]) -> bool:
"""A rubric like 'Policy & Safety', not a sentence and not a story.
Deliberately strict. Anything that gets through becomes the heading for
every chapter after it, so a false positive mislabels the rest of the
episode — where a false negative just leaves a few items unfiled.
"""
return (
not links
and len(text) <= 60
and len(text.split()) <= 6
and not text.endswith((".", "!", "?"))
)
def _text_lines(body: str) -> list[tuple[str, list[Link]]]:
"""Plain-text notes — a YouTube description. Bare URLs become links."""
lines = []
for raw in body.splitlines():
line = raw.strip()
if not line:
continue
links = [Link(url=u) for u in _URL.findall(line)]
lines.append((_URL.sub("", line).strip(), links))
return lines
def _html_lines(body: str) -> list[tuple[str, list[Link]]]:
parser = _Outline()
parser.feed(body)
parser.close()
return parser.lines
class _Outline(HTMLParser):
"""HTML to (line, links) pairs, without pulling in a parser library.
A full HTML tree would be nicer and is not worth a dependency for this:
the only structure that matters is where the line breaks fall and which
anchors landed on which line. `convert_charrefs` (on by default) means
entities arrive already decoded, so `Tools &amp; Apps` is `Tools & Apps`
by the time it reaches `handle_data`.
"""
def __init__(self) -> None:
super().__init__()
self.lines: list[tuple[str, list[Link]]] = []
self._text: list[str] = []
self._links: list[Link] = []
self._href = ""
self._anchor: list[str] = []
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if tag in _BLOCKS:
self._flush()
elif tag == "a":
self._href = dict(attrs).get("href") or ""
self._anchor = []
def handle_endtag(self, tag: str) -> None:
if tag in _BLOCKS:
self._flush()
elif tag == "a":
if self._href:
self._links.append(
Link(url=self._href, title=_squash("".join(self._anchor)))
)
self._href, self._anchor = "", []
def handle_data(self, data: str) -> None:
self._text.append(data)
if self._href:
self._anchor.append(data)
def _flush(self) -> None:
text = _squash("".join(self._text))
if text or self._links:
self.lines.append((text, self._links))
self._text, self._links = [], []
def close(self) -> None:
super().close()
self._flush()
def _squash(text: str) -> str:
return re.sub(r"\s+", " ", text or "").strip()
+113
View File
@@ -0,0 +1,113 @@
"""Where a video actually lives, and how to link into the middle of it.
Every segment newsdesk stores has to answer one question: *press play at
1:08:27 — where?* Today the answer is a YouTube URL. It will not always be.
The same episode may sit on disk, ripped or recorded; a show may take its
uploads down; a video may only ever have existed as a local file.
So nothing in this codebase stores a YouTube URL as a fact. It stores a
`MediaRef` — a kind and a key — and asks this module for a link when it needs
one. Swapping YouTube for a copy on disk is then a line of config rather than
an UPDATE across every item that mentioned it.
video:
local_media:
GUYEofBebDc: D:/media/lwiai/255.mp4
`resolve` applies that map. Everything downstream keeps working because it was
never holding a URL in the first place.
The one thing this deliberately does not do is *find* local copies. Matching
"is this file the same episode as that video" by filename or duration is a
guess, and a wrong guess points a citation at the wrong recording. The mapping
is written by a person, or it does not exist.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path, PurePath
#: A local file, playable in the reading pane's <video> element. Anything else
#: on disk (mkv, wav) still resolves; the pane just may not play it.
PLAYABLE = frozenset({".mp4", ".m4v", ".webm", ".ogg", ".mp3", ".m4a", ".wav"})
@dataclass(frozen=True, slots=True)
class MediaRef:
"""A video, identified by where it is rather than by a URL.
`kind` is what the key means: `youtube` is an 11-character video id,
`spotify` is an episode id, `file` is a path on this machine, `url` is
anything else somebody handed us. `key` is stable — it is what
`Segment.external_id` is built from — so re-pointing a video at a local
copy must *not* change it. That is why `resolve` returns a separate
playback ref instead of rewriting this one.
"""
kind: str
key: str
@property
def playable_locally(self) -> bool:
return self.kind == "file" and PurePath(self.key).suffix.lower() in PLAYABLE
def at(self, seconds: float = 0) -> str:
"""A link that opens this video at that moment.
The media fragment (`#t=`) is the same idea in every case: it is what
YouTube's `&t=` and Spotify's `?t=` do, what a `<video>` element
honours natively, and what most desktop players understand from a
`file://` URL.
"""
offset = max(0, int(seconds))
if self.kind == "youtube":
return f"https://www.youtube.com/watch?v={self.key}&t={offset}s"
if self.kind == "spotify":
return f"https://open.spotify.com/episode/{self.key}?t={offset}"
if self.kind == "file":
return Path(self.key).absolute().as_uri() + (f"#t={offset}" if offset else "")
return self.key + (f"#t={offset}" if offset else "")
def embed(self, seconds: float = 0) -> str:
"""The player a reading pane can frame at that moment, or "".
Only the hosts that publish a player built for framing have one: a
YouTube watch page and a Spotify episode page both refuse to be
framed, and each has an `/embed/` form that does not. A local file
needs none — the pane plays it in a `<video>` element — and an
arbitrary URL is not known to allow it.
"""
offset = max(0, int(seconds))
if self.kind == "youtube":
return f"https://www.youtube.com/embed/{self.key}?start={offset}"
if self.kind == "spotify":
return f"https://open.spotify.com/embed/episode/{self.key}?t={offset}"
return ""
def youtube(video_id: str) -> MediaRef:
return MediaRef(kind="youtube", key=video_id)
def local(path: str | Path) -> MediaRef:
return MediaRef(kind="file", key=str(path))
def resolve(ref: MediaRef, overrides: dict[str, str] | None = None) -> MediaRef:
"""The ref to *play*, after applying the local-media map.
Keyed on the original key, so a mapping written once keeps working however
the item was ingested. Returns the ref unchanged when there is no override
or the file has since gone — a stale mapping should fall back to the
original rather than hand out a link to a file that isn't there.
"""
if not overrides:
return ref
replacement = overrides.get(ref.key)
if not replacement:
return ref
path = Path(replacement).expanduser()
if not path.exists():
return ref
return local(path)
+261
View File
@@ -0,0 +1,261 @@
"""A video, and the pieces it breaks into.
This is the generic half and it knows nothing about podcasts, feeds or
newsdesk's tables. A `Video` is something with a transcript and a duration; a
`Segment` is a span of one, with a title, a start, an end and the words spoken
inside it. Cutting one into the other needs two inputs and no model:
Video --+-- transcript -> timed cues -----+
| +-> cut() -> Segment[]
+-- chapter marks (marks.best) ----+
What supplies those inputs is somebody else's problem, and deliberately so.
YouTube supplies both today. A show's own web page supplies better marks, with
the article links attached — that is what `desk/sources/podcast.py` does,
by passing its own marks in. A local recording plus a Whisper transcript would
supply both without touching this file.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass, field
from datetime import UTC, datetime
from typing import Any
from desk.models import ItemLink
from desk.tools.video import marks as marks_mod
from desk.tools.video import youtube
from desk.tools.video.marks import Chapter, Link
from desk.tools.video.media import MediaRef
log = logging.getLogger(__name__)
@dataclass(slots=True)
class Video:
"""One video, before anything has been done to it."""
ref: MediaRef
title: str = ""
#: The page it was published on, when that is a different thing from the
#: video itself — a Substack post, a show's episode page. Empty for a video
#: that is only a video.
url: str = ""
author: str = ""
published_at: datetime = field(default_factory=lambda: datetime.now(UTC))
duration: float = 0.0
#: Whatever text came with it and might carry chapter marks: a YouTube
#: description, a feed body, show notes as HTML.
notes: str = ""
@property
def key(self) -> str:
"""What identifies this video for the life of the archive.
The media *ref's* key, not a URL, and never the resolved playback
location — pointing an item at a local copy must not change its
identity or every segment of that video re-ingests as a new item.
"""
return self.ref.key
@dataclass(slots=True)
class Segment:
"""One span of a video: the unit that becomes a newsdesk item."""
video: Video
index: int
start: int
end: int
title: str
heading: str = "" # the rubric it was filed under
links: list[Link] = field(default_factory=list)
transcript: str = "" # verbatim, this span only
@property
def external_id(self) -> str:
"""Stable across re-ingests of the same video.
Keyed on the video and the start time rather than the position in the
list, so a show that inserts a chapter it forgot doesn't renumber —
and therefore doesn't duplicate — everything after it. Re-cutting
against corrected timestamps *does* produce new ids, which is the
honest outcome: they are different spans of the recording.
"""
return f"{self.video.key}#t={self.start}"
def url(self, overrides: dict[str, str] | None = None) -> str:
"""A link that plays this segment, wherever the video currently lives.
Resolved on the way out rather than stored, which is the whole point of
`media.MediaRef`: mapping a YouTube id to a file on disk changes every
link in the archive and rewrites nothing.
"""
from desk.tools.video import media
return media.resolve(self.video.ref, overrides).at(self.start)
@property
def duration(self) -> int:
return max(0, self.end - self.start)
@property
def timecode(self) -> str:
return marks_mod.hhmmss(self.start)
def record(self) -> dict[str, Any]:
"""The segment as a JSON-able dict: what newsdesk stores in `items.raw`.
One definition of the layout, so the two writers (a podcast segment,
a video added by hand) and the readers (the reading pane, the
summariser's query) agree on the keys. `media` is the reference and
never a URL — see `media.py`; `duration` is denormalised so a queue row
can show a length chip without arithmetic across two JSON extractions.
The articles are not in here: they are a typed collection of their own
now — see `reading`. `raw` is what no shared field has a name for, and
a link the whole reader understands is not that.
"""
return {
"media": {"kind": self.video.ref.kind, "key": self.video.ref.key},
"start": self.start,
"end": self.end,
"duration": self.duration,
"index": self.index,
}
def reading(self) -> list[ItemLink]:
"""What the hosts were reading, as links the whole app understands.
The outline's links with a role on them, which is the difference
between "a URL this show happened to print" and "the article this
stretch is about": the reader lists them under the player, the
summariser names them in the prompt, and neither has to know a podcast
produced them.
A line of show notes is whatever an editor typed, so a relative href
or a `mailto:` is an ordinary outcome rather than a fault — dropped
with a note, because one malformed link in the outline must not cost
the segment it sits on.
"""
out = []
for link in self.links:
try:
out.append(ItemLink(url=link.url, title=link.title, role="reading"))
except ValueError:
log.debug("%s: dropping unusable outline link %r",
self.external_id, link.url[:120])
return out
def cut(video: Video, cues: list[youtube.Cue],
chapters: list[Chapter] | None = None) -> list[Segment]:
"""Cut a video into one segment per chapter mark.
A chapter runs until the next one starts; the last one runs to the end of
the recording. That is the only boundary rule, and it is exact, because the
marks were written by the people who recorded it.
With no chapter marks at all the video comes back as a single segment
holding the whole transcript. That is deliberately not a failure: an
unsegmented video is still worth having, and inventing boundaries where
nobody published any is a job for a model and a later stage, not for a
slicing function.
"""
chapters = chapters if chapters is not None else marks_mod.parse(video.notes)
end_of_video = int(video.duration or (cues[-1].end if cues else 0))
if not chapters:
return [Segment(
video=video, index=0, start=0, end=end_of_video,
title=video.title, transcript=youtube.text_between(cues, 0),
)]
ordered = sorted(chapters, key=lambda c: c.start)
out = []
for index, chapter in enumerate(ordered):
following = (ordered[index + 1].start if index + 1 < len(ordered)
else end_of_video)
# A show that mistypes a timestamp can put one mark after the next.
# Clamping keeps the slice empty rather than reversed, and leaves the
# neighbouring segments intact.
end = max(chapter.start, following)
out.append(Segment(
video=video,
index=index,
start=chapter.start,
end=end,
title=chapter.title,
heading=chapter.heading,
links=chapter.links,
transcript=youtube.text_between(cues, chapter.start, end or None),
))
empty = sum(1 for s in out if not s.transcript)
if empty:
log.warning("%s: %d of %d segments have no transcript under them",
video.key, empty, len(out))
return out
# ------------------------------------------------------ transcript as text
_SENTENCE_END = re.compile(r"(?<=[.!?])\s+")
def opening(transcript: str, sentences: int = 2, cap: int = 400) -> str:
"""The first sentence or two, as a stand-in for a summary that isn't
written yet. An empty `text` is an empty row in the queue."""
if not transcript:
return ""
parts = _SENTENCE_END.split(transcript.strip())
return " ".join(parts[:sentences])[:cap]
def paragraphs(text: str, target: int = 700) -> str:
"""Break a captions stream into paragraphs a reading pane can render.
Captions arrive as one unbroken run, and the reader splits a body on blank
lines — so without this a forty-minute talk is one paragraph. Sentences
are grouped until roughly `target` characters, then a break: arbitrary,
but readable, and the alternative is a wall.
Auto-generated captions often have no punctuation at all, so one
"sentence" can be the whole talk. Those are broken on a word boundary
instead — worse than a real sentence end, better than the wall.
"""
text = " ".join((text or "").split())
if not text:
return ""
out: list[str] = []
current: list[str] = []
length = 0
for sentence in _SENTENCE_END.split(text):
for piece in _words(sentence, target * 2):
current.append(piece)
length += len(piece) + 1
if length >= target:
out.append(" ".join(current))
current, length = [], 0
if current:
out.append(" ".join(current))
return "\n\n".join(out)
def _words(run: str, limit: int) -> list[str]:
"""Split an over-long run at spaces, keeping every piece under `limit`."""
if len(run) <= limit:
return [run]
pieces: list[str] = []
current: list[str] = []
length = 0
for word in run.split(" "):
if current and length + len(word) + 1 > limit:
pieces.append(" ".join(current))
current, length = [], 0
current.append(word)
length += len(word) + 1
if current:
pieces.append(" ".join(current))
return pieces
+600
View File
@@ -0,0 +1,600 @@
"""Captions, with their timings, for one YouTube video.
YouTube has always published a machine-readable transcript for anything with
captions — including the auto-generated ones, which is most podcasts. It is the
cheapest transcript in existence: somebody else already paid for the ASR.
Getting at it is the fiddly part, and it is fiddly for a reason worth writing
down. The caption URLs embedded in a `watch` page are signed for a *browser*
session, and fetching one from anywhere else now returns `200 OK` with an empty
body — no error, no redirect, nothing to catch. So the obvious implementation
(scrape the watch page, follow `baseUrl`) fails silently, which is the worst
way for an ingest step to fail.
What works is asking the same endpoint the mobile app asks. `/youtubei/v1/player`
with a mobile client context hands back a player response whose caption URLs are
not tied to a browser session, and those fetch fine. That is the whole trick,
and it is why this module talks to an undocumented endpoint instead of scraping
a page: the scrape is the fragile one.
Two clients are tried in order, because they fail differently — `IOS` returns
JSON3 when asked and `ANDROID` returns SRV3 whatever you ask for, so both
formats are parsed. If Google breaks both, the fallbacks are `yt-dlp`, which
tracks this arms race full-time, and transcribing the audio yourself. Neither
is in here — `__init__.py` says where they would go. Until one of them is
written, a break in both clients raises `TranscriptUnavailable` rather than
storing an empty transcript: loudly, which is the whole point of the paragraph
above.
**A refusal is not an absence.** YouTube gates by address: from a VPN exit or
a datacentre range a client is answered `LOGIN_REQUIRED` ("Sign in to confirm
you're not a bot") for every video, captioned or not. Measured on 2026-09-21,
one machine, one video, seven clients, each asked bare and again with the
`visitorData` a watch page hands out:
VPN on `IOS`, `ANDROID`, `WEB`, `MWEB`, `TVHTML5` and `ANDROID_VR` all
get the bot check, and `WEB_EMBEDDED_PLAYER` an error. No tracks
from any of them, and `visitorData` changes none of it.
`youtube-transcript-api` raises `RequestBlocked`.
VPN off `IOS` and `ANDROID` return tracks that fetch. The rest are no
use even here: `WEB`, `MWEB` and `WEB_EMBEDDED_PLAYER` are
unplayable without a PO token, and `TVHTML5` and `ANDROID_VR`
are answered with the bot check from a clean address too.
So there is no anonymous client that gets past the gate and none to add: the
fix is on the network. That case raises `Blocked`, a `TranscriptUnavailable` a
caller can tell apart, because "wait for the captions" is the wrong answer to
it and would be given for ever.
"""
from __future__ import annotations
import html
import json
import logging
import re
import xml.etree.ElementTree as ET
from collections.abc import Iterator
from dataclasses import dataclass
from typing import Any
from urllib.parse import parse_qs, urlsplit
import httpx
from desk.tools.http import session
log = logging.getLogger(__name__)
INNERTUBE = "https://www.youtube.com/youtubei/v1/player?prettyPrint=false"
#: Only for the watch page, which is read for comments. A caption fetch
#: deliberately identifies as a mobile app instead — see CLIENTS.
BROWSER_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
#: Tried in order. Each is (client context, the User-Agent that goes with it) —
#: they have to match, or the response comes back playable but captionless.
CLIENTS = (
(
{"clientName": "IOS", "clientVersion": "20.10.4",
"deviceModel": "iPhone16,2", "hl": "en"},
"com.google.ios.youtube/20.10.4 (iPhone16,2; U; CPU iOS 18_3_2 like Mac OS X)",
),
(
{"clientName": "ANDROID", "clientVersion": "20.10.38",
"androidSdkVersion": 30, "hl": "en"},
"com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip",
),
)
_VIDEO_ID = re.compile(r"^[\w-]{11}$")
#: Every shape a YouTube link comes in, including the one Substack embeds
#: (`<div class="youtube-wrap" id="youtube2-VIDEOID">`).
_IN_TEXT = re.compile(
r"(?:youtu\.be/|youtube\.com/(?:watch\?(?:[^\"'\s]*&)?v=|embed/|live/|shorts/)"
r"|youtube2-)([\w-]{11})"
)
class TranscriptUnavailable(Exception):
"""No captions to be had — yet, or at all.
Raised rather than returning `[]` because the two cases an ingest has to
tell apart both look like an empty list: a video with no captions, and a
video whose captions are still being generated. The message says which,
and the caller decides whether to retry on the next pull.
"""
class Blocked(TranscriptUnavailable):
"""YouTube refused this machine, whatever the video has.
A subclass, so a caller that only wants "no captions, carry on" — a link
added by hand, a repair — is unchanged. A caller that waits for captions
has to catch this first: a refusal is not lifted by waiting, it answers
every video the same way, and retrying it quietly is how a show stores
nothing for a fortnight with nobody told. See `_refused`.
"""
@dataclass(frozen=True, slots=True)
class Cue:
"""One caption line: when it starts, how long it lasts, what was said."""
start: float
duration: float
text: str
@property
def end(self) -> float:
return self.start + self.duration
def video_id(value: str) -> str:
"""The 11-character id, from a bare id, any YouTube URL, or a blob of HTML.
Deliberately tolerant about its input: the id arrives from three different
places in this pipeline — typed by hand, in a feed entry's link, and buried
in the HTML of a Substack post — and each of them would otherwise need its
own extraction.
"""
value = (value or "").strip()
if _VIDEO_ID.match(value):
return value
if match := _IN_TEXT.search(value):
return match.group(1)
# A URL shape the pattern above doesn't cover, but with ?v= in the query.
found = parse_qs(urlsplit(value).query).get("v")
if found and _VIDEO_ID.match(found[0]):
return found[0]
raise ValueError(f"no YouTube video id in {value[:120]!r}")
def transcript(video: str, *, language: str = "en",
client: httpx.Client | None = None) -> list[Cue]:
"""Every caption cue for one video, in order.
`language` is matched as a prefix, so "en" accepts "en-US" — which matters,
because a channel that uploads a real transcript files it under a regional
code while the auto-generated one is plain "en".
Two backends, tried in that order. `youtube-transcript-api` first — it is
a dependency, and it is somebody's full-time job to keep up with YouTube's
changes. The built-in fetcher below when the library can't get the track,
so one broken path doesn't cost the transcript.
"""
cues = _via_library(video, language)
if cues is not None:
return cues
return _via_innertube(video, language, client)
#: Seconds one `requests` call inside `youtube-transcript-api` gets. Roughly
#: what `desk/tools/http.py` gives an httpx call, for the same reason.
LIBRARY_TIMEOUT = 30
def _timed_client() -> Any:
"""A `requests` session with a deadline on it, for a library that owns its HTTP.
`youtube-transcript-api` does its own fetching, on `requests`, and a
`requests` call with no `timeout=` waits on a socket read for ever. What
that costs is not a slow transcript. The work runs in a worker thread;
cancellation is cooperative, so a thread parked in a read is one the Kill
button cannot reach, and a restart re-queues the job to park on the same
video. The library takes a session of ours through `http_client`, which is
its own supported way of being told this; a `requests.Session` has no
timeout of its own to set, so the default goes on one method down.
**This is why there is no `socket.setdefaulttimeout` anywhere.** A
process-wide default is the tempting one-liner for library code that hides
its HTTP, and it is the wrong trade here: it would be inherited by every
socket the process opens, uvicorn's listener and each websocket included,
to buy a deadline for call sites that can each be told one directly. There
were two. The other was feedparser, and it is handled the same way — by
fetching where a timeout can be set; see `desk/sources/rss.py`.
Typed as `Any` because `requests` is only imported inside the guard below.
It is a `requests.Session`.
"""
import requests
class Timed(requests.Session):
def request(self, *args: Any, **kwargs: Any) -> Any:
kwargs.setdefault("timeout", LIBRARY_TIMEOUT)
return super().request(*args, **kwargs)
return Timed()
def _via_library(video: str, language: str) -> list[Cue] | None:
"""`youtube-transcript-api`. None means "not this way".
Every failure is None rather than an exception, including a video with no
captions at all: the fallback below raises the error that gets reported, so
there is one message for "no transcript" instead of two that depend on
which packages happen to be present.
"""
try:
from youtube_transcript_api import YouTubeTranscriptApi
client = _timed_client()
except ImportError:
return None
try:
tracks = YouTubeTranscriptApi(http_client=client).list(video)
wanted = [language, f"{language}-US", f"{language}-GB"]
try:
track = tracks.find_manually_created_transcript(wanted)
except Exception:
track = tracks.find_generated_transcript(wanted)
fetched = track.fetch()
except Exception as exc:
log.debug("youtube-transcript-api could not fetch %s: %s", video, exc)
return None
cues = [
Cue(start=c.start, duration=c.duration, text=clean)
for c in fetched
if (clean := _clean(c.text))
]
log.info("%s: %d cues via youtube-transcript-api", video, len(cues))
return cues or None
def _via_innertube(video: str, language: str,
client: httpx.Client | None) -> list[Cue]:
with session(client) as http:
player = _player(video, http)
track = _pick(_tracks(player), language)
if track is None:
raise TranscriptUnavailable(
f"{video}: no {language} captions. "
"Auto-captions can take a few hours to appear on a long upload."
)
# `&fmt=json3` is a request, not a promise: the ANDROID client answers
# it with SRV3 anyway. Both are parsed, so it doesn't matter which
# arrives — see `_cues`.
response = http.get(track["baseUrl"] + "&fmt=json3",
headers={"User-Agent": player.get("_caption_agent", BROWSER_UA)})
if _throttled(response):
raise Blocked(_refusal(video, [f"captions: HTTP {response.status_code}"]))
response.raise_for_status()
body = response.text
if not body.strip():
raise TranscriptUnavailable(
f"{video}: caption track fetched empty. YouTube does this when "
"the request doesn't look like the client that asked for it."
)
cues = _cues(body)
log.info("%s: %d cues, %.0f minutes", video, len(cues),
(cues[-1].end / 60) if cues else 0)
return cues
def details(video: str, *, client: httpx.Client | None = None) -> dict[str, Any]:
"""Title, author, length and description for a video.
The description is here because for a show that publishes no newsletter
it is the only place the chapter marks live — `marks.parse` reads it as
plain text. Everything else is the metadata an episode needs when YouTube
is the only source there is.
"""
with session(client) as http:
player = _player(video, http, require_captions=False)
video_details = player.get("videoDetails", {})
return {
"title": video_details.get("title", ""),
"author": video_details.get("author", ""),
"duration": float(video_details.get("lengthSeconds") or 0),
"description": video_details.get("shortDescription", ""),
}
def comments(video: str, *, limit: int = 100,
client: httpx.Client | None = None) -> list[str]:
"""Top-level comments, newest-and-most-liked first, as plain text.
Here for one reason: when a show does not publish an outline, somebody in
the comments usually has. A viewer typing out "0:00 intro / 3:32 Gemini /
13:11 Grok" is doing the same work the show notes would have done, and
YouTube itself treats such a comment as authoritative enough to draw
chapter markers from it. `marks.parse` reads these exactly as it reads a
description — see `marks.best`.
Returns `[]` rather than raising when comments are off, or when YouTube
changes the shape again. A missing fallback is not worth failing an ingest
that already has a transcript.
"""
try:
with session(client) as http:
page = http.get(f"https://www.youtube.com/watch?v={video}",
headers={"User-Agent": BROWSER_UA,
"Accept-Language": "en-US,en"}).text
context = _json_after(page, '"INNERTUBE_CONTEXT":')
initial = _json_after(page, "ytInitialData = ")
if not context or not initial:
return []
out: list[str] = []
# Several continuations live in the page (the related rail, the
# description); only one of them is the comment feed, and which
# index that is moves around. Trying them in order and keeping the
# first that yields comments is cheaper than pinning a path
# through a structure Google reshuffles.
for token in _continuations(initial)[:4]:
response = http.post(
"https://www.youtube.com/youtubei/v1/next?prettyPrint=false",
json={"context": context, "continuation": token},
headers={"Content-Type": "application/json",
"User-Agent": BROWSER_UA},
)
out = _comment_texts(response.json())
if out:
break
except Exception as exc:
log.debug("could not read comments for %s: %s", video, exc)
return []
log.info("%s: %d comments read", video, len(out))
return out[:limit]
def text_between(cues: list[Cue], start: float, end: float | None = None) -> str:
"""The words spoken in a span, as one paragraph.
A cue counts as inside the span when it *starts* inside it. Overlap would
be the other defensible rule, but it double-counts the sentence that
straddles a boundary — and these boundaries come from chapter marks a human
typed, so a sentence of drift at the seam is already there and doesn't need
doubling.
"""
return " ".join(
cue.text for cue in cues
if cue.start >= start and (end is None or cue.start < end)
).strip()
# ------------------------------------------------- internals: the watch page
def _json_after(page: str, marker: str) -> dict[str, Any] | None:
"""The JSON object that follows `marker` in a watch page.
Brace-matched rather than matched with a regex. The two blobs this reads —
`ytInitialData` and `INNERTUBE_CONTEXT` — are hundreds of kilobytes of
nested JSON containing every bracket character inside string values, and a
non-greedy regex stops at the first `}` inside a caption or a title. This
walks the string properly, which is about fifteen lines and always right.
"""
start = page.find(marker)
if start < 0:
return None
start = page.find("{", start + len(marker))
if start < 0:
return None
depth, in_string, escaped = 0, False, False
for index in range(start, len(page)):
char = page[index]
if in_string:
if escaped:
escaped = False
elif char == "\\":
escaped = True
elif char == '"':
in_string = False
continue
if char == '"':
in_string = True
elif char == "{":
depth += 1
elif char == "}":
depth -= 1
if depth == 0:
try:
return json.loads(page[start:index + 1])
except ValueError:
return None
return None
def _walk(node: Any, key: str) -> Iterator[Any]:
"""Every value stored under `key`, anywhere in a nested structure."""
if isinstance(node, dict):
if key in node:
yield node[key]
for value in node.values():
yield from _walk(value, key)
elif isinstance(node, list):
for value in node:
yield from _walk(value, key)
def _continuations(initial: dict[str, Any]) -> list[str]:
return [
token for renderer in _walk(initial, "continuationItemRenderer")
if (token := (renderer.get("continuationEndpoint", {})
.get("continuationCommand", {}) or {}).get("token"))
]
def _comment_texts(response: dict[str, Any]) -> list[str]:
"""Comment bodies out of a `next` response.
Comments arrive as `commentEntityPayload` entities in a flat batch beside
the renderers that position them, so reading them needs no understanding of
the layout at all — which is the part that keeps changing.
"""
out = []
for payload in _walk(response, "commentEntityPayload"):
text = (payload.get("properties", {}).get("content", {}) or {}).get("content", "")
if text:
out.append(text)
return out
# ------------------------------------------------------------------ internals
def _player(video: str, http: httpx.Client,
*, require_captions: bool = True) -> dict[str, Any]:
"""The player response, from whichever client answers with captions.
`require_captions` is what separates the two callers. A transcript wants
the first client that actually offers caption tracks; metadata wants the
first that answers at all, and would otherwise fail on a video that plays
fine and simply has no captions yet.
Every client's answer is kept for the error, not only the last one's: the
two fail differently, and which of them said what is the first thing
anyone asks when this breaks.
"""
said: list[str] = []
refused = False
for context, agent in CLIENTS:
name = context["clientName"]
payload = {"context": {"client": context}, "videoId": video,
"contentCheckOk": True, "racyCheckOk": True}
try:
response = http.post(
INNERTUBE, json=payload,
headers={"Content-Type": "application/json", "User-Agent": agent},
)
if _throttled(response):
refused = True
said.append(f"{name}: HTTP {response.status_code}")
continue
data = response.json()
except (httpx.HTTPError, ValueError) as exc:
said.append(f"{name}: {exc}")
continue
playability = data.get("playabilityStatus", {})
status = playability.get("status", "?")
if _tracks(data) or (not require_captions and status == "OK"):
# Caption URLs are bound to the client that obtained the player.
data["_caption_agent"] = agent
return data
refused = refused or _refused(playability)
reason = playability.get("reason") or ""
said.append(f"{name}: {status}" + (f" ({reason})" if reason else "")
+ ", no caption tracks")
log.debug("innertube %s", said[-1])
if refused:
raise Blocked(_refusal(video, said))
raise TranscriptUnavailable(
f"{video}: {'; '.join(said) or 'no client returned captions'}")
def _refused(playability: dict[str, Any]) -> bool:
"""Whether a player response is YouTube declining to serve this machine.
The status alone, not the reason beside it. The bot check reads "Sign in
to confirm you're not a bot" today, and a match on that sentence would go
back to reporting a refusal as captions not being ready on the day it is
reworded — silently, which is the failure this exists to end. The status
is the stable half, and it is sufficient: nothing asked anonymously is
ever let past `LOGIN_REQUIRED` by being asked again later.
That also takes in a private or age-gated video, where the refusal is
about the video rather than the address. It is still not a wait, and the
reason is quoted in the error for whoever reads it.
"""
return playability.get("status") == "LOGIN_REQUIRED"
def _throttled(response: httpx.Response) -> bool:
"""The same refusal, made in HTTP: a 403, a 429, or Google's `/sorry/`
interstitial, which arrives as a redirect to a captcha page."""
return response.status_code in (403, 429) or "/sorry/" in response.url.path
def _refusal(video: str, said: list[str]) -> str:
"""`Blocked`'s sentence. "YouTube refused" is what `desk/notices.py` reads
to tell this from a feed that failed, so the wording is load-bearing."""
return (f"{video}: YouTube refused this address and wants a sign-in — "
+ "; ".join(said))
def _tracks(player: dict[str, Any]) -> list[dict[str, Any]]:
return (player.get("captions", {})
.get("playerCaptionsTracklistRenderer", {})
.get("captionTracks", []) or [])
def _pick(tracks: list[dict[str, Any]], language: str) -> dict[str, Any] | None:
"""Best track for a language: a real transcript over an auto-generated one.
A channel that uploads its own captions has punctuation, speaker turns and
correctly spelled names; ASR has none of those and spells "Qwen" six ways.
Worth preferring, and never worth requiring — most shows only have ASR.
"""
wanted = [t for t in tracks
if (t.get("languageCode") or "").lower().startswith(language.lower())]
wanted.sort(key=lambda t: t.get("kind") == "asr")
return wanted[0] if wanted else None
def _cues(body: str) -> list[Cue]:
"""Parse whichever of YouTube's three caption formats turned up."""
body = body.strip()
if body.startswith("{"):
return _from_json3(json.loads(body))
root = ET.fromstring(body)
# SRV1 is `transcript > text[start,dur]`; SRV3 is `timedtext > body > p[t,d]`.
if root.tag == "transcript":
return _from_srv1(root)
return _from_srv3(root)
def _from_json3(data: dict[str, Any]) -> list[Cue]:
cues = []
for event in data.get("events", []):
text = "".join(seg.get("utf8", "") for seg in event.get("segs", []))
clean = _clean(text)
if not clean:
continue
cues.append(Cue(
start=event.get("tStartMs", 0) / 1000,
duration=event.get("dDurationMs", 0) / 1000,
text=clean,
))
return cues
def _from_srv1(root: ET.Element) -> list[Cue]:
cues = []
for node in root.iter("text"):
clean = _clean(node.text or "")
if not clean:
continue
cues.append(Cue(
start=float(node.get("start", 0)),
duration=float(node.get("dur", 0)),
text=clean,
))
return cues
def _from_srv3(root: ET.Element) -> list[Cue]:
cues = []
for node in root.iter("p"):
clean = _clean("".join(node.itertext()))
if not clean:
continue
cues.append(Cue(
start=int(node.get("t", 0)) / 1000,
duration=int(node.get("d", 0)) / 1000,
text=clean,
))
return cues
def _clean(text: str) -> str:
"""Entities out, whitespace collapsed, sound cues dropped.
Unescaped twice on purpose: SRV1 sends `&amp;#39;` — an entity for an
entity — because the caption text was escaped once when it was stored and
once again on the way into the XML.
"""
text = html.unescape(html.unescape(text or ""))
text = re.sub(r"\[(?:Music|Applause|Laughter)\]", " ", text)
return re.sub(r"\s+", " ", text).strip()
+128
View File
@@ -0,0 +1,128 @@
"""Did we fetch the article, or the page the outlet shows instead of it?
Some sites answer every article URL with the same thing: a consent gate, an
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
faithfully — it is real text on a real page — and we store it as the body. That
is worse than storing nothing, because the reader prefers the body to the feed's
summary, so a wall doesn't merely fail to add anything: it *hides* the two
readable sentences the feed already gave us.
The tell is repetition. Different articles cannot have the same body; a wall
always does. That is the whole detector — no keyword list, no per-site rules,
nothing to keep up to date. It learns golem.de's German consent page and a
Cloudflare challenge with the same line of code, and it costs no extra
requests, because the evidence is a by-product of a fetch we were making
anyway.
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
requests to one host earns a throttle or challenge page, that page is served to
whichever two articles asked next, and the two fingerprint the same. Refetching
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
was about our request rate and not about the outlet. A wall repeats on every
article, so a third body costs one extra fetch per walled host, once; a throttle
page that repeats three consecutive times is the site answering nothing else.
The verdict is per host, because the wall is: the site serves it, it isn't
chosen per article. Once a host is known to answer with one, we stop fetching
from it altogether — which makes pulls faster, not just more correct. That is
also why the count matters: `known` short-circuits the request, so a wall
verdict is the last evidence a host ever gets to give. Nothing can disprove it
except `forget`.
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
in a test), because a tool has no store of its own to remember things in. The
one way this is wrong is an outlet that really does publish the same body
`STRIKES` times running; the cost is that its articles fall back to their feed
summaries until someone calls `forget`, which is what the re-check button in
the Sources tab is for.
"""
from __future__ import annotations
import hashlib
import logging
from collections.abc import Mapping, MutableMapping
from desk.tools.framing import host_of
log = logging.getLogger(__name__)
#: Sits alongside `embed:` in the same table. The two verdicts answer different
#: questions about a host — will it let us frame it, will it give up its text —
#: and neither implies the other.
CACHE_PREFIX = "body:"
_WALL = "wall:"
REASON = "every article comes back as the same page"
#: How many consecutive identical bodies settle it. See the module docstring:
#: two was one throttle page away from retiring a working outlet for good.
STRIKES = 3
def _fingerprint(body: str) -> str:
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
def _seen(value: str) -> tuple[str, int]:
"""`(fingerprint, how many in a row)` from a stored value.
A bare fingerprint is one sighting, which is what every row written before
`STRIKES` existed holds — so an upgraded database is read correctly and
needs no migration of its own.
"""
mark, _, run = value.partition("*")
return mark, (int(run) if run.isdigit() else 1)
def known(cache: Mapping[str, str], url: str) -> str:
"""Why this host's bodies aren't worth fetching, or "" if they are.
Asked before the request, so a settled verdict costs nothing at all. Takes
a read-only mapping, because this one only ever reads — which lets the
Sources tab hand over the snapshot it already loaded rather than the live
table, and says in the signature that opening a panel changes nothing.
"""
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
"""Record a body we just extracted. Returns a reason if it proves a wall.
Only the previous body from the host is kept and how many articles running
have handed it over — not every body ever seen. A wall repeats on the
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
and one key per host beats a growing set per host.
One distinct body resets the run to one. That is what makes the count mean
"in a row" rather than "ever", and it is the whole difference between a
site that answers with a gate and a site that answered twice from a cache.
"""
host = host_of(url)
if not host or not body.strip():
return ""
key = CACHE_PREFIX + host
stored = cache.get(key, "")
if stored.startswith(_WALL):
return stored[len(_WALL):]
mark = _fingerprint(body)
seen, run = _seen(stored)
run = run + 1 if seen == mark else 1
if run >= STRIKES:
cache[key] = _WALL + REASON
log.info("%s serves a wall, not articles: %s (%d in a row)",
host, REASON, run)
return REASON
cache[key] = mark if run == 1 else f"{mark}*{run}"
return ""
def forget(cache: MutableMapping[str, str], url: str) -> None:
"""Drop what we learned about a host, so the next fetch decides again.
Outlets change their minds — a wall goes up for a quarter and comes down
again — and a verdict with no way back would be a permanent decision made
on two HTTP responses.
"""
cache.pop(CACHE_PREFIX + host_of(url), None)