The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9.6 KiB
desk.tools — the machinery
What it is
Things a pipeline stage picks up and puts down: plain inputs, plain outputs, no idea what the app is for. The test for belonging here is would this make sense in a different application.
| Tool | What it does | Needs |
|---|---|---|
agent |
ask Claude, get JSON back | anthropic or claude-agent-sdk (both optional) |
article |
supplied HTML to typed structured content, preserving tables and code; derived plain text and acquisition provenance | trafilatura |
browser |
a page rendered in headless Chromium, past a consent dialog it knows; the cookies it earned, per host | playwright (optional, plus playwright install chromium) |
embedding |
text in, vector out (~384 floats) | sentence-transformers (optional, ~100MB+) |
failures |
what went wrong, in one word a retry policy can be written against | stdlib |
framing |
will this page load in an iframe? | httpx |
http |
one shared HTTP client per call chain | httpx |
marks |
what kind of thing is this, when it isn't an ordinary article | stdlib |
posts |
a Mastodon or Bluesky post and its author's thread, through the public API, as a page to convert; registered by those plugins | httpx |
checks |
is a fetched page worth converting, and if not, which getter to try next | stdlib |
scrape |
fetch, check, convert, and fetch another way when the check says so | trafilatura |
walls |
did we fetch the article, or the consent page in front of it | stdlib |
video/ |
a recording, read as text with timings | httpx, youtube-transcript-api |
Nothing is imported in desk/tools/__init__.py on purpose — each import is paid for by the caller that wants it. Import the module, not the package: from desk.tools import agent.
agent — the model
Two interchangeable backends behind one call. agent.backend(cfg) says which will answer, or "" for nobody — which is a failure, not a third backend.
api— one Claude API call. WantsANTHROPIC_API_KEY. The unattended path.claude_code— the Agent SDK drives theclaudeCLI on your subscription. No key, nothing billed. The development path. (Spawns a process.)off/ nothing available —backend()answers"", and the caller fails the batch with a notice naming the reason. Nothing waits to be answered by hand.
from desk.tools import agent
reply = agent.ask_json(
config.agent, # desk.config.AgentConfig
system="You tag articles.", # kept byte-identical across a run — it is what caches
question={"articles": [...]},
schema={"type": "object", ...}, # both backends take the same JSON schema
model="claude-haiku-4-5-20251001",
thinking="off", # or "adaptive"
on_partial=lambda piece: ..., # streamed text, for per-item progress
)
reply.data, reply.backend, reply.model, reply.cost_usd, reply.usage
Failures of every kind raise AgentError — treat it as "no answer this time". The claude_code path deliberately gets no tools, no CLAUDE.md, no skills, no MCP, no session: one text-in, JSON-out turn, the same shape as the API call.
The others, in one line each
article.extract_article(html, source_url) returns ArticleContent or None when extraction finds nothing. structured-v1 is a typed tree, not trusted HTML; plain supports legacy strings. plain_text and revision are derived from content. Explicit gutter nodes are removed before extraction, while meaningful numeric lines and code whitespace remain. Completeness defaults to unknown. Newsdesk's body-fetch job now uses this tool and atomically persists its content and derived plain projection; the reader renders its closed node vocabulary. Fetching, persistence, wall detection and rendering remain the caller's responsibilities. The extractor requires trafilatura 2.2 or newer, verified against the representative fixture.
from desk.tools import embedding, failures, framing, marks, video, walls
from desk.tools.http import session
vecs = embedding.load("all-MiniLM-L6-v2").encode(["some text"]) # (n, dim) ndarray
verdict = framing.check(cache, "https://x.com/a") # {embeddable, reason, host, certain}; per host
item = marks.mark(item, {"paywalled": "(g+)"}) # cuts the token out, records the mark
reason = walls.note(cache, url, body) # "" or why this body is the same page every time
why = failures.of_status(503) # 'server_error'; of_exception(exc) for a raise
wait = failures.hint(resp.headers) # the server's own Retry-After, in seconds
vid, segments = video.read("https://youtu.be/GUYEofBebDc") # Video + Segment[]
with session(client) as http: ... # borrows one if given, makes and closes one if not
failures is the third of those, and the newest. A category answers one question — would asking again get a different answer? — so TRANSIENT (timeout, unreachable, rate_limited, server_error, not_ready) and SETTLED (not_found, denied, consent_wall, js_required, empty) are the whole of the vocabulary, plus unknown for a failure nothing recognises, which is deliberately in neither. Those names are what a plugin's RetryPolicy.on is validated against, so adding one here adds one to the plugin contract. An exception is named by walking the class names in its ancestry rather than by isinstance, because httpx and urllib3 are both optional and either may be the one that raised — it costs precision and buys a tool that can name a failure raised by a library it has never imported. A consent wall is a category and is in no default retry list: it is the same answer however many times it is asked for.
marks and walls are the two that read what came back rather than fetching anything, which is why neither needs a dependency. marks turns an outlet's own notation — Golem's (g+), Hacker News's [$] — into one glyph that means the same thing whoever published it, and cuts the token out of the title. walls catches the opposite problem: a site that answers every article URL with its consent gate, which trafilatura extracts faithfully and which is worse than storing nothing. The tell is that two different articles cannot have the same body and two walls always do, so the detector is a fingerprint and a cache, with no per-site rules to keep up to date.
python -m desk.tools.video <url> prints a video's segments — a dry run with no app around it.
scrape and checks — getting the article out of a URL
scrape.scrape(url, getters) fetches with http, runs checks.check on the page, converts a page that passed with article.extract_article, and on a failed check tries the getters the Verdict names. It returns Scraped: the content or the last failure, the getter that answered, every Attempt with the signature that failed it, and the server's Retry-After.
from desk.tools import checks, scrape
got = scrape.scrape(url) # STANDARD: http only
got = scrape.scrape(url, scrape.ADVANCED) # a verdict may send it to another getter
got = scrape.scrape(url, scrape.advanced(cache, getters)) # what the body fetch uses: plugins' URL getters first,
# and HTTP sends the cookies a browser earned
verdict = checks.check(checks.Page(200, html)) # None, or Verdict(category, name, next)
browser.render(url) launches Chromium, clicks the accept button of a dialog in browser.CONSENTS if one appears within GATE_WAIT seconds, and returns Rendered: status, HTML, final URL, headers, the host's cookies and which consent it accepted. remember(cache, url, cookies), cookie_header(cache, url) and forget(cache, url) are the jar, in the same kind of mapping walls uses. One browser per call and one at a time per process; browser.installed() says whether the extra is there, and ADVANCED has browser only if it is.
How the loop runs, the signature format and how to add one: docs/architecture.md §2.1.
Limits
- All synchronous and blocking. Meant to be called from a worker thread.
ask_jsonis one turn, no tools, no conversation. No streaming API for the caller beyondon_partial(raw text pieces, not parsed JSON); chunking, retries and applying the answer aredesk.modeltask's job.- Optional deps really are optional, but "optional" is not "interchangeable". A missing model SDK is an
AgentError, and there are two backends, so the other one may well answer. A missingsentence-transformersis anImportErrorand there is nothing behind it:ranking.backend: embeddingsdoes not fall back to tfidf. The embed step logs the failure and returns zero, articles keep no vectors, and everything scores a neutral 0.5 for taste and ranks on its recency alone. Which reads a lot like a ranker that is working and is not. cost_usdcomes from theclaude_codepath, not the API one. Backwards from what you would guess: the CLI'sResultMessagecarries what those tokens would have cost, so it is reported even though a subscription bills nothing for them, and the API path — the one that really is billed — leaves the field at zero because the Messages response does not price itself. Read it as an estimate of what a run would cost on the API, and never as a bill.videois YouTube-only today. No local-file transcription yet (yt-dlp + whisper is the planned third backend); a video with no captions still yields chapters, just no words.framinganswers per host, not per article, and caches; most newspapers answer "no".- No tool knows about the store, the queue, or config beyond its own section — and
toolsmust not importpipelineorsources.