Files
jobboard/desk/tools/tools.md
T
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

9.6 KiB

desk.tools — the machinery

What it is

Things a pipeline stage picks up and puts down: plain inputs, plain outputs, no idea what the app is for. The test for belonging here is would this make sense in a different application.

Tool What it does Needs
agent ask Claude, get JSON back anthropic or claude-agent-sdk (both optional)
article supplied HTML to typed structured content, preserving tables and code; derived plain text and acquisition provenance trafilatura
browser a page rendered in headless Chromium, past a consent dialog it knows; the cookies it earned, per host playwright (optional, plus playwright install chromium)
embedding text in, vector out (~384 floats) sentence-transformers (optional, ~100MB+)
failures what went wrong, in one word a retry policy can be written against stdlib
framing will this page load in an iframe? httpx
http one shared HTTP client per call chain httpx
marks what kind of thing is this, when it isn't an ordinary article stdlib
posts a Mastodon or Bluesky post and its author's thread, through the public API, as a page to convert; registered by those plugins httpx
checks is a fetched page worth converting, and if not, which getter to try next stdlib
scrape fetch, check, convert, and fetch another way when the check says so trafilatura
walls did we fetch the article, or the consent page in front of it stdlib
video/ a recording, read as text with timings httpx, youtube-transcript-api

Nothing is imported in desk/tools/__init__.py on purpose — each import is paid for by the caller that wants it. Import the module, not the package: from desk.tools import agent.

agent — the model

Two interchangeable backends behind one call. agent.backend(cfg) says which will answer, or "" for nobody — which is a failure, not a third backend.

  • api — one Claude API call. Wants ANTHROPIC_API_KEY. The unattended path.
  • claude_code — the Agent SDK drives the claude CLI on your subscription. No key, nothing billed. The development path. (Spawns a process.)
  • off / nothing available — backend() answers "", and the caller fails the batch with a notice naming the reason. Nothing waits to be answered by hand.
from desk.tools import agent

reply = agent.ask_json(
    config.agent,                       # desk.config.AgentConfig
    system="You tag articles.",         # kept byte-identical across a run — it is what caches
    question={"articles": [...]},
    schema={"type": "object", ...},     # both backends take the same JSON schema
    model="claude-haiku-4-5-20251001",
    thinking="off",                     # or "adaptive"
    on_partial=lambda piece: ...,       # streamed text, for per-item progress
)
reply.data, reply.backend, reply.model, reply.cost_usd, reply.usage

Failures of every kind raise AgentError — treat it as "no answer this time". The claude_code path deliberately gets no tools, no CLAUDE.md, no skills, no MCP, no session: one text-in, JSON-out turn, the same shape as the API call.

The others, in one line each

article.extract_article(html, source_url) returns ArticleContent or None when extraction finds nothing. structured-v1 is a typed tree, not trusted HTML; plain supports legacy strings. plain_text and revision are derived from content. Explicit gutter nodes are removed before extraction, while meaningful numeric lines and code whitespace remain. Completeness defaults to unknown. Newsdesk's body-fetch job now uses this tool and atomically persists its content and derived plain projection; the reader renders its closed node vocabulary. Fetching, persistence, wall detection and rendering remain the caller's responsibilities. The extractor requires trafilatura 2.2 or newer, verified against the representative fixture.

from desk.tools import embedding, failures, framing, marks, video, walls
from desk.tools.http import session

vecs   = embedding.load("all-MiniLM-L6-v2").encode(["some text"])   # (n, dim) ndarray
verdict = framing.check(cache, "https://x.com/a")  # {embeddable, reason, host, certain}; per host
item   = marks.mark(item, {"paywalled": "(g+)"})  # cuts the token out, records the mark
reason = walls.note(cache, url, body)   # "" or why this body is the same page every time
why    = failures.of_status(503)        # 'server_error'; of_exception(exc) for a raise
wait   = failures.hint(resp.headers)    # the server's own Retry-After, in seconds
vid, segments = video.read("https://youtu.be/GUYEofBebDc")          # Video + Segment[]
with session(client) as http: ...       # borrows one if given, makes and closes one if not

failures is the third of those, and the newest. A category answers one question — would asking again get a different answer? — so TRANSIENT (timeout, unreachable, rate_limited, server_error, not_ready) and SETTLED (not_found, denied, consent_wall, js_required, empty) are the whole of the vocabulary, plus unknown for a failure nothing recognises, which is deliberately in neither. Those names are what a plugin's RetryPolicy.on is validated against, so adding one here adds one to the plugin contract. An exception is named by walking the class names in its ancestry rather than by isinstance, because httpx and urllib3 are both optional and either may be the one that raised — it costs precision and buys a tool that can name a failure raised by a library it has never imported. A consent wall is a category and is in no default retry list: it is the same answer however many times it is asked for.

marks and walls are the two that read what came back rather than fetching anything, which is why neither needs a dependency. marks turns an outlet's own notation — Golem's (g+), Hacker News's [$] — into one glyph that means the same thing whoever published it, and cuts the token out of the title. walls catches the opposite problem: a site that answers every article URL with its consent gate, which trafilatura extracts faithfully and which is worse than storing nothing. The tell is that two different articles cannot have the same body and two walls always do, so the detector is a fingerprint and a cache, with no per-site rules to keep up to date.

python -m desk.tools.video <url> prints a video's segments — a dry run with no app around it.

scrape and checks — getting the article out of a URL

scrape.scrape(url, getters) fetches with http, runs checks.check on the page, converts a page that passed with article.extract_article, and on a failed check tries the getters the Verdict names. It returns Scraped: the content or the last failure, the getter that answered, every Attempt with the signature that failed it, and the server's Retry-After.

from desk.tools import checks, scrape

got = scrape.scrape(url)                            # STANDARD: http only
got = scrape.scrape(url, scrape.ADVANCED)           # a verdict may send it to another getter
got = scrape.scrape(url, scrape.advanced(cache, getters))  # what the body fetch uses: plugins' URL getters first,
                                                    # and HTTP sends the cookies a browser earned
verdict = checks.check(checks.Page(200, html))      # None, or Verdict(category, name, next)

browser.render(url) launches Chromium, clicks the accept button of a dialog in browser.CONSENTS if one appears within GATE_WAIT seconds, and returns Rendered: status, HTML, final URL, headers, the host's cookies and which consent it accepted. remember(cache, url, cookies), cookie_header(cache, url) and forget(cache, url) are the jar, in the same kind of mapping walls uses. One browser per call and one at a time per process; browser.installed() says whether the extra is there, and ADVANCED has browser only if it is.

How the loop runs, the signature format and how to add one: docs/architecture.md §2.1.

Limits

  • All synchronous and blocking. Meant to be called from a worker thread.
  • ask_json is one turn, no tools, no conversation. No streaming API for the caller beyond on_partial (raw text pieces, not parsed JSON); chunking, retries and applying the answer are desk.modeltask's job.
  • Optional deps really are optional, but "optional" is not "interchangeable". A missing model SDK is an AgentError, and there are two backends, so the other one may well answer. A missing sentence-transformers is an ImportError and there is nothing behind it: ranking.backend: embeddings does not fall back to tfidf. The embed step logs the failure and returns zero, articles keep no vectors, and everything scores a neutral 0.5 for taste and ranks on its recency alone. Which reads a lot like a ranker that is working and is not.
  • cost_usd comes from the claude_code path, not the API one. Backwards from what you would guess: the CLI's ResultMessage carries what those tokens would have cost, so it is reported even though a subscription bills nothing for them, and the API path — the one that really is billed — leaves the field at zero because the Messages response does not price itself. Read it as an estimate of what a run would cost on the API, and never as a bill.
  • video is YouTube-only today. No local-file transcription yet (yt-dlp + whisper is the planned third backend); a video with no captions still yields chapters, just no words.
  • framing answers per host, not per article, and caches; most newspapers answer "no".
  • No tool knows about the store, the queue, or config beyond its own section — and tools must not import pipeline or sources.