Files
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

101 lines
9.6 KiB
Markdown

# `desk.tools` — the machinery
## What it is
Things a pipeline stage picks up and puts down: plain inputs, plain outputs, no idea what the app is for. The test for belonging here is *would this make sense in a different application*.
| Tool | What it does | Needs |
|---|---|---|
| `agent` | ask Claude, get JSON back | `anthropic` **or** `claude-agent-sdk` (both optional) |
| `article` | supplied HTML to typed structured content, preserving tables and code; derived plain text and acquisition provenance | `trafilatura` |
| `browser` | a page rendered in headless Chromium, past a consent dialog it knows; the cookies it earned, per host | `playwright` (optional, plus `playwright install chromium`) |
| `embedding` | text in, vector out (~384 floats) | `sentence-transformers` (optional, ~100MB+) |
| `failures` | what went wrong, in one word a retry policy can be written against | stdlib |
| `framing` | will this page load in an iframe? | `httpx` |
| `http` | one shared HTTP client per call chain | `httpx` |
| `marks` | what kind of thing is this, when it isn't an ordinary article | stdlib |
| `posts` | a Mastodon or Bluesky post and its author's thread, through the public API, as a page to convert; registered by those plugins | `httpx` |
| `checks` | is a fetched page worth converting, and if not, which getter to try next | stdlib |
| `scrape` | fetch, check, convert, and fetch another way when the check says so | `trafilatura` |
| `walls` | did we fetch the article, or the consent page in front of it | stdlib |
| `video/` | a recording, read as text with timings | `httpx`, `youtube-transcript-api` |
**Nothing is imported in `desk/tools/__init__.py` on purpose** — each import is paid for by the caller that wants it. Import the module, not the package: `from desk.tools import agent`.
## `agent` — the model
Two interchangeable backends behind one call. `agent.backend(cfg)` says which will answer, or `""` for nobody — which is a failure, not a third backend.
- `api` — one Claude API call. Wants `ANTHROPIC_API_KEY`. The unattended path.
- `claude_code` — the Agent SDK drives the `claude` CLI on your subscription. No key, nothing billed. The development path. (**Spawns a process.**)
- `off` / nothing available — `backend()` answers `""`, and the caller fails the batch with a notice naming the reason. Nothing waits to be answered by hand.
```python
from desk.tools import agent
reply = agent.ask_json(
config.agent, # desk.config.AgentConfig
system="You tag articles.", # kept byte-identical across a run — it is what caches
question={"articles": [...]},
schema={"type": "object", ...}, # both backends take the same JSON schema
model="claude-haiku-4-5-20251001",
thinking="off", # or "adaptive"
on_partial=lambda piece: ..., # streamed text, for per-item progress
)
reply.data, reply.backend, reply.model, reply.cost_usd, reply.usage
```
Failures of every kind raise `AgentError` — treat it as "no answer this time". The `claude_code` path deliberately gets **no tools, no CLAUDE.md, no skills, no MCP, no session**: one text-in, JSON-out turn, the same shape as the API call.
## The others, in one line each
`article.extract_article(html, source_url)` returns `ArticleContent` or `None` when extraction finds nothing. `structured-v1` is a typed tree, not trusted HTML; `plain` supports legacy strings. `plain_text` and `revision` are derived from content. Explicit gutter nodes are removed before extraction, while meaningful numeric lines and code whitespace remain. Completeness defaults to unknown. Newsdesk's body-fetch job now uses this tool and atomically persists its content and derived plain projection; the reader renders its closed node vocabulary. Fetching, persistence, wall detection and rendering remain the caller's responsibilities. The extractor requires trafilatura 2.2 or newer, verified against the representative fixture.
```python
from desk.tools import embedding, failures, framing, marks, video, walls
from desk.tools.http import session
vecs = embedding.load("all-MiniLM-L6-v2").encode(["some text"]) # (n, dim) ndarray
verdict = framing.check(cache, "https://x.com/a") # {embeddable, reason, host, certain}; per host
item = marks.mark(item, {"paywalled": "(g+)"}) # cuts the token out, records the mark
reason = walls.note(cache, url, body) # "" or why this body is the same page every time
why = failures.of_status(503) # 'server_error'; of_exception(exc) for a raise
wait = failures.hint(resp.headers) # the server's own Retry-After, in seconds
vid, segments = video.read("https://youtu.be/GUYEofBebDc") # Video + Segment[]
with session(client) as http: ... # borrows one if given, makes and closes one if not
```
`failures` is the third of those, and the newest. A category answers one question — *would asking again get a different answer?* — so `TRANSIENT` (`timeout`, `unreachable`, `rate_limited`, `server_error`, `not_ready`) and `SETTLED` (`not_found`, `denied`, `consent_wall`, `js_required`, `empty`) are the whole of the vocabulary, plus `unknown` for a failure nothing recognises, which is deliberately in neither. Those names are what a plugin's `RetryPolicy.on` is validated against, so adding one here adds one to the plugin contract. An exception is named by walking the class names in its ancestry rather than by `isinstance`, because `httpx` and `urllib3` are both optional and either may be the one that raised — it costs precision and buys a tool that can name a failure raised by a library it has never imported. A consent wall is a category and is in no default retry list: it is the same answer however many times it is asked for.
`marks` and `walls` are the two that read what came back rather than fetching anything, which is why neither needs a dependency. `marks` turns an outlet's own notation — Golem's `(g+)`, Hacker News's `[$]` — into one glyph that means the same thing whoever published it, and cuts the token out of the title. `walls` catches the opposite problem: a site that answers every article URL with its consent gate, which trafilatura extracts faithfully and which is worse than storing nothing. The tell is that two different articles cannot have the same body and two walls always do, so the detector is a fingerprint and a cache, with no per-site rules to keep up to date.
`python -m desk.tools.video <url>` prints a video's segments — a dry run with no app around it.
## `scrape` and `checks` — getting the article out of a URL
`scrape.scrape(url, getters)` fetches with `http`, runs `checks.check` on the page, converts a page that passed with `article.extract_article`, and on a failed check tries the getters the `Verdict` names. It returns `Scraped`: the content or the last failure, the getter that answered, every `Attempt` with the signature that failed it, and the server's `Retry-After`.
```python
from desk.tools import checks, scrape
got = scrape.scrape(url) # STANDARD: http only
got = scrape.scrape(url, scrape.ADVANCED) # a verdict may send it to another getter
got = scrape.scrape(url, scrape.advanced(cache, getters)) # what the body fetch uses: plugins' URL getters first,
# and HTTP sends the cookies a browser earned
verdict = checks.check(checks.Page(200, html)) # None, or Verdict(category, name, next)
```
`browser.render(url)` launches Chromium, clicks the accept button of a dialog in `browser.CONSENTS` if one appears within `GATE_WAIT` seconds, and returns `Rendered`: status, HTML, final URL, headers, the host's cookies and which consent it accepted. `remember(cache, url, cookies)`, `cookie_header(cache, url)` and `forget(cache, url)` are the jar, in the same kind of mapping `walls` uses. One browser per call and one at a time per process; `browser.installed()` says whether the extra is there, and `ADVANCED` has `browser` only if it is.
How the loop runs, the signature format and how to add one: `docs/architecture.md` §2.1.
## Limits
- **All synchronous and blocking.** Meant to be called from a worker thread.
- **`ask_json` is one turn, no tools, no conversation.** No streaming API for the caller beyond `on_partial` (raw text pieces, not parsed JSON); chunking, retries and applying the answer are `desk.modeltask`'s job.
- **Optional deps really are optional, but "optional" is not "interchangeable".** A missing model SDK is an `AgentError`, and there are two backends, so the other one may well answer. A missing `sentence-transformers` is an `ImportError` and there is nothing behind it: `ranking.backend: embeddings` does **not** fall back to tfidf. The embed step logs the failure and returns zero, articles keep no vectors, and everything scores a neutral 0.5 for taste and ranks on its recency alone. Which reads a lot like a ranker that is working and is not.
- **`cost_usd` comes from the `claude_code` path, not the API one.** Backwards from what you would guess: the CLI's `ResultMessage` carries what those tokens *would* have cost, so it is reported even though a subscription bills nothing for them, and the API path — the one that really is billed — leaves the field at zero because the Messages response does not price itself. Read it as an estimate of what a run would cost on the API, and never as a bill.
- **`video` is YouTube-only today.** No local-file transcription yet (yt-dlp + whisper is the planned third backend); a video with no captions still yields chapters, just no words.
- **`framing` answers per host, not per article**, and caches; most newspapers answer "no".
- **No tool knows about the store, the queue, or config beyond its own section** — and `tools` must not import `pipeline` or `sources`.