Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9d666d0937
commit
248ea1636d
36 files changed
+8393
No files matched your search
+161
@@ -0,0 +1,161 @@
|
||||
"""The settings sections an app composes into its own config.
|
||||
|
||||
`Section` is the base: a block of YAML validated into a model, with an
|
||||
unrecognised key being an error rather than a silent no-op. `AgentConfig`
|
||||
is how a model is reached and `ModelTaskConfig` is how a task for one is
|
||||
paced — both read by the pipeline, so both live here rather than in any one
|
||||
app's config. An app declares its own `Config(Section)` with these as fields
|
||||
and whatever else it needs next to them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal, Protocol, get_args, get_origin
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
#: The bare words YAML 1.1 resolves to each boolean, in the order a field gets
|
||||
#: asked about them. See `Section._spell_yaml_booleans_back`.
|
||||
_YAML_WORDS = {False: ("off", "no", "false"), True: ("on", "yes", "true")}
|
||||
|
||||
|
||||
def _literal_strings(annotation: Any) -> set[str]:
|
||||
"""Every string a `Literal` annotation accepts, through any wrapping."""
|
||||
if get_origin(annotation) is Literal:
|
||||
return {a for a in get_args(annotation) if isinstance(a, str)}
|
||||
return {s for arg in get_args(annotation) for s in _literal_strings(arg)}
|
||||
|
||||
|
||||
class Section(BaseModel):
|
||||
"""A block of settings.
|
||||
|
||||
`extra="forbid"` is the whole point: an unrecognised key is a mistake — a
|
||||
typo, or a setting that was renamed and never updated — and saying so is
|
||||
more useful than ignoring it. `protected_namespaces=()` because pydantic
|
||||
reserves the `model_` prefix for its own methods and two settings here are
|
||||
honestly named `model` and `model_weight`.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid", protected_namespaces=())
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def _spell_yaml_booleans_back(cls, data: Any) -> Any:
|
||||
"""Let a setting whose value is spelled `off` be written as `off`.
|
||||
|
||||
YAML 1.1 reads the bare words `off no false on yes true` as booleans,
|
||||
so `backend: off` arrives here as `False` and fails against a `str`
|
||||
field — with an error that helpfully lists `off` as one of the values
|
||||
it will accept. The documented spelling being the one spelling that
|
||||
doesn't work is not a thing to answer with a note in the example file;
|
||||
`agent.backend: off` is what the example and the module that reads it
|
||||
both tell you to write.
|
||||
|
||||
So a boolean landing on a field whose `Literal` lists a word for that
|
||||
boolean is read as the word. Nothing else moves: `off` on a field that
|
||||
has no `off` to offer is still an error, and a real `bool` field is
|
||||
untouched, because `True` and `False` are what it asked for.
|
||||
"""
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
spelled = data
|
||||
for name, value in data.items():
|
||||
if not isinstance(value, bool):
|
||||
continue
|
||||
field = cls.model_fields.get(name)
|
||||
if field is None:
|
||||
continue
|
||||
choices = _literal_strings(field.annotation)
|
||||
word = next((w for w in _YAML_WORDS[value] if w in choices), None)
|
||||
if word is not None:
|
||||
# A copy, because this dict is the caller's — `load` hands the
|
||||
# parsed YAML straight in, and reading a file shouldn't edit it.
|
||||
spelled = dict(spelled) if spelled is data else spelled
|
||||
spelled[name] = word
|
||||
return spelled
|
||||
|
||||
|
||||
class AgentConfig(Section):
|
||||
"""Who answers when a job needs a model. See desk/tools/agent.py."""
|
||||
|
||||
#: `auto` prefers the API when there are credentials and otherwise drives
|
||||
#: the `claude` CLI on your subscription through the Claude Agent SDK.
|
||||
#: Naming one skips the detection; `off` means no model is reachable, so
|
||||
#: every batch that needs one fails with a notice saying which switch did
|
||||
#: it. A stage whose batches all fail still reports and the cascade
|
||||
#: carries on — turning the model off stops the model, not the pull.
|
||||
backend: Literal["auto", "api", "claude_code", "off"] = "auto"
|
||||
#: Overrides the per-task model everywhere. Empty means each job picks its
|
||||
#: own — tagging asks for Haiku, because it is Haiku work.
|
||||
model: str = ""
|
||||
#: Seconds for one call, either backend. The Claude Code path spawns a
|
||||
#: process and is the slower of the two by a few seconds.
|
||||
timeout: float = Field(300.0, gt=0)
|
||||
max_tokens: int = Field(8192, ge=1)
|
||||
#: Where the `claude` binary is, if it isn't on PATH.
|
||||
cli_path: str = ""
|
||||
credentials_database: str = Field("", exclude=True, repr=False)
|
||||
|
||||
|
||||
class ModelTaskConfig(Section):
|
||||
"""What every task for the model is governed by. See desk/modeltask.py."""
|
||||
|
||||
#: Off is off everywhere: the pull, the button and `run("tag")` alike all
|
||||
#: find nothing to do and say so. Not a refusal — a task that is off has
|
||||
#: an empty backlog, so a cascade carries straight on past it.
|
||||
enabled: bool = True
|
||||
#: Whether a *pull* runs it, and nothing else — the button never reads
|
||||
#: this. `auto` runs it during a pull when `agent.backend` resolves to
|
||||
#: something and skips it quietly when it does not, so a machine with no
|
||||
#: model pulls in silence rather than filing a failure per batch about
|
||||
#: being set up the way it was meant to be. `always` runs it either way,
|
||||
#: and a missing backend fails each batch saying so. `never` keeps it out
|
||||
#: of the pull altogether, leaving the button.
|
||||
on_ingest: Literal["auto", "always", "never"] = "auto"
|
||||
model: str = ""
|
||||
#: Whether the model reasons before it answers. Per *task* rather than per
|
||||
#: backend on purpose: both backends take the same shapes, so how hard to
|
||||
#: think is a property of the work, not of how the model is reached.
|
||||
#:
|
||||
#: `off` by default because neither task here is thinking work and the cost
|
||||
#: is not marginal — tagging twenty-five headlines measured 128s with
|
||||
#: thinking and 17s without, for tags that came back the same. Turn it on
|
||||
#: for a task whose answer is a judgement rather than a lookup.
|
||||
#:
|
||||
#: `effort` is deliberately not offered. The CLI has it and the Messages
|
||||
#: API does not, and a setting that silently means nothing on one backend
|
||||
#: is exactly the asymmetry this seam exists to avoid.
|
||||
thinking: Literal["off", "adaptive"] = "off"
|
||||
#: Rows per model call — how the task slices its own backlog, and the one
|
||||
#: number here that is genuinely the task's. Twenty-five is what one call
|
||||
#: and one schema hold; it is prompt economics, and no generic queue could
|
||||
#: know it.
|
||||
#:
|
||||
#: How many run at once is not here and should not be: that is how many
|
||||
#: workers there are, and two tasks that both want a model share them
|
||||
#: rather than each helping itself. See `workers` on the app's config.
|
||||
batch_size: int = Field(25, ge=1)
|
||||
|
||||
|
||||
class WorkersConfig(Section):
|
||||
"""How many workers of each kind the queue has.
|
||||
|
||||
A property of the machine and the subscription, not of any one task, which
|
||||
is why it is here and not next to `batch_size`. `llm` is what one Claude
|
||||
subscription answers without queueing behind itself; `http` is wider
|
||||
because a fetch is mostly waiting; `cpu` is one because the ranker and the
|
||||
embedder want the machine. A resource nobody names gets one worker.
|
||||
"""
|
||||
|
||||
llm: int = Field(4, ge=1)
|
||||
http: int = Field(8, ge=1)
|
||||
cpu: int = Field(1, ge=1)
|
||||
|
||||
|
||||
class HasAgent(Protocol):
|
||||
"""What the pipeline asks of an app's config: that it has an `agent`
|
||||
section. `modeltask` reads nothing else from it — the rest is handed to
|
||||
the task's own `settings` callable, which knows the app."""
|
||||
|
||||
@property
|
||||
def agent(self) -> AgentConfig: ...
|
||||
Reference in new issue
Block a user