"""Turning a failure into one line worth putting on screen. The whole of this module's job is a handoff. Something broke; the log has the traceback and always will; what the log cannot do is *tell you it happened* while you are looking at the app wondering why nothing appeared. So a notice is pushed to whoever is watching — and, since nobody usually is, written down. **It is stored, and it was not always.** The first design kept nothing: a notices table would be a second record of what the log already records, with rows that go stale and a dismiss button to maintain. That argument assumed somebody was watching when the notice was raised, and somebody who could read a server's log when they were not. Neither holds for a pull that runs on a schedule and a reader holding a phone. What it cost was three things, each reported as a defect of its own: a line that faded was gone twelve seconds, then nowhere to find it nothing open, nothing heard the usual case, and the notice was lost every run said it again "already said" was known only to a tab So `Jobs.record_notice` keeps one row per `(code, key)`, and the three are one fix. The same failure said again is that row with a higher count, which is how a client tells news from a repeat: it interrupts for the first and only counts the rest. A line somebody dismissed is still in the log, so dismissing means later rather than never. A row goes when somebody clears it, or a fortnight after it was last said — by which time it has stopped being true, which is the only staleness the first design was right to worry about. The table is not the log and does not try to be. No traceback, no history of each occurrence: a sentence, what to do, the raw error, how often, and when. This module is only the part the queue itself needs: the notice every failed job gets, the check that stops it being said twice, and the rendering a terminal wants. **Classifying a failure is the app's job**, because a useful sentence names things only the app knows — its model backend, its feeds, its config keys. An app writes its own vocabulary next to its handlers and passes notices in through `RunContext.notice`; newsdesk's is `desk/notices.py`. What a notice has to carry is not the full story but enough to orient you: text what happened, in a sentence, in terms of what you lost by it. kind whether this is the world misbehaving — an expired login, a feed gone 403 — or a bug of ours. That is the fork in what you do next, and guessing it from an error string is exactly the work you should not have to do at a glance. fix the command, when there is one. """ from __future__ import annotations import logging from collections.abc import Iterable from pipeline.models import Notice, NoticeKind log = logging.getLogger(__name__) #: Words that mean the failure happened at the other end of a wire. An app's #: own classifier will want more than this; `whose_fault` is the cheap default #: for the failures the queue meets on its own. THEIR_FAULT = ( "http", "url", "timeout", "connection", "ssl", "certificate", "socket", "dns", "resolve", "403", "404", "429", "500", "502", "503", "refused", "unreachable", "forbidden", "not found", "unavailable", "moved", ) def job_failed(kind: str, label: str, error: BaseException | str) -> Notice: """A job that ended in `failed`. A handler raising something it did not expect is the definition of ours to fix, so this is the one place that says `bug` outright rather than working it out. """ return Notice( code="job.failed", level="error", kind="bug", key=kind, text=f"{label or kind} failed.", detail=str(error).strip(), where=f"the `job … ({kind}) failed` traceback in the log", ) def already_said(said: Iterable[Notice], error: str) -> bool: """Whether one of these notices is about this failure. The caller is the worker, which announces `job_failed` for every job that ends in `failed` — and must not, when the handler already announced a classified notice about the very exception that ended it. "agent.backend is `off`, so nothing that needs a model runs" followed by "Tagging a batch failed — looks like a bug" is two contradictory lines about one failure, and the wrong one is the louder: it sends you to a traceback about a machine that is doing exactly what it was configured to do. Matched on `detail` rather than on identity, because what the handler classified is an exception and what reaches the worker is that same exception rendered `Type: message` — the same string with a prefix. Which is also the line in the right place: a notice about something else the job met on the way, a source that 403'd during a pull that then crashed over something unrelated, does not match, and that crash still gets its own line as the unexplained failure it is. """ return any(notice.detail.strip() and notice.detail.strip() in error for notice in said) def render(notice: Notice) -> str: """The same notice, for a terminal. The browser is the first-class destination, but it is not the only one: a handler run from a terminal hits every one of these failures and deserves the same sentence, the same command to run, and the same "is this mine to fix". Without this the CLI got the code and the text and silently lost the two fields that make a notice worth having. Rendered rather than logged in pieces so that the wording lives in one place. An app changes a rule and both destinations change together. """ lines = [notice.text] if notice.fix: lines.append(f" -> {notice.fix}") if notice.kind == "bug": lines.append(f" this looks like a bug rather than a bad day; see {notice.where}" if notice.where else " this looks like a bug rather than a bad day") return "\n".join(lines) def whose_fault(error: BaseException | str) -> NoticeKind: """Whether a failure is the other end's or ours. A 403 is the outlet's business and there is nothing here to fix. A `KeyError: 'entries'` is us assuming a shape the other end didn't have, and it will keep happening until someone changes the code — so it belongs in front of you as a defect, not as weather. """ text = f"{type(error).__name__} {error}".lower() if isinstance( error, BaseException) else str(error).lower() return "expected" if any(hint in text for hint in THEIR_FAULT) else "bug"