"""What went wrong, in one word a retry policy can be written against. Acquisition fails in ways that mean opposite things. A host that took too long will very likely answer in a minute; a page that is gone will be gone in a minute, in an hour, and tomorrow. Until this existed the difference had nowhere to live: `fetch_bodies` caught every exception in one `except`, wrote an empty body, and the row said only that something had been tried — so the one policy the archive could have was "never ask again", and it was the policy for the timeout as much as for the 404. A category is an answer to exactly one question: **would asking again get a different answer?** `TRANSIENT` is yes, `SETTLED` is no, and `unknown` is "we do not know", which is why nothing retries it unless a plugin names it. from desk.tools import failures failures.of_status(503) # 'server_error' failures.of_exception(exc) # 'timeout' failures.hint(response.headers) # 120.0, when the server named its own wait Kept away from any HTTP library on purpose. `httpx` and `urllib3` are both optional here and either may be the one that raised, so an exception is named by walking the class names in its ancestry rather than by importing the packages that define them. It costs the precision of an `isinstance` and buys a tool that can name a failure raised by a library it has never imported — which is what a plugin's own fetch will be. The names are the vocabulary `RetryPolicy.on` is validated against, so adding one here is adding one to the plugin contract: do it deliberately, and only where the new name would lead somewhere different from an existing one. """ from __future__ import annotations import logging from collections.abc import Mapping from email.utils import parsedate_to_datetime from datetime import UTC, datetime log = logging.getLogger(__name__) #: Failures worth another attempt: the world was busy, not unwilling. TRANSIENT = frozenset({ # The request did not finish in the time it was given. "timeout", # It never got an answer at all: DNS, a refused connection, a reset, TLS. # One name rather than four because nothing downstream would do anything # different about them, and because the downloader that reports most of # them does not say which it was. "unreachable", # 429. The one failure that usually comes with the server's own answer to # "when, then" — see `hint`. "rate_limited", # 5xx. The publisher's own machine is having a bad minute. "server_error", # Asked for before it existed. Not an HTTP failure at all: auto-captions # on a two-hour upload take hours to appear, so a recording that has none # yet is a wait rather than a refusal — and the same word covers anything # else a source publishes late. See `segment_profile` in # `desk/sources/podcast.py`, which is what declares a bound on that wait. "not_ready", }) #: Failures that will say the same thing next time. Retrying one of these is #: how a source ends up fetched four times an evening for nothing. SETTLED = frozenset({ # 404, 410. The address is not an article and will not become one. "not_found", # 401, 403, 451. Refused rather than broken — a login, a bot check, a # jurisdiction. Another identical request is refused identically. "denied", # The page came back and it was the consent gate the host shows instead of # the article. Deliberately here and never in a default retry list: it is # the same answer however many times it is asked for. See `desk.tools.walls`. "consent_wall", # The page was an app shell or a bot challenge: a placeholder for what a # script would have rendered. The same request gets the same placeholder, # so it is settled for a retry; a different way of fetching, a browser, is # what can answer it. See `desk.tools.checks`. "js_required", # Fetched, extracted, and there was no article in it. "empty", }) #: Neither, and on purpose. A failure nothing here recognises is not declared #: transient on its behalf — a policy that wants it retried has to say so. UNKNOWN = "unknown" #: Everything a plugin may name in `RetryPolicy.on`. CATEGORIES = TRANSIENT | SETTLED | frozenset({UNKNOWN}) #: Substrings of an exception class name, in the order they are tried, and the #: category each means. Ordered because `ReadTimeoutError` is both a timeout #: and a connection error by name, and it is a timeout. _BY_NAME: tuple[tuple[str, str], ...] = ( ("timeout", "timeout"), ("nameresolution", "unreachable"), ("ssl", "unreachable"), ("connect", "unreachable"), ("connection", "unreachable"), ("protocol", "unreachable"), ("transport", "unreachable"), ("network", "unreachable"), ) #: The longest server hint worth waiting on. A `Retry-After` of a day is not a #: wait, it is a refusal with a number on it, and honouring it would leave a #: job row holding this item's place until tomorrow. Beyond this the job's own #: backoff is used instead, which ends in a bounded number of attempts. MAX_HINT = 6 * 60 * 60 def of_status(status: int) -> str: """The category of an HTTP status that is not a success. A 2xx or 3xx is not a failure and has no category — it is `""`, so a caller can pass any status and ask. """ if 200 <= status < 400: return "" if status == 429: return "rate_limited" if status in (401, 403, 451): return "denied" if status in (404, 410): return "not_found" if 500 <= status < 600: return "server_error" return UNKNOWN def of_exception(exc: BaseException) -> str: """The category of a raised failure, named from its ancestry. `TimeoutError` is the builtin every socket timeout already is; the rest are matched on the class names in the exception's MRO, so `httpx.ConnectError` and `urllib3.exceptions.NewConnectionError` both come out `unreachable` without this module importing either package. A cause is followed, because a wrapper around the real failure is the usual shape: `ProtocolError` with a `TimeoutError` underneath it is a timeout. """ if isinstance(exc, TimeoutError): return "timeout" names = [cls.__name__.lower() for cls in type(exc).__mro__] for fragment, category in _BY_NAME: if any(fragment in name for name in names): return category cause = exc.__cause__ or exc.__context__ if cause is not None and cause is not exc: return of_exception(cause) return UNKNOWN def hint(headers: Mapping[str, str] | None) -> float | None: """The wait a server named for itself, in seconds, or None for no answer. `Retry-After` comes in two forms and both are in use: a number of seconds, and an HTTP date. Honouring it is the point — guessing costs either a minute of doing nothing when the service was ready again, or another refusal charged against a bound that is ours rather than the server's. Unreadable, negative, or longer than `MAX_HINT` is no answer: this returns None and the job's own backoff decides. A header nobody can parse is not worth failing over. """ if not headers: return None raw = next((v for k, v in headers.items() if k.lower() == "retry-after"), "").strip() if not raw: return None try: seconds = float(raw) except ValueError: try: when = parsedate_to_datetime(raw) except (TypeError, ValueError): log.debug("unreadable Retry-After: %r", raw) return None if when.tzinfo is None: when = when.replace(tzinfo=UTC) seconds = (when - datetime.now(UTC)).total_seconds() if seconds <= 0 or seconds > MAX_HINT: return None return seconds