"""Did we fetch the article, or the page the outlet shows instead of it? Some sites answer every article URL with the same thing: a consent gate, an adblock notice, a "prove you're human" interstitial. trafilatura extracts it faithfully — it is real text on a real page — and we store it as the body. That is worse than storing nothing, because the reader prefers the body to the feed's summary, so a wall doesn't merely fail to add anything: it *hides* the two readable sentences the feed already gave us. The tell is repetition. Different articles cannot have the same body; a wall always does. That is the whole detector — no keyword list, no per-site rules, nothing to keep up to date. It learns golem.de's German consent page and a Cloudflare challenge with the same line of code, and it costs no extra requests, because the evidence is a by-product of a fetch we were making anyway. **It takes `STRIKES` bodies in a row, not two.** Two was enough until a real pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of requests to one host earns a throttle or challenge page, that page is served to whichever two articles asked next, and the two fingerprint the same. Refetching the identical 25 URLs afterwards returned 25 distinct articles, so the verdict was about our request rate and not about the outlet. A wall repeats on every article, so a third body costs one extra fetch per walled host, once; a throttle page that repeats three consecutive times is the site answering nothing else. The verdict is per host, because the wall is: the site serves it, it isn't chosen per article. Once a host is known to answer with one, we stop fetching from it altogether — which makes pulls faster, not just more correct. That is also why the count matters: `known` short-circuits the request, so a wall verdict is the last evidence a host ever gets to give. Nothing can disprove it except `forget`. Kept in the same kind of mapping `framing` uses (the app's state table, a dict in a test), because a tool has no store of its own to remember things in. The one way this is wrong is an outlet that really does publish the same body `STRIKES` times running; the cost is that its articles fall back to their feed summaries until someone calls `forget`, which is what the re-check button in the Sources tab is for. """ from __future__ import annotations import hashlib import logging from collections.abc import Mapping, MutableMapping from desk.tools.framing import host_of log = logging.getLogger(__name__) #: Sits alongside `embed:` in the same table. The two verdicts answer different #: questions about a host — will it let us frame it, will it give up its text — #: and neither implies the other. CACHE_PREFIX = "body:" _WALL = "wall:" REASON = "every article comes back as the same page" #: How many consecutive identical bodies settle it. See the module docstring: #: two was one throttle page away from retiring a working outlet for good. STRIKES = 3 def _fingerprint(body: str) -> str: return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16] def _seen(value: str) -> tuple[str, int]: """`(fingerprint, how many in a row)` from a stored value. A bare fingerprint is one sighting, which is what every row written before `STRIKES` existed holds — so an upgraded database is read correctly and needs no migration of its own. """ mark, _, run = value.partition("*") return mark, (int(run) if run.isdigit() else 1) def known(cache: Mapping[str, str], url: str) -> str: """Why this host's bodies aren't worth fetching, or "" if they are. Asked before the request, so a settled verdict costs nothing at all. Takes a read-only mapping, because this one only ever reads — which lets the Sources tab hand over the snapshot it already loaded rather than the live table, and says in the signature that opening a panel changes nothing. """ verdict = cache.get(CACHE_PREFIX + host_of(url), "") return verdict[len(_WALL):] if verdict.startswith(_WALL) else "" def note(cache: MutableMapping[str, str], url: str, body: str) -> str: """Record a body we just extracted. Returns a reason if it proves a wall. Only the previous body from the host is kept and how many articles running have handed it over — not every body ever seen. A wall repeats on the *next* article, so consecutive comparison finds it in `STRIKES` fetches, and one key per host beats a growing set per host. One distinct body resets the run to one. That is what makes the count mean "in a row" rather than "ever", and it is the whole difference between a site that answers with a gate and a site that answered twice from a cache. """ host = host_of(url) if not host or not body.strip(): return "" key = CACHE_PREFIX + host stored = cache.get(key, "") if stored.startswith(_WALL): return stored[len(_WALL):] mark = _fingerprint(body) seen, run = _seen(stored) run = run + 1 if seen == mark else 1 if run >= STRIKES: cache[key] = _WALL + REASON log.info("%s serves a wall, not articles: %s (%d in a row)", host, REASON, run) return REASON cache[key] = mark if run == 1 else f"{mark}*{run}" return "" def forget(cache: MutableMapping[str, str], url: str) -> None: """Drop what we learned about a host, so the next fetch decides again. Outlets change their minds — a wall goes up for a quarter and comes down again — and a verdict with no way back would be a permanent decision made on two HTTP responses. """ cache.pop(CACHE_PREFIX + host_of(url), None)