Files
jobboard/desk/tools/walls.py
T
WolfandClaude Opus 5.5 248ea1636d Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied
rather than reached through an editable install of news, which also exposes
a package named pipeline. news stays the source of truth for desk.

Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:29:17 +02:00

129 lines
5.6 KiB
Python

"""Did we fetch the article, or the page the outlet shows instead of it?
Some sites answer every article URL with the same thing: a consent gate, an
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
faithfully — it is real text on a real page — and we store it as the body. That
is worse than storing nothing, because the reader prefers the body to the feed's
summary, so a wall doesn't merely fail to add anything: it *hides* the two
readable sentences the feed already gave us.
The tell is repetition. Different articles cannot have the same body; a wall
always does. That is the whole detector — no keyword list, no per-site rules,
nothing to keep up to date. It learns golem.de's German consent page and a
Cloudflare challenge with the same line of code, and it costs no extra
requests, because the evidence is a by-product of a fetch we were making
anyway.
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
requests to one host earns a throttle or challenge page, that page is served to
whichever two articles asked next, and the two fingerprint the same. Refetching
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
was about our request rate and not about the outlet. A wall repeats on every
article, so a third body costs one extra fetch per walled host, once; a throttle
page that repeats three consecutive times is the site answering nothing else.
The verdict is per host, because the wall is: the site serves it, it isn't
chosen per article. Once a host is known to answer with one, we stop fetching
from it altogether — which makes pulls faster, not just more correct. That is
also why the count matters: `known` short-circuits the request, so a wall
verdict is the last evidence a host ever gets to give. Nothing can disprove it
except `forget`.
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
in a test), because a tool has no store of its own to remember things in. The
one way this is wrong is an outlet that really does publish the same body
`STRIKES` times running; the cost is that its articles fall back to their feed
summaries until someone calls `forget`, which is what the re-check button in
the Sources tab is for.
"""
from __future__ import annotations
import hashlib
import logging
from collections.abc import Mapping, MutableMapping
from desk.tools.framing import host_of
log = logging.getLogger(__name__)
#: Sits alongside `embed:` in the same table. The two verdicts answer different
#: questions about a host — will it let us frame it, will it give up its text —
#: and neither implies the other.
CACHE_PREFIX = "body:"
_WALL = "wall:"
REASON = "every article comes back as the same page"
#: How many consecutive identical bodies settle it. See the module docstring:
#: two was one throttle page away from retiring a working outlet for good.
STRIKES = 3
def _fingerprint(body: str) -> str:
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
def _seen(value: str) -> tuple[str, int]:
"""`(fingerprint, how many in a row)` from a stored value.
A bare fingerprint is one sighting, which is what every row written before
`STRIKES` existed holds — so an upgraded database is read correctly and
needs no migration of its own.
"""
mark, _, run = value.partition("*")
return mark, (int(run) if run.isdigit() else 1)
def known(cache: Mapping[str, str], url: str) -> str:
"""Why this host's bodies aren't worth fetching, or "" if they are.
Asked before the request, so a settled verdict costs nothing at all. Takes
a read-only mapping, because this one only ever reads — which lets the
Sources tab hand over the snapshot it already loaded rather than the live
table, and says in the signature that opening a panel changes nothing.
"""
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
"""Record a body we just extracted. Returns a reason if it proves a wall.
Only the previous body from the host is kept and how many articles running
have handed it over — not every body ever seen. A wall repeats on the
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
and one key per host beats a growing set per host.
One distinct body resets the run to one. That is what makes the count mean
"in a row" rather than "ever", and it is the whole difference between a
site that answers with a gate and a site that answered twice from a cache.
"""
host = host_of(url)
if not host or not body.strip():
return ""
key = CACHE_PREFIX + host
stored = cache.get(key, "")
if stored.startswith(_WALL):
return stored[len(_WALL):]
mark = _fingerprint(body)
seen, run = _seen(stored)
run = run + 1 if seen == mark else 1
if run >= STRIKES:
cache[key] = _WALL + REASON
log.info("%s serves a wall, not articles: %s (%d in a row)",
host, REASON, run)
return REASON
cache[key] = mark if run == 1 else f"{mark}*{run}"
return ""
def forget(cache: MutableMapping[str, str], url: str) -> None:
"""Drop what we learned about a host, so the next fetch decides again.
Outlets change their minds — a wall goes up for a quarter and comes down
again — and a verdict with no way back would be a permanent decision made
on two HTTP responses.
"""
cache.pop(CACHE_PREFIX + host_of(url), None)