The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
129 lines
5.6 KiB
Python
129 lines
5.6 KiB
Python
"""Did we fetch the article, or the page the outlet shows instead of it?
|
|
|
|
Some sites answer every article URL with the same thing: a consent gate, an
|
|
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
|
|
faithfully — it is real text on a real page — and we store it as the body. That
|
|
is worse than storing nothing, because the reader prefers the body to the feed's
|
|
summary, so a wall doesn't merely fail to add anything: it *hides* the two
|
|
readable sentences the feed already gave us.
|
|
|
|
The tell is repetition. Different articles cannot have the same body; a wall
|
|
always does. That is the whole detector — no keyword list, no per-site rules,
|
|
nothing to keep up to date. It learns golem.de's German consent page and a
|
|
Cloudflare challenge with the same line of code, and it costs no extra
|
|
requests, because the evidence is a by-product of a fetch we were making
|
|
anyway.
|
|
|
|
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
|
|
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
|
|
requests to one host earns a throttle or challenge page, that page is served to
|
|
whichever two articles asked next, and the two fingerprint the same. Refetching
|
|
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
|
|
was about our request rate and not about the outlet. A wall repeats on every
|
|
article, so a third body costs one extra fetch per walled host, once; a throttle
|
|
page that repeats three consecutive times is the site answering nothing else.
|
|
|
|
The verdict is per host, because the wall is: the site serves it, it isn't
|
|
chosen per article. Once a host is known to answer with one, we stop fetching
|
|
from it altogether — which makes pulls faster, not just more correct. That is
|
|
also why the count matters: `known` short-circuits the request, so a wall
|
|
verdict is the last evidence a host ever gets to give. Nothing can disprove it
|
|
except `forget`.
|
|
|
|
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
|
|
in a test), because a tool has no store of its own to remember things in. The
|
|
one way this is wrong is an outlet that really does publish the same body
|
|
`STRIKES` times running; the cost is that its articles fall back to their feed
|
|
summaries until someone calls `forget`, which is what the re-check button in
|
|
the Sources tab is for.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import logging
|
|
from collections.abc import Mapping, MutableMapping
|
|
|
|
from desk.tools.framing import host_of
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
#: Sits alongside `embed:` in the same table. The two verdicts answer different
|
|
#: questions about a host — will it let us frame it, will it give up its text —
|
|
#: and neither implies the other.
|
|
CACHE_PREFIX = "body:"
|
|
_WALL = "wall:"
|
|
REASON = "every article comes back as the same page"
|
|
|
|
#: How many consecutive identical bodies settle it. See the module docstring:
|
|
#: two was one throttle page away from retiring a working outlet for good.
|
|
STRIKES = 3
|
|
|
|
|
|
def _fingerprint(body: str) -> str:
|
|
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
|
|
|
|
|
|
def _seen(value: str) -> tuple[str, int]:
|
|
"""`(fingerprint, how many in a row)` from a stored value.
|
|
|
|
A bare fingerprint is one sighting, which is what every row written before
|
|
`STRIKES` existed holds — so an upgraded database is read correctly and
|
|
needs no migration of its own.
|
|
"""
|
|
mark, _, run = value.partition("*")
|
|
return mark, (int(run) if run.isdigit() else 1)
|
|
|
|
|
|
def known(cache: Mapping[str, str], url: str) -> str:
|
|
"""Why this host's bodies aren't worth fetching, or "" if they are.
|
|
|
|
Asked before the request, so a settled verdict costs nothing at all. Takes
|
|
a read-only mapping, because this one only ever reads — which lets the
|
|
Sources tab hand over the snapshot it already loaded rather than the live
|
|
table, and says in the signature that opening a panel changes nothing.
|
|
"""
|
|
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
|
|
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
|
|
|
|
|
|
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
|
|
"""Record a body we just extracted. Returns a reason if it proves a wall.
|
|
|
|
Only the previous body from the host is kept and how many articles running
|
|
have handed it over — not every body ever seen. A wall repeats on the
|
|
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
|
|
and one key per host beats a growing set per host.
|
|
|
|
One distinct body resets the run to one. That is what makes the count mean
|
|
"in a row" rather than "ever", and it is the whole difference between a
|
|
site that answers with a gate and a site that answered twice from a cache.
|
|
"""
|
|
host = host_of(url)
|
|
if not host or not body.strip():
|
|
return ""
|
|
key = CACHE_PREFIX + host
|
|
stored = cache.get(key, "")
|
|
if stored.startswith(_WALL):
|
|
return stored[len(_WALL):]
|
|
mark = _fingerprint(body)
|
|
seen, run = _seen(stored)
|
|
run = run + 1 if seen == mark else 1
|
|
if run >= STRIKES:
|
|
cache[key] = _WALL + REASON
|
|
log.info("%s serves a wall, not articles: %s (%d in a row)",
|
|
host, REASON, run)
|
|
return REASON
|
|
cache[key] = mark if run == 1 else f"{mark}*{run}"
|
|
return ""
|
|
|
|
|
|
def forget(cache: MutableMapping[str, str], url: str) -> None:
|
|
"""Drop what we learned about a host, so the next fetch decides again.
|
|
|
|
Outlets change their minds — a wall goes up for a quarter and comes down
|
|
again — and a verdict with no way back would be a permanent decision made
|
|
on two HTTP responses.
|
|
"""
|
|
cache.pop(CACHE_PREFIX + host_of(url), None)
|