Bring in desk from news
The plugin contract, the model task and the tools roledesk imports, copied rather than reached through an editable install of news, which also exposes a package named pipeline. news stays the source of truth for desk. Refreshed from news/desk at a275c38a1aa0fe80a7ec7dd5781ecccf683f71c6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9d666d0937
commit
248ea1636d
36 files changed
+8393
No files matched your search
@@ -0,0 +1,128 @@
|
||||
"""Did we fetch the article, or the page the outlet shows instead of it?
|
||||
|
||||
Some sites answer every article URL with the same thing: a consent gate, an
|
||||
adblock notice, a "prove you're human" interstitial. trafilatura extracts it
|
||||
faithfully — it is real text on a real page — and we store it as the body. That
|
||||
is worse than storing nothing, because the reader prefers the body to the feed's
|
||||
summary, so a wall doesn't merely fail to add anything: it *hides* the two
|
||||
readable sentences the feed already gave us.
|
||||
|
||||
The tell is repetition. Different articles cannot have the same body; a wall
|
||||
always does. That is the whole detector — no keyword list, no per-site rules,
|
||||
nothing to keep up to date. It learns golem.de's German consent page and a
|
||||
Cloudflare challenge with the same line of code, and it costs no extra
|
||||
requests, because the evidence is a by-product of a fetch we were making
|
||||
anyway.
|
||||
|
||||
**It takes `STRIKES` bodies in a row, not two.** Two was enough until a real
|
||||
pull tombstoned theguardian.com and arstechnica.com on the same run: a burst of
|
||||
requests to one host earns a throttle or challenge page, that page is served to
|
||||
whichever two articles asked next, and the two fingerprint the same. Refetching
|
||||
the identical 25 URLs afterwards returned 25 distinct articles, so the verdict
|
||||
was about our request rate and not about the outlet. A wall repeats on every
|
||||
article, so a third body costs one extra fetch per walled host, once; a throttle
|
||||
page that repeats three consecutive times is the site answering nothing else.
|
||||
|
||||
The verdict is per host, because the wall is: the site serves it, it isn't
|
||||
chosen per article. Once a host is known to answer with one, we stop fetching
|
||||
from it altogether — which makes pulls faster, not just more correct. That is
|
||||
also why the count matters: `known` short-circuits the request, so a wall
|
||||
verdict is the last evidence a host ever gets to give. Nothing can disprove it
|
||||
except `forget`.
|
||||
|
||||
Kept in the same kind of mapping `framing` uses (the app's state table, a dict
|
||||
in a test), because a tool has no store of its own to remember things in. The
|
||||
one way this is wrong is an outlet that really does publish the same body
|
||||
`STRIKES` times running; the cost is that its articles fall back to their feed
|
||||
summaries until someone calls `forget`, which is what the re-check button in
|
||||
the Sources tab is for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
from collections.abc import Mapping, MutableMapping
|
||||
|
||||
from desk.tools.framing import host_of
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
#: Sits alongside `embed:` in the same table. The two verdicts answer different
|
||||
#: questions about a host — will it let us frame it, will it give up its text —
|
||||
#: and neither implies the other.
|
||||
CACHE_PREFIX = "body:"
|
||||
_WALL = "wall:"
|
||||
REASON = "every article comes back as the same page"
|
||||
|
||||
#: How many consecutive identical bodies settle it. See the module docstring:
|
||||
#: two was one throttle page away from retiring a working outlet for good.
|
||||
STRIKES = 3
|
||||
|
||||
|
||||
def _fingerprint(body: str) -> str:
|
||||
return hashlib.sha1(body.strip().encode(errors="replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _seen(value: str) -> tuple[str, int]:
|
||||
"""`(fingerprint, how many in a row)` from a stored value.
|
||||
|
||||
A bare fingerprint is one sighting, which is what every row written before
|
||||
`STRIKES` existed holds — so an upgraded database is read correctly and
|
||||
needs no migration of its own.
|
||||
"""
|
||||
mark, _, run = value.partition("*")
|
||||
return mark, (int(run) if run.isdigit() else 1)
|
||||
|
||||
|
||||
def known(cache: Mapping[str, str], url: str) -> str:
|
||||
"""Why this host's bodies aren't worth fetching, or "" if they are.
|
||||
|
||||
Asked before the request, so a settled verdict costs nothing at all. Takes
|
||||
a read-only mapping, because this one only ever reads — which lets the
|
||||
Sources tab hand over the snapshot it already loaded rather than the live
|
||||
table, and says in the signature that opening a panel changes nothing.
|
||||
"""
|
||||
verdict = cache.get(CACHE_PREFIX + host_of(url), "")
|
||||
return verdict[len(_WALL):] if verdict.startswith(_WALL) else ""
|
||||
|
||||
|
||||
def note(cache: MutableMapping[str, str], url: str, body: str) -> str:
|
||||
"""Record a body we just extracted. Returns a reason if it proves a wall.
|
||||
|
||||
Only the previous body from the host is kept and how many articles running
|
||||
have handed it over — not every body ever seen. A wall repeats on the
|
||||
*next* article, so consecutive comparison finds it in `STRIKES` fetches,
|
||||
and one key per host beats a growing set per host.
|
||||
|
||||
One distinct body resets the run to one. That is what makes the count mean
|
||||
"in a row" rather than "ever", and it is the whole difference between a
|
||||
site that answers with a gate and a site that answered twice from a cache.
|
||||
"""
|
||||
host = host_of(url)
|
||||
if not host or not body.strip():
|
||||
return ""
|
||||
key = CACHE_PREFIX + host
|
||||
stored = cache.get(key, "")
|
||||
if stored.startswith(_WALL):
|
||||
return stored[len(_WALL):]
|
||||
mark = _fingerprint(body)
|
||||
seen, run = _seen(stored)
|
||||
run = run + 1 if seen == mark else 1
|
||||
if run >= STRIKES:
|
||||
cache[key] = _WALL + REASON
|
||||
log.info("%s serves a wall, not articles: %s (%d in a row)",
|
||||
host, REASON, run)
|
||||
return REASON
|
||||
cache[key] = mark if run == 1 else f"{mark}*{run}"
|
||||
return ""
|
||||
|
||||
|
||||
def forget(cache: MutableMapping[str, str], url: str) -> None:
|
||||
"""Drop what we learned about a host, so the next fetch decides again.
|
||||
|
||||
Outlets change their minds — a wall goes up for a quarter and comes down
|
||||
again — and a verdict with no way back would be a permanent decision made
|
||||
on two HTTP responses.
|
||||
"""
|
||||
cache.pop(CACHE_PREFIX + host_of(url), None)
|
||||
Reference in new issue
Block a user