Require READS, search params and a dated robots record; record robots.txt per requested path

Every reader declares, so undeclared(), the scope() fallback, robots_page and
ROBOTS_ANY_CASE go. A search provider needs PARAMS and clean_params, a row may
carry company, places gains none. Each robots record names the paths the
reader requests, from robots.txt read again on 2026-09-30. Platsbanken states
remote, HR Manager states no place.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-30 15:51:13 +02:00
1 parent 4e827665dd
commit 2349fc424b
46 files changed
+323 -100

No files matched your search

+3 -1
View File
@@ -52,7 +52,9 @@ READS = Reads(
scope="board", listing="paged", page_size=20, ceiling=2000, text="detail",
complete="stated", places="many", filters=Filters(country="ids", remote="ids", query="one"),
fields=Fields(posted="trusted", remote="stated", employment="stated"),
robots="read: robots.txt allows /<Site>/ (checked 2026-09-29)",
robots=("read: robots.txt allows /<Site>/ and disallows /refreshFacet/ and has no rule "
"on /wday/cxs/<tenant>/<site>/, the listing's POST .../jobs and each role's "
"JSON (zendesk.wd1, lego.wd103, saxobank.wd3, checked 2026-09-30)"),
)
#: One group for `<tenant>.<wdN>` and one for the site, which discovery joins