Require READS, search params and a dated robots record; record robots.txt per requested path
Every reader declares, so undeclared(), the scope() fallback, robots_page and ROBOTS_ANY_CASE go. A search provider needs PARAMS and clean_params, a row may carry company, places gains none. Each robots record names the paths the reader requests, from robots.txt read again on 2026-09-30. Platsbanken states remote, HR Manager states no place. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
4e827665dd
commit
2349fc424b
46 files changed
+323
-100
No files matched your search
@@ -52,7 +52,10 @@ READS = Reads(
|
||||
scope="board", listing="paged", page_size=100, text="detail", teaser=True,
|
||||
complete="stated", places="many", filters=Filters(country="many", query="one"),
|
||||
fields=Fields(posted="trusted", remote="stated"),
|
||||
robots="read: robots.txt allows /<country>/<lang>/search-results (checked 2026-09-29)",
|
||||
robots=("read: robots.txt has no rule on POST /widgets or /<country>/<lang>/job/<id>; "
|
||||
"it disallows */apply, */px-widgets, */jobcart and the like (careers.king.com, "
|
||||
"careers.justeattakeaway.com; www.careers.philips.com redirects /robots.txt to "
|
||||
"its careers page; checked 2026-09-30)"),
|
||||
)
|
||||
|
||||
#: The page size asked for. 100 is what the site's own search sends; `size`
|
||||
|
||||
Reference in new issue
Block a user