Require READS, search params and a dated robots record; record robots.txt per requested path

Every reader declares, so undeclared(), the scope() fallback, robots_page and
ROBOTS_ANY_CASE go. A search provider needs PARAMS and clean_params, a row may
carry company, places gains none. Each robots record names the paths the
reader requests, from robots.txt read again on 2026-09-30. Platsbanken states
remote, HR Manager states no place.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-30 15:51:13 +02:00
1 parent 4e827665dd
commit 2349fc424b
46 files changed
+323 -100

No files matched your search

-1
View File
@@ -289,7 +289,6 @@ def test_the_careers_page_is_the_search_page() -> None:
url = "https://careers.king.com/us/en/search-results"
assert phenom.careers_url(SLUG) == ats.careers_url(f"phenom:{SLUG}") == url
assert phenom.allowed(SLUG)(httpx.URL(url).host)
assert not hasattr(phenom, "robots_page") # decided when built, see READS.robots
assert phenom.careers_url("www.careers.philips.com/global/en") == \
"https://www.careers.philips.com/global/en/search-results"