Split the All tag into two company overrides: All countries and All roles (WP L)

Fetching has two filters, and a company now overrides each on its own.
All countries (companies.all_countries) is the old All roles tag: its big
boards are read with the agents' wide reach. All roles (companies.all_roles,
new meaning) drops the agents' fetch terms and any later role filter.

- Migration 24 copies the old tag to all_countries and keeps all_roles for
  those companies, and moves a board's {"terms": false} to its company.
- The runner no longer reads {"terms": false}; the plan leaves the terms out
  for a company set to All roles.
- POST /api/companies/{key}/all-countries and /all-roles, each queueing a
  pull when turned on; CompanyOut carries both.
- Two toggles replace All on the Companies row, the reader and the Roles
  strip; the reach line states countries and role types in plain words;
  the filter select gets both options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 12:50:45 +02:00
1 parent 795ca3628c
commit e52857dea5
23 files changed
+602 -215

No files matched your search

+2 -2
View File
@@ -69,9 +69,9 @@ Numbers are from the live database after the pull of 2026-09-29. For the "list,
| robots.txt | read before the list for the 14 hosts that name a page: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Jibe, Gem, BambooHR, Pinpoint, Deel, Jobylon, Rippling, HiBob, HR Manager; kept for one day per board | A disallow fails the board with "robots.txt disallows it" and counts toward the 21 days. An unreachable file (5xx, 429, timeout) fails the board for this pull and is asked again next pull. A 4xx means no rules, which allows. `Crawl-delay` is not honoured. The eight API hosts (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor) and the search providers read no robots.txt. |
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a per-role read: the per-role reads of that board stop until the next pull | Nothing is closed on a list cut short by a 429. A search that fails is skipped for the pull. |
How far a board is read: a pull asks each board over 300 roles for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company tagged All roles is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of 30 times, unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application.
How far a board is read: a pull asks each board over 300 roles for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of 30 times, unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application.
Fetch terms narrow a wide read by words. Each agent has a list of them (`fetch_terms`, set in the agent editor); the terms of a pull are the union over the agents that are on, and none when one of those agents has none. They go only to a board read with the wide reach: a company tagged All roles, or a board whose reach is more countries than a one-country host is asked for (`MAX_SERIES`, 8). The narrow reach (Denmark, Sweden) leaves at most about 150 roles on any big board, so it takes no terms, and nor does a board under 300 roles, one never read whole, or one with its own reach. `{"terms": false}` in a board's params switches them off. A host that takes a keyword gets one series per term, unioned by id, each combined with the country series (an AND on every host the survey tested). When terms times the country series exceed 8, the place part of the reach goes first, whole, and the terms stay; more than 8 terms read the board unfiltered. Every host matches the whole text, not only the title, so a term like `engineer` cuts less than it looks. A read with terms is never the whole board: it records no size, closes no role it did not list, and the 10% guard holds it only to a read with the same terms and places. The search providers take no terms: each saved search has its own query.
Fetch terms narrow a wide read by words. Each agent has a list of them (`fetch_terms`, set in the agent editor); the terms of a pull are the union over the agents that are on, and none when one of those agents has none. They go only to a board read with the wide reach: a company set to All countries, or a board whose reach is more countries than a one-country host is asked for (`MAX_SERIES`, 8). The narrow reach (Denmark, Sweden) leaves at most about 150 roles on any big board, so it takes no terms, and nor does a board under 300 roles, one never read whole, or one with its own reach. A company set to All roles takes none on any of its boards. A host that takes a keyword gets one series per term, unioned by id, each combined with the country series (an AND on every host the survey tested). When terms times the country series exceed 8, the place part of the reach goes first, whole, and the terms stay; more than 8 terms read the board unfiltered. Every host matches the whole text, not only the title, so a term like `engineer` cuts less than it looks. A read with terms is never the whole board: it records no size, closes no role it did not list, and the 10% guard holds it only to a read with the same terms and places. The search providers take no terms: each saved search has its own query.
What no source does:
+6 -5
View File
@@ -340,10 +340,11 @@ def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None =
A module without `READS` gets `Ask()`.
Fetch terms (`reach["terms"]`) go only to a board reader that declares
`query="one"`, and only on a wide read: `reach["wide"]` (a company
tagged All roles), or a reach this rule just dropped as too many
countries. A board whose params say `"terms": false` or name a query of
their own takes none. Terms times the reach's series (`_cost`) may cost
`query="one"`, and only on a wide read: `reach["wide"]` (a company set
to All countries), or a reach this rule just dropped as too many
countries. A company set to All roles gets no terms in its reach (see
`app/jobs/reach.py`), and a board whose params name a query of their
own takes none. Terms times the reach's series (`_cost`) may cost
at most `MAX_SERIES`: over it, the place part of the reach goes first,
whole, since remote without its countries would read only the remote
roles; over it with terms alone, the terms go too and the board is read
@@ -365,7 +366,7 @@ def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None =
wanted = {name: value for name, value in wanted.items() if name in declared_now}
terms = _terms(given.get("terms"))
if (not wide or reads.scope != "board" or reads.filters.query != "one"
or params.get("terms") is False or "query" in wanted):
or "query" in wanted):
terms = ()
if terms and len(terms) * _cost(reads, wanted) > MAX_SERIES:
wanted = {name: value for name, value in wanted.items()