Files
jobboard/app/jobs/reach.py
T
WolfandClaude Opus 5.5 e52857dea5 Split the All tag into two company overrides: All countries and All roles (WP L)
Fetching has two filters, and a company now overrides each on its own.
All countries (companies.all_countries) is the old All roles tag: its big
boards are read with the agents' wide reach. All roles (companies.all_roles,
new meaning) drops the agents' fetch terms and any later role filter.

- Migration 24 copies the old tag to all_countries and keeps all_roles for
  those companies, and moves a board's {"terms": false} to its company.
- The runner no longer reads {"terms": false}; the plan leaves the terms out
  for a company set to All roles.
- POST /api/companies/{key}/all-countries and /all-roles, each queueing a
  pull when turned on; CompanyOut carries both.
- Two toggles replace All on the Companies row, the reader and the Roles
  strip; the reach line states countries and role types in plain words;
  the filter select gets both options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 12:50:45 +02:00

395 lines
17 KiB
Python

"""The reach of a pull: which countries, and whether remote roles, each
company board is asked for, derived from the search agents that are switched
on. Computed once per pull (`plan`) and handed to each board's batch as the
`reach` the runner narrows to its reader's filters
(`plugins/readers/_runner.py`, `ask_for`).
- **Base reach**: the union of the enabled agents' `any` gates on country,
place (a town or region, mapped to its country through the gazetteer) and
remote. An agent whose `any` group is empty, or holds any other member (a
function, a remote value other than `remote`, a place the gazetteer does
not know), can take a role from anywhere, so the base reach is then
everywhere.
- **Wide reach, for the companies set to All countries**
(`companies.all_countries`, global): the union over the enabled agents of
their `require` reach, the intersection of each one's `require` gates on
country and place, Europe today, expanded to its countries through the
gazetteer, and the base reach, so the override never reads less than the
board would without it. An agent with no `require` gate on place makes it
everywhere. A role in Freiburg may be open to remote, or worth asking the
company about for Denmark. Favourite (affinity 2) is scoring only, where
it skips the `any` gates; it does not widen what is fetched.
- **Board override**: a board whose subscription `params` name a country, a
place or remote gets no reach here; `ask_for` reads the params instead.
- **Small boards**: a board that listed under `SMALL` roles when last read
whole gets no filter; filtering it saves little and risks closing roles.
A reach of more countries than a `one`-filter host should be asked one by
one (the wide reach's Europe on SmartRecruiters) is dropped by `ask_for`,
which knows the reader (`MAX_SERIES`), so such a board is read whole.
**Fetch terms** (`Agent.fetch_terms`) narrow a wide read by words, so a big
board sends the roles that match one of them rather than every role:
- The terms of a pull are the union over the enabled agents. An enabled
agent with none wants every role, so then there are none.
- They go only with the wide reach: a company set to All countries (`wide`
in the reach), and a board whose reach `ask_for` drops as too many
countries for a `one` host. The narrow reach (DK, SE) already leaves at
most about 150 roles on any big board, and small, unknown and overridden
boards get no reach at all, so none of those take terms.
- A company set to All roles (`companies.all_roles`, global) takes none on
any of its boards, and no later role filter (category, level) either. It
changes nothing on the narrow reach today, which takes no terms.
- Only a reader that declares `query="one"` takes them, one series a term;
`ask_for` has the rule for terms times the reach's series.
- A read with terms is never the whole board: it sets no `ats:size`, and
the pull closes nothing against it (`app/jobs/pull.py`).
- The search providers take none. Every saved search has a query of its
own, which terms would replace; a search is a query already.
"""
from __future__ import annotations
from collections.abc import Iterable, Mapping
from dataclasses import dataclass, field
from typing import Any
from app import places
from app.models import Agent, Gate, Subscription, company_key
from app.storage import Store
from plugins import readers
from plugins.readers import _runner
#: A board that listed fewer roles than this when last read whole is read
#: whole.
SMALL = 300
#: `state` key of a board's size, `ats:size:<target>`: the roles its last
#: unfiltered read listed, written by `plugins/ats.py`.
SIZE = "ats:size:"
#: The params that make a board's own reach (see `_runner._PARAM_DIMENSION`).
OVERRIDE_KEYS = frozenset(("countries", "country", "places", "place", "remote"))
@dataclass(frozen=True)
class Reach:
"""Countries (ISO codes) and remote, or everywhere."""
countries: frozenset[str] = frozenset()
remote: bool = False
everywhere: bool = False
def __or__(self, other: Reach) -> Reach:
if self.everywhere or other.everywhere:
return EVERYWHERE
return Reach(self.countries | other.countries, self.remote or other.remote)
def params(self) -> dict[str, Any] | None:
"""What `ask_for` takes as `reach`; None for everywhere."""
if self.everywhere:
return None
out: dict[str, Any] = {"countries": sorted(self.countries)}
if self.remote:
out["remote"] = True
return out
def describe(self) -> str:
""""DK, SE, remote", "31 countries", "everywhere"."""
if self.everywhere:
return "everywhere"
codes = sorted(self.countries)
words = [", ".join(codes)] if len(codes) <= 4 else [f"{len(codes)} countries"]
if self.remote:
words.append("remote")
return ", ".join(w for w in words if w) or "nothing"
EVERYWHERE = Reach(everywhere=True)
NOWHERE = Reach()
def _countries_of(ids: Iterable[int], gaz: places.Gazetteer) -> set[str]:
"""The ISO codes of the countries the places lie in, a continent's every
country."""
out: set[str] = set()
for place_id in ids:
place = gaz.places.get(place_id)
if place is None:
continue
if place.kind == "continent":
out |= {gaz.places[p].key for p in gaz.under((place_id,))
if gaz.places[p].kind == "country"}
elif code := gaz.country(place_id):
out.add(code)
return out
def gate_reach(gate: Gate, gaz: places.Gazetteer) -> Reach:
"""Where a role that meets `gate` can be: its countries, remote, or
everywhere for a gate that says nothing of place or names one the
gazetteer does not know."""
values = [str(v).strip() for v in gate.values if str(v).strip()]
if gate.field == "remote":
return Reach(remote=True) if values and all(v.lower() == "remote" for v in values) \
else EVERYWHERE
if gate.field not in ("country", "place") or not values:
return EVERYWHERE
found: set[str] = set()
for value in values:
named = gaz.lookup(f"country:{value}" if gate.field == "country" else value)
if named is None:
if gate.field == "country" and len(value) == 2 and value.isalpha():
found.add(value.upper())
continue
return EVERYWHERE
found |= _countries_of(named, gaz)
return Reach(frozenset(found)) if found else EVERYWHERE
def any_reach(agent: Agent, gaz: places.Gazetteer) -> Reach:
"""The union of the agent's `any` gates; everywhere with none."""
group = [g for g in agent.gates if g.mode == "any"]
if not group:
return EVERYWHERE
out = NOWHERE
for gate in group:
out = out | gate_reach(gate, gaz)
return out
def require_reach(agent: Agent, gaz: places.Gazetteer) -> Reach:
"""The intersection of the agent's `require` gates on country and place,
or remote alone for a `require` of remote only; everywhere with neither,
or when the intersection is empty."""
kept: frozenset[str] | None = None
remote = False
for gate in agent.gates:
if gate.mode != "require":
continue
reach = gate_reach(gate, gaz)
if reach.everywhere:
continue
if reach.countries:
kept = reach.countries if kept is None else kept & reach.countries
elif reach.remote:
remote = True
if kept:
return Reach(kept)
return Reach(remote=True) if remote and kept is None else EVERYWHERE
def fetch_terms(agents: Iterable[Agent]) -> tuple[str, ...]:
"""The fetch terms of a pull: the union of the agents' terms, one of each
regardless of case, sorted; none when an agent has none, or with no
agent."""
found: dict[str, str] = {}
listed = False
for agent in agents:
listed = True
terms = [t.strip() for t in agent.fetch_terms if t.strip()]
if not terms:
return ()
for term in terms:
found.setdefault(term.casefold(), term)
return tuple(found[k] for k in sorted(found)) if listed else ()
def wide_names(agents: Iterable[Agent], gaz: places.Gazetteer) -> str:
"""The wide reach as the agents' `require` gates name it, for the
Companies list: "Europe" for a gate on that place, and for a gate that
lists countries one by one, the continent they all lie in; up to three
names as written; "everywhere" when an agent has no such gate; "" when
none of that fits, which `line` says as a count."""
names: dict[str, str] = {}
for agent in agents:
gates = [g for g in agent.gates if g.mode == "require"
and g.field in ("country", "place") and not gate_reach(g, gaz).everywhere]
if not gates:
return "everywhere"
for gate in gates:
values = [str(v).strip() for v in gate.values if str(v).strip()]
if len(values) > 3:
values = [_continent(gate_reach(gate, gaz).countries, gaz)]
for text in values:
names.setdefault(text.casefold(), text)
return ", ".join(names.values()) if len(names) <= 3 and "" not in names else ""
def _continent(codes: Iterable[str], gaz: places.Gazetteer) -> str:
"""The continent every one of `codes` lies in, "" for none. A code the
gazetteer has no country for (EU) is skipped."""
known = {p.key for p in gaz.places.values() if p.kind == "country"}
wanted = set(codes) & known
for place in gaz.places.values():
if place.kind != "continent" or not wanted:
continue
under = {gaz.places[p].key for p in gaz.under((place.id,))
if gaz.places[p].kind == "country"}
if wanted <= under:
return place.name
return ""
@dataclass
class Plan:
"""The reach of one pull: `base` for every board, `wide` for the boards
of the companies in `all_countries`, each board's size by target, the
fetch `terms` of the wide reads, which skip the companies in
`all_roles`, and `wide_names`, what the agents call the wide reach."""
base: Reach
wide: Reach = EVERYWHERE
sizes: dict[str, int] = field(default_factory=dict)
all_countries: frozenset[str] = frozenset()
terms: tuple[str, ...] = ()
all_roles: frozenset[str] = frozenset()
wide_names: str = ""
def why(self, sub: Subscription) -> tuple[Reach, str]:
"""The reach `sub` is read with, and why: `override`, `small`,
`unknown` (a board never read whole), `countries` (its company is
set to All countries) or `agents`."""
if OVERRIDE_KEYS & set(sub.params or {}):
return EVERYWHERE, "override"
size = self.sizes.get(sub.target)
if size is None:
return EVERYWHERE, "unknown"
if size < SMALL:
return EVERYWHERE, "small"
if company_key(sub.company or sub.label) in self.all_countries:
return self.wide, "countries"
return self.base, "agents"
def narrows_roles(self, sub: Subscription) -> bool:
"""Whether `sub`'s board may be narrowed by role type: its company is
not set to All roles."""
return company_key(sub.company or sub.label) not in self.all_roles
def for_subscription(self, sub: Subscription) -> dict[str, Any] | None:
"""The `reach` a board's batch hands `ask_for`; None for none. With
fetch terms, and unless its company is set to All roles, a board on
the wide reach carries them and `wide`, and a board on the agents'
reach carries them for `ask_for` to use if it drops the countries as
too many (see the module docstring)."""
found, why = self.why(sub)
out = found.params()
if self.terms and why in ("countries", "agents") and self.narrows_roles(sub):
out = (out or {}) | {"terms": list(self.terms)}
if why == "countries":
out["wide"] = True
return out
def sizes(store: Store, subs: Iterable[Subscription]) -> dict[str, int]:
"""Each board's size: the roles its last unfiltered read listed
(`ats:size:<target>`), else, for a board read before that was kept and
never filtered, its open roles. A board never read has none."""
state = store.state
counts = store.open_countries_by_subscription()
out: dict[str, int] = {}
for sub in subs:
kept = state.get(f"{SIZE}{sub.target}", "")
if kept.isdigit():
out[sub.target] = int(kept)
elif (state.get(f"ats:ok:{sub.target}") and not state.get(f"ats:filtered:{sub.target}")
and not state.get(f"ats:narrowed:{sub.target}")):
out[sub.target] = sum(counts.get(sub.id, {}).values())
return out
def plan(store: Store, subs: Iterable[Subscription] | None = None) -> Plan:
"""The reach of a pull, from the enabled agents and the companies' two
overrides, and the sizes of `subs` (every `ats` subscription when
None)."""
gaz = store.gazetteer()
agents = store.agents(enabled_only=True)
base = wide = NOWHERE
for agent in agents:
base = base | any_reach(agent, gaz)
wide = wide | require_reach(agent, gaz)
if not agents:
base = wide = EVERYWHERE
subs = list(store.subscriptions(source="ats") if subs is None else subs)
rows = store.companies()
# All countries never reads less than the board would without it: the
# base's remote, which a `require` of Europe does not name, is kept.
return Plan(base, wide | base, sizes(store, subs),
all_countries=frozenset(c.key for c in rows if c.all_countries),
terms=fetch_terms(agents),
all_roles=frozenset(c.key for c in rows if c.all_roles),
wide_names=wide_names(agents, gaz) if agents else "everywhere")
def reaches(store: Store, subs: Iterable[Subscription]) -> Mapping[str, dict[str, Any]]:
"""`target -> reach` for the boards that get one, what a pull puts in
its batches' payloads."""
subs = list(subs)
found = plan(store, subs)
return {s.target: r for s in subs if (r := found.for_subscription(s)) is not None}
#: Why a board is read whole, as the Companies list says it.
_WHOLE = {"small": f"under {SMALL} roles", "unknown": "size not known yet",
"override": "set here", "countries": "all countries (everywhere)",
"agents": "an agent takes any place"}
def _places(ask: readers.Ask) -> str:
words = sorted(ask.countries) if len(ask.countries) <= 4 \
else [f"{len(ask.countries)} countries"]
if ask.remote:
words.append("remote")
return ", ".join(words)
def _roles(ask: readers.Ask) -> str:
"""The role-type half of the line: the agents' terms the board is
asked for, or all roles."""
if not ask.terms:
return "all roles"
terms = ", ".join(ask.terms) if len(ask.terms) <= 5 else f"{len(ask.terms)} terms"
return f"follows agents' terms: {terms}"
def line(found: Plan, sub: Subscription) -> str:
"""What a pull asks `sub`'s board for, once its reader has narrowed the
reach, in plain words, countries first and role types second:
"follows agents: DK, SE, remote · all roles", "all countries (Europe) ·
follows agents' terms: engineer, product, data", "all countries (Europe)
· all roles", "set here: DE · all roles", "all places, too many countries
to ask · follows agents' terms: engineer", or, for a board read whole,
why: "under 300 roles: read whole", "Greenhouse cannot filter: read
whole"."""
module = readers.HOSTS.get(sub.target.partition(":")[0])
if module is None:
return ""
given, why = found.why(sub)
ask = _runner.ask_for(module, sub.params, found.for_subscription(sub))
roles = _roles(ask)
if _runner.filtered(ask):
if why == "countries":
named = found.wide_names or f"{len(found.wide.countries)} countries"
return f"all countries ({named}) · {roles}"
if why == "override":
return f"set here: {_places(ask)} · {roles}"
return f"follows agents: {_places(ask)} · {roles}"
if ask.terms:
if given.everywhere:
return f"all countries (everywhere) · {roles}"
return f"all places, too many countries to ask · {roles}"
if why in ("agents", "countries") and given.everywhere:
return f"{_WHOLE[why]}: read whole"
if why in ("agents", "countries", "override") and not module.READS.filters.declared():
return f"{module.LABEL} cannot filter: read whole"
reads = module.READS.filters
if why in ("agents", "countries") and reads.country == "one" \
and len(given.countries) > _runner.MAX_SERIES:
return "too many countries to ask: read whole"
if why in ("agents", "countries"):
return f"{module.LABEL} cannot ask for these: read whole"
return f"{_WHOLE[why]}: read whole"