A European seed of 362 employers, readable or not; discover finds a company's board from its site

The seed was jobhunt-hq's 186 US boards, chosen because they could be read. It is now a hand-picked list of European tech employers, with `board` filled by the new `app.jobs.discover` from each company's own careers site and every board fetched once before it landed: 175 have one, and the rest keep the ATS host their site runs on (Teamtailor 32, Workday 12, SuccessFactors 9, ...) and a careers URL, as rows the app cannot read yet. Companies shows them as "not readable yet" with a link, behind a filter, so the next parser to write is the host that filter shows most.

`app companies discover NAME DOMAIN [--add]` reads the home page, its careers links and the usual paths for a link into a known host, one level down on a careers page, and probes Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio by slug when the pages link nothing. It is what the LinkedIn flow will call.

Migration 7 adds companies.ats, country and city. The Workable parser now reads the v3 shape its list endpoint actually sends (location object, department list, shortcode and no URL or text); the first populated board showed it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-28 15:27:14 +02:00
1 parent c4f2bcc0c9
commit 51d1e67ff9
19 files changed
+3362 -1394

No files matched your search

+4 -1
View File
@@ -8,7 +8,7 @@ Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeli
```bash
uv sync --all-extras
uv run app companies seed # 186 companies from seed/companies.json, one board each
uv run app companies seed # 362 European employers from seed/companies.json, 175 with a readable board
uv run app job pull # read every board and search, then extract, embed and score
uv run app serve # http://127.0.0.1:8790
```
@@ -24,8 +24,11 @@ uv run app searches add jobindex "Copenhagen golang" --q "golang go-udvikler" --
uv run app searches add adzuna "Berlin backend" --q backend --country de --where Berlin
uv run app searches list
uv run app companies prune --in DK,SE,NO,EU # lists; add --apply to unsubscribe
uv run app companies discover "Pleo" pleo.io --add # find its board on its own site, then follow it
```
The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, Teamtailor, its own pages) is still a row, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint when the pages link nothing, and with `--add` follows what it found the same way the seed does. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles; the next pull after switching it back on reopens the ones it still lists. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
## The tabs
+58 -18
View File
@@ -3,7 +3,7 @@
app serve [--host H] [--port P] [--reload]
app job pull|extract|embed|score [--payload JSON]
app worker [--workers http=8,cpu=1] [--until-idle]
app companies seed|add|check|prune
app companies seed|add|discover|check|prune
app searches add|list|remove
app db init
@@ -67,11 +67,17 @@ def build_parser() -> argparse.ArgumentParser:
companies = sub.add_parser("companies", help="the target company list")
companies_sub = companies.add_subparsers(dest="action", required=True)
seed = companies_sub.add_parser(
"seed", help="import seed/companies.json, subscribing to each board")
"seed", help="import seed/companies.json, subscribing to each readable board")
seed.add_argument("--path", default="seed/companies.json")
add = companies_sub.add_parser("add", help="add one company from its board")
add.add_argument("board", help="host:slug, e.g. greenhouse:acme")
add.add_argument("name")
discover = companies_sub.add_parser(
"discover", help="find a company's board from its website; --add follows it")
discover.add_argument("name")
discover.add_argument("domain", help="its website, e.g. pleo.io")
discover.add_argument("--add", action="store_true",
help="add the company, with its board when one is readable")
companies_sub.add_parser("check", help="report boards that do not answer")
prune = companies_sub.add_parser(
"prune", help="unsubscribe every board with no open role in the region")
@@ -234,42 +240,76 @@ def run_companies(args: argparse.Namespace) -> int:
return 0
if args.action == "add":
return add_company(store, args.board, args.name)
if args.action == "discover":
return discover_company(store, args.name, args.domain, add=args.add)
if args.action == "prune":
return prune_companies(store, args.region, apply=args.apply)
return check_companies(store)
def seed_companies(store: Store, path: str) -> dict[str, int]:
"""Import `seed/companies.json`: one `companies` row and one `ats`
subscription per entry, skipping a key that already exists. Not
idempotent on edits to an existing row — re-running it only ever adds
what is missing."""
from plugins.ats import is_known_target
"""Import `seed/companies.json`: one `companies` row per entry, and one
`ats` subscription for each entry with a readable `board`. An entry
without one is the point of the list as much as one with: a company worth
following whose careers site the app cannot read yet keeps its `ats`
host and `careers_url`, and the Companies section says so. A key that
already exists is skipped, so re-running only ever adds what is missing;
it never edits a row."""
from app.models import company_key
from plugins.ats import careers_url, is_known_target
data = json.loads(Path(path).read_text(encoding="utf-8"))
added = skipped = 0
added = readable = skipped = 0
for row in data.get("companies", []):
key, name, board = row["key"], row["name"], row["board"]
name = row["name"]
key = row.get("key") or company_key(name)
if store.company(key) is not None:
skipped += 1
continue
if not is_known_target(board):
print(f"companies seed: {key}: {board!r} is not a known board; skipped",
board = row.get("board") or ""
if board and not is_known_target(board):
print(f"companies seed: {key}: {board!r} is not a known board; kept without it",
file=sys.stderr)
skipped += 1
board = ""
host = board.partition(":")[0]
store.add_company(
name, key=key, domain=row.get("domain", ""),
careers_url=row.get("careers_url") or (careers_url(board) if board else ""),
ats=row.get("ats") or host, country=row.get("country", ""),
city=row.get("city", ""))
added += 1
if not board:
continue
# A seed's `affinity` is not read: affinity is per agent now, and a
# seed has no agent to set it for.
store.add_company(name, key=key, careers_url=row.get("careers_url", ""))
host, _, _slug = board.partition(":")
try:
store.add_subscription("ats", board, label=name, company=name, board=host)
readable += 1
except ValueError as exc:
# The company row landed; the subscription is the only thing
# missing, and worth saying rather than losing silently.
print(f"companies seed: {key}: {exc}", file=sys.stderr)
added += 1
return {"added": added, "skipped": skipped, "total": len(data.get("companies", []))}
return {"added": added, "readable": readable, "skipped": skipped,
"total": len(data.get("companies", []))}
def discover_company(store: Store, name: str, domain: str, *, add: bool) -> int:
"""`app companies discover "Pleo" pleo.io [--add]`: what the company's
site says its board is, as JSON. With `--add`, the company is added the
way the seed adds it: with its board when one is readable, as a row the
app cannot read yet otherwise."""
from app import jobs
found = jobs.discover.discover(name, domain)
out: dict[str, Any] = {"name": name, "domain": domain, "board": found.board,
"ats": found.ats, "careers_url": found.careers_url,
"evidence": found.evidence}
if add:
try:
out["key"], out["roles"] = jobs.sources.add_discovered(store, name, domain, found)
except ValueError as exc:
print(f"companies discover: {exc}", file=sys.stderr)
return 1
print(json.dumps(out))
return 0
def add_company(store: Store, board: str, name: str) -> int:
+4 -3
View File
@@ -10,13 +10,14 @@ task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
offers, the `pull` job a schedule fires, and that schedule
sources.py what a pull reads, changed by hand: boards and searches
added, switched on or off, pruned by region
discover.py which ATS a company's careers site runs on, from its pages
"""
from __future__ import annotations
from typing import Any
from app.jobs import cascade, embed, extract, pull, score, sources, summarise
from app.jobs import cascade, discover, embed, extract, pull, score, sources, summarise
from app.jobs.pull import search_target, seed_searches
from app.jobs.score import seed_agents
@@ -35,5 +36,5 @@ def bootstrap(config_path: str = "") -> tuple[Any, Any]:
return config, Store(config.database)
__all__ = ["BOOTSTRAP", "bootstrap", "cascade", "embed", "extract", "pull", "score",
"search_target", "seed_agents", "seed_searches", "sources", "summarise"]
__all__ = ["BOOTSTRAP", "bootstrap", "cascade", "discover", "embed", "extract", "pull",
"score", "search_target", "seed_agents", "seed_searches", "sources", "summarise"]
+343
View File
@@ -0,0 +1,343 @@
"""Board discovery: which applicant tracking system a company's careers site
runs on, read off its own pages.
The seed list and the LinkedIn flow both start from a company name and a
website, not from a board. This module gets from the one to the other: it
fetches the site's careers page, or the pages the home page links to as
careers, and looks for a link into one of the hosts in `HOSTS`. A readable
host (one the `ats` plugin parses) becomes a `board` target the pull can
read; any other host is recorded as `ats`, so the company is kept as one the
app cannot read yet rather than dropped.
When no page names a host, the three hosts with a cheap per-slug endpoint
(Greenhouse, Lever, Ashby) are probed with the company's own slug, because a
site that draws its jobs from an iframe or a script leaves no link to find.
found = discover("Pleo", "pleo.io")
found.board # "greenhouse:pleo", or "" when nothing readable
found.ats # "greenhouse", "workday", ... or "" when unknown
found.careers_url # the page that named the host, or the careers page
found.evidence # the URL that named it
Only https is fetched, redirects are followed at most five hops, a body is
cut at `MAX_BYTES`, and every failure is one page skipped. A company's site
is content the app was pointed at by its owner, not an ATS API, so nothing
here trusts what it finds beyond a host name matching a fixed pattern.
"""
from __future__ import annotations
import logging
import re
from collections.abc import Callable, Iterable
from dataclasses import dataclass
from urllib.parse import urljoin
import httpx
from desk.sources import USER_AGENT
log = logging.getLogger(__name__)
TIMEOUT = 10.0
MAX_BYTES = 2 << 20
#: How many pages of one site are read before giving up on links.
MAX_PAGES = 6
#: The hosts the `ats` plugin parses; a match on one of these is a `board`.
READABLE = ("greenhouse", "lever", "ashby", "workable", "smartrecruiters",
"recruitee", "personio")
#: host -> pattern over a URL. Group 1, where there is one, is the slug the
#: host names the company by; a host with no group is recognised but its
#: board has no slug the app could use. Order matters only for readability:
#: the readable hosts come first so the first match on a page is the one the
#: app can read.
HOSTS: tuple[tuple[str, re.Pattern[str]], ...] = tuple(
(host, re.compile(pattern, re.IGNORECASE)) for host, pattern in (
("greenhouse", r"(?:boards|job-boards|api)\.(?:eu\.)?greenhouse\.io/(?:v1/boards/|embed/job_board\?[^\"'\s]*for=)?([A-Za-z0-9_-]+)"),
("greenhouse", r"greenhouse\.io/embed/job_board/js\?for=([A-Za-z0-9_-]+)"),
("lever", r"jobs\.(?:eu\.)?lever\.co/([A-Za-z0-9_-]+)"),
("lever", r"api\.(?:eu\.)?lever\.co/v0/postings/([A-Za-z0-9_-]+)"),
("ashby", r"jobs\.ashbyhq\.com/([A-Za-z0-9_.-]+)"),
("ashby", r"api\.ashbyhq\.com/posting-api/job-board/([A-Za-z0-9_.-]+)"),
("workable", r"apply\.workable\.com/(?:api/v\d/accounts/)?([A-Za-z0-9_-]+)"),
("workable", r"https?://(?!apply\.|www\.)([a-z0-9-]+)\.workable\.com"),
("smartrecruiters", r"(?:jobs|careers|api)\.smartrecruiters\.com/(?:v1/companies/)?([A-Za-z0-9_-]+)"),
("recruitee", r"https?://([a-z0-9-]+)\.recruitee\.com"),
("personio", r"https?://([a-z0-9-]+)\.jobs\.personio\.(?:de|com)"),
# Not readable. The slug, where the URL has one, is kept for the day
# a parser exists.
("workday", r"https?://([a-z0-9-]+)\.wd\d+\.myworkdayjobs\.com"),
("successfactors", r"(?:career|jobs|performancemanager)\d*\.successfactors\.(?:eu|com)|\.sapsf\.(?:eu|com)"),
("teamtailor", r"https?://([a-z0-9-]+)\.teamtailor\.com"),
("teamtailor", r"cdn\.teamtailor\.com|teamtailor-cdn|teamtailor\.com/assets"),
("icims", r"https?://([a-z0-9-]+)\.icims\.com"),
("taleo", r"\.taleo\.net"),
("jobvite", r"jobs\.jobvite\.com/([a-z0-9-]+)"),
("bamboohr", r"https?://([a-z0-9-]+)\.bamboohr\.com"),
("join", r"join\.com/companies/([a-z0-9-]+)"),
("softgarden", r"https?://([a-z0-9-]+)\.softgarden\.io"),
("pinpoint", r"https?://([a-z0-9-]+)\.pinpointhq\.com"),
("rippling", r"ats\.rippling\.com/([a-z0-9-]+)"),
("gem", r"jobs\.gem\.com/([a-z0-9-]+)"),
("breezy", r"https?://([a-z0-9-]+)\.breezy\.hr"),
("homerun", r"https?://([a-z0-9-]+)\.homerun\.co"),
("jazzhr", r"https?://([a-z0-9-]+)\.applytojob\.com"),
("welcometothejungle", r"welcometothejungle\.com/[a-z]{2}/companies/([a-z0-9-]+)"),
("eightfold", r"\.eightfold\.ai"),
("phenom", r"phenompeople\.com|phenom\.com/careers"),
("avature", r"\.avature\.net"),
("oracle", r"oraclecloud\.com/hcmUI"),
("comeet", r"comeet\.com/jobs/([a-z0-9-]+)"),
("hibob", r"https?://([a-z0-9-]+)\.hibob\.com"),
))
#: Paths a careers page usually lives at, tried after the home page's own
#: links so the site's word for it wins over a guess.
CAREERS_PATHS = ("/careers", "/careers/", "/jobs", "/en/careers", "/company/careers",
"/about/careers", "/join-us", "/careers/jobs", "/en/jobs")
#: Subdomains careers pages move to.
CAREERS_HOSTS = ("careers.", "jobs.", "career.")
#: A link text or href that points at a careers page.
_CAREERS_LINK = re.compile(
r"career|jobs|join[-_ ]?us|work[-_ ]?with[-_ ]?us|open[-_ ]?positions|vacanc|"
r"we[-_ ]?are[-_ ]?hiring|ledige[-_ ]?stillinger|stellen|karriere|lediga[-_ ]?jobb",
re.IGNORECASE)
_HREF = re.compile(r"""(?:href|src|action|content)\s*=\s*["']([^"'#][^"']*)["']""",
re.IGNORECASE)
_ANCHOR = re.compile(r"<a\b[^>]*href\s*=\s*[\"']([^\"'#][^\"']*)[\"'][^>]*>(.*?)</a>",
re.IGNORECASE | re.DOTALL)
_TAGS = re.compile(r"<[^>]+>")
#: Words a pattern's slug group can catch that name no board: the hosts'
#: own path segments, and the analytics subdomain Recruitee sites embed.
_NOT_SLUGS = frozenset({"embed", "boards", "v1", "api", "www", "jobs", "careers", "apply",
"careers-analytics"})
Found = tuple[str, str, str] # (host, slug, the URL that named it)
@dataclass(frozen=True)
class Discovery:
"""What discovery learned about one company; every field may be empty."""
board: str = ""
ats: str = ""
careers_url: str = ""
evidence: str = ""
@property
def readable(self) -> bool:
return bool(self.board)
# ------------------------------------------------------------------ parsing
def boards_in(text: str) -> list[Found]:
"""Every ATS host named in `text` (a page, a script, a mail), in order of
first appearance, once each. `slug` is "" for a host whose URL carries
none."""
seen: set[tuple[str, str]] = set()
out: list[Found] = []
hits: list[tuple[int, str, str, str]] = []
for host, pattern in HOSTS:
for m in pattern.finditer(text):
slug = (m.group(1) if m.groups() and m.group(1) else "").lower()
if slug in _NOT_SLUGS:
continue
hits.append((m.start(), host, slug, m.group(0)))
for _pos, host, slug, url in sorted(hits):
if (host, slug) in seen:
continue
seen.add((host, slug))
out.append((host, slug, url))
return out
def careers_links(html: str, base: str) -> list[str]:
"""The links on a page that read as its careers page, absolute, in page
order, once each. Nothing off-site is followed: a careers page on another
domain is what `boards_in` finds by its host."""
out: list[str] = []
site = httpx.URL(base).host or ""
for href, label in _ANCHOR.findall(html):
text = _TAGS.sub(" ", label)
if not (_CAREERS_LINK.search(href) or _CAREERS_LINK.search(text)):
continue
url = urljoin(base, href.strip())
parsed = httpx.URL(url)
if parsed.scheme not in ("http", "https"):
continue
host = parsed.host or ""
if host != site and not host.endswith("." + _apex(site)):
continue
url = str(parsed.copy_with(scheme="https", fragment=None))
if url not in out:
out.append(url)
return out
def _apex(host: str) -> str:
parts = host.split(".")
return ".".join(parts[-2:]) if len(parts) > 2 else host
def slug_guesses(name: str, domain: str) -> list[str]:
"""The slugs a host most likely knows the company by: its key, the key
without hyphens, and the domain's own label."""
from app.models import company_key
key = company_key(name)
label = domain.lower().split("/")[0]
for prefix in ("www.",):
label = label.removeprefix(prefix)
label = label.split(".")[0]
out = []
for guess in (key, key.replace("-", ""), label, label.replace("-", "")):
if guess and guess not in out:
out.append(guess)
return out
# ----------------------------------------------------------------- fetching
def fetch_page(client: httpx.Client, url: str) -> tuple[str, str]:
"""`(final URL, text)` for one page, or `("", "")` when it did not answer
with a 2xx text body within the limits. Never raises."""
try:
response = client.get(
url, timeout=TIMEOUT, follow_redirects=True,
headers={"User-Agent": USER_AGENT,
"Accept": "text/html,application/xhtml+xml,*/*;q=0.5"})
if not response.is_success:
return "", ""
body = response.content[:MAX_BYTES]
return str(response.url), body.decode(response.encoding or "utf-8", "replace")
except Exception as exc: # one page skipped, whatever went wrong
log.debug("discover: %s: %s", url, exc)
return "", ""
#: host -> a cheap URL that answers 2xx only when the slug is a board there.
#: Workable is left out: its API sits behind a rate limit that a run of
#: guesses trips, and a Workable site links its board plainly anyway.
_PROBES = {
"greenhouse": "https://boards-api.greenhouse.io/v1/boards/{slug}",
"lever": "https://api.lever.co/v0/postings/{slug}?limit=1&mode=json",
"ashby": "https://api.ashbyhq.com/posting-api/job-board/{slug}",
"smartrecruiters": "https://api.smartrecruiters.com/v1/companies/{slug}/postings?limit=1",
"recruitee": "https://{slug}.recruitee.com/api/offers/",
"personio": "https://{slug}.jobs.personio.de/xml",
}
def probe(client: httpx.Client, host: str, slug: str) -> bool:
"""Whether `host` has a board called `slug`, by its cheapest endpoint.
Ashby and SmartRecruiters answer an empty listing for an unknown slug,
so there a listing without jobs counts as no board."""
url = _PROBES[host].format(slug=slug)
try:
response = client.get(url, timeout=TIMEOUT, follow_redirects=False,
headers={"User-Agent": USER_AGENT})
except Exception:
return False
if not response.is_success:
return False
body = response.content
if host == "ashby":
return b'"jobs":[' in body and b'"jobs":[]' not in body
if host == "smartrecruiters":
return b'"totalFound":' in body and b'"totalFound":0' not in body
if host == "recruitee":
return b'"offers":[' in body and b'"offers":[]' not in body
if host == "personio":
return b"<position>" in body
return True
Fetch = Callable[[httpx.Client, str], tuple[str, str]]
Probe = Callable[[httpx.Client, str, str], bool]
def _site(domain: str) -> str:
domain = domain.strip().lower()
domain = re.sub(r"^https?://", "", domain).split("/")[0]
return domain.removeprefix("www.")
def _pages(domain: str, home_links: Iterable[str]) -> list[str]:
site = _site(domain)
urls = list(home_links)
urls += [f"https://{site}{path}" for path in CAREERS_PATHS]
urls += [f"https://{sub}{site}/" for sub in CAREERS_HOSTS]
out: list[str] = []
for url in urls:
if url not in out:
out.append(url)
return out
def discover(name: str, domain: str, *, fetch: Fetch = fetch_page,
probe_slug: Probe = probe, client: httpx.Client | None = None) -> Discovery:
"""Where `name`'s careers site draws its jobs from; see the module
docstring for the order pages are read in."""
site = _site(domain)
if not site:
return Discovery()
own = client or httpx.Client()
try:
return _discover(name, site, own, fetch, probe_slug)
finally:
if client is None:
own.close()
def _discover(name: str, site: str, client: httpx.Client, fetch: Fetch,
probe_slug: Probe) -> Discovery:
home_url, home = fetch(client, f"https://{site}/")
unreadable: Discovery | None = None
careers_page = ""
def first_readable(found: list[Found], page: str) -> Discovery | None:
nonlocal unreadable
for host, slug, url in found:
if host in READABLE and slug:
return Discovery(board=f"{host}:{slug}", ats=host, careers_url=page,
evidence=url)
if unreadable is None and found:
host, slug, url = found[0]
unreadable = Discovery(ats=host, careers_url=page, evidence=url)
return None
if home and (hit := first_readable(boards_in(home), home_url)):
return hit
# The home page's own careers links, then the usual paths; a careers page's
# own links (a "See open positions" one level down) go to the front of the
# queue. `MAX_PAGES` fetches in all, whatever the queue holds.
queue = _pages(site, careers_links(home, home_url) if home else [])
seen = {home_url, f"https://{site}/"}
budget = MAX_PAGES
while queue and budget > 0:
url = queue.pop(0)
if url in seen:
continue
seen.add(url)
budget -= 1
final, text = fetch(client, url)
if not text:
continue
seen.add(final)
careers_page = careers_page or final
if hit := first_readable(boards_in(text), final):
return hit
queue[:0] = [link for link in careers_links(text, final)[:2] if link not in seen]
for slug in slug_guesses(name, site):
for host in _PROBES:
if probe_slug(client, host, slug):
return Discovery(board=f"{host}:{slug}", ats=host,
careers_url=careers_page, evidence=_PROBES[host].format(slug=slug))
if unreadable is not None:
return unreadable
return Discovery(careers_url=careers_page)
+26 -1
View File
@@ -11,13 +11,16 @@ roles; switched back on, the next pull reopens every role it still lists.
from __future__ import annotations
import os
from typing import Any
from typing import TYPE_CHECKING, Any
from app.config import Config
from app.models import EUROPE, Subscription, company_key
from app.storage import Store
from app.jobs.pull import search_target
if TYPE_CHECKING:
from app.jobs.discover import Discovery
#: The providers `plugins.search` runs, and which of them needs a key.
PROVIDERS = ("jobindex", "adzuna")
@@ -91,6 +94,28 @@ def add_board(store: Store, board: str, name: str) -> tuple[Subscription, int]:
return sub, len(rows)
def add_discovered(store: Store, name: str, domain: str,
found: Discovery) -> tuple[str, int]:
"""Follow a company from what `app.jobs.discover` found: through
`add_board` when the board is readable, else as a row with its `ats` host
and careers page, so the company is kept as one the app cannot read yet.
Returns the company key and how many roles its board lists (0 with no
board). Refuses a company that exists, as `add_board` does."""
name = name.strip()
if not name:
raise ValueError("a company needs a name")
key = company_key(name)
if found.board:
_sub, roles = add_board(store, found.board, name)
store.update_company(key, domain=domain, ats=found.ats)
return key, roles
if store.company(key) is not None:
raise ValueError(f"{name} is already followed")
store.add_company(name, key=key, domain=domain, ats=found.ats,
careers_url=found.careers_url)
return key, 0
def search_params(provider: str, q: str, *, area: int = 0, country: str = "",
where: str = "", pages: int = 0) -> dict[str, Any]:
"""The `params` a search subscription stores, with only the keys its
+6
View File
@@ -314,6 +314,12 @@ class Company(Model):
status: str = "researching"
careers_url: str = ""
notes: str = ""
#: The host the careers site runs on, readable or not; "" when unknown.
#: A company with an `ats` and no board subscription is one the app
#: cannot read yet. See `app.jobs.discover`.
ats: str = ""
country: str = ""
city: str = ""
created_at: str
updated_at: str
+19 -1
View File
@@ -54,9 +54,25 @@
badge.className = "badge";
badge.textContent = company.board;
sub.appendChild(badge);
} else if (company.ats || company.careers_url) {
// Followed, but its careers site runs on nothing the app reads yet.
const badge = document.createElement("span");
badge.className = "badge";
badge.textContent = company.ats || "own site";
sub.appendChild(badge);
sub.append(" · not readable yet");
} else {
sub.append("from job boards");
}
if (company.careers_url && !company.board) {
const link = document.createElement("a");
link.href = company.careers_url;
link.target = "_blank";
link.rel = "noopener";
link.textContent = "careers page";
sub.append(" · ");
sub.appendChild(link);
}
sub.append(` · ${count(company.open_roles, "open role")}`);
return sub;
}
@@ -94,6 +110,7 @@
case "hidden": return company.affinity === -1;
case "on": return company.on === true;
case "off": return company.on === false;
case "unreadable": return company.subscription_id == null && Boolean(company.ats || company.careers_url);
default: return true;
}
}
@@ -102,7 +119,8 @@
const q = $("#c-q").value.trim().toLowerCase();
const filter = $("#c-filter").value;
const rows = [...companies.values()]
.filter((c) => !q || c.name.toLowerCase().includes(q) || c.board.includes(q))
.filter((c) => !q || c.name.toLowerCase().includes(q) || c.board.includes(q)
|| (c.ats || "").includes(q) || (c.country || "").toLowerCase() === q)
.filter((c) => matchesFilter(c, filter))
.sort((a, b) => b.open_roles - a.open_roles || a.name.localeCompare(b.name));
const body = $("#companies-list");
+13 -1
View File
@@ -91,6 +91,10 @@ CREATE TABLE IF NOT EXISTS role_tags (
-- A company as a target, whether or not any board of it is read. key is
-- models.company_key(name), the slug roles.company_key holds.
-- ats is the host the careers site runs on, readable by the ats plugin or
-- not (workday, successfactors, ...): what discovery found, '' unknown. A
-- company with an ats and no board subscription is one the app cannot read
-- yet. country is an ISO code and city the head office's.
CREATE TABLE IF NOT EXISTS companies (
key TEXT PRIMARY KEY,
name TEXT NOT NULL,
@@ -99,7 +103,10 @@ CREATE TABLE IF NOT EXISTS companies (
careers_url TEXT NOT NULL DEFAULT '',
notes TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
updated_at TEXT NOT NULL,
ats TEXT NOT NULL DEFAULT '',
country TEXT NOT NULL DEFAULT '',
city TEXT NOT NULL DEFAULT ''
);
-- The targets desk's runner hands to plugins. newsdesk's shape, with its two
@@ -319,6 +326,11 @@ SELECT (SELECT id FROM agents ORDER BY enabled DESC, id LIMIT 1), key, affinity,
ALTER TABLE roles DROP COLUMN vote;
ALTER TABLE roles DROP COLUMN voted_at;
ALTER TABLE companies DROP COLUMN affinity;
"""),
(7, "companies.ats, country and city: a company the app cannot read yet is still a row", """
ALTER TABLE companies ADD COLUMN ats TEXT NOT NULL DEFAULT '';
ALTER TABLE companies ADD COLUMN country TEXT NOT NULL DEFAULT '';
ALTER TABLE companies ADD COLUMN city TEXT NOT NULL DEFAULT '';
"""),
)
+7 -5
View File
@@ -69,7 +69,7 @@ _VOTED_SCORE = ("MIN(1.0, s.score * CASE v.vote WHEN -1 THEN 0.5 "
#: nothing, so `vote` and `voted_at` are NULL: one query shape either way.
_WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?"
_COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes")
_COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes", "ats", "country", "city")
#: A company as the store serves it: the row and the asked-for agent's affinity.
_COMPANY_SELECT = """SELECT c.*, COALESCE(ac.affinity, 0) AS affinity FROM companies c
@@ -944,7 +944,8 @@ class Store:
def add_company(self, name: str, *, key: str = "", domain: str = "",
status: str = "researching", careers_url: str = "",
notes: str = "") -> Company:
notes: str = "", ats: str = "", country: str = "",
city: str = "") -> Company:
"""Raises ValueError on a key already taken."""
name = name.strip()
key = key or company_key(name)
@@ -954,9 +955,10 @@ class Store:
try:
self._write(
"""INSERT INTO companies (key, name, domain, status, careers_url, notes,
created_at, updated_at)
VALUES (?,?,?,?,?,?,?,?)""",
(key, name, domain, status, careers_url, notes, stamp, stamp))
ats, country, city, created_at, updated_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?)""",
(key, name, domain, status, careers_url, notes, ats, country, city,
stamp, stamp))
except sqlite3.IntegrityError as exc:
raise ValueError(f"company {key!r} already exists") from exc
return self._company(key)
+2 -1
View File
@@ -86,7 +86,7 @@
</form>
<form class="filters" autocomplete="off" onsubmit="return false">
<input type="search" id="c-q" placeholder="Filter by name or board" aria-label="Filter companies">
<input type="search" id="c-q" placeholder="Filter by name, board or country code" aria-label="Filter companies">
<select id="c-filter" aria-label="Filter">
<option value="">Any</option>
<option value="favourite">Favourite</option>
@@ -94,6 +94,7 @@
<option value="hidden">Hidden</option>
<option value="on">On</option>
<option value="off">Off</option>
<option value="unreadable">Not readable yet</option>
</select>
{# Liked and hidden are per search agent: this picks whose the list shows
and whose the buttons set. companies.js fills it from /api/agents. #}
+8 -2
View File
@@ -262,7 +262,9 @@ class CompanyOut(BaseModel):
"""A company row plus what only the store can join in: the ats host it is
read from (empty for a company known only from a search), the id of that
board's subscription, whether it is on, and how many of its roles are
still open."""
still open. `ats` is the host its careers site runs on whether or not the
app reads it, so a followed company with no `board` and an `ats` is one
the app cannot read yet; `careers_url` is where a person can."""
key: str
name: str
@@ -271,6 +273,9 @@ class CompanyOut(BaseModel):
on: bool | None = None
affinity: int
open_roles: int
ats: str = ""
careers_url: str = ""
country: str = ""
def _company_boards(store: Store) -> dict[str, tuple[int, str, bool]]:
@@ -296,7 +301,8 @@ def list_companies(store: StoreDep, agent: int | None = None) -> list[CompanyOut
sub_id, host, on = boards.get(c.key, (None, "", None))
out.append(CompanyOut(key=c.key, name=c.name, board=host,
subscription_id=sub_id, on=on, affinity=c.affinity,
open_roles=counts.get(c.key, 0)))
open_roles=counts.get(c.key, 0), ats=c.ats,
careers_url=c.careers_url, country=c.country))
return out
+7 -1
View File
@@ -42,6 +42,12 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Boards follow companies; searches find them.** `ats` only reads companies someone listed, so it cannot surface a role at a company nobody knew of, and the seed list came from a US-centred project: of 29,000 open roles, 31 were in Denmark. The `search` plugin runs saved queries on boards that span companies, as `search` subscriptions whose `params` are the query, and pulls them in the same task as the boards.
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list.
**`discover` reads the site before it guesses.** A careers page that links its board is the authority, so the home page, its own careers links, then the usual paths are read for a link into a known host, and a page's own "open positions" link one level down. Only when no page names a host are Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio probed with the company's slug, because a board drawn by a script leaves no link. A probe hit on a common name can be another tenant (`recruitee:google` exists, and is not Google), which is why the seed's boards were fetched and read before they landed, and why `discover --add` reports the roles it found.
**A company row carries `ats`, `country` and `city`.** `ats` is what discovery found whether or not the app reads it; a company with an `ats` and no board subscription is one it cannot read yet. `country` and `city` come from the seed for now and are the first of the company facts the feed will filter by; funding, size and industry come from a source that has them, not from a column filled by hand.
**Jobindex and Adzuna, not Jobnet or Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jobnet's search endpoint answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. A search whose results ran out before its page limit did list everything, and closes like a board; the rest close once no search has listed them for 21 days. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists.
@@ -58,7 +64,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
**Workable's shape is confirmed, not a populated example.** Every slug tried during the build returned the documented empty `{"total": 0, "results": []}` before Cloudflare rate-limited the session, so the parser follows Workable's published job object. Recheck it against a live board with open roles the next time the plugin is touched.
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`, and `description` stays empty as for SmartRecruiters rather than costing one request per role.
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs.
+25 -19
View File
@@ -11,7 +11,8 @@ The field map, the redirect allow-list and the 10 s timeout are ported from
`classify`'s nine functions and their keywords. Workable, SmartRecruiters,
Recruitee and Personio are new: jobhunt-hq only read Greenhouse, Lever and
Ashby, so those four are written from one real board of each host, fetched
while this file was written (2026-09-23), rather than from the Go source.
while this file was written (2026-09-23; Workable on 2026-09-28, when the
first board with roles was seen), rather than from the Go source.
The body cap is not a straight port — see `MAX_BYTES` below.
@@ -256,31 +257,36 @@ def _parse_ashby(body: bytes) -> list[Row]:
return rows
def _parse_workable(body: bytes) -> list[Row]:
"""https://apply.workable.com/api/v3/accounts/{slug}/jobs (POST, empty
JSON body) -> `{"total": int, "results": [...]}`. The shape is confirmed
live (2026-09-23); the fields are Workable's long-documented job object,
since every slug this session tried on it happened to list nothing at
the moment of fetching — see the plugin's docstring and the build report."""
def _parse_workable(body: bytes, slug: str) -> list[Row]:
"""POST https://apply.workable.com/api/v3/accounts/{slug}/jobs with `{}` ->
`{"total": int, "results": [...]}`, as Hugging Face's board answered on
2026-09-28. A result carries no URL and no text: the job page is
`https://apply.workable.com/{slug}/j/{shortcode}/`, and the description
is only on the per-job endpoint, one request per role, so `description`
stays empty here as it does for SmartRecruiters. `location` is an object
and `department` a list; `remote` is a flag beside `workplace`
(remote, hybrid, on_site)."""
data = json.loads(body)
rows = []
for j in data.get("results") or []:
title = (j.get("title") or "").strip()
url = (j.get("url") or j.get("shortlink") or j.get("application_url") or "").strip()
if not title or not _is_web_url(url):
shortcode = (j.get("shortcode") or "").strip()
if not title or not shortcode:
continue
location = ", ".join(p for p in (j.get("city"), j.get("region"), j.get("country")) if p)
description = " ".join(p for p in (
j.get("description"), j.get("requirements"), j.get("benefits")) if p)
loc = j.get("location") or {}
location = ", ".join(p for p in (loc.get("city"), loc.get("region"), loc.get("country")) if p)
dept = j.get("department") or ""
if isinstance(dept, list):
dept = ", ".join(d for d in dept if d)
rows.append({
"source_id": str(j.get("id") or j.get("shortcode") or ""),
"source_id": str(j.get("id") or shortcode),
"title": title,
"location": location,
"url": url,
"posted": _parse_dt(j.get("published_on") or j.get("created_at")),
"department": (j.get("department") or "").strip(),
"description": strip_html(description),
"remote": j.get("telecommute"),
"url": f"https://apply.workable.com/{slug}/j/{shortcode}/",
"posted": _parse_dt(j.get("published")),
"department": dept.strip(),
"description": "",
"remote": bool(j.get("remote")) or j.get("workplace") == "remote",
"salary": "",
})
return rows
@@ -375,7 +381,7 @@ _PARSERS: dict[str, Callable[..., list[Row]]] = {
"greenhouse": lambda body, slug: _parse_greenhouse(body),
"lever": lambda body, slug: _parse_lever(body),
"ashby": lambda body, slug: _parse_ashby(body),
"workable": lambda body, slug: _parse_workable(body),
"workable": _parse_workable,
"smartrecruiters": _parse_smartrecruiters,
"recruitee": lambda body, slug: _parse_recruitee(body),
"personio": _parse_personio,
+2569 -1305
View File
File diff suppressed because it is too large. Load diff
+34 -25
View File
@@ -2,33 +2,42 @@
"total": 2,
"results": [
{
"id": "4001",
"title": "Backend Engineer",
"department": "Engineering",
"city": "Berlin",
"region": "",
"country": "Germany",
"telecommute": true,
"published_on": "2026-09-01",
"created_at": "2026-08-30T10:00:00.000Z",
"url": "https://workable.example/1",
"shortlink": "https://workable.example/short/1",
"description": "<p>Ship our API.</p>",
"requirements": "<p>5+ years.</p>",
"benefits": ""
"id": 6125026,
"shortcode": "DB4D7C0EC8",
"title": "Senior Open-Source Python Engineer, ML Developer Tools - EMEA Remote",
"remote": true,
"location": {
"country": "France",
"countryCode": "FR",
"city": "",
"region": null
},
"state": "published",
"published": "2026-09-21T00:00:00.000Z",
"type": "full",
"department": [
"Product"
],
"workplace": "remote"
},
{
"id": "4002",
"shortcode": "ABCDEF",
"title": "Account Manager",
"department": "Sales",
"city": "Munich",
"region": "",
"country": "Germany",
"telecommute": false,
"published_on": "2026-08-15",
"url": "https://workable.example/2",
"description": ""
"id": 6125041,
"shortcode": "F88446C814",
"title": "Senior Open-Source Python Engineer, ML Developer Tools - US Remote",
"remote": false,
"location": {
"country": "Germany",
"countryCode": "DE",
"city": "Berlin",
"region": null
},
"state": "published",
"published": "2026-09-21T00:00:00.000Z",
"type": "full",
"department": [
"Product"
],
"workplace": "hybrid"
}
]
}
+9 -5
View File
@@ -59,12 +59,16 @@ def test_ashby_parses_is_remote_and_compensation() -> None:
assert rows[1]["remote"] is True
def test_workable_parses_location_parts_and_telecommute() -> None:
rows = _rows("workable")
assert [r["source_id"] for r in rows] == ["4001", "4002"]
assert rows[0]["location"] == "Berlin, Germany"
def test_workable_builds_the_job_url_and_reads_location_and_remote() -> None:
rows = _rows("workable", slug="huggingface")
assert [r["source_id"] for r in rows] == ["6125026", "6125041"]
assert rows[0]["url"] == "https://apply.workable.com/huggingface/j/DB4D7C0EC8/"
assert rows[0]["location"] == "France"
assert rows[0]["remote"] is True
assert "Ship our API." in rows[0]["description"]
assert rows[0]["department"] == "Product"
assert rows[1]["location"] == "Berlin, Germany"
assert rows[1]["remote"] is False
assert rows[0]["description"] == "" # the list endpoint carries no text
def test_smartrecruiters_builds_the_job_url_from_id_and_slug() -> None:
+211
View File
@@ -0,0 +1,211 @@
"""Board discovery: which ATS a careers site names, and in what order the
site's pages are read."""
from __future__ import annotations
import json
from pathlib import Path
import httpx
import pytest
from app.cli import seed_companies
from app.jobs import discover as mod
from app.jobs.discover import Discovery, boards_in, careers_links, discover, slug_guesses
from app.storage import Store
# ------------------------------------------------------------------ parsing
def test_boards_in_names_readable_hosts_with_their_slug() -> None:
page = """
<a href="https://job-boards.greenhouse.io/pleo">Open roles</a>
<script src="https://boards.greenhouse.io/embed/job_board/js?for=pleo"></script>
<a href="https://jobs.lever.co/spotify/abc-123">One role</a>
<iframe src="https://jobs.ashbyhq.com/DeepL/embed"></iframe>
<a href="https://apply.workable.com/acme/">Workable</a>
<a href="https://jobs.smartrecruiters.com/Bosch/123">SR</a>
<a href="https://acme.recruitee.com/o/backend">Recruitee</a>
<a href="https://acme-gmbh.jobs.personio.de/job/1">Personio</a>
"""
assert [(h, s) for h, s, _url in boards_in(page)] == [
("greenhouse", "pleo"), ("lever", "spotify"), ("ashby", "deepl"),
("workable", "acme"), ("smartrecruiters", "bosch"), ("recruitee", "acme"),
("personio", "acme-gmbh")]
def test_boards_in_keeps_hosts_the_app_cannot_read() -> None:
page = """
<a href="https://novo.wd3.myworkdayjobs.com/en-US/careers">Workday</a>
<a href="https://performancemanager.successfactors.eu/career?company=x">SF</a>
<a href="https://acme.teamtailor.com/jobs">TT</a>
"""
assert boards_in(page) == [
("workday", "novo", "https://novo.wd3.myworkdayjobs.com"),
("successfactors", "", "performancemanager.successfactors.eu"),
("teamtailor", "acme", "https://acme.teamtailor.com")]
def test_boards_in_dedupes_and_skips_the_api_paths_own_words() -> None:
page = ('<a href="https://boards.greenhouse.io/embed/job_board?for=acme">a</a>'
'<a href="https://boards-api.greenhouse.io/v1/boards/acme/jobs">b</a>')
assert [(h, s) for h, s, _ in boards_in(page)] == [("greenhouse", "acme")]
assert boards_in("nothing here") == []
def test_careers_links_keeps_on_site_links_that_read_as_careers() -> None:
html = """
<a href="/about">About</a>
<a href="/careers">Careers</a>
<a href="https://jobs.acme.com/">We are hiring</a>
<a href="https://other.example/jobs">Partner jobs</a>
<a href="/en/karriere#top"><span>Karriere</span></a>
<a href="mailto:jobs@acme.com">Mail</a>
<a href="/careers">Careers again</a>
"""
assert careers_links(html, "https://www.acme.com/") == [
"https://www.acme.com/careers", "https://jobs.acme.com/",
"https://www.acme.com/en/karriere"]
def test_slug_guesses_come_from_the_key_and_the_domain() -> None:
assert slug_guesses("Trade Republic Bank GmbH", "www.traderepublic.com") == [
"trade-republic-bank", "traderepublicbank", "traderepublic"]
assert slug_guesses("Pleo", "pleo.io") == ["pleo"]
# ------------------------------------------------------------------- discover
class Site:
"""A fake web: `pages` maps a URL to `(final URL, html)`; anything else
is a page that did not answer. Remembers the order of fetches."""
def __init__(self, pages: dict[str, tuple[str, str] | str],
boards: set[tuple[str, str]] = frozenset()) -> None:
self.pages = {k: (v if isinstance(v, tuple) else (k, v)) for k, v in pages.items()}
self.boards = boards
self.fetched: list[str] = []
self.probed: list[tuple[str, str]] = []
def fetch(self, client: httpx.Client, url: str) -> tuple[str, str]:
self.fetched.append(url)
return self.pages.get(url, ("", ""))
def probe(self, client: httpx.Client, host: str, slug: str) -> bool:
self.probed.append((host, slug))
return (host, slug) in self.boards
def run(site: Site, name: str = "Acme", domain: str = "acme.com") -> Discovery:
with httpx.Client() as client:
return discover(name, domain, fetch=site.fetch, probe_slug=site.probe, client=client)
def test_a_board_named_on_the_home_page_ends_discovery_there() -> None:
site = Site({"https://acme.com/": '<a href="https://jobs.lever.co/acme">Jobs</a>'})
assert run(site) == Discovery(board="lever:acme", ats="lever",
careers_url="https://acme.com/",
evidence="jobs.lever.co/acme")
assert site.fetched == ["https://acme.com/"]
assert site.probed == []
def test_the_home_pages_own_careers_link_is_read_before_the_usual_paths() -> None:
site = Site({
"https://acme.com/": '<a href="/company/join-us">Join us</a>',
"https://acme.com/company/join-us": ('https://acme.com/company/join-us',
'<a href="https://acme.recruitee.com/">Open</a>'),
"https://acme.com/careers": '<a href="https://jobs.lever.co/wrong">no</a>',
})
found = run(site)
assert found.board == "recruitee:acme"
assert found.careers_url == "https://acme.com/company/join-us"
assert site.fetched == ["https://acme.com/", "https://acme.com/company/join-us"]
def test_a_careers_pages_own_link_is_followed_one_level_down() -> None:
site = Site({
"https://acme.com/": "<p>no links</p>",
"https://acme.com/careers": '<a href="/careers/open-positions">See open positions</a>',
"https://acme.com/careers/open-positions":
'<iframe src="https://jobs.ashbyhq.com/acme"></iframe>',
})
assert run(site).board == "ashby:acme"
assert site.fetched[:3] == ["https://acme.com/", "https://acme.com/careers",
"https://acme.com/careers/open-positions"]
def test_an_unreadable_host_is_kept_when_nothing_readable_turns_up() -> None:
site = Site({
"https://acme.com/": "<p>welcome</p>",
"https://acme.com/careers": '<a href="https://acme.wd3.myworkdayjobs.com/x">Jobs</a>',
})
found = run(site)
assert found == Discovery(ats="workday", careers_url="https://acme.com/careers",
evidence="https://acme.wd3.myworkdayjobs.com")
assert not found.readable
# Every slug was still tried on the hosts with a probe, in case a readable
# board exists that the pages do not link.
assert ("greenhouse", "acme") in site.probed
def test_slug_probes_find_a_board_the_pages_never_link() -> None:
site = Site({"https://acme.com/": "<p>a script draws the jobs</p>",
"https://acme.com/jobs": "<div id=app></div>"},
boards={("ashby", "acme")})
found = run(site)
assert found.board == "ashby:acme"
assert found.careers_url == "https://acme.com/jobs"
assert found.evidence == "https://api.ashbyhq.com/posting-api/job-board/acme"
def test_a_site_that_never_answers_gives_an_empty_discovery() -> None:
site = Site({})
assert run(site) == Discovery()
assert len(site.fetched) <= mod.MAX_PAGES + 1
def test_a_domain_is_folded_to_its_site() -> None:
site = Site({"https://acme.com/": '<a href="https://jobs.lever.co/acme">Jobs</a>'})
with httpx.Client() as client:
found = discover("Acme", "https://www.acme.com/some/page", fetch=site.fetch,
probe_slug=site.probe, client=client)
assert found.board == "lever:acme"
assert discover("Acme", "", fetch=site.fetch, probe_slug=site.probe) == Discovery()
# ---------------------------------------------------------------- the seed
@pytest.fixture
def store(tmp_path: Path) -> Store:
return Store(tmp_path / "t.db")
def test_the_seed_keeps_a_company_the_app_cannot_read(store: Store, tmp_path: Path,
capsys: pytest.CaptureFixture[str]) -> None:
seed = tmp_path / "companies.json"
seed.write_text(json.dumps({"companies": [
{"name": "Pleo", "domain": "pleo.io", "country": "DK", "city": "Copenhagen",
"board": "ashby:pleo"},
{"name": "Novo Nordisk", "domain": "novonordisk.com", "country": "DK",
"city": "Bagsværd", "ats": "successfactors",
"careers_url": "https://www.novonordisk.com/careers.html"},
{"name": "Odd", "board": "nosuch:odd"},
]}))
assert seed_companies(store, str(seed)) == {"added": 3, "readable": 1, "skipped": 0,
"total": 3}
pleo = store.company("pleo")
assert pleo is not None
assert (pleo.ats, pleo.country, pleo.city, pleo.careers_url) == (
"ashby", "DK", "Copenhagen", "https://jobs.ashbyhq.com/pleo")
[sub] = store.subscriptions(source="ats")
assert (sub.target, sub.company) == ("ashby:pleo", "Pleo")
novo = store.company("novo-nordisk")
assert novo is not None
assert (novo.ats, novo.careers_url) == ("successfactors",
"https://www.novonordisk.com/careers.html")
odd = store.company("odd")
assert odd is not None and odd.ats == ""
assert "not a known board" in capsys.readouterr().err
# Re-running adds nothing and edits nothing.
assert seed_companies(store, str(seed))["skipped"] == 3
+7 -3
View File
@@ -93,6 +93,8 @@ def test_a_v1_file_migrates_to_the_shape_a_new_file_has(tmp_path: Path) -> None:
# v1 had affinity on the company; the vote columns v4 added and v6
# drops again are not in a fresh file, so there is nothing to put back.
conn.execute("ALTER TABLE companies ADD COLUMN affinity INTEGER NOT NULL DEFAULT 0")
for column in ("ats", "country", "city"): # v7
conn.execute(f"ALTER TABLE companies DROP COLUMN {column}")
for table in ("vectors", "role_tags", "agent_companies", "role_votes"):
conn.execute(f"DROP TABLE {table}")
conn.execute("PRAGMA user_version = 1")
@@ -113,6 +115,8 @@ def _downgrade_to_v4(path: Path) -> None:
conn.execute("ALTER TABLE roles ADD COLUMN vote INTEGER CHECK (vote IN (-1, 0, 1))")
conn.execute("ALTER TABLE roles ADD COLUMN voted_at TEXT")
conn.execute("ALTER TABLE companies ADD COLUMN affinity INTEGER NOT NULL DEFAULT 0")
for column in ("ats", "country", "city"): # v7
conn.execute(f"ALTER TABLE companies DROP COLUMN {column}")
conn.execute("DROP TABLE agent_companies")
conn.execute("DROP TABLE role_votes")
conn.execute("PRAGMA user_version = 4")
@@ -139,7 +143,7 @@ def test_v6_moves_the_votes_and_affinities_to_the_default_agent(tmp_path: Path)
migrated = Store(path)
assert migrated.migrations_applied == [5, 6]
assert migrated.migrations_applied == list(range(5, LATEST + 1))
assert [a.name for a in migrated.agents()] == ["First", "Second"] # none added
assert migrated.votes(first.id) == {posting("liked").uid: 1, posting("passed").uid: -1}
assert migrated.votes(second.id) == {}
@@ -181,7 +185,7 @@ def test_v6_makes_no_agent_when_there_is_nothing_to_move(tmp_path: Path) -> None
store.add_company("Acme", key="acme")
_downgrade_to_v4(path)
assert Store(path).migrations_applied == [5, 6]
assert Store(path).migrations_applied == list(range(5, LATEST + 1))
assert Store(path).agents() == []
@@ -191,7 +195,7 @@ def test_a_v4_file_migrates_read_state_and_tags(tmp_path: Path) -> None:
Store(old)
_downgrade_to_v4(old)
assert Store(old).migrations_applied == [5, 6]
assert Store(old).migrations_applied == list(range(5, LATEST + 1))
assert sorted(_columns(old, "roles")) == sorted(_columns(fresh, "roles"))
with sqlite3.connect(old) as conn:
tables = {r[0] for r in conn.execute(
+10 -3
View File
@@ -384,17 +384,24 @@ def test_the_roles_page_offers_the_judged_filter_and_hidden_companies(
def test_companies_lists_board_and_open_role_count(
client: TestClient, store: Store) -> None:
sub = store.add_subscription("ats", "greenhouse:acme", company="Acme", board="greenhouse")
store.add_company("Acme", key="acme")
store.add_company("Acme", key="acme", ats="greenhouse", country="DK")
store.upsert_role(posting("1", company="Acme"), sub)
store.add_company("Other", key="other") # known only from a search: no board
store.add_company("Novo", key="novo", ats="successfactors", # followed, not readable
careers_url="https://novo.example/careers")
res = client.get("/api/companies")
assert res.status_code == 200
[row] = [c for c in res.json() if c["key"] == "acme"]
assert row == {"key": "acme", "name": "Acme", "board": "greenhouse",
"subscription_id": sub.id, "on": True, "affinity": 0, "open_roles": 1}
"subscription_id": sub.id, "on": True, "affinity": 0, "open_roles": 1,
"ats": "greenhouse", "careers_url": "", "country": "DK"}
[other] = [c for c in res.json() if c["key"] == "other"]
assert (other["board"], other["subscription_id"], other["on"]) == ("", None, None)
assert (other["board"], other["subscription_id"], other["on"], other["ats"]) == (
"", None, None, "")
[novo] = [c for c in res.json() if c["key"] == "novo"]
assert (novo["board"], novo["subscription_id"], novo["ats"], novo["careers_url"]) == (
"", None, "successfactors", "https://novo.example/careers")
def test_set_affinity_validates_updates_and_enqueues_a_score(