A European seed of 362 employers, readable or not; discover finds a company's board from its site
The seed was jobhunt-hq's 186 US boards, chosen because they could be read. It is now a hand-picked list of European tech employers, with `board` filled by the new `app.jobs.discover` from each company's own careers site and every board fetched once before it landed: 175 have one, and the rest keep the ATS host their site runs on (Teamtailor 32, Workday 12, SuccessFactors 9, ...) and a careers URL, as rows the app cannot read yet. Companies shows them as "not readable yet" with a link, behind a filter, so the next parser to write is the host that filter shows most. `app companies discover NAME DOMAIN [--add]` reads the home page, its careers links and the usual paths for a link into a known host, one level down on a careers page, and probes Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio by slug when the pages link nothing. It is what the LinkedIn flow will call. Migration 7 adds companies.ats, country and city. The Workable parser now reads the v3 shape its list endpoint actually sends (location object, department list, shortcode and no URL or text); the first populated board showed it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
c4f2bcc0c9
commit
51d1e67ff9
19 files changed
+3362
-1394
No files matched your search
@@ -8,7 +8,7 @@ Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeli
|
||||
|
||||
```bash
|
||||
uv sync --all-extras
|
||||
uv run app companies seed # 186 companies from seed/companies.json, one board each
|
||||
uv run app companies seed # 362 European employers from seed/companies.json, 175 with a readable board
|
||||
uv run app job pull # read every board and search, then extract, embed and score
|
||||
uv run app serve # http://127.0.0.1:8790
|
||||
```
|
||||
@@ -24,8 +24,11 @@ uv run app searches add jobindex "Copenhagen golang" --q "golang go-udvikler" --
|
||||
uv run app searches add adzuna "Berlin backend" --q backend --country de --where Berlin
|
||||
uv run app searches list
|
||||
uv run app companies prune --in DK,SE,NO,EU # lists; add --apply to unsubscribe
|
||||
uv run app companies discover "Pleo" pleo.io --add # find its board on its own site, then follow it
|
||||
```
|
||||
|
||||
The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, Teamtailor, its own pages) is still a row, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint when the pages link nothing, and with `--add` follows what it found the same way the seed does. A probe hit is a guess by name: check the roles it lists before trusting it for a common name.
|
||||
|
||||
A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles; the next pull after switching it back on reopens the ones it still lists. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys.
|
||||
|
||||
## The tabs
|
||||
|
||||
+58
-18
@@ -3,7 +3,7 @@
|
||||
app serve [--host H] [--port P] [--reload]
|
||||
app job pull|extract|embed|score [--payload JSON]
|
||||
app worker [--workers http=8,cpu=1] [--until-idle]
|
||||
app companies seed|add|check|prune
|
||||
app companies seed|add|discover|check|prune
|
||||
app searches add|list|remove
|
||||
app db init
|
||||
|
||||
@@ -67,11 +67,17 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
companies = sub.add_parser("companies", help="the target company list")
|
||||
companies_sub = companies.add_subparsers(dest="action", required=True)
|
||||
seed = companies_sub.add_parser(
|
||||
"seed", help="import seed/companies.json, subscribing to each board")
|
||||
"seed", help="import seed/companies.json, subscribing to each readable board")
|
||||
seed.add_argument("--path", default="seed/companies.json")
|
||||
add = companies_sub.add_parser("add", help="add one company from its board")
|
||||
add.add_argument("board", help="host:slug, e.g. greenhouse:acme")
|
||||
add.add_argument("name")
|
||||
discover = companies_sub.add_parser(
|
||||
"discover", help="find a company's board from its website; --add follows it")
|
||||
discover.add_argument("name")
|
||||
discover.add_argument("domain", help="its website, e.g. pleo.io")
|
||||
discover.add_argument("--add", action="store_true",
|
||||
help="add the company, with its board when one is readable")
|
||||
companies_sub.add_parser("check", help="report boards that do not answer")
|
||||
prune = companies_sub.add_parser(
|
||||
"prune", help="unsubscribe every board with no open role in the region")
|
||||
@@ -234,42 +240,76 @@ def run_companies(args: argparse.Namespace) -> int:
|
||||
return 0
|
||||
if args.action == "add":
|
||||
return add_company(store, args.board, args.name)
|
||||
if args.action == "discover":
|
||||
return discover_company(store, args.name, args.domain, add=args.add)
|
||||
if args.action == "prune":
|
||||
return prune_companies(store, args.region, apply=args.apply)
|
||||
return check_companies(store)
|
||||
|
||||
|
||||
def seed_companies(store: Store, path: str) -> dict[str, int]:
|
||||
"""Import `seed/companies.json`: one `companies` row and one `ats`
|
||||
subscription per entry, skipping a key that already exists. Not
|
||||
idempotent on edits to an existing row — re-running it only ever adds
|
||||
what is missing."""
|
||||
from plugins.ats import is_known_target
|
||||
"""Import `seed/companies.json`: one `companies` row per entry, and one
|
||||
`ats` subscription for each entry with a readable `board`. An entry
|
||||
without one is the point of the list as much as one with: a company worth
|
||||
following whose careers site the app cannot read yet keeps its `ats`
|
||||
host and `careers_url`, and the Companies section says so. A key that
|
||||
already exists is skipped, so re-running only ever adds what is missing;
|
||||
it never edits a row."""
|
||||
from app.models import company_key
|
||||
from plugins.ats import careers_url, is_known_target
|
||||
|
||||
data = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
added = skipped = 0
|
||||
added = readable = skipped = 0
|
||||
for row in data.get("companies", []):
|
||||
key, name, board = row["key"], row["name"], row["board"]
|
||||
name = row["name"]
|
||||
key = row.get("key") or company_key(name)
|
||||
if store.company(key) is not None:
|
||||
skipped += 1
|
||||
continue
|
||||
if not is_known_target(board):
|
||||
print(f"companies seed: {key}: {board!r} is not a known board; skipped",
|
||||
board = row.get("board") or ""
|
||||
if board and not is_known_target(board):
|
||||
print(f"companies seed: {key}: {board!r} is not a known board; kept without it",
|
||||
file=sys.stderr)
|
||||
skipped += 1
|
||||
board = ""
|
||||
host = board.partition(":")[0]
|
||||
store.add_company(
|
||||
name, key=key, domain=row.get("domain", ""),
|
||||
careers_url=row.get("careers_url") or (careers_url(board) if board else ""),
|
||||
ats=row.get("ats") or host, country=row.get("country", ""),
|
||||
city=row.get("city", ""))
|
||||
added += 1
|
||||
if not board:
|
||||
continue
|
||||
# A seed's `affinity` is not read: affinity is per agent now, and a
|
||||
# seed has no agent to set it for.
|
||||
store.add_company(name, key=key, careers_url=row.get("careers_url", ""))
|
||||
host, _, _slug = board.partition(":")
|
||||
try:
|
||||
store.add_subscription("ats", board, label=name, company=name, board=host)
|
||||
readable += 1
|
||||
except ValueError as exc:
|
||||
# The company row landed; the subscription is the only thing
|
||||
# missing, and worth saying rather than losing silently.
|
||||
print(f"companies seed: {key}: {exc}", file=sys.stderr)
|
||||
added += 1
|
||||
return {"added": added, "skipped": skipped, "total": len(data.get("companies", []))}
|
||||
return {"added": added, "readable": readable, "skipped": skipped,
|
||||
"total": len(data.get("companies", []))}
|
||||
|
||||
|
||||
def discover_company(store: Store, name: str, domain: str, *, add: bool) -> int:
|
||||
"""`app companies discover "Pleo" pleo.io [--add]`: what the company's
|
||||
site says its board is, as JSON. With `--add`, the company is added the
|
||||
way the seed adds it: with its board when one is readable, as a row the
|
||||
app cannot read yet otherwise."""
|
||||
from app import jobs
|
||||
|
||||
found = jobs.discover.discover(name, domain)
|
||||
out: dict[str, Any] = {"name": name, "domain": domain, "board": found.board,
|
||||
"ats": found.ats, "careers_url": found.careers_url,
|
||||
"evidence": found.evidence}
|
||||
if add:
|
||||
try:
|
||||
out["key"], out["roles"] = jobs.sources.add_discovered(store, name, domain, found)
|
||||
except ValueError as exc:
|
||||
print(f"companies discover: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(json.dumps(out))
|
||||
return 0
|
||||
|
||||
|
||||
def add_company(store: Store, board: str, name: str) -> int:
|
||||
|
||||
@@ -10,13 +10,14 @@ task in `pipeline.TASKS` and the default schedule in `pipeline.SCHEDULES`.
|
||||
offers, the `pull` job a schedule fires, and that schedule
|
||||
sources.py what a pull reads, changed by hand: boards and searches
|
||||
added, switched on or off, pruned by region
|
||||
discover.py which ATS a company's careers site runs on, from its pages
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.jobs import cascade, embed, extract, pull, score, sources, summarise
|
||||
from app.jobs import cascade, discover, embed, extract, pull, score, sources, summarise
|
||||
from app.jobs.pull import search_target, seed_searches
|
||||
from app.jobs.score import seed_agents
|
||||
|
||||
@@ -35,5 +36,5 @@ def bootstrap(config_path: str = "") -> tuple[Any, Any]:
|
||||
return config, Store(config.database)
|
||||
|
||||
|
||||
__all__ = ["BOOTSTRAP", "bootstrap", "cascade", "embed", "extract", "pull", "score",
|
||||
"search_target", "seed_agents", "seed_searches", "sources", "summarise"]
|
||||
__all__ = ["BOOTSTRAP", "bootstrap", "cascade", "discover", "embed", "extract", "pull",
|
||||
"score", "search_target", "seed_agents", "seed_searches", "sources", "summarise"]
|
||||
@@ -0,0 +1,343 @@
|
||||
"""Board discovery: which applicant tracking system a company's careers site
|
||||
runs on, read off its own pages.
|
||||
|
||||
The seed list and the LinkedIn flow both start from a company name and a
|
||||
website, not from a board. This module gets from the one to the other: it
|
||||
fetches the site's careers page, or the pages the home page links to as
|
||||
careers, and looks for a link into one of the hosts in `HOSTS`. A readable
|
||||
host (one the `ats` plugin parses) becomes a `board` target the pull can
|
||||
read; any other host is recorded as `ats`, so the company is kept as one the
|
||||
app cannot read yet rather than dropped.
|
||||
|
||||
When no page names a host, the three hosts with a cheap per-slug endpoint
|
||||
(Greenhouse, Lever, Ashby) are probed with the company's own slug, because a
|
||||
site that draws its jobs from an iframe or a script leaves no link to find.
|
||||
|
||||
found = discover("Pleo", "pleo.io")
|
||||
found.board # "greenhouse:pleo", or "" when nothing readable
|
||||
found.ats # "greenhouse", "workday", ... or "" when unknown
|
||||
found.careers_url # the page that named the host, or the careers page
|
||||
found.evidence # the URL that named it
|
||||
|
||||
Only https is fetched, redirects are followed at most five hops, a body is
|
||||
cut at `MAX_BYTES`, and every failure is one page skipped. A company's site
|
||||
is content the app was pointed at by its owner, not an ATS API, so nothing
|
||||
here trusts what it finds beyond a host name matching a fixed pattern.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Callable, Iterable
|
||||
from dataclasses import dataclass
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import httpx
|
||||
|
||||
from desk.sources import USER_AGENT
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
TIMEOUT = 10.0
|
||||
MAX_BYTES = 2 << 20
|
||||
#: How many pages of one site are read before giving up on links.
|
||||
MAX_PAGES = 6
|
||||
|
||||
#: The hosts the `ats` plugin parses; a match on one of these is a `board`.
|
||||
READABLE = ("greenhouse", "lever", "ashby", "workable", "smartrecruiters",
|
||||
"recruitee", "personio")
|
||||
|
||||
#: host -> pattern over a URL. Group 1, where there is one, is the slug the
|
||||
#: host names the company by; a host with no group is recognised but its
|
||||
#: board has no slug the app could use. Order matters only for readability:
|
||||
#: the readable hosts come first so the first match on a page is the one the
|
||||
#: app can read.
|
||||
HOSTS: tuple[tuple[str, re.Pattern[str]], ...] = tuple(
|
||||
(host, re.compile(pattern, re.IGNORECASE)) for host, pattern in (
|
||||
("greenhouse", r"(?:boards|job-boards|api)\.(?:eu\.)?greenhouse\.io/(?:v1/boards/|embed/job_board\?[^\"'\s]*for=)?([A-Za-z0-9_-]+)"),
|
||||
("greenhouse", r"greenhouse\.io/embed/job_board/js\?for=([A-Za-z0-9_-]+)"),
|
||||
("lever", r"jobs\.(?:eu\.)?lever\.co/([A-Za-z0-9_-]+)"),
|
||||
("lever", r"api\.(?:eu\.)?lever\.co/v0/postings/([A-Za-z0-9_-]+)"),
|
||||
("ashby", r"jobs\.ashbyhq\.com/([A-Za-z0-9_.-]+)"),
|
||||
("ashby", r"api\.ashbyhq\.com/posting-api/job-board/([A-Za-z0-9_.-]+)"),
|
||||
("workable", r"apply\.workable\.com/(?:api/v\d/accounts/)?([A-Za-z0-9_-]+)"),
|
||||
("workable", r"https?://(?!apply\.|www\.)([a-z0-9-]+)\.workable\.com"),
|
||||
("smartrecruiters", r"(?:jobs|careers|api)\.smartrecruiters\.com/(?:v1/companies/)?([A-Za-z0-9_-]+)"),
|
||||
("recruitee", r"https?://([a-z0-9-]+)\.recruitee\.com"),
|
||||
("personio", r"https?://([a-z0-9-]+)\.jobs\.personio\.(?:de|com)"),
|
||||
# Not readable. The slug, where the URL has one, is kept for the day
|
||||
# a parser exists.
|
||||
("workday", r"https?://([a-z0-9-]+)\.wd\d+\.myworkdayjobs\.com"),
|
||||
("successfactors", r"(?:career|jobs|performancemanager)\d*\.successfactors\.(?:eu|com)|\.sapsf\.(?:eu|com)"),
|
||||
("teamtailor", r"https?://([a-z0-9-]+)\.teamtailor\.com"),
|
||||
("teamtailor", r"cdn\.teamtailor\.com|teamtailor-cdn|teamtailor\.com/assets"),
|
||||
("icims", r"https?://([a-z0-9-]+)\.icims\.com"),
|
||||
("taleo", r"\.taleo\.net"),
|
||||
("jobvite", r"jobs\.jobvite\.com/([a-z0-9-]+)"),
|
||||
("bamboohr", r"https?://([a-z0-9-]+)\.bamboohr\.com"),
|
||||
("join", r"join\.com/companies/([a-z0-9-]+)"),
|
||||
("softgarden", r"https?://([a-z0-9-]+)\.softgarden\.io"),
|
||||
("pinpoint", r"https?://([a-z0-9-]+)\.pinpointhq\.com"),
|
||||
("rippling", r"ats\.rippling\.com/([a-z0-9-]+)"),
|
||||
("gem", r"jobs\.gem\.com/([a-z0-9-]+)"),
|
||||
("breezy", r"https?://([a-z0-9-]+)\.breezy\.hr"),
|
||||
("homerun", r"https?://([a-z0-9-]+)\.homerun\.co"),
|
||||
("jazzhr", r"https?://([a-z0-9-]+)\.applytojob\.com"),
|
||||
("welcometothejungle", r"welcometothejungle\.com/[a-z]{2}/companies/([a-z0-9-]+)"),
|
||||
("eightfold", r"\.eightfold\.ai"),
|
||||
("phenom", r"phenompeople\.com|phenom\.com/careers"),
|
||||
("avature", r"\.avature\.net"),
|
||||
("oracle", r"oraclecloud\.com/hcmUI"),
|
||||
("comeet", r"comeet\.com/jobs/([a-z0-9-]+)"),
|
||||
("hibob", r"https?://([a-z0-9-]+)\.hibob\.com"),
|
||||
))
|
||||
|
||||
#: Paths a careers page usually lives at, tried after the home page's own
|
||||
#: links so the site's word for it wins over a guess.
|
||||
CAREERS_PATHS = ("/careers", "/careers/", "/jobs", "/en/careers", "/company/careers",
|
||||
"/about/careers", "/join-us", "/careers/jobs", "/en/jobs")
|
||||
#: Subdomains careers pages move to.
|
||||
CAREERS_HOSTS = ("careers.", "jobs.", "career.")
|
||||
|
||||
#: A link text or href that points at a careers page.
|
||||
_CAREERS_LINK = re.compile(
|
||||
r"career|jobs|join[-_ ]?us|work[-_ ]?with[-_ ]?us|open[-_ ]?positions|vacanc|"
|
||||
r"we[-_ ]?are[-_ ]?hiring|ledige[-_ ]?stillinger|stellen|karriere|lediga[-_ ]?jobb",
|
||||
re.IGNORECASE)
|
||||
_HREF = re.compile(r"""(?:href|src|action|content)\s*=\s*["']([^"'#][^"']*)["']""",
|
||||
re.IGNORECASE)
|
||||
_ANCHOR = re.compile(r"<a\b[^>]*href\s*=\s*[\"']([^\"'#][^\"']*)[\"'][^>]*>(.*?)</a>",
|
||||
re.IGNORECASE | re.DOTALL)
|
||||
_TAGS = re.compile(r"<[^>]+>")
|
||||
|
||||
#: Words a pattern's slug group can catch that name no board: the hosts'
|
||||
#: own path segments, and the analytics subdomain Recruitee sites embed.
|
||||
_NOT_SLUGS = frozenset({"embed", "boards", "v1", "api", "www", "jobs", "careers", "apply",
|
||||
"careers-analytics"})
|
||||
|
||||
Found = tuple[str, str, str] # (host, slug, the URL that named it)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Discovery:
|
||||
"""What discovery learned about one company; every field may be empty."""
|
||||
|
||||
board: str = ""
|
||||
ats: str = ""
|
||||
careers_url: str = ""
|
||||
evidence: str = ""
|
||||
|
||||
@property
|
||||
def readable(self) -> bool:
|
||||
return bool(self.board)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ parsing
|
||||
|
||||
def boards_in(text: str) -> list[Found]:
|
||||
"""Every ATS host named in `text` (a page, a script, a mail), in order of
|
||||
first appearance, once each. `slug` is "" for a host whose URL carries
|
||||
none."""
|
||||
seen: set[tuple[str, str]] = set()
|
||||
out: list[Found] = []
|
||||
hits: list[tuple[int, str, str, str]] = []
|
||||
for host, pattern in HOSTS:
|
||||
for m in pattern.finditer(text):
|
||||
slug = (m.group(1) if m.groups() and m.group(1) else "").lower()
|
||||
if slug in _NOT_SLUGS:
|
||||
continue
|
||||
hits.append((m.start(), host, slug, m.group(0)))
|
||||
for _pos, host, slug, url in sorted(hits):
|
||||
if (host, slug) in seen:
|
||||
continue
|
||||
seen.add((host, slug))
|
||||
out.append((host, slug, url))
|
||||
return out
|
||||
|
||||
|
||||
def careers_links(html: str, base: str) -> list[str]:
|
||||
"""The links on a page that read as its careers page, absolute, in page
|
||||
order, once each. Nothing off-site is followed: a careers page on another
|
||||
domain is what `boards_in` finds by its host."""
|
||||
out: list[str] = []
|
||||
site = httpx.URL(base).host or ""
|
||||
for href, label in _ANCHOR.findall(html):
|
||||
text = _TAGS.sub(" ", label)
|
||||
if not (_CAREERS_LINK.search(href) or _CAREERS_LINK.search(text)):
|
||||
continue
|
||||
url = urljoin(base, href.strip())
|
||||
parsed = httpx.URL(url)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
continue
|
||||
host = parsed.host or ""
|
||||
if host != site and not host.endswith("." + _apex(site)):
|
||||
continue
|
||||
url = str(parsed.copy_with(scheme="https", fragment=None))
|
||||
if url not in out:
|
||||
out.append(url)
|
||||
return out
|
||||
|
||||
|
||||
def _apex(host: str) -> str:
|
||||
parts = host.split(".")
|
||||
return ".".join(parts[-2:]) if len(parts) > 2 else host
|
||||
|
||||
|
||||
def slug_guesses(name: str, domain: str) -> list[str]:
|
||||
"""The slugs a host most likely knows the company by: its key, the key
|
||||
without hyphens, and the domain's own label."""
|
||||
from app.models import company_key
|
||||
|
||||
key = company_key(name)
|
||||
label = domain.lower().split("/")[0]
|
||||
for prefix in ("www.",):
|
||||
label = label.removeprefix(prefix)
|
||||
label = label.split(".")[0]
|
||||
out = []
|
||||
for guess in (key, key.replace("-", ""), label, label.replace("-", "")):
|
||||
if guess and guess not in out:
|
||||
out.append(guess)
|
||||
return out
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- fetching
|
||||
|
||||
def fetch_page(client: httpx.Client, url: str) -> tuple[str, str]:
|
||||
"""`(final URL, text)` for one page, or `("", "")` when it did not answer
|
||||
with a 2xx text body within the limits. Never raises."""
|
||||
try:
|
||||
response = client.get(
|
||||
url, timeout=TIMEOUT, follow_redirects=True,
|
||||
headers={"User-Agent": USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,*/*;q=0.5"})
|
||||
if not response.is_success:
|
||||
return "", ""
|
||||
body = response.content[:MAX_BYTES]
|
||||
return str(response.url), body.decode(response.encoding or "utf-8", "replace")
|
||||
except Exception as exc: # one page skipped, whatever went wrong
|
||||
log.debug("discover: %s: %s", url, exc)
|
||||
return "", ""
|
||||
|
||||
|
||||
#: host -> a cheap URL that answers 2xx only when the slug is a board there.
|
||||
#: Workable is left out: its API sits behind a rate limit that a run of
|
||||
#: guesses trips, and a Workable site links its board plainly anyway.
|
||||
_PROBES = {
|
||||
"greenhouse": "https://boards-api.greenhouse.io/v1/boards/{slug}",
|
||||
"lever": "https://api.lever.co/v0/postings/{slug}?limit=1&mode=json",
|
||||
"ashby": "https://api.ashbyhq.com/posting-api/job-board/{slug}",
|
||||
"smartrecruiters": "https://api.smartrecruiters.com/v1/companies/{slug}/postings?limit=1",
|
||||
"recruitee": "https://{slug}.recruitee.com/api/offers/",
|
||||
"personio": "https://{slug}.jobs.personio.de/xml",
|
||||
}
|
||||
|
||||
|
||||
def probe(client: httpx.Client, host: str, slug: str) -> bool:
|
||||
"""Whether `host` has a board called `slug`, by its cheapest endpoint.
|
||||
Ashby and SmartRecruiters answer an empty listing for an unknown slug,
|
||||
so there a listing without jobs counts as no board."""
|
||||
url = _PROBES[host].format(slug=slug)
|
||||
try:
|
||||
response = client.get(url, timeout=TIMEOUT, follow_redirects=False,
|
||||
headers={"User-Agent": USER_AGENT})
|
||||
except Exception:
|
||||
return False
|
||||
if not response.is_success:
|
||||
return False
|
||||
body = response.content
|
||||
if host == "ashby":
|
||||
return b'"jobs":[' in body and b'"jobs":[]' not in body
|
||||
if host == "smartrecruiters":
|
||||
return b'"totalFound":' in body and b'"totalFound":0' not in body
|
||||
if host == "recruitee":
|
||||
return b'"offers":[' in body and b'"offers":[]' not in body
|
||||
if host == "personio":
|
||||
return b"<position>" in body
|
||||
return True
|
||||
|
||||
|
||||
Fetch = Callable[[httpx.Client, str], tuple[str, str]]
|
||||
Probe = Callable[[httpx.Client, str, str], bool]
|
||||
|
||||
|
||||
def _site(domain: str) -> str:
|
||||
domain = domain.strip().lower()
|
||||
domain = re.sub(r"^https?://", "", domain).split("/")[0]
|
||||
return domain.removeprefix("www.")
|
||||
|
||||
|
||||
def _pages(domain: str, home_links: Iterable[str]) -> list[str]:
|
||||
site = _site(domain)
|
||||
urls = list(home_links)
|
||||
urls += [f"https://{site}{path}" for path in CAREERS_PATHS]
|
||||
urls += [f"https://{sub}{site}/" for sub in CAREERS_HOSTS]
|
||||
out: list[str] = []
|
||||
for url in urls:
|
||||
if url not in out:
|
||||
out.append(url)
|
||||
return out
|
||||
|
||||
|
||||
def discover(name: str, domain: str, *, fetch: Fetch = fetch_page,
|
||||
probe_slug: Probe = probe, client: httpx.Client | None = None) -> Discovery:
|
||||
"""Where `name`'s careers site draws its jobs from; see the module
|
||||
docstring for the order pages are read in."""
|
||||
site = _site(domain)
|
||||
if not site:
|
||||
return Discovery()
|
||||
own = client or httpx.Client()
|
||||
try:
|
||||
return _discover(name, site, own, fetch, probe_slug)
|
||||
finally:
|
||||
if client is None:
|
||||
own.close()
|
||||
|
||||
|
||||
def _discover(name: str, site: str, client: httpx.Client, fetch: Fetch,
|
||||
probe_slug: Probe) -> Discovery:
|
||||
home_url, home = fetch(client, f"https://{site}/")
|
||||
unreadable: Discovery | None = None
|
||||
careers_page = ""
|
||||
|
||||
def first_readable(found: list[Found], page: str) -> Discovery | None:
|
||||
nonlocal unreadable
|
||||
for host, slug, url in found:
|
||||
if host in READABLE and slug:
|
||||
return Discovery(board=f"{host}:{slug}", ats=host, careers_url=page,
|
||||
evidence=url)
|
||||
if unreadable is None and found:
|
||||
host, slug, url = found[0]
|
||||
unreadable = Discovery(ats=host, careers_url=page, evidence=url)
|
||||
return None
|
||||
|
||||
if home and (hit := first_readable(boards_in(home), home_url)):
|
||||
return hit
|
||||
# The home page's own careers links, then the usual paths; a careers page's
|
||||
# own links (a "See open positions" one level down) go to the front of the
|
||||
# queue. `MAX_PAGES` fetches in all, whatever the queue holds.
|
||||
queue = _pages(site, careers_links(home, home_url) if home else [])
|
||||
seen = {home_url, f"https://{site}/"}
|
||||
budget = MAX_PAGES
|
||||
while queue and budget > 0:
|
||||
url = queue.pop(0)
|
||||
if url in seen:
|
||||
continue
|
||||
seen.add(url)
|
||||
budget -= 1
|
||||
final, text = fetch(client, url)
|
||||
if not text:
|
||||
continue
|
||||
seen.add(final)
|
||||
careers_page = careers_page or final
|
||||
if hit := first_readable(boards_in(text), final):
|
||||
return hit
|
||||
queue[:0] = [link for link in careers_links(text, final)[:2] if link not in seen]
|
||||
for slug in slug_guesses(name, site):
|
||||
for host in _PROBES:
|
||||
if probe_slug(client, host, slug):
|
||||
return Discovery(board=f"{host}:{slug}", ats=host,
|
||||
careers_url=careers_page, evidence=_PROBES[host].format(slug=slug))
|
||||
if unreadable is not None:
|
||||
return unreadable
|
||||
return Discovery(careers_url=careers_page)
|
||||
+26
-1
@@ -11,13 +11,16 @@ roles; switched back on, the next pull reopens every role it still lists.
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.config import Config
|
||||
from app.models import EUROPE, Subscription, company_key
|
||||
from app.storage import Store
|
||||
from app.jobs.pull import search_target
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.jobs.discover import Discovery
|
||||
|
||||
#: The providers `plugins.search` runs, and which of them needs a key.
|
||||
PROVIDERS = ("jobindex", "adzuna")
|
||||
|
||||
@@ -91,6 +94,28 @@ def add_board(store: Store, board: str, name: str) -> tuple[Subscription, int]:
|
||||
return sub, len(rows)
|
||||
|
||||
|
||||
def add_discovered(store: Store, name: str, domain: str,
|
||||
found: Discovery) -> tuple[str, int]:
|
||||
"""Follow a company from what `app.jobs.discover` found: through
|
||||
`add_board` when the board is readable, else as a row with its `ats` host
|
||||
and careers page, so the company is kept as one the app cannot read yet.
|
||||
Returns the company key and how many roles its board lists (0 with no
|
||||
board). Refuses a company that exists, as `add_board` does."""
|
||||
name = name.strip()
|
||||
if not name:
|
||||
raise ValueError("a company needs a name")
|
||||
key = company_key(name)
|
||||
if found.board:
|
||||
_sub, roles = add_board(store, found.board, name)
|
||||
store.update_company(key, domain=domain, ats=found.ats)
|
||||
return key, roles
|
||||
if store.company(key) is not None:
|
||||
raise ValueError(f"{name} is already followed")
|
||||
store.add_company(name, key=key, domain=domain, ats=found.ats,
|
||||
careers_url=found.careers_url)
|
||||
return key, 0
|
||||
|
||||
|
||||
def search_params(provider: str, q: str, *, area: int = 0, country: str = "",
|
||||
where: str = "", pages: int = 0) -> dict[str, Any]:
|
||||
"""The `params` a search subscription stores, with only the keys its
|
||||
|
||||
@@ -314,6 +314,12 @@ class Company(Model):
|
||||
status: str = "researching"
|
||||
careers_url: str = ""
|
||||
notes: str = ""
|
||||
#: The host the careers site runs on, readable or not; "" when unknown.
|
||||
#: A company with an `ats` and no board subscription is one the app
|
||||
#: cannot read yet. See `app.jobs.discover`.
|
||||
ats: str = ""
|
||||
country: str = ""
|
||||
city: str = ""
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
+19
-1
@@ -54,9 +54,25 @@
|
||||
badge.className = "badge";
|
||||
badge.textContent = company.board;
|
||||
sub.appendChild(badge);
|
||||
} else if (company.ats || company.careers_url) {
|
||||
// Followed, but its careers site runs on nothing the app reads yet.
|
||||
const badge = document.createElement("span");
|
||||
badge.className = "badge";
|
||||
badge.textContent = company.ats || "own site";
|
||||
sub.appendChild(badge);
|
||||
sub.append(" · not readable yet");
|
||||
} else {
|
||||
sub.append("from job boards");
|
||||
}
|
||||
if (company.careers_url && !company.board) {
|
||||
const link = document.createElement("a");
|
||||
link.href = company.careers_url;
|
||||
link.target = "_blank";
|
||||
link.rel = "noopener";
|
||||
link.textContent = "careers page";
|
||||
sub.append(" · ");
|
||||
sub.appendChild(link);
|
||||
}
|
||||
sub.append(` · ${count(company.open_roles, "open role")}`);
|
||||
return sub;
|
||||
}
|
||||
@@ -94,6 +110,7 @@
|
||||
case "hidden": return company.affinity === -1;
|
||||
case "on": return company.on === true;
|
||||
case "off": return company.on === false;
|
||||
case "unreadable": return company.subscription_id == null && Boolean(company.ats || company.careers_url);
|
||||
default: return true;
|
||||
}
|
||||
}
|
||||
@@ -102,7 +119,8 @@
|
||||
const q = $("#c-q").value.trim().toLowerCase();
|
||||
const filter = $("#c-filter").value;
|
||||
const rows = [...companies.values()]
|
||||
.filter((c) => !q || c.name.toLowerCase().includes(q) || c.board.includes(q))
|
||||
.filter((c) => !q || c.name.toLowerCase().includes(q) || c.board.includes(q)
|
||||
|| (c.ats || "").includes(q) || (c.country || "").toLowerCase() === q)
|
||||
.filter((c) => matchesFilter(c, filter))
|
||||
.sort((a, b) => b.open_roles - a.open_roles || a.name.localeCompare(b.name));
|
||||
const body = $("#companies-list");
|
||||
|
||||
+13
-1
@@ -91,6 +91,10 @@ CREATE TABLE IF NOT EXISTS role_tags (
|
||||
|
||||
-- A company as a target, whether or not any board of it is read. key is
|
||||
-- models.company_key(name), the slug roles.company_key holds.
|
||||
-- ats is the host the careers site runs on, readable by the ats plugin or
|
||||
-- not (workday, successfactors, ...): what discovery found, '' unknown. A
|
||||
-- company with an ats and no board subscription is one the app cannot read
|
||||
-- yet. country is an ISO code and city the head office's.
|
||||
CREATE TABLE IF NOT EXISTS companies (
|
||||
key TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
@@ -99,7 +103,10 @@ CREATE TABLE IF NOT EXISTS companies (
|
||||
careers_url TEXT NOT NULL DEFAULT '',
|
||||
notes TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
updated_at TEXT NOT NULL,
|
||||
ats TEXT NOT NULL DEFAULT '',
|
||||
country TEXT NOT NULL DEFAULT '',
|
||||
city TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
|
||||
-- The targets desk's runner hands to plugins. newsdesk's shape, with its two
|
||||
@@ -319,6 +326,11 @@ SELECT (SELECT id FROM agents ORDER BY enabled DESC, id LIMIT 1), key, affinity,
|
||||
ALTER TABLE roles DROP COLUMN vote;
|
||||
ALTER TABLE roles DROP COLUMN voted_at;
|
||||
ALTER TABLE companies DROP COLUMN affinity;
|
||||
"""),
|
||||
(7, "companies.ats, country and city: a company the app cannot read yet is still a row", """
|
||||
ALTER TABLE companies ADD COLUMN ats TEXT NOT NULL DEFAULT '';
|
||||
ALTER TABLE companies ADD COLUMN country TEXT NOT NULL DEFAULT '';
|
||||
ALTER TABLE companies ADD COLUMN city TEXT NOT NULL DEFAULT '';
|
||||
"""),
|
||||
)
|
||||
|
||||
|
||||
@@ -69,7 +69,7 @@ _VOTED_SCORE = ("MIN(1.0, s.score * CASE v.vote WHEN -1 THEN 0.5 "
|
||||
#: nothing, so `vote` and `voted_at` are NULL: one query shape either way.
|
||||
_WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?"
|
||||
|
||||
_COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes")
|
||||
_COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes", "ats", "country", "city")
|
||||
|
||||
#: A company as the store serves it: the row and the asked-for agent's affinity.
|
||||
_COMPANY_SELECT = """SELECT c.*, COALESCE(ac.affinity, 0) AS affinity FROM companies c
|
||||
@@ -944,7 +944,8 @@ class Store:
|
||||
|
||||
def add_company(self, name: str, *, key: str = "", domain: str = "",
|
||||
status: str = "researching", careers_url: str = "",
|
||||
notes: str = "") -> Company:
|
||||
notes: str = "", ats: str = "", country: str = "",
|
||||
city: str = "") -> Company:
|
||||
"""Raises ValueError on a key already taken."""
|
||||
name = name.strip()
|
||||
key = key or company_key(name)
|
||||
@@ -954,9 +955,10 @@ class Store:
|
||||
try:
|
||||
self._write(
|
||||
"""INSERT INTO companies (key, name, domain, status, careers_url, notes,
|
||||
created_at, updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?)""",
|
||||
(key, name, domain, status, careers_url, notes, stamp, stamp))
|
||||
ats, country, city, created_at, updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
(key, name, domain, status, careers_url, notes, ats, country, city,
|
||||
stamp, stamp))
|
||||
except sqlite3.IntegrityError as exc:
|
||||
raise ValueError(f"company {key!r} already exists") from exc
|
||||
return self._company(key)
|
||||
|
||||
@@ -86,7 +86,7 @@
|
||||
</form>
|
||||
|
||||
<form class="filters" autocomplete="off" onsubmit="return false">
|
||||
<input type="search" id="c-q" placeholder="Filter by name or board" aria-label="Filter companies">
|
||||
<input type="search" id="c-q" placeholder="Filter by name, board or country code" aria-label="Filter companies">
|
||||
<select id="c-filter" aria-label="Filter">
|
||||
<option value="">Any</option>
|
||||
<option value="favourite">Favourite</option>
|
||||
@@ -94,6 +94,7 @@
|
||||
<option value="hidden">Hidden</option>
|
||||
<option value="on">On</option>
|
||||
<option value="off">Off</option>
|
||||
<option value="unreadable">Not readable yet</option>
|
||||
</select>
|
||||
{# Liked and hidden are per search agent: this picks whose the list shows
|
||||
and whose the buttons set. companies.js fills it from /api/agents. #}
|
||||
|
||||
+8
-2
@@ -262,7 +262,9 @@ class CompanyOut(BaseModel):
|
||||
"""A company row plus what only the store can join in: the ats host it is
|
||||
read from (empty for a company known only from a search), the id of that
|
||||
board's subscription, whether it is on, and how many of its roles are
|
||||
still open."""
|
||||
still open. `ats` is the host its careers site runs on whether or not the
|
||||
app reads it, so a followed company with no `board` and an `ats` is one
|
||||
the app cannot read yet; `careers_url` is where a person can."""
|
||||
|
||||
key: str
|
||||
name: str
|
||||
@@ -271,6 +273,9 @@ class CompanyOut(BaseModel):
|
||||
on: bool | None = None
|
||||
affinity: int
|
||||
open_roles: int
|
||||
ats: str = ""
|
||||
careers_url: str = ""
|
||||
country: str = ""
|
||||
|
||||
|
||||
def _company_boards(store: Store) -> dict[str, tuple[int, str, bool]]:
|
||||
@@ -296,7 +301,8 @@ def list_companies(store: StoreDep, agent: int | None = None) -> list[CompanyOut
|
||||
sub_id, host, on = boards.get(c.key, (None, "", None))
|
||||
out.append(CompanyOut(key=c.key, name=c.name, board=host,
|
||||
subscription_id=sub_id, on=on, affinity=c.affinity,
|
||||
open_roles=counts.get(c.key, 0)))
|
||||
open_roles=counts.get(c.key, 0), ats=c.ats,
|
||||
careers_url=c.careers_url, country=c.country))
|
||||
return out
|
||||
|
||||
|
||||
|
||||
+7
-1
@@ -42,6 +42,12 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**Boards follow companies; searches find them.** `ats` only reads companies someone listed, so it cannot surface a role at a company nobody knew of, and the seed list came from a US-centred project: of 29,000 open roles, 31 were in Denmark. The `search` plugin runs saved queries on boards that span companies, as `search` subscriptions whose `params` are the query, and pulls them in the same task as the boards.
|
||||
|
||||
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list.
|
||||
|
||||
**`discover` reads the site before it guesses.** A careers page that links its board is the authority, so the home page, its own careers links, then the usual paths are read for a link into a known host, and a page's own "open positions" link one level down. Only when no page names a host are Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio probed with the company's slug, because a board drawn by a script leaves no link. A probe hit on a common name can be another tenant (`recruitee:google` exists, and is not Google), which is why the seed's boards were fetched and read before they landed, and why `discover --add` reports the roles it found.
|
||||
|
||||
**A company row carries `ats`, `country` and `city`.** `ats` is what discovery found whether or not the app reads it; a company with an `ats` and no board subscription is one it cannot read yet. `country` and `city` come from the seed for now and are the first of the company facts the feed will filter by; funding, size and industry come from a source that has them, not from a column filled by hand.
|
||||
|
||||
**Jobindex and Adzuna, not Jobnet or Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jobnet's search endpoint answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
|
||||
|
||||
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. A search whose results ran out before its page limit did list everything, and closes like a board; the rest close once no search has listed them for 21 days. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists.
|
||||
@@ -58,7 +64,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
|
||||
|
||||
**Workable's shape is confirmed, not a populated example.** Every slug tried during the build returned the documented empty `{"total": 0, "results": []}` before Cloudflare rate-limited the session, so the parser follows Workable's published job object. Recheck it against a live board with open roles the next time the plugin is touched.
|
||||
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`, and `description` stays empty as for SmartRecruiters rather than costing one request per role.
|
||||
|
||||
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs.
|
||||
|
||||
|
||||
+25
-19
@@ -11,7 +11,8 @@ The field map, the redirect allow-list and the 10 s timeout are ported from
|
||||
`classify`'s nine functions and their keywords. Workable, SmartRecruiters,
|
||||
Recruitee and Personio are new: jobhunt-hq only read Greenhouse, Lever and
|
||||
Ashby, so those four are written from one real board of each host, fetched
|
||||
while this file was written (2026-09-23), rather than from the Go source.
|
||||
while this file was written (2026-09-23; Workable on 2026-09-28, when the
|
||||
first board with roles was seen), rather than from the Go source.
|
||||
|
||||
The body cap is not a straight port — see `MAX_BYTES` below.
|
||||
|
||||
@@ -256,31 +257,36 @@ def _parse_ashby(body: bytes) -> list[Row]:
|
||||
return rows
|
||||
|
||||
|
||||
def _parse_workable(body: bytes) -> list[Row]:
|
||||
"""https://apply.workable.com/api/v3/accounts/{slug}/jobs (POST, empty
|
||||
JSON body) -> `{"total": int, "results": [...]}`. The shape is confirmed
|
||||
live (2026-09-23); the fields are Workable's long-documented job object,
|
||||
since every slug this session tried on it happened to list nothing at
|
||||
the moment of fetching — see the plugin's docstring and the build report."""
|
||||
def _parse_workable(body: bytes, slug: str) -> list[Row]:
|
||||
"""POST https://apply.workable.com/api/v3/accounts/{slug}/jobs with `{}` ->
|
||||
`{"total": int, "results": [...]}`, as Hugging Face's board answered on
|
||||
2026-09-28. A result carries no URL and no text: the job page is
|
||||
`https://apply.workable.com/{slug}/j/{shortcode}/`, and the description
|
||||
is only on the per-job endpoint, one request per role, so `description`
|
||||
stays empty here as it does for SmartRecruiters. `location` is an object
|
||||
and `department` a list; `remote` is a flag beside `workplace`
|
||||
(remote, hybrid, on_site)."""
|
||||
data = json.loads(body)
|
||||
rows = []
|
||||
for j in data.get("results") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("url") or j.get("shortlink") or j.get("application_url") or "").strip()
|
||||
if not title or not _is_web_url(url):
|
||||
shortcode = (j.get("shortcode") or "").strip()
|
||||
if not title or not shortcode:
|
||||
continue
|
||||
location = ", ".join(p for p in (j.get("city"), j.get("region"), j.get("country")) if p)
|
||||
description = " ".join(p for p in (
|
||||
j.get("description"), j.get("requirements"), j.get("benefits")) if p)
|
||||
loc = j.get("location") or {}
|
||||
location = ", ".join(p for p in (loc.get("city"), loc.get("region"), loc.get("country")) if p)
|
||||
dept = j.get("department") or ""
|
||||
if isinstance(dept, list):
|
||||
dept = ", ".join(d for d in dept if d)
|
||||
rows.append({
|
||||
"source_id": str(j.get("id") or j.get("shortcode") or ""),
|
||||
"source_id": str(j.get("id") or shortcode),
|
||||
"title": title,
|
||||
"location": location,
|
||||
"url": url,
|
||||
"posted": _parse_dt(j.get("published_on") or j.get("created_at")),
|
||||
"department": (j.get("department") or "").strip(),
|
||||
"description": strip_html(description),
|
||||
"remote": j.get("telecommute"),
|
||||
"url": f"https://apply.workable.com/{slug}/j/{shortcode}/",
|
||||
"posted": _parse_dt(j.get("published")),
|
||||
"department": dept.strip(),
|
||||
"description": "",
|
||||
"remote": bool(j.get("remote")) or j.get("workplace") == "remote",
|
||||
"salary": "",
|
||||
})
|
||||
return rows
|
||||
@@ -375,7 +381,7 @@ _PARSERS: dict[str, Callable[..., list[Row]]] = {
|
||||
"greenhouse": lambda body, slug: _parse_greenhouse(body),
|
||||
"lever": lambda body, slug: _parse_lever(body),
|
||||
"ashby": lambda body, slug: _parse_ashby(body),
|
||||
"workable": lambda body, slug: _parse_workable(body),
|
||||
"workable": _parse_workable,
|
||||
"smartrecruiters": _parse_smartrecruiters,
|
||||
"recruitee": lambda body, slug: _parse_recruitee(body),
|
||||
"personio": _parse_personio,
|
||||
|
||||
+2569
-1305
File diff suppressed because it is too large.
Load diff
Vendored
+34
-25
@@ -2,33 +2,42 @@
|
||||
"total": 2,
|
||||
"results": [
|
||||
{
|
||||
"id": "4001",
|
||||
"title": "Backend Engineer",
|
||||
"department": "Engineering",
|
||||
"city": "Berlin",
|
||||
"region": "",
|
||||
"country": "Germany",
|
||||
"telecommute": true,
|
||||
"published_on": "2026-09-01",
|
||||
"created_at": "2026-08-30T10:00:00.000Z",
|
||||
"url": "https://workable.example/1",
|
||||
"shortlink": "https://workable.example/short/1",
|
||||
"description": "<p>Ship our API.</p>",
|
||||
"requirements": "<p>5+ years.</p>",
|
||||
"benefits": ""
|
||||
"id": 6125026,
|
||||
"shortcode": "DB4D7C0EC8",
|
||||
"title": "Senior Open-Source Python Engineer, ML Developer Tools - EMEA Remote",
|
||||
"remote": true,
|
||||
"location": {
|
||||
"country": "France",
|
||||
"countryCode": "FR",
|
||||
"city": "",
|
||||
"region": null
|
||||
},
|
||||
"state": "published",
|
||||
"published": "2026-09-21T00:00:00.000Z",
|
||||
"type": "full",
|
||||
"department": [
|
||||
"Product"
|
||||
],
|
||||
"workplace": "remote"
|
||||
},
|
||||
{
|
||||
"id": "4002",
|
||||
"shortcode": "ABCDEF",
|
||||
"title": "Account Manager",
|
||||
"department": "Sales",
|
||||
"city": "Munich",
|
||||
"region": "",
|
||||
"country": "Germany",
|
||||
"telecommute": false,
|
||||
"published_on": "2026-08-15",
|
||||
"url": "https://workable.example/2",
|
||||
"description": ""
|
||||
"id": 6125041,
|
||||
"shortcode": "F88446C814",
|
||||
"title": "Senior Open-Source Python Engineer, ML Developer Tools - US Remote",
|
||||
"remote": false,
|
||||
"location": {
|
||||
"country": "Germany",
|
||||
"countryCode": "DE",
|
||||
"city": "Berlin",
|
||||
"region": null
|
||||
},
|
||||
"state": "published",
|
||||
"published": "2026-09-21T00:00:00.000Z",
|
||||
"type": "full",
|
||||
"department": [
|
||||
"Product"
|
||||
],
|
||||
"workplace": "hybrid"
|
||||
}
|
||||
]
|
||||
}
|
||||
+9
-5
@@ -59,12 +59,16 @@ def test_ashby_parses_is_remote_and_compensation() -> None:
|
||||
assert rows[1]["remote"] is True
|
||||
|
||||
|
||||
def test_workable_parses_location_parts_and_telecommute() -> None:
|
||||
rows = _rows("workable")
|
||||
assert [r["source_id"] for r in rows] == ["4001", "4002"]
|
||||
assert rows[0]["location"] == "Berlin, Germany"
|
||||
def test_workable_builds_the_job_url_and_reads_location_and_remote() -> None:
|
||||
rows = _rows("workable", slug="huggingface")
|
||||
assert [r["source_id"] for r in rows] == ["6125026", "6125041"]
|
||||
assert rows[0]["url"] == "https://apply.workable.com/huggingface/j/DB4D7C0EC8/"
|
||||
assert rows[0]["location"] == "France"
|
||||
assert rows[0]["remote"] is True
|
||||
assert "Ship our API." in rows[0]["description"]
|
||||
assert rows[0]["department"] == "Product"
|
||||
assert rows[1]["location"] == "Berlin, Germany"
|
||||
assert rows[1]["remote"] is False
|
||||
assert rows[0]["description"] == "" # the list endpoint carries no text
|
||||
|
||||
|
||||
def test_smartrecruiters_builds_the_job_url_from_id_and_slug() -> None:
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Board discovery: which ATS a careers site names, and in what order the
|
||||
site's pages are read."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.cli import seed_companies
|
||||
from app.jobs import discover as mod
|
||||
from app.jobs.discover import Discovery, boards_in, careers_links, discover, slug_guesses
|
||||
from app.storage import Store
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ parsing
|
||||
|
||||
def test_boards_in_names_readable_hosts_with_their_slug() -> None:
|
||||
page = """
|
||||
<a href="https://job-boards.greenhouse.io/pleo">Open roles</a>
|
||||
<script src="https://boards.greenhouse.io/embed/job_board/js?for=pleo"></script>
|
||||
<a href="https://jobs.lever.co/spotify/abc-123">One role</a>
|
||||
<iframe src="https://jobs.ashbyhq.com/DeepL/embed"></iframe>
|
||||
<a href="https://apply.workable.com/acme/">Workable</a>
|
||||
<a href="https://jobs.smartrecruiters.com/Bosch/123">SR</a>
|
||||
<a href="https://acme.recruitee.com/o/backend">Recruitee</a>
|
||||
<a href="https://acme-gmbh.jobs.personio.de/job/1">Personio</a>
|
||||
"""
|
||||
assert [(h, s) for h, s, _url in boards_in(page)] == [
|
||||
("greenhouse", "pleo"), ("lever", "spotify"), ("ashby", "deepl"),
|
||||
("workable", "acme"), ("smartrecruiters", "bosch"), ("recruitee", "acme"),
|
||||
("personio", "acme-gmbh")]
|
||||
|
||||
|
||||
def test_boards_in_keeps_hosts_the_app_cannot_read() -> None:
|
||||
page = """
|
||||
<a href="https://novo.wd3.myworkdayjobs.com/en-US/careers">Workday</a>
|
||||
<a href="https://performancemanager.successfactors.eu/career?company=x">SF</a>
|
||||
<a href="https://acme.teamtailor.com/jobs">TT</a>
|
||||
"""
|
||||
assert boards_in(page) == [
|
||||
("workday", "novo", "https://novo.wd3.myworkdayjobs.com"),
|
||||
("successfactors", "", "performancemanager.successfactors.eu"),
|
||||
("teamtailor", "acme", "https://acme.teamtailor.com")]
|
||||
|
||||
|
||||
def test_boards_in_dedupes_and_skips_the_api_paths_own_words() -> None:
|
||||
page = ('<a href="https://boards.greenhouse.io/embed/job_board?for=acme">a</a>'
|
||||
'<a href="https://boards-api.greenhouse.io/v1/boards/acme/jobs">b</a>')
|
||||
assert [(h, s) for h, s, _ in boards_in(page)] == [("greenhouse", "acme")]
|
||||
assert boards_in("nothing here") == []
|
||||
|
||||
|
||||
def test_careers_links_keeps_on_site_links_that_read_as_careers() -> None:
|
||||
html = """
|
||||
<a href="/about">About</a>
|
||||
<a href="/careers">Careers</a>
|
||||
<a href="https://jobs.acme.com/">We are hiring</a>
|
||||
<a href="https://other.example/jobs">Partner jobs</a>
|
||||
<a href="/en/karriere#top"><span>Karriere</span></a>
|
||||
<a href="mailto:jobs@acme.com">Mail</a>
|
||||
<a href="/careers">Careers again</a>
|
||||
"""
|
||||
assert careers_links(html, "https://www.acme.com/") == [
|
||||
"https://www.acme.com/careers", "https://jobs.acme.com/",
|
||||
"https://www.acme.com/en/karriere"]
|
||||
|
||||
|
||||
def test_slug_guesses_come_from_the_key_and_the_domain() -> None:
|
||||
assert slug_guesses("Trade Republic Bank GmbH", "www.traderepublic.com") == [
|
||||
"trade-republic-bank", "traderepublicbank", "traderepublic"]
|
||||
assert slug_guesses("Pleo", "pleo.io") == ["pleo"]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- discover
|
||||
|
||||
class Site:
|
||||
"""A fake web: `pages` maps a URL to `(final URL, html)`; anything else
|
||||
is a page that did not answer. Remembers the order of fetches."""
|
||||
|
||||
def __init__(self, pages: dict[str, tuple[str, str] | str],
|
||||
boards: set[tuple[str, str]] = frozenset()) -> None:
|
||||
self.pages = {k: (v if isinstance(v, tuple) else (k, v)) for k, v in pages.items()}
|
||||
self.boards = boards
|
||||
self.fetched: list[str] = []
|
||||
self.probed: list[tuple[str, str]] = []
|
||||
|
||||
def fetch(self, client: httpx.Client, url: str) -> tuple[str, str]:
|
||||
self.fetched.append(url)
|
||||
return self.pages.get(url, ("", ""))
|
||||
|
||||
def probe(self, client: httpx.Client, host: str, slug: str) -> bool:
|
||||
self.probed.append((host, slug))
|
||||
return (host, slug) in self.boards
|
||||
|
||||
|
||||
def run(site: Site, name: str = "Acme", domain: str = "acme.com") -> Discovery:
|
||||
with httpx.Client() as client:
|
||||
return discover(name, domain, fetch=site.fetch, probe_slug=site.probe, client=client)
|
||||
|
||||
|
||||
def test_a_board_named_on_the_home_page_ends_discovery_there() -> None:
|
||||
site = Site({"https://acme.com/": '<a href="https://jobs.lever.co/acme">Jobs</a>'})
|
||||
assert run(site) == Discovery(board="lever:acme", ats="lever",
|
||||
careers_url="https://acme.com/",
|
||||
evidence="jobs.lever.co/acme")
|
||||
assert site.fetched == ["https://acme.com/"]
|
||||
assert site.probed == []
|
||||
|
||||
|
||||
def test_the_home_pages_own_careers_link_is_read_before_the_usual_paths() -> None:
|
||||
site = Site({
|
||||
"https://acme.com/": '<a href="/company/join-us">Join us</a>',
|
||||
"https://acme.com/company/join-us": ('https://acme.com/company/join-us',
|
||||
'<a href="https://acme.recruitee.com/">Open</a>'),
|
||||
"https://acme.com/careers": '<a href="https://jobs.lever.co/wrong">no</a>',
|
||||
})
|
||||
found = run(site)
|
||||
assert found.board == "recruitee:acme"
|
||||
assert found.careers_url == "https://acme.com/company/join-us"
|
||||
assert site.fetched == ["https://acme.com/", "https://acme.com/company/join-us"]
|
||||
|
||||
|
||||
def test_a_careers_pages_own_link_is_followed_one_level_down() -> None:
|
||||
site = Site({
|
||||
"https://acme.com/": "<p>no links</p>",
|
||||
"https://acme.com/careers": '<a href="/careers/open-positions">See open positions</a>',
|
||||
"https://acme.com/careers/open-positions":
|
||||
'<iframe src="https://jobs.ashbyhq.com/acme"></iframe>',
|
||||
})
|
||||
assert run(site).board == "ashby:acme"
|
||||
assert site.fetched[:3] == ["https://acme.com/", "https://acme.com/careers",
|
||||
"https://acme.com/careers/open-positions"]
|
||||
|
||||
|
||||
def test_an_unreadable_host_is_kept_when_nothing_readable_turns_up() -> None:
|
||||
site = Site({
|
||||
"https://acme.com/": "<p>welcome</p>",
|
||||
"https://acme.com/careers": '<a href="https://acme.wd3.myworkdayjobs.com/x">Jobs</a>',
|
||||
})
|
||||
found = run(site)
|
||||
assert found == Discovery(ats="workday", careers_url="https://acme.com/careers",
|
||||
evidence="https://acme.wd3.myworkdayjobs.com")
|
||||
assert not found.readable
|
||||
# Every slug was still tried on the hosts with a probe, in case a readable
|
||||
# board exists that the pages do not link.
|
||||
assert ("greenhouse", "acme") in site.probed
|
||||
|
||||
|
||||
def test_slug_probes_find_a_board_the_pages_never_link() -> None:
|
||||
site = Site({"https://acme.com/": "<p>a script draws the jobs</p>",
|
||||
"https://acme.com/jobs": "<div id=app></div>"},
|
||||
boards={("ashby", "acme")})
|
||||
found = run(site)
|
||||
assert found.board == "ashby:acme"
|
||||
assert found.careers_url == "https://acme.com/jobs"
|
||||
assert found.evidence == "https://api.ashbyhq.com/posting-api/job-board/acme"
|
||||
|
||||
|
||||
def test_a_site_that_never_answers_gives_an_empty_discovery() -> None:
|
||||
site = Site({})
|
||||
assert run(site) == Discovery()
|
||||
assert len(site.fetched) <= mod.MAX_PAGES + 1
|
||||
|
||||
|
||||
def test_a_domain_is_folded_to_its_site() -> None:
|
||||
site = Site({"https://acme.com/": '<a href="https://jobs.lever.co/acme">Jobs</a>'})
|
||||
with httpx.Client() as client:
|
||||
found = discover("Acme", "https://www.acme.com/some/page", fetch=site.fetch,
|
||||
probe_slug=site.probe, client=client)
|
||||
assert found.board == "lever:acme"
|
||||
assert discover("Acme", "", fetch=site.fetch, probe_slug=site.probe) == Discovery()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the seed
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path: Path) -> Store:
|
||||
return Store(tmp_path / "t.db")
|
||||
|
||||
|
||||
def test_the_seed_keeps_a_company_the_app_cannot_read(store: Store, tmp_path: Path,
|
||||
capsys: pytest.CaptureFixture[str]) -> None:
|
||||
seed = tmp_path / "companies.json"
|
||||
seed.write_text(json.dumps({"companies": [
|
||||
{"name": "Pleo", "domain": "pleo.io", "country": "DK", "city": "Copenhagen",
|
||||
"board": "ashby:pleo"},
|
||||
{"name": "Novo Nordisk", "domain": "novonordisk.com", "country": "DK",
|
||||
"city": "Bagsværd", "ats": "successfactors",
|
||||
"careers_url": "https://www.novonordisk.com/careers.html"},
|
||||
{"name": "Odd", "board": "nosuch:odd"},
|
||||
]}))
|
||||
assert seed_companies(store, str(seed)) == {"added": 3, "readable": 1, "skipped": 0,
|
||||
"total": 3}
|
||||
pleo = store.company("pleo")
|
||||
assert pleo is not None
|
||||
assert (pleo.ats, pleo.country, pleo.city, pleo.careers_url) == (
|
||||
"ashby", "DK", "Copenhagen", "https://jobs.ashbyhq.com/pleo")
|
||||
[sub] = store.subscriptions(source="ats")
|
||||
assert (sub.target, sub.company) == ("ashby:pleo", "Pleo")
|
||||
novo = store.company("novo-nordisk")
|
||||
assert novo is not None
|
||||
assert (novo.ats, novo.careers_url) == ("successfactors",
|
||||
"https://www.novonordisk.com/careers.html")
|
||||
odd = store.company("odd")
|
||||
assert odd is not None and odd.ats == ""
|
||||
assert "not a known board" in capsys.readouterr().err
|
||||
# Re-running adds nothing and edits nothing.
|
||||
assert seed_companies(store, str(seed))["skipped"] == 3
|
||||
@@ -93,6 +93,8 @@ def test_a_v1_file_migrates_to_the_shape_a_new_file_has(tmp_path: Path) -> None:
|
||||
# v1 had affinity on the company; the vote columns v4 added and v6
|
||||
# drops again are not in a fresh file, so there is nothing to put back.
|
||||
conn.execute("ALTER TABLE companies ADD COLUMN affinity INTEGER NOT NULL DEFAULT 0")
|
||||
for column in ("ats", "country", "city"): # v7
|
||||
conn.execute(f"ALTER TABLE companies DROP COLUMN {column}")
|
||||
for table in ("vectors", "role_tags", "agent_companies", "role_votes"):
|
||||
conn.execute(f"DROP TABLE {table}")
|
||||
conn.execute("PRAGMA user_version = 1")
|
||||
@@ -113,6 +115,8 @@ def _downgrade_to_v4(path: Path) -> None:
|
||||
conn.execute("ALTER TABLE roles ADD COLUMN vote INTEGER CHECK (vote IN (-1, 0, 1))")
|
||||
conn.execute("ALTER TABLE roles ADD COLUMN voted_at TEXT")
|
||||
conn.execute("ALTER TABLE companies ADD COLUMN affinity INTEGER NOT NULL DEFAULT 0")
|
||||
for column in ("ats", "country", "city"): # v7
|
||||
conn.execute(f"ALTER TABLE companies DROP COLUMN {column}")
|
||||
conn.execute("DROP TABLE agent_companies")
|
||||
conn.execute("DROP TABLE role_votes")
|
||||
conn.execute("PRAGMA user_version = 4")
|
||||
@@ -139,7 +143,7 @@ def test_v6_moves_the_votes_and_affinities_to_the_default_agent(tmp_path: Path)
|
||||
|
||||
migrated = Store(path)
|
||||
|
||||
assert migrated.migrations_applied == [5, 6]
|
||||
assert migrated.migrations_applied == list(range(5, LATEST + 1))
|
||||
assert [a.name for a in migrated.agents()] == ["First", "Second"] # none added
|
||||
assert migrated.votes(first.id) == {posting("liked").uid: 1, posting("passed").uid: -1}
|
||||
assert migrated.votes(second.id) == {}
|
||||
@@ -181,7 +185,7 @@ def test_v6_makes_no_agent_when_there_is_nothing_to_move(tmp_path: Path) -> None
|
||||
store.add_company("Acme", key="acme")
|
||||
_downgrade_to_v4(path)
|
||||
|
||||
assert Store(path).migrations_applied == [5, 6]
|
||||
assert Store(path).migrations_applied == list(range(5, LATEST + 1))
|
||||
assert Store(path).agents() == []
|
||||
|
||||
|
||||
@@ -191,7 +195,7 @@ def test_a_v4_file_migrates_read_state_and_tags(tmp_path: Path) -> None:
|
||||
Store(old)
|
||||
_downgrade_to_v4(old)
|
||||
|
||||
assert Store(old).migrations_applied == [5, 6]
|
||||
assert Store(old).migrations_applied == list(range(5, LATEST + 1))
|
||||
assert sorted(_columns(old, "roles")) == sorted(_columns(fresh, "roles"))
|
||||
with sqlite3.connect(old) as conn:
|
||||
tables = {r[0] for r in conn.execute(
|
||||
|
||||
+10
-3
@@ -384,17 +384,24 @@ def test_the_roles_page_offers_the_judged_filter_and_hidden_companies(
|
||||
def test_companies_lists_board_and_open_role_count(
|
||||
client: TestClient, store: Store) -> None:
|
||||
sub = store.add_subscription("ats", "greenhouse:acme", company="Acme", board="greenhouse")
|
||||
store.add_company("Acme", key="acme")
|
||||
store.add_company("Acme", key="acme", ats="greenhouse", country="DK")
|
||||
store.upsert_role(posting("1", company="Acme"), sub)
|
||||
store.add_company("Other", key="other") # known only from a search: no board
|
||||
store.add_company("Novo", key="novo", ats="successfactors", # followed, not readable
|
||||
careers_url="https://novo.example/careers")
|
||||
|
||||
res = client.get("/api/companies")
|
||||
assert res.status_code == 200
|
||||
[row] = [c for c in res.json() if c["key"] == "acme"]
|
||||
assert row == {"key": "acme", "name": "Acme", "board": "greenhouse",
|
||||
"subscription_id": sub.id, "on": True, "affinity": 0, "open_roles": 1}
|
||||
"subscription_id": sub.id, "on": True, "affinity": 0, "open_roles": 1,
|
||||
"ats": "greenhouse", "careers_url": "", "country": "DK"}
|
||||
[other] = [c for c in res.json() if c["key"] == "other"]
|
||||
assert (other["board"], other["subscription_id"], other["on"]) == ("", None, None)
|
||||
assert (other["board"], other["subscription_id"], other["on"], other["ats"]) == (
|
||||
"", None, None, "")
|
||||
[novo] = [c for c in res.json() if c["key"] == "novo"]
|
||||
assert (novo["board"], novo["subscription_id"], novo["ats"], novo["careers_url"]) == (
|
||||
"", None, "successfactors", "https://novo.example/careers")
|
||||
|
||||
|
||||
def test_set_affinity_validates_updates_and_enqueues_a_score(
|
||||
|
||||
Reference in new issue
Block a user