diff --git a/README.md b/README.md index d2edee1..6cc9a7e 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Built on the [boilerplate](../boilerplate): its job queue and scheduler (`pipeli ```bash uv sync --all-extras -uv run app companies seed # 362 European employers from seed/companies.json, 175 with a readable board +uv run app companies seed # 362 European employers from seed/companies.json, 175 with a readable board; a re-run adds new rows and boards uv run app job pull # read every board and search, embed and score them, extract the best, rescore, summarise uv run app serve # http://127.0.0.1:8790 ``` @@ -24,9 +24,11 @@ uv run app searches add jobindex "Copenhagen golang" --q "golang go-udvikler" -- uv run app searches add adzuna "Berlin backend" --q backend --country de --where Berlin uv run app searches list uv run app companies discover "Pleo" pleo.io --add # find its board on its own site, then follow it +uv run app companies set-board lunar greenhouse:lunar # attach a board to a company, or replace the one it has +uv run app companies detach lunar # stop reading its board; the company stays ``` -The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, Teamtailor, its own pages) is still a row, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint when the pages link nothing, and with `--add` follows what it found the same way the seed does. A probe hit is a guess by name: check the roles it lists before trusting it for a common name. +The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, Teamtailor, its own pages) is still a row, with that host and its slug there where one was found, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles; a board is read once before it is kept, and one that lists no roles yet is kept too. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name. A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles; the next pull after switching it back on reopens the ones it still lists. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys. @@ -44,7 +46,7 @@ The cog opens `/settings`: a rail of pages on the left, the page beside it, stic | | | |---|---| | **Search agents** | a card per agent, newest first: name, an On/Off switch, what it looks for, and its weights. Edit turns the card into the form in place (name, intent, weights, require/any/prefer gates behind an Advanced disclosure); New agent puts a blank card on top. Saving or switching on rescores; deleting one takes its votes, liked companies and scores with it | -| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex or Adzuna: try, save, edit, remove), company boards (every company whose board is read, its open roles and its affinity for the picked agent, following a company) and runs (Run the pull now, what is running, the schedule, and the Problems log) | +| **Sources** | one page showing one section at a time, picked in the rail or by the URL fragment: job boards (queries on Jobindex or Adzuna: try, save, edit, remove), company boards (every company, its board or the host it cannot be read from, its open roles and its affinity for the picked agent, following a company, setting or changing a company's board) and runs (Run the pull now, what is running, the schedule, and the Problems log) | | **Account** | the password, sessions and passkeys. Open on loopback, locked on anything else | Likes, hidden companies and votes belong to a search agent, not to the account: two agents rank the same roles differently, and one may hide a company the other favours. An agent also learns from votes: its `taste` weight scores a role by how close it is to the roles the agent liked minus the ones it passed. diff --git a/app/cli.py b/app/cli.py index 109d79e..ea85c7b 100644 --- a/app/cli.py +++ b/app/cli.py @@ -3,7 +3,7 @@ app serve [--host H] [--port P] [--reload] app job pull|extract|embed|score [--payload JSON] app worker [--workers http=8,cpu=1] [--until-idle] - app companies seed|add|discover|check + app companies seed|add|set-board|detach|discover|check app searches add|list|remove app db init @@ -67,17 +67,26 @@ def build_parser() -> argparse.ArgumentParser: companies = sub.add_parser("companies", help="the target company list") companies_sub = companies.add_subparsers(dest="action", required=True) seed = companies_sub.add_parser( - "seed", help="import seed/companies.json, subscribing to each readable board") + "seed", help="import seed/companies.json, and the boards it names for rows without one") seed.add_argument("--path", default="seed/companies.json") - add = companies_sub.add_parser("add", help="add one company from its board") + add = companies_sub.add_parser( + "add", help="follow a company by its board, attaching or replacing an existing one's") add.add_argument("board", help="host:slug, e.g. greenhouse:acme") add.add_argument("name") + set_board = companies_sub.add_parser( + "set-board", help="attach a board to a company, or replace the one it has") + set_board.add_argument("key", help="the company key, e.g. trade-republic") + set_board.add_argument("board", help="host:slug, e.g. greenhouse:acme") + detach = companies_sub.add_parser( + "detach", help="stop reading a company's board and close its roles; the row stays") + detach.add_argument("key") discover = companies_sub.add_parser( "discover", help="find a company's board from its website; --add follows it") discover.add_argument("name") discover.add_argument("domain", help="its website, e.g. pleo.io") discover.add_argument("--add", action="store_true", - help="add the company, with its board when one is readable") + help="add or update the company: its board when one is readable, " + "else the host it runs on") companies_sub.add_parser("check", help="report boards that do not answer") searches = sub.add_parser("searches", help="saved searches on cross-company job boards") @@ -230,69 +239,41 @@ def run_companies(args: argparse.Namespace) -> int: config = config_mod.load(args.config) store = Store(config.database) if args.action == "seed": - print(json.dumps(seed_companies(store, args.path))) - return 0 + return seed_companies(store, args.path) if args.action == "add": return add_company(store, args.board, args.name) + if args.action == "set-board": + return set_board(store, args.key, args.board) + if args.action == "detach": + return detach_board(store, args.key) if args.action == "discover": return discover_company(store, args.name, args.domain, add=args.add) return check_companies(store) -def seed_companies(store: Store, path: str) -> dict[str, int]: - """Import `seed/companies.json`: one `companies` row per entry, and one - `ats` subscription for each entry with a readable `board`. An entry - without one is the point of the list as much as one with: a company worth - following whose careers site the app cannot read yet keeps its `ats` - host and `careers_url`, and the Companies section says so. A key that - already exists is skipped, so re-running only ever adds what is missing; - it never edits a row.""" - from app.models import company_key - from plugins.ats import careers_url, is_known_target +def seed_companies(store: Store, path: str) -> int: + """`app companies seed`: `app.jobs.sources.seed_companies`, its counts as + JSON and each entry that did not land whole on stderr.""" + from app import jobs - data = json.loads(Path(path).read_text(encoding="utf-8")) - added = readable = skipped = 0 - for row in data.get("companies", []): - name = row["name"] - key = row.get("key") or company_key(name) - if store.company(key) is not None: - skipped += 1 - continue - board = row.get("board") or "" - if board and not is_known_target(board): - print(f"companies seed: {key}: {board!r} is not a known board; kept without it", - file=sys.stderr) - board = "" - host = board.partition(":")[0] - store.add_company( - name, key=key, domain=row.get("domain", ""), - careers_url=row.get("careers_url") or (careers_url(board) if board else ""), - ats=row.get("ats") or host, country=row.get("country", ""), - city=row.get("city", "")) - added += 1 - if not board: - continue - try: - store.add_subscription("ats", board, label=name, company=name, board=host) - readable += 1 - except ValueError as exc: - # The company row landed; the subscription is the only thing - # missing, and worth saying rather than losing silently. - print(f"companies seed: {key}: {exc}", file=sys.stderr) - return {"added": added, "readable": readable, "skipped": skipped, - "total": len(data.get("companies", []))} + result = jobs.sources.seed_companies(store, path) + for problem in result.pop("problems"): + print(f"companies seed: {problem}", file=sys.stderr) + print(json.dumps(result)) + return 0 def discover_company(store: Store, name: str, domain: str, *, add: bool) -> int: """`app companies discover "Pleo" pleo.io [--add]`: what the company's - site says its board is, as JSON. With `--add`, the company is added the - way the seed adds it: with its board when one is readable, as a row the - app cannot read yet otherwise.""" + site says its board is, as JSON. With `--add`, the company is added or + updated: its board attached or replaced when one is readable, else the + host and slug it runs on recorded; see `add_discovered`.""" from app import jobs found = jobs.discover.discover(name, domain) out: dict[str, Any] = {"name": name, "domain": domain, "board": found.board, - "ats": found.ats, "careers_url": found.careers_url, + "ats": found.ats, "ats_slug": found.ats_slug, + "careers_url": found.careers_url, "evidence": found.evidence} if add: try: @@ -306,8 +287,8 @@ def discover_company(store: Store, name: str, domain: str, *, add: bool) -> int: def add_company(store: Store, board: str, name: str) -> int: """`app companies add greenhouse:acme "Acme"`: fetch the board first - and refuse if it does not answer, is not a known host, or the company - already exists. Port of seedgen's `-add`.""" + and refuse if it does not answer or is not a known host. A company that + exists gets the board attached or replaced. Port of seedgen's `-add`.""" from app import jobs from app.models import company_key @@ -320,12 +301,42 @@ def add_company(store: Store, board: str, name: str) -> int: return 0 +def set_board(store: Store, key: str, board: str) -> int: + """`app companies set-board trade-republic greenhouse:traderepublic`: + attach the board to an existing company or replace the one it has.""" + from app import jobs + + try: + sub, found, closed = jobs.sources.set_board(store, key, board) + except ValueError as exc: + print(f"companies set-board: {exc}", file=sys.stderr) + return 1 + print(json.dumps({"key": key, "board": sub.target, "roles": found, "closed": closed})) + return 0 + + +def detach_board(store: Store, key: str) -> int: + """`app companies detach acme`: stop reading its board, close its roles.""" + from app import jobs + + try: + closed = jobs.sources.detach_board(store, key) + except KeyError: + print(f"companies detach: no company {key!r}", file=sys.stderr) + return 1 + except ValueError as exc: + print(f"companies detach: {exc}", file=sys.stderr) + return 1 + print(json.dumps({"key": key, "closed": closed})) + return 0 + + def check_companies(store: Store) -> int: """`app companies check`: every `ats` board, fetched once. Prints a line per board that did not answer and exits 1 if any did not.""" from plugins.ats import fetch_board - subs = store.subscriptions(source="ats") + subs = [s for s in store.subscriptions(source="ats") if s.show] dead = [] for sub in subs: try: diff --git a/app/jobs/discover.py b/app/jobs/discover.py index b934755..c118418 100644 --- a/app/jobs/discover.py +++ b/app/jobs/discover.py @@ -6,16 +6,20 @@ website, not from a board. This module gets from the one to the other: it fetches the site's careers page, or the pages the home page links to as careers, and looks for a link into one of the hosts in `HOSTS`. A readable host (one the `ats` plugin parses) becomes a `board` target the pull can -read; any other host is recorded as `ats`, so the company is kept as one the -app cannot read yet rather than dropped. +read; any other host is recorded as `ats`, with its slug where the URL +carries one, so the company is kept as one the app cannot read yet rather +than dropped. -When no page names a host, the three hosts with a cheap per-slug endpoint -(Greenhouse, Lever, Ashby) are probed with the company's own slug, because a -site that draws its jobs from an iframe or a script leaves no link to find. +Only when no page names a host at all are the hosts with a cheap per-slug +endpoint (`_PROBES`) probed with the company's own slug, because a site that +draws its jobs from an iframe or a script leaves no link to find. A page that +names an unreadable host is the answer: a probe on a common name can hit +another tenant's board. found = discover("Pleo", "pleo.io") found.board # "greenhouse:pleo", or "" when nothing readable found.ats # "greenhouse", "workday", ... or "" when unknown + found.ats_slug # the name that host knows the company by, or "" found.careers_url # the page that named the host, or the careers page found.evidence # the URL that named it @@ -30,7 +34,7 @@ from __future__ import annotations import logging import re from collections.abc import Callable, Iterable -from dataclasses import dataclass +from dataclasses import dataclass, replace from urllib.parse import urljoin import httpx @@ -112,9 +116,10 @@ _ANCHOR = re.compile(r"]*href\s*=\s*[\"']([^\"'#][^\"']*)[\"'][^>]*>(.*?) _TAGS = re.compile(r"<[^>]+>") #: Words a pattern's slug group can catch that name no board: the hosts' -#: own path segments, and the analytics subdomain Recruitee sites embed. +#: own path segments, the analytics subdomain Recruitee sites embed, and +#: the CDN Teamtailor sites load from (cdn.teamtailor.com). _NOT_SLUGS = frozenset({"embed", "boards", "v1", "api", "www", "jobs", "careers", "apply", - "careers-analytics"}) + "careers-analytics", "cdn"}) Found = tuple[str, str, str] # (host, slug, the URL that named it) @@ -125,6 +130,7 @@ class Discovery: board: str = "" ats: str = "" + ats_slug: str = "" careers_url: str = "" evidence: str = "" @@ -301,14 +307,22 @@ def _discover(name: str, site: str, client: httpx.Client, fetch: Fetch, careers_page = "" def first_readable(found: list[Found], page: str) -> Discovery | None: + """The first readable board on a page. Otherwise the first host named + is remembered, with the first slug any page gives that host: a page + may name teamtailor by its CDN before it links acme.teamtailor.com.""" nonlocal unreadable for host, slug, url in found: if host in READABLE and slug: - return Discovery(board=f"{host}:{slug}", ats=host, careers_url=page, - evidence=url) + return Discovery(board=f"{host}:{slug}", ats=host, ats_slug=slug, + careers_url=page, evidence=url) if unreadable is None and found: host, slug, url = found[0] - unreadable = Discovery(ats=host, careers_url=page, evidence=url) + unreadable = Discovery(ats=host, ats_slug=slug, careers_url=page, evidence=url) + if unreadable is not None and not unreadable.ats_slug: + for host, slug, url in found: + if host == unreadable.ats and slug: + unreadable = replace(unreadable, ats_slug=slug, evidence=url) + break return None if home and (hit := first_readable(boards_in(home), home_url)): @@ -333,11 +347,13 @@ def _discover(name: str, site: str, client: httpx.Client, fetch: Fetch, if hit := first_readable(boards_in(text), final): return hit queue[:0] = [link for link in careers_links(text, final)[:2] if link not in seen] + # A page named a host the app cannot read: that is the answer, and a + # probe could only add another tenant's board of the same name. + if unreadable is not None: + return unreadable for slug in slug_guesses(name, site): for host in _PROBES: if probe_slug(client, host, slug): - return Discovery(board=f"{host}:{slug}", ats=host, + return Discovery(board=f"{host}:{slug}", ats=host, ats_slug=slug, careers_url=careers_page, evidence=_PROBES[host].format(slug=slug)) - if unreadable is not None: - return unreadable return Discovery(careers_url=careers_page) diff --git a/app/jobs/sources.py b/app/jobs/sources.py index 8637670..a72318d 100644 --- a/app/jobs/sources.py +++ b/app/jobs/sources.py @@ -1,6 +1,8 @@ """What a pull reads, changed by hand: add a board or a search, switch one on -or off. `app companies`, `app searches` and the settings pages all call these, -so the command line and the pages cannot disagree about what "off" means. +or off, attach, replace or detach a company's board, import the seed. +`app companies`, `app searches` and the settings pages all call these, so the +command line and the pages cannot disagree about what "off" means. Every way +a company gets a board is `set_board`. A board or search switched off has its open roles closed at once. Nothing else would ever close them, since only a pull that reads a target closes its @@ -9,7 +11,9 @@ roles; switched back on, the next pull reopens every role it still lists. from __future__ import annotations +import json import os +from pathlib import Path from typing import TYPE_CHECKING, Any from app.config import Config @@ -32,55 +36,171 @@ def set_fetch(store: Store, sub_id: int, fetch: bool) -> tuple[Subscription, int return sub, closed -def add_board(store: Store, board: str, name: str) -> tuple[Subscription, int]: - """Follow one company's ATS board: fetch it first, and refuse if the host - is unknown, the company exists, the board does not answer or lists - nothing. Returns the subscription and how many roles the board lists.""" +def set_board(store: Store, key: str, board: str, *, name: str = "", + check: bool = True, **fields: str) -> tuple[Subscription, int | None, int]: + """Read company `key` from `board` (`host:slug`): attach it to a company + that has none, replace the one it has, or create the company from `name` + when there is no row. The board is fetched first and refused if its host + is unknown or it does not answer; a board listing zero roles is kept, since + a company between openings is still one to follow. `check=False` skips the + fetch, for the seed, whose boards were read once when it was written. + + The row's `ats` and `ats_slug` become the board's host and slug, and its + `careers_url` the board's page where it was empty or the old board's. + `fields` (domain, country, city, ...) fill only columns that are empty, so + nothing a person or the seed wrote is overwritten. Row and subscription + change in one transaction (`Store.set_company_board`), and replacing a + board closes the old one's open roles. Returns the subscription, how many + roles the board lists (None unchecked) and how many roles closed.""" from plugins.ats import careers_url, fetch_board, is_known_target board, name = board.strip(), name.strip() - if not name: - raise ValueError("a company needs a name") + key = key.strip() or company_key(name) if not is_known_target(board): raise ValueError(f"{board!r} is not host:slug for a known ATS host, " "e.g. greenhouse:acme or ashby:acme") - key = company_key(name) - if store.company(key) is not None: - raise ValueError(f"{name} is already followed") - try: - rows = fetch_board(board) - except Exception as exc: - raise ValueError(f"{board} did not answer: {exc}") from exc - if not rows: - raise ValueError(f"{board} answered but lists no roles; nothing added") - host = board.partition(":")[0] - store.add_company(name, key=key, careers_url=careers_url(board)) - sub = store.add_subscription("ats", board, label=name, company=name, board=host) - return sub, len(rows) + company = store.company(key) + if company is None: + if not name: + raise ValueError(f"no company {key!r}; a new company needs a name") + if company_key(name) != key: + raise ValueError(f"{name!r} is not the company {key!r}") + rows: list[Any] | None = None + if check: + try: + rows = fetch_board(board) + except Exception as exc: + raise ValueError(f"{board} did not answer: {exc}") from exc + host, _, slug = board.partition(":") + writes: dict[str, str] = {k: v for k, v in fields.items() if v} + if company is not None: + writes = {k: v for k, v in writes.items() if not getattr(company, k)} + old = store.company_subscription(key) + if not company.careers_url or (old is not None + and company.careers_url == careers_url(old.target)): + writes["careers_url"] = careers_url(board) + else: + writes.setdefault("careers_url", careers_url(board)) + writes.update(ats=host, ats_slug=slug) + _company, sub, closed = store.set_company_board(key, board, host=host, name=name, + fields=writes) + assert sub is not None + return sub, None if rows is None else len(rows), closed + + +def detach_board(store: Store, key: str) -> int: + """Stop reading company `key`'s board: its open roles close and the row + stays, with its `ats` and `ats_slug`, as one the app does not read. + Returns how many roles closed. Raises KeyError for an unknown company and + ValueError for one with no board.""" + if store.company(key) is None: + raise KeyError(key) + sub = store.company_subscription(key) + if sub is None or not sub.show: + raise ValueError(f"{key} has no board to detach") + _company, _sub, closed = store.set_company_board(key, "") + return closed + + +def add_board(store: Store, board: str, name: str) -> tuple[Subscription, int]: + """Follow one company by its ATS board: `set_board` keyed by `name`, so + a company that exists gets the board attached or replaced. Returns the + subscription and how many roles the board lists.""" + name = name.strip() + if not name: + raise ValueError("a company needs a name") + sub, roles, _closed = set_board(store, company_key(name), board, name=name) + return sub, roles or 0 def add_discovered(store: Store, name: str, domain: str, found: Discovery) -> tuple[str, int]: - """Follow a company from what `app.jobs.discover` found: through - `add_board` when the board is readable, else as a row with its `ats` host - and careers page, so the company is kept as one the app cannot read yet. - Returns the company key and how many roles its board lists (0 with no - board). Refuses a company that exists, as `add_board` does.""" + """Follow a company from what `app.jobs.discover` found. A readable + board goes through `set_board`, which attaches it, replaces the one the + company has, or creates the company. Otherwise the row records `ats`, + `ats_slug` and the careers page, as one the app cannot read yet; a + company that already has a board keeps it and this raises, since + discovery finding nothing readable is weaker evidence than a board that + answers. Returns the company key and how many roles its board lists (0 + with no board).""" name = name.strip() if not name: raise ValueError("a company needs a name") key = company_key(name) if found.board: - _sub, roles = add_board(store, found.board, name) - store.update_company(key, domain=domain, ats=found.ats) - return key, roles - if store.company(key) is not None: - raise ValueError(f"{name} is already followed") - store.add_company(name, key=key, domain=domain, ats=found.ats, - careers_url=found.careers_url) + _sub, roles, _closed = set_board(store, key, found.board, name=name, domain=domain) + return key, roles or 0 + company = store.company(key) + if company is None: + store.add_company(name, key=key, domain=domain, ats=found.ats, + ats_slug=found.ats_slug, careers_url=found.careers_url) + return key, 0 + sub = store.company_subscription(key) + if sub is not None and sub.show: + raise ValueError(f"{name} is read from {sub.target} and discovery found no " + "readable board; nothing changed") + store.update_company(key, ats=found.ats, ats_slug=found.ats_slug, + careers_url=found.careers_url or company.careers_url, + **({"domain": domain} if not company.domain else {})) return key, 0 +def seed_companies(store: Store, path: str | Path) -> dict[str, Any]: + """Import `seed/companies.json`: one `companies` row per entry, and its + `board` through `set_board` where the entry names one. An entry without + a board is the point of the list as much as one with: a company whose + careers site the app cannot read yet keeps its `ats` host, `ats_slug` + and `careers_url`, and the Companies section says so. + + Re-running adds what is missing: a new entry becomes a row, and an + existing row that never had a board gets the one the seed now names, so a + seed update that gains boards reaches a database made before it. A row's + board, once it has one (detached counts), and its affinity, notes and + status are never touched. The seed's boards are not fetched again: they + were read once when the seed was written, and `app companies check` + finds the dead ones. Returns counts, and `problems`, one line per entry + that could not land whole.""" + from plugins.ats import is_known_target + + data = json.loads(Path(path).read_text(encoding="utf-8")) + entries = data.get("companies", []) + added = boards = skipped = 0 + problems: list[str] = [] + for row in entries: + name = row["name"].strip() + key = row.get("key") or company_key(name) + board = (row.get("board") or "").strip() + if board and not is_known_target(board): + problems.append(f"{key}: {board!r} is not a known board; kept without it") + board = "" + facts = {f: str(row.get(f) or "") for f in ("domain", "country", "city", + "careers_url", "ats", "ats_slug")} + exists = store.company(key) is not None + if exists and (not board or store.company_subscription(key) is not None): + skipped += 1 + continue + if board: + try: + set_board(store, key, board, name=name, check=False, + **{f: v for f, v in facts.items() if f not in ("ats", "ats_slug")}) + boards += 1 + added += not exists + continue + except ValueError as exc: + problems.append(f"{key}: {exc}") + if exists: + skipped += 1 + continue + # No board, or one that clashed: the company is still a row, and a + # clashing board's host and slug are what discovery found. + if board: + facts["ats"], _, facts["ats_slug"] = board.partition(":") + store.add_company(name, key=key, **facts) + added += 1 + return {"added": added, "boards": boards, "skipped": skipped, "total": len(entries), + "problems": problems} + + def search_params(provider: str, q: str, *, area: int = 0, country: str = "", where: str = "", pages: int = 0) -> dict[str, Any]: """The `params` a search subscription stores, with only the keys its diff --git a/app/models.py b/app/models.py index 70d44e6..1c91b85 100644 --- a/app/models.py +++ b/app/models.py @@ -335,6 +335,9 @@ class Company(Model): ats: str = "" country: str = "" city: str = "" + #: The name `ats` knows the company by, the `acme` of acme.teamtailor.com; + #: "" where discovery found none. Kept for a host no parser reads yet. + ats_slug: str = "" created_at: str updated_at: str diff --git a/app/static/app.css b/app/static/app.css index eaff7e5..40b38ee 100644 --- a/app/static/app.css +++ b/app/static/app.css @@ -1200,6 +1200,17 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di gap: .4rem; font-size: .8rem; color: var(--dim); } .srow[data-muted="true"], .crow[data-hidden="true"] { opacity: .6; } +/* A company row's Set board / Change editor: its own line under `.sub`, + wrapping like `.sub` does, the boxes styled as `.source-form`'s. */ +.crow .board-edit { + grid-column: 1 / -1; display: flex; flex-wrap: wrap; align-items: center; + gap: .4rem; padding-top: .25rem; +} +.crow .board-edit select, .crow .board-edit input { + background: var(--bg); border: 1px solid var(--line); + border-radius: .3rem; padding: .35rem .5rem; +} +.crow .board-edit input { flex: 1 1 8rem; min-width: 0; } .linky { padding: 0; border: 0; background: none; font-size: .8rem; @@ -1282,7 +1293,8 @@ table.list select { padding: .25rem .4rem; } button, .top nav a { min-height: 2.75rem; } .top nav a { display: inline-flex; align-items: center; justify-content: center; } .filters input[type="search"], .filters input[type="text"], .filters select, - .agent-form input, .source-form select { min-height: 2.75rem; } + .agent-form input, .source-form select, + .crow .board-edit input, .crow .board-edit select { min-height: 2.75rem; } .affinity button { min-width: 2.75rem; } .check input { width: 1.25rem; height: 1.25rem; } } diff --git a/app/static/companies.js b/app/static/companies.js index 8c8aab5..54740c0 100644 --- a/app/static/companies.js +++ b/app/static/companies.js @@ -11,6 +11,12 @@ * server's default, the enabled agent with the lowest id; with no agent at * all there is nothing to like, and the section says so instead of drawing * the buttons. + * + * Each row's "Set board" (no board yet) or "Change" (a board) opens a small + * editor in the row: the host select, with the options the server rendered + * into the Follow form's `#b-host`, and a slug. Saving posts to + * `/api/companies/{key}/board`, which reads the board once before keeping + * it; Detach stops reading it and keeps the company. */ (function () { "use strict"; @@ -21,6 +27,9 @@ /** @type {object[]} the search agents, by id */ let agents = []; + /** The key of the row whose board editor is open, or null. One at a time. */ + let editing = null; + /** The id in `#c-agent`, or null while there is no agent to pick. */ function agentId() { const value = $("#c-agent").value; @@ -45,7 +54,7 @@ // ---------------------------------------------------------------- the list - function subLine(company) { + function subLine(company, rerender) { const sub = document.createElement("span"); sub.className = "sub"; if (company.board) { @@ -57,11 +66,15 @@ // Followed, but its careers site runs on nothing the app reads yet. const badge = document.createElement("span"); badge.className = "badge"; - badge.textContent = company.ats || "own site"; + badge.textContent = company.ats + ? (company.ats_slug ? `${company.ats}:${company.ats_slug}` : company.ats) + : "own site"; sub.appendChild(badge); sub.append(" · not readable yet"); } else { - sub.append("from job boards"); + // Every row comes from the seed, Follow or discover: none of them + // found where this company posts its roles. + sub.append("board not found"); } if (company.careers_url && !company.board) { const link = document.createElement("a"); @@ -72,11 +85,92 @@ sub.append(" · "); sub.appendChild(link); } - sub.append(` · ${count(company.open_roles, "open role")}`); + sub.append(` · ${count(company.open_roles, "open role")} `); + const edit = document.createElement("button"); + edit.type = "button"; + edit.className = "linky"; + edit.textContent = company.subscription_id == null ? "Set board" : "Change"; + edit.setAttribute("aria-expanded", String(editing === company.key)); + edit.addEventListener("click", () => { + editing = editing === company.key ? null : company.key; + rerender(); + }); + sub.appendChild(edit); return sub; } - /** The board switch, or nothing for a company known only from a search: + /** The open row's board editor: host and slug, Save, Cancel, and Detach + for a row that has a board. */ + function boardEditor(company, rerender) { + const form = document.createElement("form"); + form.className = "board-edit"; + form.autocomplete = "off"; + const host = document.createElement("select"); + host.setAttribute("aria-label", `${company.name}'s board host`); + for (const option of $("#b-host").options) host.appendChild(option.cloneNode(true)); + const known = [...host.options].map((o) => o.value); + const current = company.board || company.ats; + if (known.includes(current)) host.value = current; + const slug = document.createElement("input"); + slug.type = "text"; + slug.required = true; + slug.spellcheck = false; + slug.placeholder = "slug"; + slug.value = company.ats_slug || ""; + slug.setAttribute("aria-label", `${company.name}'s slug on that host`); + const save = document.createElement("button"); + save.type = "submit"; + save.textContent = company.subscription_id == null ? "Set board" : "Change board"; + const cancel = document.createElement("button"); + cancel.type = "button"; + cancel.className = "linky"; + cancel.textContent = "Cancel"; + cancel.addEventListener("click", () => { editing = null; rerender(); }); + form.append(host, slug, save, cancel); + + async function done() { + editing = null; + await loadCompanies(agentId()); + rerender(); + } + + form.addEventListener("submit", async (event) => { + event.preventDefault(); + save.disabled = true; + try { + const result = await api(`/api/companies/${encodeURIComponent(company.key)}/board`, + { method: "POST", body: { board: `${host.value}:${slug.value.trim()}` } }); + const closed = result.closed ? `, ${count(result.closed, "old role")} closed` : ""; + window.toast.info(`${company.name}: reading ${result.source.target}, ` + + `${count(result.found, "role")} listed${closed}.`); + await done(); + } catch (err) { + save.disabled = false; + } + }); + if (company.subscription_id != null) { + const detach = document.createElement("button"); + detach.type = "button"; + detach.className = "linky"; + detach.textContent = "Detach"; + detach.addEventListener("click", async () => { + detach.disabled = true; + try { + const result = await api(`/api/companies/${encodeURIComponent(company.key)}/board`, + { method: "DELETE" }); + window.toast.info(`${company.name}: board detached, ` + + `${count(result.closed, "open role")} closed.`); + await done(); + } catch (err) { + detach.disabled = false; + } + }); + form.append(detach); + } + return form; + } + + /** The board switch, or nothing for a company with no board: there is no subscription to switch. */ function onSwitch(company, rerender) { if (company.subscription_id == null) return null; @@ -109,7 +203,7 @@ case "hidden": return company.affinity === -1; case "on": return company.on === true; case "off": return company.on === false; - case "unreadable": return company.subscription_id == null && Boolean(company.ats || company.careers_url); + case "unreadable": return company.subscription_id == null; default: return true; } } @@ -140,7 +234,8 @@ if (agent != null) { row.appendChild(affinityControl(company.key, company.affinity, async () => render(), agent)); } - row.appendChild(subLine(company)); + row.appendChild(subLine(company, render)); + if (editing === company.key) row.appendChild(boardEditor(company, render)); body.appendChild(row); } const open = rows.reduce((sum, c) => sum + c.open_roles, 0); diff --git a/app/storage/schema.py b/app/storage/schema.py index 75d7711..f9dc207 100644 --- a/app/storage/schema.py +++ b/app/storage/schema.py @@ -94,7 +94,9 @@ CREATE TABLE IF NOT EXISTS role_tags ( -- ats is the host the careers site runs on, readable by the ats plugin or -- not (workday, successfactors, ...): what discovery found, '' unknown. A -- company with an ats and no board subscription is one the app cannot read --- yet. country is an ISO code and city the head office's. +-- yet. country is an ISO code and city the head office's. ats_slug is the +-- name that host knows the company by (the acme of acme.teamtailor.com), +-- '' where its URL carries none, kept for the parser written later. CREATE TABLE IF NOT EXISTS companies ( key TEXT PRIMARY KEY, name TEXT NOT NULL, @@ -106,7 +108,8 @@ CREATE TABLE IF NOT EXISTS companies ( updated_at TEXT NOT NULL, ats TEXT NOT NULL DEFAULT '', country TEXT NOT NULL DEFAULT '', - city TEXT NOT NULL DEFAULT '' + city TEXT NOT NULL DEFAULT '', + ats_slug TEXT NOT NULL DEFAULT '' ); -- The targets desk's runner hands to plugins. newsdesk's shape, with its two @@ -337,6 +340,9 @@ ALTER TABLE companies ADD COLUMN city TEXT NOT NULL DEFAULT ''; """), (8, "agents.limits: how deep into its ranking the model stages read is each agent's own", """ ALTER TABLE agents ADD COLUMN limits TEXT NOT NULL DEFAULT '{}'; +"""), + (9, "companies.ats_slug: the slug of a host the app cannot read yet is kept", """ +ALTER TABLE companies ADD COLUMN ats_slug TEXT NOT NULL DEFAULT ''; """), ) diff --git a/app/storage/store.py b/app/storage/store.py index c463971..ac98409 100644 --- a/app/storage/store.py +++ b/app/storage/store.py @@ -69,7 +69,8 @@ _VOTED_SCORE = ("MIN(1.0, s.score * CASE v.vote WHEN -1 THEN 0.5 " #: nothing, so `vote` and `voted_at` are NULL: one query shape either way. _WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?" -_COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes", "ats", "country", "city") +_COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes", "ats", "country", "city", + "ats_slug") #: A company as the store serves it: the row and the asked-for agent's affinity. _COMPANY_SELECT = """SELECT c.*, COALESCE(ac.affinity, 0) AS affinity FROM companies c @@ -979,24 +980,31 @@ class Store: def add_company(self, name: str, *, key: str = "", domain: str = "", status: str = "researching", careers_url: str = "", notes: str = "", ats: str = "", country: str = "", - city: str = "") -> Company: + city: str = "", ats_slug: str = "") -> Company: """Raises ValueError on a key already taken.""" name = name.strip() key = key or company_key(name) if not key: raise ValueError("a company needs a name") - stamp = self.now() try: - self._write( - """INSERT INTO companies (key, name, domain, status, careers_url, notes, - ats, country, city, created_at, updated_at) - VALUES (?,?,?,?,?,?,?,?,?,?,?)""", - (key, name, domain, status, careers_url, notes, ats, country, city, - stamp, stamp)) + with self.transaction() as conn: + self._insert_company(conn, key, name, { + "domain": domain, "status": status, "careers_url": careers_url, + "notes": notes, "ats": ats, "country": country, "city": city, + "ats_slug": ats_slug}) except sqlite3.IntegrityError as exc: raise ValueError(f"company {key!r} already exists") from exc return self._company(key) + def _insert_company(self, conn: sqlite3.Connection, key: str, name: str, + fields: dict[str, Any]) -> None: + stamp = self.now() + columns = ["key", "name", *fields, "created_at", "updated_at"] + conn.execute( + f"INSERT INTO companies ({', '.join(columns)}) " + f"VALUES ({', '.join('?' * len(columns))})", + (key, name, *fields.values(), stamp, stamp)) + def company(self, key: str, agent_id: int | None = None) -> Company | None: """One company, with `affinity` as `agent_id` set it; 0 without an agent.""" row = self._row(f"{_COMPANY_SELECT} WHERE c.key=?", (agent_id, key)) @@ -1029,8 +1037,8 @@ class Store: return self._company(key, agent_id) def update_company(self, key: str, **fields: Any) -> Company: - """Change any of name, domain, status, careers_url, notes. Affinity is - per agent: `set_affinity`.""" + """Change any of `_COMPANY_FIELDS`. Affinity is per agent: + `set_affinity`.""" unknown = set(fields) - set(_COMPANY_FIELDS) if unknown: raise ValueError(f"not a company field: {', '.join(sorted(unknown))}") @@ -1072,13 +1080,94 @@ class Store: """`company_key -> ats host`, read off each company's own subscription. `Subscription.company` is the display name it and the company row were both added with, so `company_key` of one finds the - other; a company added by hand with no subscription is simply - absent.""" + other; a company with no subscription, or a detached one (`show` 0, + see `set_company_board`), is simply absent.""" boards: dict[str, str] = {} for sub in self.subscriptions(source="ats"): - boards.setdefault(company_key(sub.company), sub.board) + if sub.show: + boards.setdefault(company_key(sub.company), sub.board) return boards + def company_subscription(self, key: str) -> Subscription | None: + """Company `key`'s own `ats` subscription, detached or not; None when + it never had one. A company has at most one: `set_company_board` + rewrites it in place rather than adding a second.""" + with closing(self._connect()) as conn: + return self._company_subscription(conn, key) + + @staticmethod + def _company_subscription(conn: sqlite3.Connection, key: str) -> Subscription | None: + rows = conn.execute("SELECT * FROM subscriptions WHERE source = 'ats' " + "ORDER BY show DESC, id").fetchall() + return next((Subscription.from_row(r) for r in rows + if company_key(r["company"]) == key), None) + + def set_company_board(self, key: str, target: str, *, host: str = "", name: str = "", + fields: dict[str, Any] | None = None, + ) -> tuple[Company, Subscription | None, int]: + """Point company `key` at the `ats` board `target`, or at none when + `target` is "", in one transaction, so a clash on either table leaves + nothing half written. + + A missing company is created from `name` and `fields`; an existing one + has `fields` written over it. Its subscription is rewritten in place: + a new target closes the old target's open roles, as switching a board + off does, and "" detaches it (off, hidden, its roles closed) while the + row stays for the next board. Returns the company, its subscription + (None when there is none) and how many roles closed. Raises KeyError + for a missing company with no board, ValueError on a UNIQUE clash.""" + fields = dict(fields or {}) + unknown = set(fields) - set(_COMPANY_FIELDS) + if unknown: + raise ValueError(f"not a company field: {', '.join(sorted(unknown))}") + stamp = self.now() + closed = 0 + try: + with self.transaction() as conn: + row = conn.execute("SELECT name FROM companies WHERE key=?", (key,)).fetchone() + if row is None: + name = fields.pop("name", "") or name.strip() + if not target: + raise KeyError(key) + if not name: + raise ValueError("a company needs a name") + self._insert_company(conn, key, name, fields) + else: + name = fields.get("name") or row["name"] + if fields: + sets = ", ".join(f"{f}=?" for f in fields) + conn.execute(f"UPDATE companies SET {sets}, updated_at=? WHERE key=?", + (*fields.values(), stamp, key)) + sub = self._company_subscription(conn, key) + sub_id = sub.id if sub else None + if sub is not None and sub.target != target: + closed = conn.execute( + "UPDATE roles SET closed_at = ? " + "WHERE subscription_id = ? AND closed_at IS NULL", + (stamp, sub.id)).rowcount + if not target: + if sub is not None: + conn.execute("UPDATE subscriptions SET subscribed=0, fetch=0, show=0 " + "WHERE id=?", (sub.id,)) + elif sub is not None: + conn.execute( + """UPDATE subscriptions + SET target=?, board=?, label=?, company=?, subscribed=1, + fetch=1, show=1, + target_id=CASE WHEN target=? THEN target_id ELSE '' END + WHERE id=?""", + (target, host, name, name, target, sub.id)) + else: + sub_id = conn.execute( + """INSERT INTO subscriptions (source, target, label, company, board, + params, added_at) + VALUES ('ats',?,?,?,?,'{}',?)""", + (target, name, name, host, stamp)).lastrowid + except sqlite3.IntegrityError as exc: + raise ValueError(f"another board already has {target!r} or the label " + f"{name!r}") from exc + return (self._company(key), self.subscription(sub_id) if sub_id else None, closed) + # -------------------------------------------------------- subscriptions def subscriptions(self, source: str | None = None, diff --git a/app/templates/settings/sources.html b/app/templates/settings/sources.html index 386b62f..1e8efbc 100644 --- a/app/templates/settings/sources.html +++ b/app/templates/settings/sources.html @@ -68,14 +68,11 @@

Follow a company