diff --git a/README.md b/README.md index 7e55246..b921f40 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,12 @@ uv run app searches list uv run app companies discover "Pleo" pleo.io --add # find its board on its own site, then follow it uv run app companies set-board lunar greenhouse:lunar # attach a board to a company, or replace the one it has uv run app companies detach lunar # stop reading its board; the company stays +uv run app companies add "Revolut" --careers-url https://www.revolut.com/careers --reason blocked # one word: a company to read by hand +uv run app companies add greenhouse:lunar Lunar # two: a board and its company, which follows the company +uv run app companies by-hand # the companies read by hand, never opened first, with the Watchlist's rating +uv run app companies edit revolut --note "needs a login" # name, link, note or reason +uv run app companies checked revolut # its careers page was looked at just now; the Watchlist shows the date +uv run app companies remove revolut # refused while a board is on or it has open roles ``` The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, its own pages) is still a row, with that host and its slug there where one was found, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles, and asks first when there are open roles to lose; a board is read once before it is kept, and one that lists no roles yet is kept too. Setting a board queues a pull right after, so its roles are read by the one already running or the very next one, rather than waiting for the schedule. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name. diff --git a/app/cli.py b/app/cli.py index 6017af6..fe92c5d 100644 --- a/app/cli.py +++ b/app/cli.py @@ -3,7 +3,7 @@ app serve [--host H] [--port P] [--reload] app job pull|extract|embed|score [--payload JSON] app worker [--workers http=8,cpu=1] [--until-idle] - app companies seed|add|set-board|detach|discover|check + app companies seed|add|edit|checked|remove|by-hand|set-board|detach|discover|check app searches add|list|remove app agents list|prompt app stages [extract|summarise on|off] @@ -44,6 +44,8 @@ if str(_ROOT) not in sys.path: def build_parser() -> argparse.ArgumentParser: + from app.models import HAND_REASONS + parser = argparse.ArgumentParser( prog="app", description="roledesk: a job-search engine on desk and the pipeline queue.") parser.add_argument("-c", "--config", default=None, @@ -71,10 +73,34 @@ def build_parser() -> argparse.ArgumentParser: seed = companies_sub.add_parser( "seed", help="import seed/companies.json, and the boards it names for rows without one") seed.add_argument("--path", default="seed/companies.json") + # One word after `add` is a company for the list checked by hand, two are + # a board and its company, which is what `add` meant before that list. add = companies_sub.add_parser( - "add", help="follow a company by its board, attaching or replacing an existing one's") - add.add_argument("board", help="host:slug, e.g. greenhouse:acme") - add.add_argument("name") + "add", help="add NAME to the list checked by hand, or with BOARD NAME follow a " + "company by its board, attaching or replacing an existing one's") + add.add_argument("first", metavar="NAME|BOARD", + help="the company, or host:slug, e.g. greenhouse:acme, before its name") + add.add_argument("second", metavar="NAME", nargs="?") + add.add_argument("--careers-url", default="", help="its careers page, http or https") + add.add_argument("--note", default="") + add.add_argument("--reason", default="added", choices=HAND_REASONS, metavar="REASON", + help=f"one of {', '.join(repr(r) for r in HAND_REASONS)}") + edit = companies_sub.add_parser("edit", help="change a company's name, link, note or reason") + edit.add_argument("key") + edit.add_argument("--name") + edit.add_argument("--careers-url") + edit.add_argument("--note") + edit.add_argument("--reason", choices=HAND_REASONS, metavar="REASON", + help=f"one of {', '.join(repr(r) for r in HAND_REASONS)}") + checked = companies_sub.add_parser( + "checked", help="record that its careers page was looked at just now") + checked.add_argument("key") + remove = companies_sub.add_parser( + "remove", help="delete a company with no board on and no open roles") + remove.add_argument("key") + companies_sub.add_parser( + "by-hand", help="the companies checked by hand, never checked first, then the " + "longest ago") set_board = companies_sub.add_parser( "set-board", help="attach a board to a company, or replace the one it has") set_board.add_argument("key", help="the company key, e.g. trade-republic") @@ -274,7 +300,15 @@ def run_companies(args: argparse.Namespace) -> int: if args.action == "seed": return seed_companies(store, args.path) if args.action == "add": - return add_company(store, args.board, args.name) + return add_company(store, args) + if args.action == "edit": + return edit_company(store, args) + if args.action == "checked": + return company_checked(store, args.key) + if args.action == "remove": + return remove_company(store, args.key) + if args.action == "by-hand": + return list_by_hand(store) if args.action == "set-board": return set_board(store, args.key, args.board) if args.action == "detach": @@ -318,7 +352,25 @@ def discover_company(store: Store, name: str, domain: str, *, add: bool) -> int: return 0 -def add_company(store: Store, board: str, name: str) -> int: +def add_company(store: Store, args: argparse.Namespace) -> int: + """`app companies add "Acme" [--careers-url URL] [--note TEXT] [--reason + REASON]` puts a company with no board on the list checked by hand; + `app companies add greenhouse:acme "Acme"` follows it by its board.""" + if args.second is not None: + return add_company_by_board(store, args.first, args.second) + from app import jobs + + try: + company = jobs.sources.add_by_hand(store, args.first, careers_url=args.careers_url, + notes=args.note, hand_reason=args.reason) + except ValueError as exc: + print(f"companies add: {exc}", file=sys.stderr) + return 1 + print(json.dumps({"key": company.key, "hand_reason": company.hand_reason})) + return 0 + + +def add_company_by_board(store: Store, board: str, name: str) -> int: """`app companies add greenhouse:acme "Acme"`: fetch the board first and refuse if it does not answer or is not a known host. A company that exists gets the board attached or replaced. Port of seedgen's `-add`.""" @@ -334,6 +386,73 @@ def add_company(store: Store, board: str, name: str) -> int: return 0 +def edit_company(store: Store, args: argparse.Namespace) -> int: + """`app companies edit acme --careers-url https://acme.example/jobs`: only + the options given change; see `jobs.sources.edit_company`.""" + from app import jobs + + given = {"name": args.name, "careers_url": args.careers_url, "notes": args.note, + "hand_reason": args.reason} + try: + company = jobs.sources.edit_company( + store, args.key, **{f: v for f, v in given.items() if v is not None}) + except KeyError: + print(f"companies edit: no company {args.key!r}", file=sys.stderr) + return 1 + except ValueError as exc: + print(f"companies edit: {exc}", file=sys.stderr) + return 1 + print(json.dumps({"key": company.key, "name": company.name, + "careers_url": company.careers_url, "notes": company.notes, + "hand_reason": company.hand_reason})) + return 0 + + +def company_checked(store: Store, key: str) -> int: + """`app companies checked acme`: its careers page was looked at just now, + which the Watchlist keeps as `careers_opened_at`, the same stamp opening + the page from there sets.""" + if store.company(key) is None: + print(f"companies checked: no company {key!r}", file=sys.stderr) + return 1 + company = store.update_company(key, careers_opened_at=store.now()) + print(json.dumps({"key": key, "careers_opened_at": company.careers_opened_at})) + return 0 + + +def remove_company(store: Store, key: str) -> int: + """`app companies remove acme`: refused for a company still read, see + `jobs.sources.remove_company`.""" + from app import jobs + + try: + jobs.sources.remove_company(store, key) + except KeyError: + print(f"companies remove: no company {key!r}", file=sys.stderr) + return 1 + except ValueError as exc: + print(f"companies remove: {exc}", file=sys.stderr) + return 1 + print(json.dumps({"key": key, "removed": True})) + return 0 + + +def list_by_hand(store: Store) -> int: + """`app companies by-hand`: the companies with no board shown that are not + known from a search only, one line each, never opened first and then the + longest ago; `watch` is the Watchlist's rating, yes, no or a dash for + unrated. See `jobs.sources.by_hand`.""" + from app import jobs + + rows = jobs.sources.by_hand(store) + for c in rows: + watch = "-" if c.watch is None else "yes" if c.watch else "no" + opened = c.careers_opened_at[:10] or "never" + print(f"{c.name:30} {c.hand_reason or '-':8} {watch:4} {opened:10} {c.careers_url}") + print(f"{len(rows)} by hand") + return 0 + + def set_board(store: Store, key: str, board: str) -> int: """`app companies set-board trade-republic greenhouse:traderepublic`: attach the board to an existing company or replace the one it has.""" diff --git a/app/jobs/sources.py b/app/jobs/sources.py index 4eedbea..3eab0bb 100644 --- a/app/jobs/sources.py +++ b/app/jobs/sources.py @@ -15,9 +15,10 @@ import json import os from pathlib import Path from typing import TYPE_CHECKING, Any +from urllib.parse import urlsplit from app.config import Config -from app.models import Subscription, company_key +from app.models import HAND_REASONS, Company, Subscription, company_key from app.storage import Store from app.jobs.pull import FAILING, search_target @@ -45,8 +46,9 @@ def set_board(store: Store, key: str, board: str, *, name: str = "", a company between openings is still one to follow. `check=False` skips the fetch, for the seed, whose boards were read once when it was written. - The row's `ats` and `ats_slug` become the board's host and slug, and its - `careers_url` the board's page where it was empty or the old board's. + The row's `ats` and `ats_slug` become the board's host and slug, its + `careers_url` the board's page where it was empty or the old board's, and + its `hand_reason` empty, since a company that is read is not read by hand. `fields` (domain, country, city, ...) fill only columns that are empty, so nothing a person or the seed wrote is overwritten. Row and subscription change in one transaction (`Store.set_company_board`), and replacing a @@ -81,7 +83,7 @@ def set_board(store: Store, key: str, board: str, *, name: str = "", writes["careers_url"] = careers_url(board) else: writes.setdefault("careers_url", careers_url(board)) - writes.update(ats=host, ats_slug=slug) + writes.update(ats=host, ats_slug=slug, hand_reason="") before = store.company_subscription(key) _company, sub, closed = store.set_company_board(key, board, host=host, name=name, fields=writes) @@ -150,12 +152,116 @@ def add_discovered(store: Store, name: str, domain: str, if sub is not None and sub.show: raise ValueError(f"{name} is read from {sub.target} and discovery found no " "readable board; nothing changed") + # A link somebody typed is kept: discovery's is only a guess at the page. store.update_company(key, ats=found.ats, ats_slug=found.ats_slug, - careers_url=found.careers_url or company.careers_url, + careers_url=company.careers_url or found.careers_url, **({"domain": domain} if not company.domain else {})) return key, 0 +def known_from_search_only(company: Company, ever_boarded: set[str]) -> bool: + """A row that exists only because a search listed its roles and somebody + judged it (`Store.set_affinity`): no board, not even a detached one + (`ever_boarded`, the keys of every `ats` subscription), and none of the + facts every way of following a company or listing it by hand writes.""" + return not (company.key in ever_boarded or company.ats or company.careers_url + or company.domain or company.hand_reason) + + +def by_hand(store: Store) -> list[Company]: + """The list checked by hand: companies with no board shown that are not + known from a search only. Never opened first, then the longest ago + (`careers_opened_at`, the Watchlist's stamp); the name breaks a tie.""" + boards = store.company_boards() + ever_boarded = {company_key(s.company) for s in store.subscriptions(source="ats")} + rows = [c for c in store.companies() + if c.key not in boards and not known_from_search_only(c, ever_boarded)] + return sorted(rows, key=lambda c: (c.careers_opened_at, c.name.casefold())) + + +class Conflict(ValueError): + """A change refused because of what the company already is, not because of + what was asked for: the pages answer it with 409, not 422.""" + + +def check_careers_url(url: str) -> str: + """`url` stripped; ValueError unless it is empty or an http(s) link. The + page puts it in an `href`, where `javascript:` would run.""" + url = url.strip() + if url: + parts = urlsplit(url) + if parts.scheme not in ("http", "https") or not parts.netloc: + raise ValueError("The careers link must start with http:// or https://.") + return url + + +def add_by_hand(store: Store, name: str, *, careers_url: str = "", notes: str = "", + hand_reason: str = "added") -> Company: + """A company with no board, put on the list checked by hand. Raises + Conflict when the key its name folds to is taken, ValueError for a name + with no letters, a link that is no http(s) URL or a reason outside + `HAND_REASONS`.""" + link = check_careers_url(careers_url) + key = company_key(name) + if key and store.company(key) is not None: + raise Conflict(f"{name.strip()} is on the list already as {key}.") + return store.add_company(name, key=key, careers_url=link, notes=notes, + hand_reason=hand_reason) + + +def edit_company(store: Store, key: str, **fields: str) -> Company: + """Change a company's `name`, `careers_url`, `notes` or `hand_reason`; the + key never changes. A new name is Conflict for a company with a board + subscription, shown or not, since the subscription is found by the key + its company name folds to and would be lost. Raises KeyError for an + unknown company and ValueError for a blank name, a link that is no http(s) + URL or a reason outside `HAND_REASONS`.""" + unknown = set(fields) - {"name", "careers_url", "notes", "hand_reason"} + if unknown: + raise ValueError(f"not a field to edit: {', '.join(sorted(unknown))}") + company = store.company(key) + if company is None: + raise KeyError(key) + if "careers_url" in fields: + fields["careers_url"] = check_careers_url(fields["careers_url"]) + if "name" in fields: + fields["name"] = fields["name"].strip() + if not fields["name"]: + raise ValueError("A company needs a name.") + if fields["name"] != company.name and store.company_subscription(key) is not None: + raise Conflict(f"{company.name} has a board, which is found by its name; " + "detach the board first.") + return store.update_company(key, **fields) + + +def remove_company(store: Store, key: str) -> None: + """Delete company `key` and every agent's stance on it. Raises KeyError + for an unknown company and Conflict for one still read: a company with + a board on (`show`, as `detach_board` decides) or with open roles, which + would otherwise vanish from the list with nothing said. A detached board + is no board, since nothing reads it and its roles are closed; its + subscription row stays, and follows the company if it comes back.""" + if store.company(key) is None: + raise KeyError(key) + sub = store.company_subscription(key) + if sub is not None and sub.show: + raise Conflict(f"{key} is read from {sub.target}; detach the board first") + if store.open_role_counts().get(key): + raise Conflict(f"{key} has open roles; detach its board first") + store.remove_company(key) + + +def _fill_empty(store: Store, key: str, **facts: str) -> None: + """Write each of `facts` to company `key` where the stored value is + empty: what a seed may add to a row that exists, since a person's link or + reason is never overwritten.""" + company = store.company(key) + assert company is not None + fill = {f: v for f, v in facts.items() if v and not getattr(company, f)} + if fill: + store.update_company(key, **fill) + + def seed_companies(store: Store, path: str | Path) -> dict[str, Any]: """Import `seed/companies.json`: one `companies` row per entry, and its `board` through `set_board` where the entry names one. An entry without @@ -167,8 +273,10 @@ def seed_companies(store: Store, path: str | Path) -> dict[str, Any]: existing row that never had a board gets the one the seed now names, so a seed update that gains boards reaches a database made before it. A row's board, once it has one (detached counts), and its affinity, notes and - status are never touched. The seed's boards are not fetched again: they - were read once when the seed was written, and `app companies check` + status are never touched; its `careers_url` and `hand_reason` are filled + where empty, so the seed can hand a row its link and its reason but never + overwrite the one a person set. The seed's boards are not fetched again: + they were read once when the seed was written, and `app companies check` finds the dead ones. Returns counts, and `problems`, one line per entry that could not land whole.""" from plugins.ats import is_known_target @@ -186,8 +294,15 @@ def seed_companies(store: Store, path: str | Path) -> dict[str, Any]: board = "" facts = {f: str(row.get(f) or "") for f in ("domain", "country", "city", "careers_url", "ats", "ats_slug")} + reason = str(row.get("hand_reason") or "") + if reason not in HAND_REASONS: + problems.append(f"{key}: {reason!r} is not a hand_reason; kept without it") + reason = "" exists = store.company(key) is not None + if exists: + _fill_empty(store, key, hand_reason=reason) if exists and (not board or store.company_subscription(key) is not None): + _fill_empty(store, key, careers_url=facts["careers_url"]) skipped += 1 continue if board: @@ -200,13 +315,14 @@ def seed_companies(store: Store, path: str | Path) -> dict[str, Any]: except ValueError as exc: problems.append(f"{key}: {exc}") if exists: + _fill_empty(store, key, careers_url=facts["careers_url"]) skipped += 1 continue # No board, or one that clashed: the company is still a row, and a # clashing board's host and slug are what discovery found. if board: facts["ats"], _, facts["ats_slug"] = board.partition(":") - store.add_company(name, key=key, **facts) + store.add_company(name, key=key, hand_reason=reason, **facts) added += 1 return {"added": added, "boards": boards, "skipped": skipped, "total": len(entries), "problems": problems} diff --git a/app/models.py b/app/models.py index f398496..119adfc 100644 --- a/app/models.py +++ b/app/models.py @@ -644,6 +644,16 @@ STATUSES: tuple[str, ...] = get_args(Status) #: -1 mute, 0 neutral, 1 interested, 2 favourite (skips gates, floors the score). AFFINITY = range(-1, 3) +#: Why a company is on the list checked by hand. `unread` is a host the app +#: could read and has no parser for yet, `robots` a site whose robots.txt +#: disallows it, `blocked` one that answers a program with a challenge, `key` +#: one whose API answers only with a key lifted from its own script, `group` a +#: company that sits inside its owner's board, `gone` one whose site is dead, +#: and `added` one put there by hand or by an import and not looked at. '' +#: is none of these: the company is read, or is not a matter for the list. +HandReason = Literal["", "unread", "robots", "blocked", "key", "group", "gone", "added"] +HAND_REASONS: tuple[str, ...] = get_args(HandReason) + # -------------------------------------------------------------- posting dates @@ -929,6 +939,9 @@ class Company(Model): watch: int | None = None #: When the careers page was last opened from the Watchlist; "" never. careers_opened_at: str = "" + #: Why the company is read by hand, one of `HAND_REASONS`; "" when it is + #: not on that list. + hand_reason: str = "" created_at: str updated_at: str diff --git a/app/storage/schema.py b/app/storage/schema.py index 3682551..ff3353c 100644 --- a/app/storage/schema.py +++ b/app/storage/schema.py @@ -109,7 +109,9 @@ CREATE TABLE IF NOT EXISTS role_tags ( -- '' where its URL carries none, kept for the parser written later. -- watch is the Watchlist's rating, global and not an agent's: NULL unrated, 1 -- worth checking by hand, 0 not; careers_opened_at when its careers page was --- last opened from there. +-- last opened from there, which is also when a person last looked at it. +-- hand_reason is why a person reads the careers page (models.HAND_REASONS), +-- '' when nobody has to. CREATE TABLE IF NOT EXISTS companies ( key TEXT PRIMARY KEY, name TEXT NOT NULL, @@ -124,7 +126,8 @@ CREATE TABLE IF NOT EXISTS companies ( city TEXT NOT NULL DEFAULT '', ats_slug TEXT NOT NULL DEFAULT '', watch INTEGER CHECK (watch IN (0, 1)), - careers_opened_at TEXT NOT NULL DEFAULT '' + careers_opened_at TEXT NOT NULL DEFAULT '', + hand_reason TEXT NOT NULL DEFAULT '' ); -- The targets desk's runner hands to plugins. newsdesk's shape, with its two @@ -436,6 +439,12 @@ ALTER TABLE scores ADD COLUMN prompt TEXT NOT NULL DEFAULT ''; "when its careers page was last opened", """ ALTER TABLE companies ADD COLUMN watch INTEGER CHECK (watch IN (0, 1)); ALTER TABLE companies ADD COLUMN careers_opened_at TEXT NOT NULL DEFAULT ''; +"""), + # Why a person reads the company's careers page by hand, one of + # models.HAND_REASONS; '' when nobody has to. When it was last looked at + # is careers_opened_at. + (16, "companies.hand_reason: why a company is read by hand", """ +ALTER TABLE companies ADD COLUMN hand_reason TEXT NOT NULL DEFAULT ''; """), ) diff --git a/app/storage/store.py b/app/storage/store.py index afa4983..c3f880e 100644 --- a/app/storage/store.py +++ b/app/storage/store.py @@ -21,7 +21,7 @@ import numpy as np from desk.models import Item from app.models import ( - AFFINITY, REMOTE, STATUSES, Agent, Application, ApplicationEvent, Candidate, + AFFINITY, HAND_REASONS, REMOTE, STATUSES, Agent, Application, ApplicationEvent, Candidate, Company, Gate, Limits, Prompt, Role, RoleRow, ScoredRole, Subscription, Weights, check_prompts, company_key, country_of, excerpt, place_key, posted_at, posted_host, remote_of, title_key, @@ -94,7 +94,7 @@ _EXCERPT_SOURCE = "COALESCE(NULLIF(roles.summary, ''), roles.description)" _EXCERPT_RAW = 240 _COMPANY_FIELDS = ("name", "domain", "status", "careers_url", "notes", "ats", "country", "city", - "ats_slug", "watch", "careers_opened_at") + "ats_slug", "watch", "careers_opened_at", "hand_reason") #: A company as the store serves it: the row and the asked-for agent's affinity. _COMPANY_SELECT = """SELECT c.*, COALESCE(ac.affinity, 0) AS affinity FROM companies c @@ -1163,18 +1163,21 @@ class Store: def add_company(self, name: str, *, key: str = "", domain: str = "", status: str = "researching", careers_url: str = "", notes: str = "", ats: str = "", country: str = "", - city: str = "", ats_slug: str = "") -> Company: - """Raises ValueError on a key already taken.""" + city: str = "", ats_slug: str = "", + hand_reason: str = "") -> Company: + """Raises ValueError on a key already taken or a `hand_reason` outside + `HAND_REASONS`.""" name = name.strip() key = key or company_key(name) if not key: raise ValueError("a company needs a name") + _check_hand_reason(hand_reason) try: with self.transaction() as conn: self._insert_company(conn, key, name, { "domain": domain, "status": status, "careers_url": careers_url, "notes": notes, "ats": ats, "country": country, "city": city, - "ats_slug": ats_slug}) + "ats_slug": ats_slug, "hand_reason": hand_reason}) except sqlite3.IntegrityError as exc: raise ValueError(f"company {key!r} already exists") from exc return self._company(key) @@ -1238,6 +1241,8 @@ class Store: unknown = set(fields) - set(_COMPANY_FIELDS) if unknown: raise ValueError(f"not a company field: {', '.join(sorted(unknown))}") + if "hand_reason" in fields: + _check_hand_reason(fields["hand_reason"]) if fields: sets = ", ".join(f"{f}=?" for f in fields) changed = self._write(f"UPDATE companies SET {sets}, updated_at=? WHERE key=?", @@ -1247,7 +1252,13 @@ class Store: return self._company(key) def remove_company(self, key: str) -> bool: - return self._write("DELETE FROM companies WHERE key=?", (key,)) > 0 + """Delete the row and every agent's stance on it, in one transaction: + `agent_companies` holds the key with no foreign key to hold it back, so + a row left behind would give a company added again under the same name + an affinity it never got. Roles and subscriptions are not touched.""" + with self.transaction() as conn: + conn.execute("DELETE FROM agent_companies WHERE company_key=?", (key,)) + return conn.execute("DELETE FROM companies WHERE key=?", (key,)).rowcount > 0 def _company(self, key: str, agent_id: int | None = None) -> Company: found = self.company(key, agent_id) @@ -1542,6 +1553,12 @@ def _check_status(status: str) -> None: f"({', '.join(STATUSES)})") +def _check_hand_reason(reason: Any) -> None: + if reason not in HAND_REASONS: + raise ValueError(f"hand_reason is one of {', '.join(repr(r) for r in HAND_REASONS)}, " + f"not {reason!r}") + + def _check_affinity(affinity: Any) -> None: if isinstance(affinity, bool) or not isinstance(affinity, int) or affinity not in AFFINITY: raise ValueError(f"affinity is -1, 0, 1 or 2, not {affinity!r}") diff --git a/app/web/api.py b/app/web/api.py index b458c0b..0aadefb 100644 --- a/app/web/api.py +++ b/app/web/api.py @@ -17,7 +17,7 @@ from pydantic import BaseModel, Field from app import jobs from app.models import ( AFFINITY, Agent, Application, Company, Gate, Limits, Prompts, Role, RoleRow, - ScoredRole, Status, Subscription, Weights, company_key, + HandReason, ScoredRole, Status, Subscription, Weights, company_key, ) from app.storage import Store from app.web.deps import ConfigDep, StoreDep @@ -303,12 +303,17 @@ class CompanyOut(BaseModel): the board has failed on every fetch and why it last did, both empty while it answers. `from_search` marks a row that exists only because a search listed its roles and somebody judged it (`Store.set_affinity`): - no board, not even a detached one, and no `ats`, `careers_url` or - `domain`, which every way of following a company writes one of. - `watch` and `careers_opened_at` are the Watchlist's (`Company`'s).""" + no board, not even a detached one, and no `ats`, `careers_url`, `domain` + or `hand_reason`, which every way of following a company or listing it by + hand writes one of (`jobs.sources.known_from_search_only`). `watch` and + `careers_opened_at` are the Watchlist's (`Company`'s), the latter also + when a person last looked at the careers page. `by_hand` marks a company + with no board shown that is not `from_search`, and `hand_reason` says why + it is read by hand (`app.models.HAND_REASONS`).""" key: str name: str + notes: str = "" board: str subscription_id: int | None = None on: bool | None = None @@ -325,6 +330,8 @@ class CompanyOut(BaseModel): failing_since: str = "" failing_reason: str = "" from_search: bool = False + hand_reason: str = "" + by_hand: bool = False def _company_boards(store: Store) -> dict[str, tuple[int, str, bool, str]]: @@ -342,30 +349,101 @@ def _company_boards(store: Store) -> dict[str, tuple[int, str, bool, str]]: return boards -@router.get("/companies") -def list_companies(store: StoreDep, agent: int | None = None) -> list[CompanyOut]: - """Every company, with `affinity` as `agent` (the default agent when - absent) has it; 0 throughout when there is no agent.""" +def _company_outs(store: Store, agent: int | None) -> list[CompanyOut]: boards = _company_boards(store) ever_boarded = {company_key(s.company) for s in store.subscriptions(source="ats")} counts = store.open_role_counts() failing = jobs.pull.failing_boards(store) out = [] - for c in store.companies(_agent_id(store, agent)): + for c in store.companies(agent): sub_id, host, on, target = boards.get(c.key, (None, "", None, "")) since, reason = failing.get(target, ("", "")) - out.append(CompanyOut(key=c.key, name=c.name, board=host, + from_search = jobs.sources.known_from_search_only(c, ever_boarded) + out.append(CompanyOut(key=c.key, name=c.name, notes=c.notes, board=host, subscription_id=sub_id, on=on, affinity=c.affinity, open_roles=counts.get(c.key, 0), ats=c.ats, ats_slug=c.ats_slug, careers_url=c.careers_url, country=c.country, city=c.city, domain=c.domain, watch=c.watch, careers_opened_at=c.careers_opened_at, failing_since=since, failing_reason=reason, - from_search=not (c.key in ever_boarded or c.ats - or c.careers_url or c.domain))) + from_search=from_search, hand_reason=c.hand_reason, + by_hand=c.key not in boards and not from_search)) return out +def _one_company(store: Store, key: str) -> CompanyOut: + """`key` as the list shows it, with the default agent's affinity.""" + return next(c for c in _company_outs(store, _agent_id(store, None)) if c.key == key) + + +@router.get("/companies") +def list_companies(store: StoreDep, agent: int | None = None) -> list[CompanyOut]: + """Every company, with `affinity` as `agent` (the default agent when + absent) has it; 0 throughout when there is no agent.""" + return _company_outs(store, _agent_id(store, agent)) + + +class NewCompanyBody(BaseModel): + name: str + careers_url: str = "" + notes: str = "" + hand_reason: HandReason = "added" + + +@router.post("/companies", status_code=201) +def add_company(body: NewCompanyBody, store: StoreDep) -> CompanyOut: + """A company put on the list checked by hand: no board, `hand_reason` + `added` unless it says another. 409 when its key, from the name, is taken.""" + try: + company = jobs.sources.add_by_hand(store, body.name, careers_url=body.careers_url, + notes=body.notes, hand_reason=body.hand_reason) + except jobs.sources.Conflict as exc: + raise HTTPException(409, str(exc)) from exc + except ValueError as exc: + raise HTTPException(422, str(exc)) from exc + return _one_company(store, company.key) + + +class CompanyPatch(BaseModel): + name: str | None = None + careers_url: str | None = None + notes: str | None = None + hand_reason: HandReason | None = None + + +@router.patch("/companies/{key}") +def edit_company(key: str, body: CompanyPatch, store: StoreDep) -> CompanyOut: + """Change any of a company's name, careers link, notes and reason; a field + left out stays. The key never changes, and `name` only for a company + without a board: 409 for one with a subscription, shown or not (see + `jobs.sources.edit_company`).""" + fields = body.model_dump(exclude_unset=True) + if any(v is None for v in fields.values()): + raise HTTPException(422, "A field is left out, not null.") + try: + jobs.sources.edit_company(store, key, **fields) + except KeyError: + raise HTTPException(404, f"no such company: {key}") from None + except jobs.sources.Conflict as exc: + raise HTTPException(409, str(exc)) from exc + except ValueError as exc: + raise HTTPException(422, str(exc)) from exc + return _one_company(store, key) + + +@router.delete("/companies/{key}", status_code=204) +def remove_company(key: str, store: StoreDep) -> Response: + """Delete a company and every agent's stance on it. 409 for one with a + board shown or with open roles; detach the board first.""" + try: + jobs.sources.remove_company(store, key) + except KeyError: + raise HTTPException(404, f"no such company: {key}") from None + except ValueError as exc: + raise HTTPException(409, str(exc)) from exc + return Response(status_code=204) + + class CompanyBoardBody(BaseModel): board: str @@ -489,10 +567,12 @@ class CareersBody(BaseModel): @router.post("/companies/{key}/careers") def set_careers_url(key: str, body: CareersBody, store: StoreDep) -> Company: """Set or correct where a company's careers page is; "" clears it. 422 - for anything but an http(s) URL, 404 for an unknown key.""" - url = body.careers_url.strip() - if url and not url.lower().startswith(("http://", "https://")): - raise HTTPException(422, "careers_url must start with http:// or https://") + for anything but an http(s) URL (`jobs.sources.check_careers_url`, the + rule the by-hand edit shares), 404 for an unknown key.""" + try: + url = jobs.sources.check_careers_url(body.careers_url) + except ValueError as exc: + raise HTTPException(422, str(exc)) from exc try: return store.update_company(key, careers_url=url) except KeyError: diff --git a/docs/plan-hosts-and-by-hand.md b/docs/plan-hosts-and-by-hand.md index e5c5cf3..fd478f6 100644 --- a/docs/plan-hosts-and-by-hand.md +++ b/docs/plan-hosts-and-by-hand.md @@ -38,7 +38,7 @@ The 49 were missed because the slug is not the company's name (`BoschGroup`, `ji **No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key that the page's JavaScript carries. They go on the by-hand list. -**The by-hand list is the `companies` table.** A company is on it when it has no shown board and is not known from a search only. `careers_url` and `notes` exist already; migration 15 adds `hand_reason` and `checked_at`. No second table, since a company has one careers link and the list needs no history of checks. +**The by-hand list is the `companies` table, and the Watchlist tab is the list.** A company is on it when it has no shown board and is not known from a search only. `careers_url`, `notes`, `watch` (the rating) and `careers_opened_at` exist already, the last being the date a person last looked at the careers page; migration 16 adds `hand_reason`. There is no `checked_at`, since opening the page from the Watchlist stamps the date already. No second table, since a company has one careers link and the list needs no history of checks. **A link typed by hand is kept.** `add_discovered` and the seed fill `careers_url` only when it is empty. @@ -75,16 +75,15 @@ def probe_ok(body: bytes) -> bool: ... Slugs: Workday `lego.wd103/LEGO_External`; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (`jobs.teradyne.com/MiR`); Oracle `/`. -Company JSON, in `GET /api/companies`: `CompanyOut` gains `domain`, `notes`, `hand_reason`, `checked_at` and `by_hand` (bool). `hand_reason` is one of `''`, `unread` (readable, no parser yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone`, `added` (by hand or by an import, not looked at). `from_search` is false for a company with a `hand_reason`. +Company JSON, in `GET /api/companies`: `CompanyOut` has `domain`, `city`, `watch` and `careers_opened_at` from the Watchlist, and gains `notes`, `hand_reason` and `by_hand` (bool, no board shown and not `from_search`). `hand_reason` is one of `''`, `unread` (readable, no parser yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone`, `added` (by hand or by an import, not looked at). `from_search` is false for a company with a `hand_reason`. | Endpoint | Body | Answer | |---|---|---| | `POST /api/companies` | `name`, `careers_url`, `notes`, `hand_reason` (default `added`) | 201 `CompanyOut`; 409 when the key is taken | | `PATCH /api/companies/{key}` | any of `name`, `careers_url`, `notes`, `hand_reason` | `CompanyOut`; the key stays; 409 on `name` for a company with a board | -| `POST /api/companies/{key}/checked` | none | `CompanyOut` with `checked_at` now | | `DELETE /api/companies/{key}` | none | 204; 409 when it has a board or open roles; its `agent_companies` rows go with it | -`careers_url` is http or https, as `safeUrl` demands on the page. +`POST /api/companies/{key}/careers` and `POST /api/companies/{key}/opened` of the Watchlist stay and answer a `Company`; `POST .../careers` and `PATCH` check the link with the same function (`check_careers_url` in `app/jobs/sources.py`). `careers_url` is http or https, as `safeUrl` demands on the page. A person looking at the page is `POST .../opened`, which stamps `careers_opened_at`. ## Work packages @@ -94,7 +93,7 @@ Wave 0, in parallel, from today's `main`: |---|---|---|---| | 0 Host modules | `wp0-hosts` | Opus | `plugins/ats.py`, `plugins/atshosts/`, `app/jobs/discover.py`, `tests/test_ats.py`, `tests/test_discover.py`, `tests/test_atshosts.py` | | 1 By-hand backend | `wp1-byhand-api` | Sonnet | `app/models.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/web/api.py`, `app/jobs/sources.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_sources.py` | -| 2 By-hand page | `wp2-byhand-ui` | Opus | `app/templates/settings/sources.html`, `app/static/byhand.js`, `app/static/companies.js`, `app/static/app.css`, `app/web/pages.py` | +| 2b By-hand editing | `wp2b-byhand-ui` | Opus | `app/static/watchlist.js`, `app/templates/watchlist.html`, `app/static/app.css`, `tests/test_pages.py` | Wave 1, in parallel, from `main` after WP0 is merged. Each owns its modules in `plugins/atshosts/`, `tests/test_ats_.py` and `tests/fixtures/ats/*`, and nothing else: @@ -111,9 +110,9 @@ Wave 2 is decided after wave 1 by the roles each company would add: a module per **WP0.** The eight hosts move into modules with no change in behaviour: the existing tests pass with edits to imports only. `careers_url`, `is_known_target`, `_allowed_host` and `_paged` lose their Teamtailor branches to the module. `_get` takes `headers`. `detail` and the robots check are added to the shared code with tests against a fake host. `discover.py` takes `READABLE`, `HOSTS`, `_PROBES` and the body checks of `probe` from the modules; the hosts it names but cannot read stay in a table of its own. Discovery gains Lever's EU host. The tests that use `workday:` or `successfactors` as the unreadable host use `taleo` instead. -**WP1.** Migration 15, the four endpoints, `CompanyOut`'s new fields, `remove_company` taking the `agent_companies` rows, `careers_url` filled only when empty in `add_discovered` and `seed_companies`, the seed reading `hand_reason`, and `app companies add`, `edit`, `checked` and `remove` in the CLI. The `workday:` and `successfactors` fixtures in its three test files become `taleo`. +**WP1.** Migration 16 with `hand_reason` only, the endpoints `POST /api/companies`, `PATCH` and `DELETE`, `CompanyOut`'s new fields, `remove_company` taking the `agent_companies` rows, `careers_url` filled only when empty in `add_discovered` and `seed_companies`, the seed reading `hand_reason`, `check_careers_url` as the one rule for a typed link, and `app companies add`, `edit`, `checked`, `by-hand` and `remove` in the CLI, where `checked` stamps `careers_opened_at` and `by-hand` lists the never opened first. The `workday:` and `successfactors` fixtures in its three test files become `taleo`. -**WP2.** A section "Check by hand" in the Sources rail, after Company boards: a count, a New company button, and rows sorted by `checked_at`, never checked first. A row shows the name, the affinity control, the reason, the careers link, the note, "checked 3 days ago", and Checked, Edit and Remove. New and Edit use the form pattern of the job boards section (`sources.js`: a `