From e6be593542a8cb8cc74c302cd9ea5a35f9d91f08 Mon Sep 17 00:00:00 2001 From: Wolf Date: Tue, 29 Sep 2026 15:18:16 +0200 Subject: [PATCH] Ask before a destructive board or search switch; queue a pull after a reopening one Switching a company's board or a job board search off, replacing a company's board and detaching one all close open roles in a tap; each now asks first, naming the count (a replace also names the old and the new board), the same as removing a search and deleting an agent already do. Switching a target back on, following a board, setting a board and saving a search used to show nothing until the schedule's next pull, up to six hours later. Each now queues a pull job right in the API route, so the CLI and any other caller of the same endpoint behave like the page; the queued pull's own refusal while one is already running is not surfaced as an error. Co-Authored-By: Claude Sonnet --- README.md | 4 +-- app/static/companies.js | 47 ++++++++++++++++++++++++++---- app/static/sources.js | 12 +++++++- app/web/api.py | 38 +++++++++++++++++++------ docs/decisions.md | 2 ++ tests/test_web.py | 63 +++++++++++++++++++++++++++++++++++++---- 6 files changed, 143 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 6cc9a7e..6a6ac1d 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,9 @@ uv run app companies set-board lunar greenhouse:lunar # attach a board to a com uv run app companies detach lunar # stop reading its board; the company stays ``` -The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, Teamtailor, its own pages) is still a row, with that host and its slug there where one was found, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles; a board is read once before it is kept, and one that lists no roles yet is kept too. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name. +The seed is a hand-picked list of European tech employers, not a list of boards: a company whose careers site runs on something the app cannot read yet (Workday, SuccessFactors, Teamtailor, its own pages) is still a row, with that host and its slug there where one was found, shown under Companies as "not readable yet" with a link to its careers page, so the gap is visible and one parser at a time closes it. A row where nothing was found reads "board not found"; the "Not readable yet" filter shows both. Each row has a Set board control (Change once it has one) to attach, replace or detach its board by hand. Replacing or detaching closes the old board's open roles, and asks first when there are open roles to lose; a board is read once before it is kept, and one that lists no roles yet is kept too. Setting a board queues a pull right after, so its roles are read by the one already running or the very next one, rather than waiting for the schedule. `companies discover` reads a company's site for a link into one of the boards it knows, probes the hosts with a cheap per-slug endpoint only when no page names any host, and with `--add` sets the board of a new or existing company, or records the host it found when nothing readable was. A probe hit is a guess by name: check the roles it lists before trusting it for a common name. -A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles; the next pull after switching it back on reopens the ones it still lists. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys. +A Jobindex query is its own syntax: words are OR, `"a b"` is a phrase, `+word` is required; `--area 1` is Region Hovedstaden. An Adzuna search needs a free key from developer.adzuna.com in `ADZUNA_APP_ID` and `ADZUNA_APP_KEY`. `searches add` runs the search once and saves it only if it finds something. Switching a board or search off closes its open roles, and asks first when there are open roles to lose; switching it back on, or saving a search, queues a pull right after, so the ones it still lists reopen without waiting for the schedule. The `embeddings` extra is what `fit` needs, `agent` is for `extract` on a model, `passkeys` for the Account tab's passkeys. ## The tabs diff --git a/app/static/companies.js b/app/static/companies.js index c228f67..e7f2eb5 100644 --- a/app/static/companies.js +++ b/app/static/companies.js @@ -147,13 +147,22 @@ form.addEventListener("submit", async (event) => { event.preventDefault(); + const board = `${host.value}:${slug.value.trim()}`; + // Replacing a different board closes its roles (see set_board in + // app/jobs/sources.py); ask first, naming both boards and the count, + // the same as Detach and a removed search already do. + if (company.board && company.board !== board + && !window.confirm(`Replace ${company.name}'s board (${company.board}) ` + + `with ${board}? Its ${count(company.open_roles, "open role")} close.`)) { + return; + } save.disabled = true; try { const result = await api(`/api/companies/${encodeURIComponent(company.key)}/board`, - { method: "POST", body: { board: `${host.value}:${slug.value.trim()}` } }); + { method: "POST", body: { board } }); const closed = result.closed ? `, ${count(result.closed, "old role")} closed` : ""; - window.toast.info(`${company.name}: reading ${result.source.target}, ` - + `${count(result.found, "role")} listed${closed}.`); + window.toast.info(`${company.name}: ${result.source.target} listed ` + + `${count(result.found, "role")}${closed}. Reading now, or with the pull right after.`); await done(); } catch (err) { save.disabled = false; @@ -165,6 +174,11 @@ detach.className = "linky"; detach.textContent = "Detach"; detach.addEventListener("click", async () => { + if (company.open_roles + && !window.confirm(`Detach ${company.name}'s board (${company.board})? ` + + `Its ${count(company.open_roles, "open role")} close.`)) { + return; + } detach.disabled = true; try { const result = await api(`/api/companies/${encodeURIComponent(company.key)}/board`, @@ -190,6 +204,14 @@ box.checked = Boolean(company.on); box.setAttribute("aria-label", `Read ${company.name}'s board`); box.addEventListener("change", async () => { + // Switching off closes every open role in one tap; ask first, naming + // the count, as removing a search and deleting an agent already do. + if (!box.checked && company.open_roles + && !window.confirm(`Switch off ${company.name}'s board? ` + + `Its ${count(company.open_roles, "open role")} close.`)) { + box.checked = true; + return; + } box.disabled = true; try { const result = await api(`/api/sources/${company.subscription_id}/switch`, @@ -197,6 +219,8 @@ company.on = result.source.on; if (!box.checked && result.closed) { window.toast.info(`${company.name}: ${count(result.closed, "open role")} closed.`); + } else if (box.checked) { + window.toast.info(`${company.name}: reading now, or with the pull right after.`); } } catch (err) { box.checked = !box.checked; @@ -260,19 +284,30 @@ event.preventDefault(); const status = $("#board-status"); const button = $("#board-form button[type=submit]"); + const name = $("#b-name").value.trim(); + const board = `${$("#b-host").value}:${$("#b-slug").value.trim()}`; + // This name may already be a company with a different board: `set_board` + // would replace it and close its roles, the same as the row's own + // editor does, so it asks first too. + const existing = [...companies.values()] + .find((c) => c.name.toLowerCase() === name.toLowerCase()); + if (existing && existing.board && existing.board !== board + && !window.confirm(`Replace ${existing.name}'s board (${existing.board}) ` + + `with ${board}? Its ${count(existing.open_roles, "open role")} close.`)) { + return; + } button.disabled = true; status.textContent = "Reading the board once before following it…"; - const board = `${$("#b-host").value}:${$("#b-slug").value.trim()}`; try { const result = await api("/api/boards", { method: "POST", - body: { board, name: $("#b-name").value.trim() }, + body: { board, name }, }); await loadCompanies(agentId()); render(); $("#board-form").reset(); status.textContent = `Following ${result.source.label}: ` - + `${count(result.found, "role")} listed. They arrive with the next pull.`; + + `${count(result.found, "role")} listed. Reading now, or with the pull right after.`; } catch (err) { status.textContent = ""; } diff --git a/app/static/sources.js b/app/static/sources.js index ff7411f..193318c 100644 --- a/app/static/sources.js +++ b/app/static/sources.js @@ -34,6 +34,14 @@ box.checked = source.on; box.setAttribute("aria-label", `Run ${source.label}`); box.addEventListener("change", async () => { + // Switching off closes every open role in one tap; ask first, naming + // the count, as removing a search and deleting an agent already do. + if (!box.checked && source.open_roles + && !window.confirm(`Switch off ${source.label}? ` + + `Its ${count(source.open_roles, "open role")} close.`)) { + box.checked = true; + return; + } box.disabled = true; try { const result = await api(`/api/sources/${source.id}/switch`, @@ -41,6 +49,8 @@ Object.assign(source, result.source); if (!box.checked && result.closed) { window.toast.info(`${source.label}: ${count(result.closed, "open role")} closed.`); + } else if (box.checked) { + window.toast.info(`${source.label}: reading now, or with the pull right after.`); } } catch (err) { box.checked = !box.checked; @@ -184,7 +194,7 @@ renderSearches(); fillSearchForm(null); status.textContent = `Saved ${result.source.label}. ${found(result)} ` - + "Its roles arrive with the next pull."; + + "Reading now, or with the pull right after."; } catch (err) { status.textContent = ""; } diff --git a/app/web/api.py b/app/web/api.py index 374ef7c..633dc59 100644 --- a/app/web/api.py +++ b/app/web/api.py @@ -324,16 +324,21 @@ class CompanyBoardBody(BaseModel): @router.post("/companies/{key}/board") -def set_company_board(key: str, body: CompanyBoardBody, store: StoreDep) -> dict[str, Any]: +def set_company_board(key: str, body: CompanyBoardBody, store: StoreDep, + queue: QueueDep) -> dict[str, Any]: """Attach a board to a company, or replace the one it has. The board is fetched first and refused if it does not answer; replacing closes the - old board's open roles.""" + old board's open roles. Queues a pull right after, so the board is read + by the one already running or the very next one rather than the + schedule, up to six hours later; a refusal because one is already + running is not an error, see `app.jobs.pull.claim_run`.""" if store.company(key) is None: raise HTTPException(404, f"no such company: {key}") try: sub, found, closed = jobs.sources.set_board(store, key, body.board) except ValueError as exc: raise HTTPException(422, str(exc)) from exc + queue.enqueue("pull") return {"source": _one_source(store, sub.id), "found": found, "closed": closed} @@ -429,12 +434,18 @@ class SwitchBody(BaseModel): @router.post("/sources/{sub_id}/switch") -def switch_source(sub_id: int, body: SwitchBody, store: StoreDep) -> dict[str, Any]: - """On or off for the next pull. Off closes the open roles at once; on - reopens the ones the next pull still finds.""" +def switch_source(sub_id: int, body: SwitchBody, store: StoreDep, + queue: QueueDep) -> dict[str, Any]: + """On or off. Off closes the open roles at once; on queues a pull right + after, so the ones it still lists reopen once the pull that is already + running reads it, or the very next one does, rather than waiting for the + schedule, up to six hours later. A refusal because one is already + running is not an error, see `app.jobs.pull.claim_run`.""" if store.subscription(sub_id) is None: raise HTTPException(404, f"no such board or search: {sub_id}") _, closed = jobs.sources.set_fetch(store, sub_id, body.on) + if body.on: + queue.enqueue("pull") return {"source": _one_source(store, sub_id), "closed": closed} @@ -471,14 +482,19 @@ def try_search(body: SearchBody, config: ConfigDep) -> dict[str, Any]: @router.post("/searches") -def save_search(body: SearchBody, store: StoreDep, config: ConfigDep) -> dict[str, Any]: +def save_search(body: SearchBody, store: StoreDep, config: ConfigDep, + queue: QueueDep) -> dict[str, Any]: """Save a search, or update the one of that provider and name. It is run - once first and refused if it finds nothing.""" + once first and refused if it finds nothing. Queues a pull right after, + so its roles are read by the one already running or the very next one + rather than the schedule, up to six hours later; a refusal because one is + already running is not an error, see `app.jobs.pull.claim_run`.""" try: sub, found, complete = jobs.sources.save_search(config, store, body.provider, body.name, _params(body)) except ValueError as exc: raise HTTPException(422, str(exc)) from exc + queue.enqueue("pull") return {"source": _one_source(store, sub.id), "found": found, "complete": complete} @@ -497,14 +513,18 @@ class BoardBody(BaseModel): @router.post("/boards", status_code=201) -def add_board(body: BoardBody, store: StoreDep) -> dict[str, Any]: +def add_board(body: BoardBody, store: StoreDep, queue: QueueDep) -> dict[str, Any]: """Follow a company by its ATS board: `set_board` keyed by the name, so an existing company gets it attached or replaced. The board is fetched - first and refused if it does not answer.""" + first and refused if it does not answer. Queues a pull right after, so + its roles are read by the one already running or the very next one + rather than the schedule, up to six hours later; a refusal because one is + already running is not an error, see `app.jobs.pull.claim_run`.""" try: sub, found = jobs.sources.add_board(store, body.board, body.name) except ValueError as exc: raise HTTPException(422, str(exc)) from exc + queue.enqueue("pull") return {"source": _one_source(store, sub.id), "found": found} diff --git a/docs/decisions.md b/docs/decisions.md index e9f9952..02218be 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -66,6 +66,8 @@ What was decided for roledesk and the one reason for each. The longer reasoning **Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch already in flight when the switch happens reopens them again, so the pull's settle closes every switched-off target once more. +**A destructive switch asks first; a reopening one queues a pull instead of waiting for the schedule.** Switching a company's board or a job board search off, replacing a company's board (the Follow form and the row's Set board/Change editor, both `set_board`) and detaching one each close open roles in a tap; the page asks first when there are roles to lose, naming the count, a replace also naming the old and the new board, the same as removing a search and deleting an agent already do. Switching a target back on, following a board, setting a board and saving a search used to answer with nothing until the schedule's next pull, up to six hours later; each now queues `pull` right after, in the API route (`/api/sources/{id}/switch`, `/api/companies/{key}/board`, `/api/boards`, `/api/searches`) rather than in the page's script, so a script or `app job pull` calling the same endpoint gets what a click gets. `pull`'s own refusal while one is already running (`claim_run`) is not surfaced as an error: the change is read by the run already in flight, or by the very next one either way. + **The Sources tab and the CLI share `app.jobs.sources`.** Adding a board or a search and switching one are one function each, called by `app companies`, `app searches` and `/api/sources*`, so the two cannot disagree about what "off" does. A search or board is run once before it is saved, and refused if it does not answer. A search that finds nothing is refused too, since that is usually a typo in the query; a board that lists nothing is kept, since a company between openings is still one to follow. **Every way a company gets a board is `app.jobs.sources.set_board`.** `app companies add|set-board|discover --add|seed`, `/api/boards` and `/api/companies/{key}/board` all call it, so attaching a board to an existing row, replacing one and creating a company are one code path; the seed importer moved out of `app/cli.py` for this. The company row and its subscription change in one transaction (`Store.set_company_board`), so a board another company already has leaves no company row behind. Its host and slug become the row's `ats` and `ats_slug`; other facts only fill empty columns, so a person's or the seed's values survive. diff --git a/tests/test_web.py b/tests/test_web.py index 9f71ed0..2344f68 100644 --- a/tests/test_web.py +++ b/tests/test_web.py @@ -870,8 +870,13 @@ def test_sources_lists_searches_and_boards_with_their_countries( assert res["searches"][0]["params"] == {"q": "backend", "area": 1} -def test_switching_a_board_off_closes_its_roles_and_on_keeps_them_closed_until_a_pull( - client: TestClient, store: Store) -> None: +#: Every non-final state a job may be in, for a "was one queued at all" check +#: that does not race a background worker draining it before the assertion. +_ANY_STATE = ("queued", "running", "done", "failed", "cancelled") + + +def test_switching_a_board_off_closes_its_roles_and_on_queues_a_pull( + client: TestClient, store: Store, queue: Any) -> None: board = store.add_subscription("ats", "greenhouse:acme", label="Acme", company="Acme", board="greenhouse") store.upsert_role(posting("1", company="Acme"), board) @@ -880,14 +885,18 @@ def test_switching_a_board_off_closes_its_roles_and_on_keeps_them_closed_until_a assert off["closed"] == 1 and off["source"]["on"] is False assert store.subscriptions(source="ats", fetchable=True) == [] assert store.roles() == [] + assert queue.jobs.last("pull", states=_ANY_STATE) is None # off reads nothing again on = client.post(f"/api/sources/{board.id}/switch", json={"on": True}).json() assert on["closed"] == 0 and on["source"]["on"] is True + # On queues a pull right after, so the reopened board is read by the one + # running or the next one rather than the schedule, up to six hours away. + assert queue.jobs.last("pull", states=_ANY_STATE) is not None assert client.post("/api/sources/999/switch", json={"on": True}).status_code == 404 def test_a_search_is_tried_saved_updated_and_removed( - client: TestClient, store: Store, monkeypatch: pytest.MonkeyPatch) -> None: + client: TestClient, store: Store, queue: Any, monkeypatch: pytest.MonkeyPatch) -> None: _fake_search(monkeypatch, 3) body = {"provider": "jobindex", "name": "Copenhagen backend", "q": "backend", "area": 1} @@ -896,10 +905,14 @@ def test_a_search_is_tried_saved_updated_and_removed( assert tried["sample"][0] == {"title": "Engineer 0", "company": "Acme", "place": "København", "url": "https://x.example/0"} assert store.subscriptions(source="search") == [] + assert queue.jobs.last("pull", states=_ANY_STATE) is None # a mere try reads nothing saved = client.post("/api/searches", json=body).json() assert saved["source"]["target"] == "jobindex:copenhagen-backend" assert saved["source"]["params"] == {"q": "backend", "area": 1} + # Saving queues a pull right after, so its roles are read by the one + # running or the next one rather than the schedule, up to six hours away. + assert queue.jobs.last("pull", states=_ANY_STATE) is not None updated = client.post("/api/searches", json={**body, "q": "python", "pages": 3}).json() assert updated["source"]["id"] == saved["source"]["id"] @@ -925,7 +938,7 @@ def test_a_search_that_finds_nothing_or_is_malformed_is_refused( def test_a_board_is_followed_only_if_it_answers( - client: TestClient, store: Store, monkeypatch: pytest.MonkeyPatch) -> None: + client: TestClient, store: Store, queue: Any, monkeypatch: pytest.MonkeyPatch) -> None: import plugins.ats as ats monkeypatch.setattr(ats, "fetch_board", lambda target: ([{"source_id": "1"}], True)) @@ -933,6 +946,10 @@ def test_a_board_is_followed_only_if_it_answers( assert res.status_code == 201 assert res.json()["found"] == 1 and res.json()["source"]["target"] == "ashby:acme" assert store.company("acme") is not None + # Following a board queues a pull right after, so its roles are read by + # the one running or the next one rather than the schedule, up to six + # hours away. + assert queue.jobs.last("pull", states=_ANY_STATE) is not None # Following a company that exists goes through `set_board`: the same # board again is the same subscription, not a refusal or a second one. @@ -952,7 +969,7 @@ def test_a_board_is_followed_only_if_it_answers( def test_a_companys_board_is_set_replaced_and_detached( - client: TestClient, store: Store, monkeypatch: pytest.MonkeyPatch) -> None: + client: TestClient, store: Store, queue: Any, monkeypatch: pytest.MonkeyPatch) -> None: import plugins.ats as ats monkeypatch.setattr(ats, "fetch_board", lambda target: ([], True)) # between openings @@ -967,16 +984,24 @@ def test_a_companys_board_is_set_replaced_and_detached( [row] = client.get("/api/companies").json() assert (row["board"], row["subscription_id"], row["ats"], row["ats_slug"]) == ( "greenhouse", sub_id, "greenhouse", "acme") + # Setting a board queues a pull right after, so it is read by the one + # running or the next one rather than the schedule, up to six hours away. + assert queue.jobs.last("pull", states=_ANY_STATE) is not None res = client.post("/api/companies/acme/board", json={"board": "lever:acme"}) assert res.json()["closed"] == 1 and res.json()["source"]["id"] == sub_id + before_detach = queue.jobs.last("pull", states=_ANY_STATE) res = client.delete("/api/companies/acme/board") assert res.status_code == 200 and res.json() == {"closed": 0} [row] = client.get("/api/companies").json() assert (row["board"], row["subscription_id"], row["ats"]) == ("", None, "lever") assert client.get("/api/sources").json()["boards"] == [] assert client.delete("/api/companies/acme/board").status_code == 409 + # Detach reads nothing, so it queues no new pull. + after_detach = queue.jobs.last("pull", states=_ANY_STATE) + assert (before_detach.id if before_detach else None) == ( + after_detach.id if after_detach else None) assert client.post("/api/companies/nope/board", json={"board": "lever:acme"}).status_code == 404 @@ -985,6 +1010,34 @@ def test_a_companys_board_is_set_replaced_and_detached( assert bad.status_code == 422 and "host:slug" in bad.json()["detail"] +def test_the_pull_a_source_switch_queues_is_refused_quietly_while_one_runs( + app: FastAPI, store: Store) -> None: + """The four actions that queue a `pull` (switching a source on, following + or setting a board, saving a search) all go through the same job, whose + own refusal while another pull is in flight is not an error — see + `app.jobs.pull.claim_run` and `app.jobs.cascade.run_pull`. Exercised here + through the switch route as a stand-in for the other three, which queue + the identical `pull` job. No lifespan on this client, so no background + worker races the assertion: the queued job waits for the explicit + `drain`, and a refused pull starts no cascade — no fetch, no embed, + nothing else here needs mocking.""" + from app.jobs import pull + + client = TestClient(app) + board = store.add_subscription("ats", "greenhouse:acme", label="Acme", company="Acme", + board="greenhouse") + running = pull.claim_run(store) + assert running + + res = client.post(f"/api/sources/{board.id}/switch", json={"on": True}) + assert res.status_code == 200 + + app.state.queue.drain(timeout=30) + job = app.state.queue.jobs.last("pull", states=("done",)) + assert job is not None + assert job.result == {"skipped": pull.ALREADY_RUNNING, "running": running} + + # ------------------------------------------------- roles: text and summary def test_the_posting_text_is_fetched_once(