Commit Graph
153 Commits
Author SHA1 Message Date
WolfandClaude Opus 5.5 4e3c3f130e Give a search agent several prompts and score a role by its best one
Migration 14 moves each non-empty agents.intent into agent_prompts as one
prompt named after the agent, with its cached vector, and drops the three
intent columns. A role's fit is the maximum over the agent's prompts of the
calibrated cosine, and scores.prompt names the prompt that won. The agents
API takes prompts and still takes intent; the role endpoint serves the
agent's score and prompt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:19:57 +02:00
WolfandClaude Opus 5.5 0996b03f8f Edit an agent's prompts as rows and its gates as a Where block and rules
The card lists each prompt and says the gates in words. The form edits
prompts as name and text rows, and the gates as a Where block (countries
required, then one of countries, remote or places, and prefer hybrid) over
a row per remaining rule. A gate the Where block cannot say lands in a row,
so a load and a save lose nothing. The gates JSON box stays under Advanced,
read only, as a view of what Save sends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:17:15 +02:00
WolfandClaude Fable 5.1 aabf2b4f1b Name an unreadable host in the two tests that used Teamtailor for one
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:15:55 +02:00
Wolf d887a91af0 Merge branch 'wp3-teamtailor' 2026-09-29 16:15:22 +02:00
WolfandClaude Sonnet 5.5 b2ab2a270d Discover Teamtailor boards as readable, own-domain sites by their host
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:14:41 +02:00
WolfandClaude Sonnet 5.5 50edf0eeeb Read Teamtailor career sites from their jobs.rss feed, paged by offset
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:14:41 +02:00
WolfandClaude Fable 5.1 063463defd Pass Platsbanken's regions to the Sources page
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:13:07 +02:00
Wolf dc7b8747b0 Merge branch 'wp2-jobboards-ui' 2026-09-29 16:12:41 +02:00
WolfandClaude Sonnet 5.5 6c63139343 Open the job board form in place of its row, and add Platsbanken to it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:11:59 +02:00
Wolf 8aa4cfdcba Merge branch 'wp1-platsbanken' 2026-09-29 16:11:11 +02:00
WolfandClaude Sonnet 5.5 00e111961e Add Platsbanken as a third search provider
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:10:45 +02:00
WolfandClaude Fable 5.1 b6afe24eaa Plan several prompts per agent, Swedish sources and the Sources and gates UI
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:06:39 +02:00
WolfandClaude Fable 5.1 92466e2cd4 Drop the retry around the queue's dedup race, fixed upstream
`_score_after_change` tried three times because `Jobs.enqueue` could re-raise an IntegrityError when the live job finished between its insert and its lookup. The queue now does both in one BEGIN IMMEDIATE transaction (boilerplate a1c6f0d), so the workaround and its test go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:02:58 +02:00
WolfandClaude Fable 5.1 2e4624cbc4 Pull pipeline/ from boilerplate a1c6f0d: a dedup key is looked up and inserted in one transaction
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:02:58 +02:00
WolfandClaude Sonnet 471ace703c Switch the extract and summarise stages on the page, over config.yaml
A stored `stage:extract` / `stage:summarise` in the state table overrides
extract.enabled / summarise.enabled; with none, the file applies. One
function, stages.stage_on, decides, and both tasks' finders ask it.
GET/PUT /api/stages and `app stages` show and set it with who would answer;
Runs has a "What the model reads" block, and a search agent's card says when
a stage is off and links to the switch.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:54:53 +02:00
Wolf a629eda8de Merge branch 'r2-web'
# Conflicts:
#	app/static/roles.js
#	app/web/api.py
#	tests/test_web.py
2026-09-29 15:31:35 +02:00
WolfandClaude Sonnet 5dfaa393be Guard every feed-, board-, discovery- or user-supplied link with safeUrl
roles.js already checked a role's original_url is http(s) before
putting it in an href, so a board that wrote something other than a
URL could not make the reader navigate to it; sources.js (a search
sample's row.url), companies.js (careers_url) and applications.js
(application.url) put the same kind of outside data straight into a
link unchecked.

safeUrl moves to app.js, beside the app's other shared helpers, and
is exported the same way; a link whose safeUrl comes back empty is
now left out of the page (plain text) instead of getting an href of
"".

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:24:54 +02:00
WolfandClaude Sonnet 4fb9e801f5 Load the feed regardless of a slow or failed agents or companies fetch
The init block awaited /api/agents and /api/companies, both of which
only dim hidden companies, draw the affinity control and pick the
score order, before ever calling loadFeed; a slow or failed one left
the feed blank for good behind a silent catch that returned early.

The feed's own request now goes out alongside them rather than after.
Agents and companies fill in and redraw what they touch (the rail,
the feed's dimming, the reader's affinity slot) whenever they land,
reloading the feed if the real agent turns out to differ from what it
first loaded with. The feed's own fetch failing gets an empty state
with a retry instead of the same silent blank.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:23:48 +02:00
Wolf 562fa3ab09 Merge branch 'r2-backend'
# Conflicts:
#	app/storage/schema.py
2026-09-29 15:22:03 +02:00
WolfandClaude Sonnet 91596300e1 Serve a row projection from GET /api/roles instead of the whole role
The feed only ever shows a title, a few badges and a short excerpt,
but the list endpoint answered with every role's full row --
description, feed_text, raw and article_text included, about 5.5 KB
a role -- so a 1000-row restore was several megabytes of text a phone
never renders.

Store.roles_page selects only the fields a row draws (RoleRow), with
excerpt built server-side off a SUBSTR of summary or description, so
the big columns never cross into the response. The reader is
unaffected: it already fetched the whole role by uid when it opened
one; the reader pane's own reliance on the list row carrying the body
is removed too, so it no longer flashes a wrong "no description"
before that fetch lands.

Measured on a copy of the live database: a 60-row page went from
392 KiB to 39 KiB, and a 1000-row restore from 6.6 MB to 631 KiB.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:22:00 +02:00
WolfandClaude Opus b2c8c08f7f Judge an employer known only from a search; the Companies list says "from job boards"
Store.set_affinity creates a minimal company row (key, and the name its roles
give most often) for a key with roles and no row, so a Jobindex or Adzuna
employer can be hidden or liked; the reader draws the control for any role
with a company_key and reloads the company list after the first change.

/api/companies derives from_search: no ats subscription (detached included)
and no ats, careers_url or domain. companies.js labels it "from job boards"
and keeps it out of "Not readable yet".

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:19:55 +02:00
WolfandClaude Sonnet e6be593542 Ask before a destructive board or search switch; queue a pull after a reopening one
Switching a company's board or a job board search off, replacing a
company's board and detaching one all close open roles in a tap; each
now asks first, naming the count (a replace also names the old and the
new board), the same as removing a search and deleting an agent
already do.

Switching a target back on, following a board, setting a board and
saving a search used to show nothing until the schedule's next pull,
up to six hours later. Each now queues a pull job right in the API
route, so the CLI and any other caller of the same endpoint behave
like the page; the queued pull's own refusal while one is already
running is not surfaced as an error.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:18:16 +02:00
WolfandClaude Opus 91d15fcc7d A change made while score runs gets one follow-up score, and the page waits on it
score is solo, so a request during a run was deduped into the running job,
which had already read the old state. _score_after_change returns a queued
score when one has not started, else enqueues a plain score and, if that
dedupes to a running job, one follow-up with depends_on it, after="done" and
dedup_key score:after:<id>. POST /api/score answers with that job and
agents.js waits on it; the affinity route answers with score_job.

The flaky affinity test was pipeline's race, not the app's: Jobs.enqueue
re-raises the UNIQUE IntegrityError when the live job it collided with
finishes before the SELECT that should hand it back. The app retries.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:16:23 +02:00
WolfandClaude Opus 3abb44e10a Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:12:14 +02:00
Wolf bbb7474cef Merge branch 'r2-salary' 2026-09-29 15:09:19 +02:00
WolfandClaude Sonnet fa4e1a657f Keep a board's stated pay as salary_text, even when it does not normalise
Every ats parser now writes one human-readable pay line whether or not it
converts to yearly salary_min/salary_max/currency: hourly and daily rates,
Greenhouse's several pay zones, and a range with no currency all survive as
text instead of being dropped outright. Stored in roles.salary_text
(migration 12), replaced whole on every pull like title, so a board that
stops stating pay clears it rather than leaving a stale line. salary_state
is derived, not stored ('normalised', 'stated', or ''), and is what the
feed's new Pay filter and badge read; gates and scoring are unchanged, still
reading only the normalised columns.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:08:45 +02:00
WolfandClaude Opus 2c4b992c50 Country lookup: every ISO country, the widest unit wins, several European countries are EU
country_of folds with _place_fold (ł, đ, ø, æ map to ASCII; _fold keeps building
company_key unchanged), knows every ISO country by English name and European
ones by their own, and the cities among the 150 most frequent unresolved
places on the live data. Within one location a country beats a US state or
Canadian province, which beats a city. A state code that is also an ISO code
counts as the state only when the line names a US place. Several countries,
all European, are EU; otherwise ''. A code in parentheses is read when no
name matched.

Migration 12 recomputes country from place for roles extract has not read
and whose board sent no country in raw. On the live data: open roles with no
country 2,827 -> 1,183 of 16,841; 1,705 open roles change, 48 from a code.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:06:15 +02:00
WolfandClaude Opus b021838959 Calibrate fit and taste per embedding model instead of min-max per run
scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants
from CALIBRATION in app/config.py, keyed by model and replaceable under
embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19
(taste from synthetic votes). An uncalibrated model falls back to min-max
and logs a warning once per run.

The extract and summarise bars default to 0.55 and 0.65, about 300 and 35
roles for the starter agent on the measured data; stored limits are kept.
An agent without an intent is left out of the model-stage scope, and each
score run names such agents in one score.no_intent notice.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:59:34 +02:00
WolfandClaude Sonnet 92ff2bc069 Remove the MOVED redirects
MOVED and _moved answered 301 for twelve paths the settings cog moved
off of a few days ago. A browser caches a 301 indefinitely, so a path
reused later would be unreachable through no fault of that page, and
a one-person app has no outstanding bookmark to protect. Drop the
table, the routes, its export from app.web, and the tests that pinned
it; the old paths now answer plain 404. Keep the test that every
section of the Sources page exists, rewritten without MOVED.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 14:50:18 +02:00
Wolf 706b127bc0 Merge branch 'worktree-agent-afad54f12aef2cccf'
# Conflicts:
#	app/storage/schema.py
#	app/storage/store.py
#	tests/test_storage.py
2026-09-29 14:32:28 +02:00
WolfandClaude Sonnet 5 44ff7c0131 Unpack structured pay from five ATS hosts; fix the function classifier
Part A: Ashby (?includeCompensation=true), Greenhouse (pay_transparency=true,
already free on the list request), Lever (salaryRange), Recruitee (salary)
and Personio (salaryInformation) all give structured pay; Workable and
SmartRecruiters do not (checked live). Every host converts to a yearly
salary_min/salary_max/currency, multiplying a monthly figure by 12 and
skipping an hourly or daily one rather than annualising a number nobody
stated. A range with no currency is ignored, and Greenhouse's own multi-zone
ranges are skipped outright since nothing says which zone the role is at.

Part B: classify() matched substrings, so " it" matched inside "Italy",
filing European sales roles as engineering; it now matches on word
boundaries (full boundary for the handful of short tokens that are
themselves the start of an unrelated word, prefix matching kept for the
stems). Title is checked before department (a hand-checked sample found
title-first right about twice as often), nothing matching returns '' rather
than "other", and a modest set of German/Dutch/Danish/Norwegian/Swedish/
French keywords covers the seed's other languages.

function is not a board field, so it gets the opposite rule from every
other raw column: extract's answer overwrites the classifier's guess
outright, and once extracted_at is set no later pull may put the guess
back. Migration 9 recomputes function for every not-yet-extracted role from
its stored title and department.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-29 14:29:12 +02:00
WolfandClaude Fable 5.1 04da463e9f A replaced or detached board takes its failure record with it
The pull records a board that fails on every fetch under ats:failing:<target>, and set_board can now move a company to another board. The old target's record would have outlived the board and counted in `app companies check` for good.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 14:17:20 +02:00
Wolf 43773efc53 Merge branch 'worktree-agent-a3bd292551395dda0'
# Conflicts:
#	app/storage/schema.py
2026-09-29 14:16:35 +02:00
Wolf c6246af994 Merge branch 'worktree-agent-ac3c9345edd29a1c6'
# Conflicts:
#	app/cli.py
#	app/jobs/sources.py
#	app/static/app.css
#	app/static/companies.js
#	app/web/api.py
#	tests/test_web.py
2026-09-29 14:15:10 +02:00
WolfandClaude Sonnet 75eca1c914 Date a role by posted_at, not just first_seen
recency and the newest feed both counted from first_seen, when the app
itself first stored a role, so a newly added board's whole backlog scored
as freshly posted for two weeks: 10,223 of 15,415 board roles were posted
more than 30 days before they were first seen (average gap in days: Ashby
78, Greenhouse 143, Lever 226, Personio 333).

Add roles.posted_at (migration 9): the earlier of first_seen and a trusted
host's published_at, where TRUSTED_POSTED_HOSTS (app/models.py) records
which of the ats and search hosts report a creation date rather than one
that moves on an edit, checked against live boards and the live database.
A future or pre-2015 date is not trusted either. upsert_role sets it on
insert and keeps the earliest on update, so an edit on the board can only
move it earlier, never later. recency, the feed's newest order, the "new
since" days filter and _top_scope's tie-break now read posted_at;
first_seen is unchanged and still what the UI shows as when the app found
a role. The row's age in roles.js now shows the posted date instead.

A partial index (posted_at DESC, uid) WHERE closed_at IS NULL serves the
newest feed; sqlite3's bundled build has no STAT4, so its planner
otherwise sorts every open role instead of using it, measured 30x slower
on the live database, so Store.roles() forces it for that query shape.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 14:13:06 +02:00
WolfandClaude Opus b43fc54565 One pull at a time: the run is claimed in the database and its marker rides in each batch
A second pull (schedule, Run button, `app job pull` in another process) could
start while one was fetching and overwrite the stored run marker, so the first
run's boards failed their stamp check or a settle closed on the other run's id
lists. `claim_run` sets `pull:run` in one conditional statement, only while it
equals `pull:settled` or is older than an hour; a refused `pull` job queues no
stage and says so in its result and a notice, and the CLI exits 1. Batches
carry the marker in their payload, the plugins stamp with it, and the settle
reads it off its own batches' payloads before releasing the run.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:12:17 +02:00
Wolf 0ea6c10d80 Merge branch 'worktree-agent-a7e6bf7e67aa710c3' 2026-09-29 14:05:35 +02:00
WolfandClaude Opus 5.5 3b209d628a Every way a company gets a board is set_board, and an unreadable host keeps its slug
`app.jobs.sources.set_board` attaches a board to an existing company,
replaces the one it has (closing the old board's open roles) or creates
the company, with row and subscription changed in one transaction by
`Store.set_company_board`. A board that lists zero roles is kept. `add`,
`discover --add`, the seed importer (moved out of app/cli.py) and the new
`POST|DELETE /api/companies/{key}/board` all go through it; a seed re-run
attaches a board to a row that never had one. A company's subscription is
rewritten in place, and detaching hides it.

`companies.ats_slug` (migration 9) keeps the slug of a host the app
cannot read yet, and discovery returns such a host before probing slugs.
The Companies rows get Set board / Change, the host select is rendered
from `plugins.ats.HOSTS`, and a row with nothing found reads "board not
found" and is in the "Not readable yet" filter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 14:04:52 +02:00
WolfandClaude Opus c7da12b93d Paged boards are read to their stated total, and a board failing for 21 days closes
Workable and SmartRecruiters page their lists, and both parsers read page one
only, so the settle closed every role past it (Delivery Hero: 980 stated, 100
open). `_paged` follows nextPage and offset, and the plugin stamps
`ats:complete` only when the distinct ids read equal the stated total; a 429
after page one keeps what was read and closes nothing. Adzuna completes on its
`count`, Jobindex counts raw items and raises on an answer with no channel.

A board whose fetch raises is recorded under `ats:failing:<target>` with the
first failure's time and the latest reason; after STALE_DAYS the settle closes
its roles, and its next answer reopens them. `/api/companies`, the Companies
list and `app companies check` show it.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:02:49 +02:00
Wolf 0b880c20fc Merge branch 'worktree-agent-a920063505aaba401' 2026-09-29 14:00:52 +02:00
WolfandClaude Sonnet b560cdce2a Switch the embed model to multilingual-e5-small for the European corpus
Measured on the live roles against the "Copenhagen engineering" agent's
intent: the English-only all-MiniLM-L6-v2 separates engineering from
non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted
to Danish-country roles (a Danish engineering role beats the rest's
non-engineering median only 68% of the time). multilingual-e5-small reaches
0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time.
paraphrase-multilingual-MiniLM-L12-v2 measured worse than the English-only
baseline (AUC 0.60) and was dropped.

e5 needs "query: "/"passage: " prefixes; app/jobs/embed.py adds them by
matching the model name so a model change can't leave a stale prefix
setting behind. Added tests for the prefixing and for the two invariants
the model change relies on: embed drops the old model's vectors before
re-embedding, and an agent's cached intent vector is keyed by model so a
model change is a cache miss, never a stale-dimension read.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 13:59:47 +02:00
WolfandClaude Opus 5.5 65ae7d5000 Run configurations call into the claudebox distro, as in boilerplate and news
The Python configurations had no interpreter, and a Windows one cannot open
the Linux .venv or share roledesk.db with the distro. Each is now a Shell
Script configuration running uv in /home/claude/cboxcode/jobboard. serve
reloads, and worker is new.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:53:35 +02:00
WolfandClaude Fable 5.1 0558ef19ed Pull pipeline/ from boilerplate e397005: the queue chip counts down a retry
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 19:43:21 +02:00
Wolf 074a41c6cf Merge branch 'main' into isolate-batches 2026-09-28 19:42:51 +02:00
WolfandClaude Fable 5.1 b73a0fc2cb The model batches are isolated with a deadline, so a Kill or a hang ends them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 19:42:28 +02:00
Wolf 6779dda888 Merge branch 'gates-any'
# Conflicts:
#	README.md
#	app/storage/store.py
2026-09-28 19:30:27 +02:00
WolfandClaude Fable 5.1 fb83ae5099 Gates reject on evidence only, any gates are one group, remote is read off the location line
A require gate on an empty column gated the role ("country unknown"), and extract, the stage that fills the column, is bounded to the roles that rank, so 2,281 of 15,556 open roles could never come back: every board that writes "Remote", "Hybrid" or "Distributed" as its location. An empty column now neither passes nor fails a gate; the role ranks, extract reads it, the rescore decides.

`any` is a third gate mode: the any gates are one group a role must match one of, so a ranking can say "in Denmark, or remote", which require alone could not. The group is gated only when some member had evidence against it.

`remote_of(place)` fills `remote` from the location line the way `country_of` fills `country`, and the startup backfill does both, since Greenhouse writes "Hybrid" as the location and gives no flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 19:30:06 +02:00
WolfandClaude Fable 5.1 c63f348dc9 Merge main: the Runs cleanup meets the queue's own idle text
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 16:24:24 +02:00
WolfandClaude Fable 5.1 3a240e51df Pull pipeline/ from boilerplate a80ad8a: cadence picker, queue badge, label refresh
The schedule cadence is a picker, the add row is behind a button and off
here, a running-jobs chip with Kill and Stop run sits in every header, and
a declared schedule's label is refreshed at startup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 16:22:46 +02:00
WolfandClaude Fable 5.1 8532e4e739 Runs is the queue's elements and a Problems log; a Kill lands between boards
Run one stage goes: the cascade is the unit of work and app job keeps the
per-stage entry points. The pull button is disabled while a pull runs and
the lede no longer hardcodes the cadence. Messages folds into Runs as
Problems. pull_batch checks for a cancel before every board, one plugin per
target, and both plugins now close their HTTP client.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 16:10:38 +02:00