Commit Graph
67 Commits
Author SHA1 Message Date
WolfandClaude Opus 5.5 a91032b60f Close a role of a swept board only when two finished sweeps in a row missed it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:23:57 +02:00
WolfandClaude Opus 5.5 322db49773 Title levels on the ladder: principal, senior manager, distinguished, lead split (WP A)
from_title matches a title twice, for the highest people level and the
highest individual grade. A people level wins on its own rung; a grade
above that rung lifts it one rung (Principal Engineering Manager is a
senior manager). Senior Staff is principal, Senior Principal, Distinguished
and Technical Fellow are distinguished; Associate Director, AVP and Senior
Manager are senior managers. lead is now a team lead only (Team Lead, Shift
Lead, Supervisor, Teamleiter); Tech Lead, Lead X and X Lead(er) are staff.
A slash names two levels, so Senior/Staff is staff.

- Jobylon "lead" maps to staff; Recruitee maps student_college, manager,
  senior_manager and executive.
- extract's prompt describes the ladder and its two tracks; enum from LEVELS.
- prep VERSION 2, so the next prep maps every open role again.
- Migration 25 clears seniority_model where it is staff, lead, manager or
  director, so extract reads those roles again, and sets roles.seniority to
  the first filled of model, stated and title.
- The coverage test of tests/test_seniority.py passes; its xfail is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:11:26 +02:00
WolfandClaude Opus 5.5 e52857dea5 Split the All tag into two company overrides: All countries and All roles (WP L)
Fetching has two filters, and a company now overrides each on its own.
All countries (companies.all_countries) is the old All roles tag: its big
boards are read with the agents' wide reach. All roles (companies.all_roles,
new meaning) drops the agents' fetch terms and any later role filter.

- Migration 24 copies the old tag to all_countries and keeps all_roles for
  those companies, and moves a board's {"terms": false} to its company.
- The runner no longer reads {"terms": false}; the plan leaves the terms out
  for a company set to All roles.
- POST /api/companies/{key}/all-countries and /all-roles, each queueing a
  pull when turned on; CompanyOut carries both.
- Two toggles replace All on the Companies row, the reader and the Roles
  strip; the reach line states countries and role types in plain words;
  the filter select gets both options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 12:50:45 +02:00
WolfandClaude Opus 5.5 0fd45b333a Fetch terms narrow what big boards read with the wide reach send (WP F2)
An agent has fetch_terms (migration 23, API, a chip field in the agent
editor). The terms of a pull are the union over the enabled agents, none
when one of them has none. They go only to wide reads: a company tagged All
roles, or a reach ask_for drops as too many countries for a one host; a
board's {"terms": false} switches them off. SmartRecruiters q,
SuccessFactors q, Workday searchText, Phenom keywords, Eightfold query and
Oracle keyword declare query="one"; the runner makes one series per term
times the country series, drops the places first over MAX_SERIES, then the
terms. A term read is narrowed: it writes no ats:size, closes nothing and
prunes no per-role state, and the 10% guard holds it to its own
fingerprint. The Companies reach line counts the terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:59:27 +02:00
WolfandClaude Opus 5.5 d04ceae7e0 Prep stage: host words stored raw, mapped centrally in app/prep after every pull
upsert_role stores the host's own words (roles.*_host), the place texts
unresolved when they change, and the company key unresolved on insert, and
marks the role prep_due. app.prep maps seniority, employment, remote and
country through the host tables in app/prep/hosts, resolves places and
company aliases, lists unknown words in `unmapped`, remaps every open role
when the tables' fingerprint changes, and runs rebuild_dupes, which moves
out of the pull's settle. Migration 22 adds the columns and the table and
fills the host words from raw. Cascade: pull, prep, embed, score, enrich,
prep, embed, score, extract, score, summarise. `app prep [unmapped]
[--remap]`, /api/prep/unmapped and a fold on the Sources page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:19:22 +02:00
WolfandClaude Opus 5.5 5d5b7e7d3f Tag a company All roles to read its big boards with the agents' wide reach; favourite scores only
Migration 21 adds companies.all_roles, tags every favourite of any agent and each company whose board had params {"reach": "all"} (logged per board, the param removed). reach.plan reads a tagged company's boards with the union of the enabled agents' require reach and the base reach; favourite no longer widens the fetch. POST /api/companies/{key}/all-roles sets it and queues a pull when on. The Companies list, the reader and the company strip draw the toggle beside the affinity buttons, the list filters on it, and the board's reach editor drops its All roles option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:44:41 +02:00
WolfandClaude Opus 5.5 903b69b6d3 Share the readers' place, table-lookup and country helpers, unify the employment vocabulary, and list the enrich switch with the stages
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:13:24 +02:00
WolfandClaude Opus 5.5 ca80219bbb Merge branch 'wp-f-reach'
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:04:26 +02:00
WolfandClaude Opus 5.5 dafca5435e Show and set each board's reach on the Companies list, list the readers' declarations on Sources, and document filtering at the source
A company row says what a pull asks its board for; its editor sets the board's own reach (the agents', all roles, or countries and remote) through /api/sources/{id}/reach. /api/readers lists each reader's listing, text, filters, places and robots decision, folded under the Follow form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:03:41 +02:00
WolfandClaude Opus 5.5 3462a4abe7 Read big boards within the agents' reach: countries and remote from the enabled agents, Europe for favourites, filtered at the source by six hosts
app/jobs/reach.py derives each board's reach once per pull and the pull hands it to the runner through the batch payload. SmartRecruiters, Eightfold and SuccessFactors filter one country a request, Phenom a list, Workday and Oracle by host ids from facets. Remote beside countries is a series of its own, since two facets intersect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:55:18 +02:00
WolfandClaude Opus 5.5 38a80d5b55 Merge branch 'wp-g-enrich'
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:53:57 +02:00
WolfandClaude Opus 5.5 77b45a4b0f Read role pages in an enrich stage after scoring, best roles first
The pull reads listings only. `enrich` (app/jobs/enrich.py) takes the open
roles without text that an enabled agent ranks ungated in its best
`enrich.top` at `enrich.min_score` or above, best score first, on boards
whose reader has `detail`, within `enrich.per_board` and `enrich.per_run`,
one batch per board. `plugins.ats.read_board_details` reads them and keeps
the per-board state; a read that gave no text is read again after
`enrich.retry_days` (finding 18). Filled roles are upserted, which drops
their vectors, and embed and score follow; `cascade.step` passes over both
when nothing was filled.

The pull cascade is pull, embed, score, enrich, embed, score, extract, score,
summarise. Saving an agent and changing a company's affinity queue the
`enrich` job (enrich, embed, score) behind their rescore; a vote does not.
The stage has a switch, `stage:enrich`, over `enrich.enabled`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:53:29 +02:00
WolfandClaude Opus 5.5 7ca1f02e26 Extract as the afterburner for the level: model verdict with stated and title hints, best rank first, agreement per host
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:43:52 +02:00
WolfandClaude Opus 5.5 5f0f4ad5b7 Merge branch 'wp-e1-places'
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:33:07 +02:00
WolfandClaude Opus 5.5 bbd7522163 Places as a gazetteer, role_places, teaser and three seniority sources
Migration 19 adds places, place_aliases, place_includes (seeded from the
dictionaries in app/models.py and app/places.py, Europe as a continent over
the European countries) and role_places, one row per place of a role,
resolved by app.places.Gazetteer, NULL when unknown. upsert_role writes them
from raw["places"], else the place line cut at ";"; the migration builds them
from each role's place column.

The Place filter and the country and place gates are joins over the tree:
country:DK is any place under Denmark, region:skane any town in Skane, a
country gate is met by any place in its countries, and the EU fallback of
the word matcher is no longer used. A place: token the gazetteer does not
know still matches whole words. Store gains gazetteer, role_places,
unresolved_places, add_place_alias, add_place and relink_places.

roles.teaser is kept-filled and embedded while there is no description; a
vector is dropped when a better text arrives. seniority_model, _stated and
_title sit behind roles.seniority, which stays the first filled of them.

TRUSTED_POSTED_HOSTS gives way to trusted_posted(host), which reads
READS.fields.posted and falls back to the old set for undeclared readers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:31:45 +02:00
WolfandClaude Opus 5.5 902089a5e2 Runner: readers read from their declaration, ats and search thin over it
plugins/readers/_runner.py takes the guarded fetch, the listing (whole,
paged, one series per filter value unioned by id), completeness from READS,
the sweep of a board over the page cap across pulls, the 10% guard on a
filtered or empty="suspect" read, facets kept a day, the detail reads, the
raw an Item carries and the failure bookkeeping out of plugins/ats.py.
Ask comes from a subscription's params (ask_for, open for the reach), with
a declared key resolved into Ask.params["key"] and refused when missing.

The runtime robots.txt check goes (decided at build time, READS.robots).
plugins/search.py reads the providers of readers.PROVIDERS through the
runner and applies the saved pages param. A filtered board's close keeps
roles with a vote or an application. DETAIL_FIELDS gain places, teaser,
country and employment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:28:33 +02:00
WolfandClaude Opus 5.5 b78c9dfc67 Search providers as reader modules: jobindex, adzuna, platsbanken
Each declares READS with scope=search, keeps every place and hands the excerpt over as teaser. search_params and PROVIDERS follow the declarations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:08:25 +02:00
WolfandClaude Opus 5.5 73dec9028e Rename plugins/atshosts to plugins/readers
Every import, test and live doc follows the new path; behaviour is unchanged.
tests/test_atshosts.py becomes tests/test_readers.py. plan-readers.md and
review-ingest.md keep the old name, since they describe the state before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:48:30 +02:00
WolfandClaude Opus 5.5 82b7902eff Regroup copies when a board or search is switched off, and fold the duplicates plan into the decisions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:11:33 +02:00
WolfandClaude Opus 5.5 545c932f7b Store copies of one posting as roles.dup_of and leave them out of the feed, the counts and the model stages
Migration 18 adds roles.dup_of with a partial index and runs the first rebuild. Store.rebuild_dupes feeds app.dupes.assign the open roles of every (company_key, title_key) with more than one, rewrites dup_of where it changed, clears it on closed roles, and hands read_at and votes to a changed group's kept role. The pull's settle and `app dupes` run it. RoleRow.copies counts a row's open copies; Role.copies lists them and Role.dup_of names a copy's kept role.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:09:20 +02:00
WolfandClaude Fable 5.1 bcdb4cf06a Say the favourite's new rule where the affinity levels are listed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 21:54:19 +02:00
WolfandClaude 3ab8035d0d Gate a favourite by the require gates and drop its 0.9 floor
A favourite company skips the any group only; a require gate it fails
gates it with the usual reason. Its score is the weighted sum with the
affinity term at 1, and weights that sum to 0 give "no weights" as for
every other company.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-29 21:52:26 +02:00
Wolf a7bb6c20ae Merge branch 'main' into wp0-hosts 2026-09-29 19:49:00 +02:00
WolfandClaude Opus 5.5 7f04c10e9e Forget a replaced or detached board's robots.txt verdict and detail lists with its failure record
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:43:24 +02:00
WolfandClaude Opus 5.5 75a389d725 Keep a discovered slug's case for a host with KEEP_CASE, and check only group 1 against the hosts' own path words
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:37:26 +02:00
WolfandClaude Fable 5.1 2c3bce99e3 Clear a company's by-hand reason when a board is attached to it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 19:27:55 +02:00
WolfandClaude Sonnet 5.5 161e89c955 Sort the by-hand list by the Watchlist's careers_opened_at and print the watch rating
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 19:25:46 +02:00
WolfandClaude Opus 5.5 a30b1ff09f Take discovery's readable hosts, patterns and probes from the host modules, and find Lever boards on its EU instance
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:07:34 +02:00
WolfandClaude Sonnet 5.5 b3cb93d5aa Keep a typed careers link, seed the by-hand reason and add the by-hand commands to app companies
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 17:02:19 +02:00
Wolf 7939b99a5c Merge branch 'wp4-prompts' 2026-09-29 16:22:26 +02:00
WolfandClaude Opus 5.5 4e3c3f130e Give a search agent several prompts and score a role by its best one
Migration 14 moves each non-empty agents.intent into agent_prompts as one
prompt named after the agent, with its cached vector, and drops the three
intent columns. A role's fit is the maximum over the agent's prompts of the
calibrated cosine, and scores.prompt names the prompt that won. The agents
API takes prompts and still takes intent; the role endpoint serves the
agent's score and prompt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:19:57 +02:00
Wolf d887a91af0 Merge branch 'wp3-teamtailor' 2026-09-29 16:15:22 +02:00
WolfandClaude Sonnet 5.5 b2ab2a270d Discover Teamtailor boards as readable, own-domain sites by their host
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:14:41 +02:00
WolfandClaude Sonnet 5.5 00e111961e Add Platsbanken as a third search provider
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:10:45 +02:00
WolfandClaude Sonnet 471ace703c Switch the extract and summarise stages on the page, over config.yaml
A stored `stage:extract` / `stage:summarise` in the state table overrides
extract.enabled / summarise.enabled; with none, the file applies. One
function, stages.stage_on, decides, and both tasks' finders ask it.
GET/PUT /api/stages and `app stages` show and set it with who would answer;
Runs has a "What the model reads" block, and a search agent's card says when
a stage is off and links to the switch.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:54:53 +02:00
Wolf 562fa3ab09 Merge branch 'r2-backend'
# Conflicts:
#	app/storage/schema.py
2026-09-29 15:22:03 +02:00
WolfandClaude Opus 3abb44e10a Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:12:14 +02:00
WolfandClaude Opus b021838959 Calibrate fit and taste per embedding model instead of min-max per run
scale() maps a cosine to clip((cos - lo) / (hi - lo), 0, 1) with constants
from CALIBRATION in app/config.py, keyed by model and replaceable under
embed.calibration. multilingual-e5-small: fit 0.75..0.88, taste -0.14..0.19
(taste from synthetic votes). An uncalibrated model falls back to min-max
and logs a warning once per run.

The extract and summarise bars default to 0.55 and 0.65, about 300 and 35
roles for the starter agent on the measured data; stored limits are kept.
An agent without an intent is left out of the model-stage scope, and each
score run names such agents in one score.no_intent notice.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:59:34 +02:00
WolfandClaude Fable 5.1 04da463e9f A replaced or detached board takes its failure record with it
The pull records a board that fails on every fetch under ats:failing:<target>, and set_board can now move a company to another board. The old target's record would have outlived the board and counted in `app companies check` for good.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 14:17:20 +02:00
Wolf 43773efc53 Merge branch 'worktree-agent-a3bd292551395dda0'
# Conflicts:
#	app/storage/schema.py
2026-09-29 14:16:35 +02:00
Wolf c6246af994 Merge branch 'worktree-agent-ac3c9345edd29a1c6'
# Conflicts:
#	app/cli.py
#	app/jobs/sources.py
#	app/static/app.css
#	app/static/companies.js
#	app/web/api.py
#	tests/test_web.py
2026-09-29 14:15:10 +02:00
WolfandClaude Sonnet 75eca1c914 Date a role by posted_at, not just first_seen
recency and the newest feed both counted from first_seen, when the app
itself first stored a role, so a newly added board's whole backlog scored
as freshly posted for two weeks: 10,223 of 15,415 board roles were posted
more than 30 days before they were first seen (average gap in days: Ashby
78, Greenhouse 143, Lever 226, Personio 333).

Add roles.posted_at (migration 9): the earlier of first_seen and a trusted
host's published_at, where TRUSTED_POSTED_HOSTS (app/models.py) records
which of the ats and search hosts report a creation date rather than one
that moves on an edit, checked against live boards and the live database.
A future or pre-2015 date is not trusted either. upsert_role sets it on
insert and keeps the earliest on update, so an edit on the board can only
move it earlier, never later. recency, the feed's newest order, the "new
since" days filter and _top_scope's tie-break now read posted_at;
first_seen is unchanged and still what the UI shows as when the app found
a role. The row's age in roles.js now shows the posted date instead.

A partial index (posted_at DESC, uid) WHERE closed_at IS NULL serves the
newest feed; sqlite3's bundled build has no STAT4, so its planner
otherwise sorts every open role instead of using it, measured 30x slower
on the live database, so Store.roles() forces it for that query shape.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 14:13:06 +02:00
WolfandClaude Opus b43fc54565 One pull at a time: the run is claimed in the database and its marker rides in each batch
A second pull (schedule, Run button, `app job pull` in another process) could
start while one was fetching and overwrite the stored run marker, so the first
run's boards failed their stamp check or a settle closed on the other run's id
lists. `claim_run` sets `pull:run` in one conditional statement, only while it
equals `pull:settled` or is older than an hour; a refused `pull` job queues no
stage and says so in its result and a notice, and the CLI exits 1. Batches
carry the marker in their payload, the plugins stamp with it, and the settle
reads it off its own batches' payloads before releasing the run.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:12:17 +02:00
Wolf 0ea6c10d80 Merge branch 'worktree-agent-a7e6bf7e67aa710c3' 2026-09-29 14:05:35 +02:00
WolfandClaude Opus 5.5 3b209d628a Every way a company gets a board is set_board, and an unreadable host keeps its slug
`app.jobs.sources.set_board` attaches a board to an existing company,
replaces the one it has (closing the old board's open roles) or creates
the company, with row and subscription changed in one transaction by
`Store.set_company_board`. A board that lists zero roles is kept. `add`,
`discover --add`, the seed importer (moved out of app/cli.py) and the new
`POST|DELETE /api/companies/{key}/board` all go through it; a seed re-run
attaches a board to a row that never had one. A company's subscription is
rewritten in place, and detaching hides it.

`companies.ats_slug` (migration 9) keeps the slug of a host the app
cannot read yet, and discovery returns such a host before probing slugs.
The Companies rows get Set board / Change, the host select is rendered
from `plugins.ats.HOSTS`, and a row with nothing found reads "board not
found" and is in the "Not readable yet" filter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 14:04:52 +02:00
WolfandClaude Opus c7da12b93d Paged boards are read to their stated total, and a board failing for 21 days closes
Workable and SmartRecruiters page their lists, and both parsers read page one
only, so the settle closed every role past it (Delivery Hero: 980 stated, 100
open). `_paged` follows nextPage and offset, and the plugin stamps
`ats:complete` only when the distinct ids read equal the stated total; a 429
after page one keeps what was read and closes nothing. Adzuna completes on its
`count`, Jobindex counts raw items and raises on an answer with no channel.

A board whose fetch raises is recorded under `ats:failing:<target>` with the
first failure's time and the latest reason; after STALE_DAYS the settle closes
its roles, and its next answer reopens them. `/api/companies`, the Companies
list and `app companies check` show it.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:02:49 +02:00
WolfandClaude Sonnet b560cdce2a Switch the embed model to multilingual-e5-small for the European corpus
Measured on the live roles against the "Copenhagen engineering" agent's
intent: the English-only all-MiniLM-L6-v2 separates engineering from
non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted
to Danish-country roles (a Danish engineering role beats the rest's
non-engineering median only 68% of the time). multilingual-e5-small reaches
0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time.
paraphrase-multilingual-MiniLM-L12-v2 measured worse than the English-only
baseline (AUC 0.60) and was dropped.

e5 needs "query: "/"passage: " prefixes; app/jobs/embed.py adds them by
matching the model name so a model change can't leave a stale prefix
setting behind. Added tests for the prefixing and for the two invariants
the model change relies on: embed drops the old model's vectors before
re-embedding, and an agent's cached intent vector is keyed by model so a
model change is a cache miss, never a stale-dimension read.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 13:59:47 +02:00
Wolf 074a41c6cf Merge branch 'main' into isolate-batches 2026-09-28 19:42:51 +02:00
WolfandClaude Fable 5.1 b73a0fc2cb The model batches are isolated with a deadline, so a Kill or a hang ends them
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 19:42:28 +02:00
Wolf 6779dda888 Merge branch 'gates-any'
# Conflicts:
#	README.md
#	app/storage/store.py
2026-09-28 19:30:27 +02:00