Commit Graph
95 Commits
Author SHA1 Message Date
WolfandClaude Opus 5.5 0fd45b333a Fetch terms narrow what big boards read with the wide reach send (WP F2)
An agent has fetch_terms (migration 23, API, a chip field in the agent
editor). The terms of a pull are the union over the enabled agents, none
when one of them has none. They go only to wide reads: a company tagged All
roles, or a reach ask_for drops as too many countries for a one host; a
board's {"terms": false} switches them off. SmartRecruiters q,
SuccessFactors q, Workday searchText, Phenom keywords, Eightfold query and
Oracle keyword declare query="one"; the runner makes one series per term
times the country series, drops the places first over MAX_SERIES, then the
terms. A term read is narrowed: it writes no ats:size, closes nothing and
prunes no per-role state, and the 10% guard holds it to its own
fingerprint. The Companies reach line counts the terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:59:27 +02:00
WolfandClaude Opus 5.5 a66aac95cf Merge branch 'wp-k-all-roles'
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:45:03 +02:00
WolfandClaude Opus 5.5 5d5b7e7d3f Tag a company All roles to read its big boards with the agents' wide reach; favourite scores only
Migration 21 adds companies.all_roles, tags every favourite of any agent and each company whose board had params {"reach": "all"} (logged per board, the param removed). reach.plan reads a tagged company's boards with the union of the enabled agents' require reach and the base reach; favourite no longer widens the fetch. POST /api/companies/{key}/all-roles sets it and queues a pull when on. The Companies list, the reader and the company strip draw the toggle beside the affinity buttons, the list filters on it, and the board's reach editor drops its All roles option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:44:41 +02:00
WolfandClaude Opus 5.5 5fa05d5f73 Plan prep: narrow the terms package to what the survey found
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:40:40 +02:00
WolfandClaude Opus 5.5 535eb86f3e Survey which source-side filters the hosts take, and how towns reach the fetch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:40:06 +02:00
WolfandClaude Opus 5.5 11ba94cfd7 Plan: readers only scrape, a prep stage maps centrally, agents' terms narrow big boards
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:32:01 +02:00
WolfandClaude Opus 5.5 8d9a059e24 Read Adzuna's keyed API although its robots.txt disallows /: the owner's decision
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:14:42 +02:00
WolfandClaude Opus 5.5 b567144612 Fold the readers plan into the decisions and delete it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:06:50 +02:00
WolfandClaude Opus 5.5 779b136757 Plan readers: listing run and enrich run, and what triggers the enrich run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:16:29 +02:00
WolfandClaude Opus 5.5 4c0da58da7 Plan readers: seniority as stated, title and model values
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:57:30 +02:00
WolfandClaude Opus 5.5 73dec9028e Rename plugins/atshosts to plugins/readers
Every import, test and live doc follows the new path; behaviour is unchanged.
tests/test_atshosts.py becomes tests/test_readers.py. plan-readers.md and
review-ingest.md keep the old name, since they describe the state before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:48:30 +02:00
WolfandClaude Opus 5.5 af337ef558 Plan readers: owner decisions on reach, robots.txt at build time, places and companies as symbols, teaser kept
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:46:29 +02:00
WolfandClaude Opus 5.5 10fdd60988 Plan readers as plugins: declared traits, one API, one runner, one row per posting with all its places
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 08:34:49 +02:00
WolfandClaude Opus 5.5 667e1d35f4 Review the ingest path: text, places, dates, coverage and filtering at the source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:43:19 +02:00
WolfandClaude Opus 5.5 82b7902eff Regroup copies when a board or search is switched off, and fold the duplicates plan into the decisions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:11:33 +02:00
WolfandClaude Fable 5.1 55e353e764 Drop the open items that were done and say what seniority still lacks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 22:54:31 +02:00
WolfandClaude Fable 5.1 a9ecd1b595 Fold the feed filters plan into the decisions and name what it leaves open
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 22:33:24 +02:00
WolfandClaude Fable 5.1 30b36df2fb Describe every source in plugins/README.md: how it is read, what it delivers and its limits
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 22:23:43 +02:00
WolfandClaude Opus 5.5 b09a2f6b33 Group copies only when they all match, compare towns through places.towns, and record the rule in the plan
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:12:39 +02:00
WolfandClaude Opus 5.5 de156ed90f Plan one role per posting, across hosts and within one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:04:08 +02:00
WolfandClaude Fable 5.1 af0c4ae5b3 Match places with a registered function in the plan, since LIKE took 400 ms a region
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 22:02:07 +02:00
WolfandClaude Fable 5.1 64f367cc18 Plan starred companies, filters in history, and place and company chips
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 21:48:24 +02:00
WolfandClaude Fable 5.1 a39bd36cf9 Say what the seed is brought in line with, in the wave 2 plan
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 20:24:21 +02:00
WolfandClaude Fable 5.1 7785e62b0d Fold the hosts plan into the decisions, and plan the readers that are left
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 20:24:01 +02:00
WolfandClaude Fable 5.1 bb8a9a3b8f Read robots.txt without case for a host that serves a path in any case, as Workday does
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 20:07:14 +02:00
WolfandClaude Fable 5.1 6ae5716c62 Move the sitemap reader to wave 2 of the hosts plan and record what the planner has done
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 19:58:36 +02:00
Wolf a7bb6c20ae Merge branch 'main' into wp0-hosts 2026-09-29 19:49:00 +02:00
WolfandClaude Opus 5.5 2ff5046993 State the final host contract in the package docstring and the plan, with ranks, case, detail fields and robots.txt
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:47:09 +02:00
Wolf e51bdb269f Merge branch 'wp1-byhand-api' 2026-09-29 19:28:05 +02:00
WolfandClaude Sonnet 5.5 1ebfab1b99 Bring the by-hand plan in line with the Watchlist tab and migration 16
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 19:26:08 +02:00
WolfandClaude Fable 5.1 0d11ce0278 Record the Watchlist tab in the decisions
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 19:23:20 +02:00
WolfandClaude Fable 5.1 a584b88ead Count the Teamtailor boards attached today in the hosts plan
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:55:32 +02:00
WolfandClaude Fable 5.1 0a1d3c67c7 Plan a module per host, six packages of new hosts and the by-hand list
The survey of the 155 companies without a board is in docs/plan-hosts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:52:40 +02:00
WolfandClaude Fable 5.1 6f273cfdd6 Record the Where block as one question in the decisions
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:46:57 +02:00
WolfandClaude Fable 5.1 4b722744a7 Show Viewing as on Roles as a select, also with one agent
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:40:33 +02:00
WolfandClaude Fable 5.1 6124b00062 Fold the prompts and sources plan into the decisions and delete it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:28:59 +02:00
WolfandClaude Fable 5.1 b6afe24eaa Plan several prompts per agent, Swedish sources and the Sources and gates UI
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:06:39 +02:00
WolfandClaude Fable 5.1 92466e2cd4 Drop the retry around the queue's dedup race, fixed upstream
`_score_after_change` tried three times because `Jobs.enqueue` could re-raise an IntegrityError when the live job finished between its insert and its lookup. The queue now does both in one BEGIN IMMEDIATE transaction (boilerplate a1c6f0d), so the workaround and its test go.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:02:58 +02:00
WolfandClaude Sonnet 471ace703c Switch the extract and summarise stages on the page, over config.yaml
A stored `stage:extract` / `stage:summarise` in the state table overrides
extract.enabled / summarise.enabled; with none, the file applies. One
function, stages.stage_on, decides, and both tasks' finders ask it.
GET/PUT /api/stages and `app stages` show and set it with who would answer;
Runs has a "What the model reads" block, and a search agent's card says when
a stage is off and links to the switch.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:54:53 +02:00
Wolf a629eda8de Merge branch 'r2-web'
# Conflicts:
#	app/static/roles.js
#	app/web/api.py
#	tests/test_web.py
2026-09-29 15:31:35 +02:00
WolfandClaude Sonnet 5dfaa393be Guard every feed-, board-, discovery- or user-supplied link with safeUrl
roles.js already checked a role's original_url is http(s) before
putting it in an href, so a board that wrote something other than a
URL could not make the reader navigate to it; sources.js (a search
sample's row.url), companies.js (careers_url) and applications.js
(application.url) put the same kind of outside data straight into a
link unchecked.

safeUrl moves to app.js, beside the app's other shared helpers, and
is exported the same way; a link whose safeUrl comes back empty is
now left out of the page (plain text) instead of getting an href of
"".

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:24:54 +02:00
WolfandClaude Sonnet 4fb9e801f5 Load the feed regardless of a slow or failed agents or companies fetch
The init block awaited /api/agents and /api/companies, both of which
only dim hidden companies, draw the affinity control and pick the
score order, before ever calling loadFeed; a slow or failed one left
the feed blank for good behind a silent catch that returned early.

The feed's own request now goes out alongside them rather than after.
Agents and companies fill in and redraw what they touch (the rail,
the feed's dimming, the reader's affinity slot) whenever they land,
reloading the feed if the real agent turns out to differ from what it
first loaded with. The feed's own fetch failing gets an empty state
with a retry instead of the same silent blank.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:23:48 +02:00
Wolf 562fa3ab09 Merge branch 'r2-backend'
# Conflicts:
#	app/storage/schema.py
2026-09-29 15:22:03 +02:00
WolfandClaude Sonnet 91596300e1 Serve a row projection from GET /api/roles instead of the whole role
The feed only ever shows a title, a few badges and a short excerpt,
but the list endpoint answered with every role's full row --
description, feed_text, raw and article_text included, about 5.5 KB
a role -- so a 1000-row restore was several megabytes of text a phone
never renders.

Store.roles_page selects only the fields a row draws (RoleRow), with
excerpt built server-side off a SUBSTR of summary or description, so
the big columns never cross into the response. The reader is
unaffected: it already fetched the whole role by uid when it opened
one; the reader pane's own reliance on the list row carrying the body
is removed too, so it no longer flashes a wrong "no description"
before that fetch lands.

Measured on a copy of the live database: a 60-row page went from
392 KiB to 39 KiB, and a 1000-row restore from 6.6 MB to 631 KiB.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:22:00 +02:00
WolfandClaude Opus b2c8c08f7f Judge an employer known only from a search; the Companies list says "from job boards"
Store.set_affinity creates a minimal company row (key, and the name its roles
give most often) for a key with roles and no row, so a Jobindex or Adzuna
employer can be hidden or liked; the reader draws the control for any role
with a company_key and reloads the company list after the first change.

/api/companies derives from_search: no ats subscription (detached included)
and no ats, careers_url or domain. companies.js labels it "from job boards"
and keeps it out of "Not readable yet".

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:19:55 +02:00
WolfandClaude Sonnet e6be593542 Ask before a destructive board or search switch; queue a pull after a reopening one
Switching a company's board or a job board search off, replacing a
company's board and detaching one all close open roles in a tap; each
now asks first, naming the count (a replace also names the old and the
new board), the same as removing a search and deleting an agent
already do.

Switching a target back on, following a board, setting a board and
saving a search used to show nothing until the schedule's next pull,
up to six hours later. Each now queues a pull job right in the API
route, so the CLI and any other caller of the same endpoint behave
like the page; the queued pull's own refusal while one is already
running is not surfaced as an error.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:18:16 +02:00
WolfandClaude Opus 91d15fcc7d A change made while score runs gets one follow-up score, and the page waits on it
score is solo, so a request during a run was deduped into the running job,
which had already read the old state. _score_after_change returns a queued
score when one has not started, else enqueues a plain score and, if that
dedupes to a running job, one follow-up with depends_on it, after="done" and
dedup_key score:after:<id>. POST /api/score answers with that job and
agents.js waits on it; the affinity route answers with score_job.

The flaky affinity test was pipeline's race, not the app's: Jobs.enqueue
re-raises the UNIQUE IntegrityError when the live job it collided with
finishes before the SELECT that should hand it back. The app retries.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:16:23 +02:00
WolfandClaude Opus 3abb44e10a Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:12:14 +02:00
Wolf bbb7474cef Merge branch 'r2-salary' 2026-09-29 15:09:19 +02:00
WolfandClaude Sonnet fa4e1a657f Keep a board's stated pay as salary_text, even when it does not normalise
Every ats parser now writes one human-readable pay line whether or not it
converts to yearly salary_min/salary_max/currency: hourly and daily rates,
Greenhouse's several pay zones, and a range with no currency all survive as
text instead of being dropped outright. Stored in roles.salary_text
(migration 12), replaced whole on every pull like title, so a board that
stops stating pay clears it rather than leaving a stale line. salary_state
is derived, not stored ('normalised', 'stated', or ''), and is what the
feed's new Pay filter and badge read; gates and scoring are unchanged, still
reading only the normalised columns.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:08:45 +02:00