Commit Graph
30 Commits
Author SHA1 Message Date
WolfandClaude Opus 5.5 60136765b2 Refuse a fetch to another port than 443, an IP address or a local host name, at every hop
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:42:58 +02:00
WolfandClaude Opus 5.5 f61caddced Match a slug whole, and compare an allowed host name without case
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:42:10 +02:00
WolfandClaude Opus 5.5 0ffd0726e9 Read never-tried roles before failed ones in detail, count a 404 as read, and take only the fields a later pull keeps
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:41:39 +02:00
WolfandClaude Opus 5.5 1e47192aba Fail a board whose robots.txt cannot be read, and follow a robots.txt redirect to the host's www. twin only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:40:14 +02:00
WolfandClaude Opus 5.5 13ce99304b Match robots.txt rules by RFC 9309, the longest rule winning, with wildcards, end anchors and case-sensitive paths
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:39:11 +02:00
WolfandClaude Opus 5.5 a9fb4397ac Order the hosts by a RANK per module, which restores the select's and the probes' order from before the modules
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:38:18 +02:00
WolfandClaude Opus 5.5 75a389d725 Keep a discovered slug's case for a host with KEEP_CASE, and check only group 1 against the hosts' own path words
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:37:26 +02:00
WolfandClaude Opus 5.5 8b14d8575b Read a role's own page once for a host with detail, and fail a board whose robots.txt disallows it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:05:44 +02:00
WolfandClaude Opus 5.5 b02b72b4aa Move the eight hosts into a module each under plugins/atshosts and build the ats plugin's tables from them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:03:37 +02:00
Wolf d887a91af0 Merge branch 'wp3-teamtailor' 2026-09-29 16:15:22 +02:00
WolfandClaude Sonnet 5.5 50edf0eeeb Read Teamtailor career sites from their jobs.rss feed, paged by offset
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:14:41 +02:00
WolfandClaude Sonnet 5.5 00e111961e Add Platsbanken as a third search provider
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-29 16:10:45 +02:00
Wolf 562fa3ab09 Merge branch 'r2-backend'
# Conflicts:
#	app/storage/schema.py
2026-09-29 15:22:03 +02:00
WolfandClaude Opus 3abb44e10a Stream a board's body against MAX_BYTES and a 60 s deadline; read timeout 30 s
_get streams the body with a running byte count, so an oversized board is
refused at MAX_BYTES instead of after it is held whole, and bounds the whole
request, redirects included, by DEADLINE, checked between chunks. The
per-phase timeouts are httpx.Timeout(10, read=30), set from 84 live requests
against mongodb, feedzai and the five Greenhouse boards with most open roles,
with and without pay_transparency=true: the body arrives in under 1 s, the
first byte waited up to 16.6 s, and the flag made no measurable difference.
_paged works unchanged through it (tested over a mock transport, 429 included).

Correct pull.py's BATCH_SIZE comment: a batch reads its boards in sequence,
and the parallelism is the http pool's.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 15:12:14 +02:00
WolfandClaude Sonnet fa4e1a657f Keep a board's stated pay as salary_text, even when it does not normalise
Every ats parser now writes one human-readable pay line whether or not it
converts to yearly salary_min/salary_max/currency: hourly and daily rates,
Greenhouse's several pay zones, and a range with no currency all survive as
text instead of being dropped outright. Stored in roles.salary_text
(migration 12), replaced whole on every pull like title, so a board that
stops stating pay clears it rather than leaving a stale line. salary_state
is derived, not stored ('normalised', 'stated', or ''), and is what the
feed's new Pay filter and badge read; gates and scoring are unchanged, still
reading only the normalised columns.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-09-29 15:08:45 +02:00
Wolf 706b127bc0 Merge branch 'worktree-agent-afad54f12aef2cccf'
# Conflicts:
#	app/storage/schema.py
#	app/storage/store.py
#	tests/test_storage.py
2026-09-29 14:32:28 +02:00
WolfandClaude Sonnet 5 44ff7c0131 Unpack structured pay from five ATS hosts; fix the function classifier
Part A: Ashby (?includeCompensation=true), Greenhouse (pay_transparency=true,
already free on the list request), Lever (salaryRange), Recruitee (salary)
and Personio (salaryInformation) all give structured pay; Workable and
SmartRecruiters do not (checked live). Every host converts to a yearly
salary_min/salary_max/currency, multiplying a monthly figure by 12 and
skipping an hourly or daily one rather than annualising a number nobody
stated. A range with no currency is ignored, and Greenhouse's own multi-zone
ranges are skipped outright since nothing says which zone the role is at.

Part B: classify() matched substrings, so " it" matched inside "Italy",
filing European sales roles as engineering; it now matches on word
boundaries (full boundary for the handful of short tokens that are
themselves the start of an unrelated word, prefix matching kept for the
stems). Title is checked before department (a hand-checked sample found
title-first right about twice as often), nothing matching returns '' rather
than "other", and a modest set of German/Dutch/Danish/Norwegian/Swedish/
French keywords covers the seed's other languages.

function is not a board field, so it gets the opposite rule from every
other raw column: extract's answer overwrites the classifier's guess
outright, and once extracted_at is set no later pull may put the guess
back. Migration 9 recomputes function for every not-yet-extracted role from
its stored title and department.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-29 14:29:12 +02:00
Wolf c6246af994 Merge branch 'worktree-agent-ac3c9345edd29a1c6'
# Conflicts:
#	app/cli.py
#	app/jobs/sources.py
#	app/static/app.css
#	app/static/companies.js
#	app/web/api.py
#	tests/test_web.py
2026-09-29 14:15:10 +02:00
WolfandClaude Opus b43fc54565 One pull at a time: the run is claimed in the database and its marker rides in each batch
A second pull (schedule, Run button, `app job pull` in another process) could
start while one was fetching and overwrite the stored run marker, so the first
run's boards failed their stamp check or a settle closed on the other run's id
lists. `claim_run` sets `pull:run` in one conditional statement, only while it
equals `pull:settled` or is older than an hour; a refused `pull` job queues no
stage and says so in its result and a notice, and the CLI exits 1. Batches
carry the marker in their payload, the plugins stamp with it, and the settle
reads it off its own batches' payloads before releasing the run.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:12:17 +02:00
WolfandClaude Opus 5.5 3b209d628a Every way a company gets a board is set_board, and an unreadable host keeps its slug
`app.jobs.sources.set_board` attaches a board to an existing company,
replaces the one it has (closing the old board's open roles) or creates
the company, with row and subscription changed in one transaction by
`Store.set_company_board`. A board that lists zero roles is kept. `add`,
`discover --add`, the seed importer (moved out of app/cli.py) and the new
`POST|DELETE /api/companies/{key}/board` all go through it; a seed re-run
attaches a board to a row that never had one. A company's subscription is
rewritten in place, and detaching hides it.

`companies.ats_slug` (migration 9) keeps the slug of a host the app
cannot read yet, and discovery returns such a host before probing slugs.
The Companies rows get Set board / Change, the host select is rendered
from `plugins.ats.HOSTS`, and a row with nothing found reads "board not
found" and is in the "Not readable yet" filter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 14:04:52 +02:00
WolfandClaude Opus c7da12b93d Paged boards are read to their stated total, and a board failing for 21 days closes
Workable and SmartRecruiters page their lists, and both parsers read page one
only, so the settle closed every role past it (Delivery Hero: 980 stated, 100
open). `_paged` follows nextPage and offset, and the plugin stamps
`ats:complete` only when the distinct ids read equal the stated total; a 429
after page one keeps what was read and closes nothing. Adzuna completes on its
`count`, Jobindex counts raw items and raises on an answer with no channel.

A board whose fetch raises is recorded under `ats:failing:<target>` with the
first failure's time and the latest reason; after STALE_DAYS the settle closes
its roles, and its next answer reopens them. `/api/companies`, the Companies
list and `app companies check` show it.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
2026-09-29 14:02:49 +02:00
WolfandClaude Fable 5.1 8532e4e739 Runs is the queue's elements and a Problems log; a Kill lands between boards
Run one stage goes: the cascade is the unit of work and app job keeps the
per-stage entry points. The pull button is disabled while a pull runs and
the lede no longer hardcodes the cadence. Messages folds into Runs as
Problems. pull_batch checks for a cancel before every board, one plugin per
target, and both plugins now close their HTTP client.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 16:10:38 +02:00
WolfandClaude Fable 5.1 51d1e67ff9 A European seed of 362 employers, readable or not; discover finds a company's board from its site
The seed was jobhunt-hq's 186 US boards, chosen because they could be read. It is now a hand-picked list of European tech employers, with `board` filled by the new `app.jobs.discover` from each company's own careers site and every board fetched once before it landed: 175 have one, and the rest keep the ATS host their site runs on (Teamtailor 32, Workday 12, SuccessFactors 9, ...) and a careers URL, as rows the app cannot read yet. Companies shows them as "not readable yet" with a link, behind a filter, so the next parser to write is the host that filter shows most.

`app companies discover NAME DOMAIN [--add]` reads the home page, its careers links and the usual paths for a link into a known host, one level down on a careers page, and probes Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio by slug when the pages link nothing. It is what the LinkedIn flow will call.

Migration 7 adds companies.ats, country and city. The Workable parser now reads the v3 shape its list endpoint actually sends (location object, department list, shortcode and no URL or text); the first populated board showed it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-28 15:27:14 +02:00
WolfandClaude Opus 5.5 6b40702a9a Sources tab: searches, company boards and the prune by region, in the app
What a pull reads can now be changed on a page instead of only the command
line: try a search and see its first ten results before saving it, edit or
remove one, follow a company's ATS board, switch any board or search on or
off, and preview and apply a prune by region. Each row shows its open roles
and the countries they are in.

The page and `app companies`/`app searches` call the same functions in
app/jobs/sources.py. Switching a target off closes its open roles, and the
pull's settle closes them again if a batch in flight reopened them. Roles
pulled before `country` existed are backfilled at startup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 15:15:04 +02:00
WolfandClaude Opus 5.5 047f7cd8ad Find roles by search, not only by company: Jobindex and Adzuna searches
The ats boards only read companies someone listed, and the seed list is
US-centred: 31 of 29,000 open roles were in Denmark. A `search` plugin runs
saved queries on boards that span companies, pulled in the same task as the
boards: Jobindex (Denmark, RSS, no key) and Adzuna (UK and most of the EU,
free key, built to its documented API). Jobnet and Jooble were checked and
left out.

`country` is now filled from the board's location line by a lookup, with EU
for Europe-wide roles, so country gates work without extract. `app companies
prune --in` unsubscribes boards with no open role in a region. The starter
agents and searches are Copenhagen and Europe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:55:50 +02:00
WolfandClaude Opus 5.5 59a301c351 Wire the app like the boilerplate: door, queue, schedules, eight tabs
app/main.py assembles it in the boilerplate's order, auth.connect first,
then the queue with the scheduler and the pages. TABS is Roles,
Applications, Agents, Companies, Jobs, Schedules, Messages and Account:
the alpha page's three views become templates on the boilerplate's
base.html, driven by static/roles.js (the old app.js) through api() and
toasts.js, over the unchanged JSON routes. Companies lists every board with
its open roles and the affinity control. A 422 is one sentence, so a toast
never reads [object Object].

Config takes the boilerplate's shape: APP_CONFIG or config.yaml, a missing
file is the defaults, an unknown key is an error. roles.yaml is gone;
config.yaml is the committed default and carries the starter agents.

pull is now a job that asks for the pull cascade, because a schedule
enqueues a job kind; its default schedule, every 6h, is declared beside it
in app/jobs/cascade.py. pull_batch's targets are keyword-only so the Jobs
tab offers no batch button. `app worker` is the boilerplate's; `app job`
still drains a queue of its own and fires no schedules.

test_coupling holds pipeline, auth and desk apart (desk may import
pipeline) and plugins to desk; test_architecture keeps app's layers, now
with main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:48:12 +02:00
WolfandClaude Opus 5.5 7a63311714 Rename the package roledesk to app
The boilerplate's rule: the deploy kit's Dockerfile, the APP_CONFIG,
APP_PASSWORD and APP_DATA variables and its commands all assume a package
called app, so a project renames its display title and nothing else. The
console script is `app`, and the launch entry runs `uv run app serve --port
8790`. The domain noun stays `role`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:30:10 +02:00
WolfandClaude Sonnet 5 867ebe6405 Raise the ats body cap to 96 MB, retry Greenhouse without content=true
openings.go's 8 MB cap was written against arbitrary URLs; every URL here is
on a fixed allow-list of seven ATS hosts, and the real pull of 2026-09-23
refused Anduril, Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex
for it. A board that still exceeds 96 MB now retries once without
content=true (Greenhouse only), trading the description for the listing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 11:39:39 +02:00
WolfandClaude Sonnet 5 21ed472a87 Read Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee and
Personio boards

The `ats` plugin: one Source per host's public JSON (Personio: XML), each
parsed to a common row and yielded as a `role` item. The field map, the
redirect allow-list, the 8 MB cap and the 10 s timeout are ported from
jobhunt-hq's openings.go (MIT); the nine-function classifier is ported from
its seedgen. Workable, SmartRecruiters, Recruitee and Personio are new,
written from one real board of each fetched while this was built. Fixtures
under tests/fixtures/ats/ keep every test off the network.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 11:27:14 +02:00
WolfandClaude Opus 5.5 36dbcbb29e Skeleton: roledesk on desk and pipeline from ../news
Package layers models, config, storage, jobs, web, cli, held by
tests/test_architecture.py with desk and pipeline as external trees and
plugins/ limited to desk. roles.yaml is the default config, roledesk.db
its own file, port 8790. create_app mounts pipeline.web.connect over
Jobs(config.database); `roledesk job pull` drains a no-op pull and
`roledesk db init` creates the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 10:59:17 +02:00