diff --git a/README.md b/README.md index 71b9170..fe39995 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ uv run app job pull # read every board and search, embed and score th uv run app serve # http://127.0.0.1:8790 ``` -`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents every key, including the `extract:` and `summarise:` blocks, which set the default for the two model stages; the switch under Settings, Sources, Runs overrides it without a restart, and `app stages` does the same from the command line. +`uv run app worker` drains the queue without the web app, and `uv run pytest` runs the app's tests and the copied modules' own. `node --test tests/js/*.test.mjs` runs the chip input's. `config.yaml` is the committed default, with one starter agent and four Copenhagen searches on Jobindex; `config.example.yaml` documents every key, including the `extract:` and `summarise:` blocks, which set the default for the two model stages; the switch under Settings, Sources, Runs overrides it without a restart, and `app stages` does the same from the command line. ## Sources: job boards and company boards diff --git a/docs/decisions.md b/docs/decisions.md index 6b9107b..c6f9c74 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -12,7 +12,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning **Storage is written, not imported.** `desk` has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`. -**Layers are `models, config, storage, jobs, web, main, cli`.** `tests/test_architecture.py` holds the direction and the doors within `app`; `tests/test_coupling.py` holds that `pipeline`, `auth` and `desk` import nothing from `app` or each other, except `desk` from `pipeline`, and that a plugin imports `desk` only. +**Layers are `models, places, seniority, config, storage, jobs, web, main, cli`.** `places` and `seniority` read what a board wrote and know no database, so they sit under `config` and storage may use them. `tests/test_architecture.py` holds the direction and the doors within `app`; `tests/test_coupling.py` holds that `pipeline`, `auth` and `desk` import nothing from `app` or each other, except `desk` from `pipeline`, and that a plugin imports `desk` only. ## Data @@ -114,7 +114,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning ## Ranking -**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the gates and floors the score, `-1` mutes. +**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the `any` group, `-1` mutes. **An agent is the group: it holds several prompts, and `fit` is the best prompt's.** A hunt spans several kinds of role (AI and data leadership, customer-facing technical leadership, AI research), and one blended paragraph ranks a role by its distance to the blend. Each prompt is a named text in `agent_prompts` (migration 14, which moved every non-empty `agents.intent` into a prompt named after the agent and dropped the intent columns), embedded on its own and cached under sha1 of model and text, so an edit re-embeds one prompt. A role's fit is the maximum over the prompts of the calibrated cosine, the first prompt winning a tie, and `scores.prompt` names the winner. Votes, liked companies, gates, weights, limits and `taste` stay per agent, so a vote counts for every prompt; a market watch is a second agent with its own. No third level above the agent was added. `POST /api/agents` still takes `intent` when `prompts` is absent. On the live data of 2026-09-29 the embedder separates the three kinds of role but not seniority: an individual contributor's posting on the same subject ranks beside a head's, so seniority has to come from `extract` and a prefer gate, or from votes. @@ -130,11 +130,13 @@ What was decided for roledesk and the one reason for each. The longer reasoning **Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty. -**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 and the 0.9 floor for a favourite means the same thing whatever the weights are. +**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 whatever the weights are. **A gate rejects on evidence, not on absence, and `any` gates are one group.** A require gate used to gate a role whose column was empty ("country unknown"), and extract, which fills that column, is bounded to the roles that rank, so 2,281 of 15,556 open roles were out of reach for good: every board that writes "Remote", "Hybrid" or "Distributed" as its location. Now an empty column neither passes nor fails; the role ranks, extract reads it, and the rescore gates it on what it found. `any` is the third mode: the `any` gates form one group a role must match one of, so "in Denmark, or remote" is two `any` gates, where require alone could only say "and". `remote` is also read off the location line, as `country` is, since Greenhouse puts "Hybrid" there and nowhere else. -**Gates match loosely where the column is free text.** `place` is a case-insensitive substring, so `Berlin` matches "Berlin, Germany"; `company` compares company keys, `languages` passes when every required language is a value, `salary_min` is at least the smallest value, and the rest are case-insensitive equality. Each gate is one SQL `CASE` column, so a require gate filters while the gated row keeps every failed gate as its reason. +**A favourite passes the `any` group, not the `require` gates, and has no floor.** A favourite used to skip every gate and score at least 0.9. Starring is how the owner widens one company's roles beyond the agent's places, and with every gate skipped a starred Databricks brought 471 US roles. With the `require` gates kept, the owner's agent shows 166 Databricks roles, all in Europe, against 53 unstarred. The floor went with it: over the widened set it put every role of a starred company above all others whatever its fit. The affinity term still lifts them, and with the default weight of 0.2 Databricks held 33 of the top 60 rows on the data of 2026-09-29. + +**Gates match loosely where the column is free text.** `place` takes the Place filter's tokens on whole folded words, so `Berlin` matches "Berlin, Germany" and `Lund` no longer matches "Billund"; `company` compares company keys, `languages` passes when every required language is a value, `salary_min` is at least the smallest value, and the rest are case-insensitive equality. Each gate is one SQL `CASE` column, so a require gate filters while the gated row keeps every failed gate as its reason. **A role with no vector is scored without `fit`.** The fit weight leaves the denominator, so a role `embed` has not reached yet ranks on recency, affinity and prefer gates instead of sinking to the bottom. It is gated as "no vector" only when fit is the only weighted term, and as "no intent vector" when the intent could not be embedded. @@ -156,7 +158,21 @@ What was decided for roledesk and the one reason for each. The longer reasoning **The feed loads whether or not its enhancements do.** `/api/agents` and `/api/companies` only dim hidden companies, draw the affinity control and pick the viewing agent's score order; the init block used to await both before ever calling `loadFeed`, so a slow or failed one left the feed blank for good behind a silent `catch (err) { return; }`. Now the feed's own request goes out alongside them rather than after, agents and companies fill in and redraw what they touch (the rail, the feed's dimming, the reader's affinity slot) whenever they land, and a feed already fetched with no agent known reloads once the real one is. A failed feed fetch of its own gets an empty state with a retry (`state.feedError`) instead of the same silent blank. -**`roles(company=, place=)` match a case-insensitive substring of what the board wrote.** The API's company and place fields are free text a person types, not a slug, so "Trade Republic" finds `trade-republic-bank` without knowing the key. +**`roles(company=, place=)` match a case-insensitive substring of what the board wrote.** The API's company and place fields are free text a person types, not a slug, so "Trade Republic" finds `trade-republic-bank` without knowing the key. Other pages link with them; the rail no longer types them and shows one as a chip to take off. + +**A place is matched on whole folded words, by a function registered on the connection.** `roles.place_fold` (migration 17) holds `places.fold(place)`: casefolded, marks stripped, `ø` to `o`, `æ` to `ae`, punctuation to spaces, so `Skane` finds `Skåne` and `Kobenhavn` finds `København`. A filter is a token: `country:DK` is the country column, or a line naming the country or one of its cities while the column is empty or `EU`, which finds 1,193 roles against 1,132 on the column alone; `region:skane` is any name or town of a region in `app/places.py`; `place:malmo` is a town with its aliases. Several tokens are joined by OR. A region is up to 161 phrases, and as `LIKE` clauses over 34,538 roles `region:oresund` took 400 ms, twice per feed request. `place_match(place_fold, country, token)`, which `Store._connect` registers, takes 10 to 30 ms. + +**The regions are a hand-written list.** Skåne, the Capital Region of Denmark, Zealand, Øresund (the two capital regions), Jutland and Funen, each with its names and towns. On the data of 2026-09-29 no Danish or Skåne town with more than one role was missing. A region elsewhere is added to `REGIONS`. + +**`seniority` is read from the title until extract has read the role.** Extract had never run, so the column was empty on all 33,881 open roles and a seniority filter or gate had nothing to read. `seniority.from_title` matches whole words in English, Danish, Swedish, German and French and gives 43% of open roles a level. "X Manager" is a level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor; intern wins over a boss named in the title. Known mistakes: "VP" is always exec, "Field CTO" and "Chief of Staff" get none, "Account Director" is director. Extract overwrites the title's level, a level a board states wins over both. + +**A filter that most roles cannot answer keeps the ones that do not say.** The Seniority filter keeps roles with no level unless its box is unticked, since 57% have none. The Pay filter leaves out roles that state no pay unless its box is ticked, since 87% state none and the filter would otherwise do nothing. `GET /api/roles/facets` counts each level and pay state under the other filters, so the rail shows what a click would add or lose. + +**Company and Place are chips picked from suggestions.** Only a suggestion becomes a chip, so a filter names something the database holds: a company key, a place token. `GET /api/suggest/companies` and `/api/suggest/places` fold what is typed and match word prefixes, computed per request over 743 companies and 5,600 place lines. `app/static/chips.js` is the input, with no dependency; its tests run with `node --test tests/js/*.test.mjs`. + +**A filter is a history entry, text is not.** Filters used to replace the entry, so back left the page. Now each filter set pushes one and back takes it off: a tag, the period, a chip, a level, a company. The search box and typed text replace, or back would step through a search a letter at a time, and so do the agent and the order. Every pushed entry carries the address it was pushed over, so closing the reader or taking a company off is `history.back()` when the entry under it is that list, and a new entry otherwise. + +**A company's name is a link that adds the company to the filters on screen.** In a row and in the reader. With one company filtered a strip over the feed names it, counts its roles, links its careers page and carries the affinity control, which is where a company is starred from. After a change the strip waits for the `score_job` the route answers with and reloads the feed. **A muted company is excluded in SQL, before `LIMIT`.** Filtering after `LIMIT` shipped first and showed nothing once Databricks' 884 freshly pulled roles filled the newest 200 rows and Databricks was muted. diff --git a/docs/open.md b/docs/open.md index fdfcaa0..4a2d5f3 100644 --- a/docs/open.md +++ b/docs/open.md @@ -22,3 +22,8 @@ What is left after the phone rework of 2026-09-25, for whoever picks the repo up - **A posting fetched by hand feeds the reader only.** "Fetch the posting" stores `article_text`, which `summarise` reads and `embed` and `extract` do not. - **A list's teaser is not stored.** Phenom's and Oracle's lists carry a short description, which is dropped so that it cannot replace the full text; until the role's page is read the role has no text at all. +- **A starred company's roles crowd the top of Best first.** With the companies weight at 0.2 a favourite's roles gain about 0.13; after starring Databricks it held 33 of the top 60 rows. Lower the weight on the agent's card, or star fewer companies. +- **The owner's agent passes any role in Sweden and Germany.** Its `any` group holds `country` DK, SE, DE, which is why SAP in Garching ranks. For Denmark, remote and Skåne only, the group is `country` DK, `remote` remote and `place` `region:skane`. +- **★ means two things on the Roles tab.** On a row it saves the role, in the company strip it marks the company a favourite. +- **Place suggestions take about 110 ms**, most of it splitting 5,600 place lines into towns on every request. Caching the towns per distinct line would cut it. +- **`company_key` keeps `ø`**, so "Orsted" and "Ørsted" are two companies. Folding it changes the primary key of `companies`. diff --git a/docs/plan-feed-filters.md b/docs/plan-feed-filters.md deleted file mode 100644 index 625c876..0000000 --- a/docs/plan-feed-filters.md +++ /dev/null @@ -1,150 +0,0 @@ -# Plan: starred companies, filters in history, place and company chips - -Planned 2026-09-29 from the owner's dictated notes. This file is folded into `decisions.md` and deleted when the packages land. - -## What was asked - -1. A starred company shows its roles across Europe, every other company stays limited to the agent's places. -2. A click on a company name in the feed filters the feed to that company. Back removes the filter. -3. A company is starred from the feed, not from Settings. -4. A seniority filter that does not lose the roles with no known seniority. The same worry for Pay. -5. A place filter that finds every role in Denmark, and takes several places, e.g. Denmark and Skåne. -6. Company and place fields with autocomplete against what the database holds, each pick a chip, several chips per field. -7. `Skane` finds `Skåne`, `Malmo` finds `Malmö`, `Kobenhavn` finds `København`. - -## Measured on the live database, 2026-09-29 - -| Fact | Number | -|---|---| -| Open roles | 33,881 | -| With `seniority` | 0, extract has never run | -| With pay stated | 4,345 (13%) | -| With `remote` | 8,582 (25%) | -| `country` empty | 2,152 | -| `country = 'DK'` | 1,132 | -| Denmark or Copenhagen in `place`, `country` is `EU` or empty | 61 | -| Distinct `place` values | 5,613 | -| Companies with an affinity | 0 | -| Databricks open roles, of which in the US | 790, 471 | - -The Place filter is a substring of `place`, so `Denmark` misses `København K` and `Ballerup`. The `place` gate is a substring too, so `Lund` matches `Billund` and `Albertslund`. `place_key` keeps `ø` and `æ`, since `_fold` does not translate them. The ★ on a feed row saves the role; the company control is `affinityControl` (`app/static/app.js`), which has Favourite, Like and Hidden. - -## Decisions - -**A star is affinity 2, and a favourite passes the `any` group, not the `require` gates.** Today a favourite skips every gate, so a starred Databricks would bring 471 US roles. The owner's agent requires a European country and then any of a country, remote or a town. With the new rule a starred company's roles are gated by Europe only. An agent with no `require` gate shows a favourite's roles from everywhere, as today. - -**The 0.9 floor for a favourite is removed.** With the gates widened, the floor would put every European role of a starred company above all other roles whatever its fit. The affinity term still lifts them, and one click on the company name lists them all. No company has an affinity today, so no stored score changes meaning. - -**A filter pushes a history entry, the search box and typing replace it.** This reverses the rule in `roles.js` that a filter replaces. `popstate` reads the address into the filter and reloads. Agent and order are not filters and replace. - -**A place is matched on whole folded words.** `roles.place_fold` holds `places.fold(place)`: casefolded, marks stripped, `ø` to `o`, `æ` to `ae`, `å` to `a`, punctuation to spaces. `Lund` is the word `lund`, so it no longer finds `Billund`. The `place` gate uses the same matcher. - -**The match is a Python function registered on the connection, not `LIKE`.** A region is up to 161 phrases. As `LIKE` clauses over 34,538 roles `region:oresund` took 400 ms, `region:hovedstaden` 250 ms, and the feed asks twice, for the count and the page. `places.matches` behind `place_match(place_fold, country, ?)` took 14 ms cold and 5 ms warm. - -**A place filter is a token.** `country:DK` matches `country = 'DK'`, or a role whose country is empty or `EU` and whose `place_fold` has a name of that country. `region:skane` matches any town or name of the region in `app/places.py`. `place:malmo` matches the folded words and the aliases of that town (`copenhagen`, `kobenhavn`, `kopenhamn`). Several tokens are joined by OR. A `place` gate value accepts the same tokens, a bare value is read as `place:`. - -**Seniority comes from the title when nothing better is known.** `app/seniority.py` classifies a title into `SENIORITIES`. It fills `roles.seniority` at ingest and in the migration where the column is empty; extract overwrites it. The filter takes several levels and keeps roles with no level unless told otherwise. The rail shows how many roles each level and "no level" hold under the other filters, and the same for Pay. - -**Suggestions are computed per request.** 5,613 distinct places and 743 company keys are small enough to fold and match in Python without a cache. - -## Contract - -Every package builds against this, so waves can run before the one below has merged. - -### `app/places.py` (package A) - -```python -def fold(text: str) -> str # "Skåne län" -> "skane lan", "København K" -> "kobenhavn k" -def parse(token: str) -> Token # "country:DK", "region:skane", "place:malmo"; bare text -> place -def known(token: str) -> bool # False for region:atlantis, country:ZZ, an empty value -def matches(place_fold: str, country: str, token: str) -> bool -def register(conn: sqlite3.Connection) -> None # place_match(place_fold, country, token) -def match_sql(token: str, *, place: str = "place_fold", country: str = "country") -> tuple[str, list] -def suggest(q: str, places: Iterable[tuple[str, str, int]], limit: int = 12) -> list[Suggestion] -REGIONS: dict[str, Region] # key -> label, names, towns -``` - -`match_sql` answers `place_match(, , ?)` with the token as its parameter, so every connection that runs it has called `register`. `Token` has `kind`, `value`, `label`. `Suggestion` has `token`, `label`, `kind` (`country`, `region`, `place`) and `count`. `places` is `(place, country, open roles)`. A place suggestion is a town: the first comma part of each `;` part of `place`, grouped by its fold, labelled by its commonest spelling. `q` matches a word prefix of the fold of a label, name or alias. Countries come first, then regions, then towns by count. Regions to start with: Skåne, Capital Region of Denmark, Zealand, Øresund (the two capital regions), Jutland, Funen. - -### `app/seniority.py` (package B) - -```python -def from_title(title: str) -> str # one of SENIORITIES, "" when the title does not say -``` - -Word-bounded, on the folded title, highest level wins. English, Danish, Swedish, German and French titles. - -### `GET /api/roles` (package E) - -New parameters, the old `company` and `place` substrings stay for links from other pages: - -| Parameter | Meaning | -|---|---| -| `company_keys` | repeated, exact `company_key`, OR | -| `places` | repeated place tokens, OR | -| `seniority` | repeated levels, OR | -| `seniority_unknown` | default true: roles with no level pass the seniority filter | -| `pay_unknown` | default false: roles that state no pay pass the pay filter | - -`RoleRow` gains `seniority`. - -### New routes (package E) - -| Route | Answer | -|---|---| -| `GET /api/suggest/companies?q=&agent=&limit=` | `[{key, name, count, affinity}]`, word prefix on the folded name, by open roles | -| `GET /api/suggest/places?q=&limit=` | `[{token, label, kind, count}]` | -| `GET /api/places/labels?tokens=` | `[{token, label}]`, for chips read from the address | -| `GET /api/roles/facets?` | `{seniority: [{value, count}], pay: [{value, count}]}`, each counted with its own filter left out; value `""` is unknown | - -### `app/static/chips.js` (package D) - -```js -window.chipInput(root, { suggest, onChange, placeholder, label }) -> { values(), set(items), clear() } -``` - -`root` is an empty element. `suggest(q)` resolves to `[{value, label, hint}]`. An item is `{value, label}`. A pick becomes a chip with a remove button; Enter picks the first suggestion, arrows move, Escape shuts the list, Backspace in an empty field removes the last chip. Text that matches no suggestion becomes no chip. `onChange(items)` runs after each add or remove by the person; `set` and `clear` are the page's own and do not call it. The tests run with `node --test tests/js/*.test.mjs`. - -## Work packages - -Each owns the files named and nothing else. A package that finds it needs another file stops and reports. - -### Wave 1, in parallel - -| Package | Model | Owns | -|---|---|---| -| A, places | Sonnet | `app/places.py`, `tests/test_places.py` | -| B, seniority | Sonnet | `app/seniority.py`, `tests/test_seniority.py` | -| C, favourite rule | Sonnet | `app/jobs/score.py`, `tests/test_score.py` | -| D, chip input | Opus | `app/static/chips.js`, `app/static/chips.css`, `tests/js/chips.test.mjs` | -| F, history, company filter and star | Opus | `app/static/roles.js`, `app/templates/roles.html`, `app/static/app.css` | - -F sends `company_keys`, which the server ignores until E lands, so F is checked against a stub and again after wave 2. - -F in detail: the company name in a feed row and in the reader is a link to `/?company_keys=` and pushes; with exactly one company filtered a strip above the feed names it, counts its open roles, links its careers page and carries `affinityControl`; `setFilter` pushes and `popstate` restores. - -### Wave 2, after A, B and C are merged - -| Package | Model | Owns | -|---|---|---| -| E, storage and API | Opus | `app/storage/store.py`, `app/storage/schema.py`, `app/web/api.py`, `app/models.py`, `plugins/ats.py`, `tests/test_storage.py`, `tests/test_web.py` | - -Migration 17 adds `roles.place_fold` and fills it, and fills `seniority` from the title where it is empty. `upsert_role` writes both. `Store._connect` calls `places.register`. `_gate_sql` uses `places.match_sql`. - -### Wave 3, after D, E and F are merged - -| Package | Model | Owns | -|---|---|---| -| G, the rail | Opus | `app/static/roles.js`, `app/templates/roles.html`, `app/static/app.css` | - -Company and Place become chip inputs on the suggest routes. Seniority is a row of level buttons with "and no level" ticked, each with its facet count; Pay shows its counts and gets "and no pay stated". Chips are read from and written to the address. - -### Review - -The planner reads every diff, runs the suite, and checks the pages in a browser at 1400 and 390 pixels on a copy of the live database before each merge. - -## Not in this plan - -- Duplicates: SAP's one role shows four times in the feed. It is item 1 of the backlog. -- The owner's agent still passes any role in Sweden and Germany (`country` DK, SE, DE in the `any` group). Once E lands the group can be `country:DK`, remote and `region:skane`; the owner changes it under Settings, Search agents. -- Reading only some countries of a large board ("A filter in a target", `plan-hosts-wave-2.md`).