diff --git a/docs/plan-ranking-flow.md b/docs/plan-ranking-flow.md index 8f35882..a756410 100644 --- a/docs/plan-ranking-flow.md +++ b/docs/plan-ranking-flow.md @@ -111,3 +111,88 @@ Without the Where lines, about four in five model reads go to roles outside DK, **Today.** Summarise reads at most 30 roles scoring at least 0.65, plus the roles you saved or liked. It runs inside the model pass, on the same score. **Decided.** Summarise is not part of the preparation or the scoring. It runs once the final scores are set, on the top of the feed by the final score, with its own size on the form (50 to start, as a slider). Roles you saved or liked are summarised too, as today. + +## Planner choices + +Two points the architecture leaves open, settled by the planner so the build can start: + +- On a narrow read (the agents' places), when fetch terms times the place series exceed `MAX_SERIES`, the terms are dropped and the places kept, because the places are what the agent's fetch gate needs. A wide read keeps today's rule: the place part is dropped and the terms kept. +- Stored agent limits are reset to the new defaults by the migration. The old bars were measured on the blended score and mean nothing on plain fit. + +## Contract between the packages + +1. **`Limits`** keys: `cut_fit_top` 500, `cut_fit_min` 0.6, `cut_cv_top` 150, `cut_cv_min` 0.6, `summarise_top` 50. `extract_*`, `judge_*` and `summarise_min` are removed. +2. **`Gate`**: modes `require`, `any`, `exclude`. `prefer` and the `weight` field are removed. `Weights` is unchanged. +3. **`scores`** gets `fit REAL` (the prompt fit level, before the bonus) and `cut TEXT NOT NULL DEFAULT ''`, one of `fit`, `cv`, `kept` or empty. `score` is the final score; `cv_fit` stays. +4. **Analysis levels**: `bonus = w.affinity * affinity / 3 / (w.fit + w.recency + w.affinity + w.taste)`; the fit level is `fit + bonus` and the CV level `cv_fit + bonus`. Per agent, over roles that pass the fetch gate: the best `cut_fit_top` by fit level at or above `cut_fit_min` are `fit`. Then the best `cut_cv_top` by CV level at or above `cut_cv_min`, among the roles not already in, are `cv`. Roles this agent liked, and saved roles (`applications.role_uid`), not already in, are `kept`. Ties break on `posted_at` DESC, then `uid`. +5. **Fetch gate in scoring**: `require` gates always apply. The `any` group is skipped for a company with `companies.all_countries = 1` and no longer for a favourite. `exclude` gates on `title` are skipped for a company with `companies.all_roles = 1`. A hidden company is not scored. +6. **Final score**: today's `score_one` without prefer terms. +7. **Stage selection**: + - fetch text: open roles without text, in the cut of any enabled agent, within the existing per-board, per-run and retry budgets. + - extract: waiting roles in the cut of any enabled agent. + - judge: `(agent, role)` for roles in that agent's cut, for an enabled agent with a CV and a prompt. + - summarise: roles without a summary that have text, and are either in the best `summarise_top` by final score of an enabled agent with a prompt, or saved, or liked. +8. **Cascades**: + - `pull`: pull, prep, embed, score, fetch_text, prep, embed, score, extract, score, judge, summarise. + - `fetch_text` (after an agent save or an affinity change): fetch_text, prep, embed, score, extract, score, judge, summarise. + - The skip past prep, embed and score when fetch text filled nothing stays. +9. **Rename `enrich` to fetch text**: + - Python: module `app/jobs/fetch_text.py`; job kinds `fetch_text` and `fetch_text_batch`; `enrich_after` becomes `fetch_text_after`. + - Stage switch: name `fetch_text`, state key `stage:fetch_text`, and the `/api/stages` names extract, summarise, judge, fetch_text. + - Config: section `fetch_text:` with `enabled`, `per_board`, `per_run` and `retry_days`; `top` and `min_score` are removed. + - Job label: "Fetching the text of the roles in the cut". +10. **Feed**: when the request names an agent, every order, newest included, joins `scores` and hides roles that fail the fetch gate unless `gated=1`. +11. **Saved filters**: + - Table `saved_filters(id INTEGER PRIMARY KEY, agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE, name TEXT NOT NULL, query TEXT NOT NULL, created_at TEXT NOT NULL)`, with names unique per agent regardless of case. + - `GET /api/agents/{id}/filters` returns `[{id, name, query, created_at}]` ordered by name. + - `POST /api/agents/{id}/filters {name, query}` returns 201 with the row; 409 for a duplicate name, 404 for an unknown agent, 422 for an empty name. + - `DELETE /api/filters/{id}` returns 204, or 404. + - `query` is the feed's query string as `roles.js` `addressFor` writes it, without the agent. +12. **Migration 32**, in one step: + - rewrite every `agents.limits` to the new defaults; + - drop prefer gates and every gate's `weight` from `agents.gates`; + - add the two `scores` columns and the `saved_filters` table; + - rename the state key `stage:enrich` to `stage:fetch_text`; + - delete queued `enrich` and `enrich_batch` jobs. + +## Work packages + +Wave 1 runs in parallel. Wave 2 starts once wave 1 is merged into main. Each package works in its own worktree, made from main by the planner, and does not merge or rebase. + +### WP1: ranking core, in Python (Opus), wave 1 + +Owns `app/models.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/config.py`, `app/cli.py`, `app/web/api.py`, `app/jobs/{score,extract,judge,summarise,enrich,cascade,stages,posting,prep}.py`, `config.example.yaml`, `plugins/ats.py`, `plugins/readers/__init__.py` and `plugins/README.md` (the enrich wording only), `README.md`, and every Python test except those WP2 owns. It builds contract points 1 to 12 with tests, including a test that loading an agent stored before migration 32 works. + +### WP2: fetch gate at the host (Sonnet), wave 1 + +Owns `app/jobs/reach.py`, `plugins/readers/_runner.py`, `app/jobs/limits.py`, the Limits options in `app/templates/settings/sources.html`, and `tests/test_{fetch_terms,overrides,reach,runner,web_reach,limits,readers_filters}.py`. + +- Fetch terms go to every board with 300 or more roles (`why` is `agents` or `countries`) whose host takes `query="one"`, unless the company is set to All roles. `ask_for` no longer requires `wide`. +- Apply the planner's choice on series. +- Update the reach line, the docstrings and the limit words: "text per role" now says fetch text reads each role's page for the roles in the cut. + +### WP3: settings pages (Sonnet), wave 2 + +Owns `app/templates/settings/agents.html`, `app/static/agents.js`, `app/static/remote.js`, `tests/js/remote.test.mjs`, the Runs and Agent match wording in `app/templates/settings/sources.html`, `app/static/jobs.js`, `app/static/companies.js`, the overrides text in `app/static/app.js`, and the form assertions in `tests/test_web.py`. + +- The agent form follows Settings in `architecture.md`: + - a fetch gate block in three parts: Region (scope), Place (countries, towns, remote), Role (fetch terms, titles to leave out); + - the feed order weights and the CV match weights; + - the cut: the best N by fit, each at least X; the best M by CV closeness not already in, each at least Y; liked and saved roles always in; + - summarise the top N of the feed. +- Hybrid and prefer leave the form. The card lines follow the new limits. +- The tooltips say what All countries (place part) and All roles (role part) bypass. +- The Runs page names fetch text. + +### WP4: Roles feed (Opus), wave 2 + +Owns `app/templates/roles.html`, `app/static/roles.js`, `app/static/app.css`, `app/static/chips.js`. + +- A rail note under Viewing as says the feed is already filtered by the agent's fetch gate, places included. +- Show gated is offered in every order when an agent is chosen. +- Saved filters in the rail, per agent: save the current filters under a name, apply with one click, delete. Patterns come from `../news` (the `#active` pills). +- The company strip says which parts of the fetch gate apply to the company, and no longer says that a favourite skips the places. + +## Done when + +`uv run pytest` and `node --test tests/js/*.test.mjs` pass on main, every statement in `architecture.md` holds, the reasoning is folded into `decisions.md`, and this plan is deleted. The live server picks the change up on Wolf's restart. Migration 32 runs then, and the first pull judges the whole cut (about an hour).