Files
jobboard/docs/plan-ranking-flow.md
T

20 KiB

Plan: ranking flow

This plan takes the code from what it does today to the target in architecture.md. Each stage gives today's behaviour, what was decided on 2026-10-04 and the measurements behind it. Work packages follow once it is cut into them. When the work lands, the reasoning is folded into decisions.md and this file is deleted.

Three scores

Each agent decides about a role three times: a gate on what comes in, a gate on what the models read, and an order. Everything after that is the owner's filters, saved per agent.

  • Fetch gate, a yes or no: the agent's hard rules. These are the required countries (Europe), the Where lines (countries, towns, remote) and the titles to leave out. A company set to All roles passes everything but the required countries: its roles skip the Where lines and the titles to leave out. The gate applies at every stage: at the host when the host can filter, in the cheap pass, again after extract (now with the real location and remote area), and in the feed. A role that fails it is never read by a model and never shown.
  • Analysis score: decides which roles that pass the fetch gate make the cut, the roles that get fetch text and the model pass.
  • Final score: orders the feed.

Names: fetch text is the second HTTP read of a role's own page, with no model (the code calls it enrich). Extract, judge and summarise are the model calls. Fetch terms are words a big board must match to send a role; they narrow the read at the host and are not part of the fetch gate.

  company boards (ATS)                          search providers
  fetch gate at the host + fetch terms,         their own saved query
  only boards ≥300 roles whose host filters
            │                                         │
            ▼                                         ▼
  1  Fetch ─ list pages: title, place, teaser or text
            │
            ▼
  2  Cheap pass, no tokens ─ prep, embed, fit to the prompts, closeness to the CV, fetch gate
            │
            ▼    analysis score → the cut
  3  Fetch text ─ the role's own page, for cut roles without text → embed, rescore
            │
            ▼
  4  Model pass (Haiku) ─ extract the cut → fetch gate and cut again → judge the cut
            │
            ▼
  5  Feed ─ every role that passes the fetch gate, in two orders:
            "what I'm looking for" (final score) and "what fits me" (CV match);
            front-end filters and saved filters narrow it
            │
            ▼
  6  After the ranking ─ summarise the top of the feed

1 Fetch

Today. The Where lines become the reach: a country and remote filter the host applies. This only happens on boards with 300 or more roles whose host can filter by place. Smaller boards, and hosts that cannot filter, are read whole. Fetch terms only apply to companies set to All countries (Mistral AI and OpenAI today), and only on hosts that take a query (Workday, SuccessFactors, Oracle, Phenom, Eightfold, SmartRecruiters). Search providers send their own saved query and take neither filter.

Decided. The fetch gate's location part and the fetch terms become one block on the agent form and follow one rule: each applies to every board with 300 or more roles whose host can filter on it. Fetch terms no longer depend on All countries. Agent 2 has no fetch terms today, so nothing changes on the live server until some are set. The titles to leave out cannot be asked of a host, so that part of the fetch gate starts at the cheap pass.

Consequence. The pull never closes a role against a read narrowed by terms, because that read is not the whole board. With terms on every big board, roles there close only by age. Accepted, since Wolf ranks on recency, not on closed_at.

2 Cheap pass

Today. Every open role gets one score per agent: (0.6 fit + 0.2 newness + 0.2 company affinity + 0.2 taste + prefer gates) / sum of the weights. Fit is the calibrated cosine to the best of the agent's prompts. The gates (Where lines, titles to leave out) set a failing role's score to 0 with a reason. A gate rejects on evidence only, so a role with no known location still ranks until extract fills it in (323 of agent 2's 7,263 roles that pass). Closeness to the CV (cv_fit) is stored beside the score but is not part of it. That one score drives enrich, extract, summarise, judge and the feed order.

Decided. The one score is split into the analysis score (this section) and the final score (section 5). The analysis score uses fit to the prompts, closeness to the CV and company affinity. It leaves out newness (a pull runs every 6 hours and a role is read once), taste (votes come after the read) and prefer gates. Prefer gates leave the agent; hybrid becomes a feed filter.

Decided: the cut is two lists, joined. One list is by prompt fit. The other is by CV closeness: the roles closest to the CV that the first list leaves out. The two are not merged into one number, because they answer different questions (what I want, what I am), and a merged number would let the stronger side crowd the other out. Each list has a count and a minimum level on the form. For a broad agent the count stops the list; for a narrow agent the minimum does, so roles that do not fit are not read just to fill the count. Starting values: 500 by prompt fit and 150 by CV closeness, each at least 0.6. The minimums are levels of plain fit and plain CV closeness, not of today's blended score. Among agent 2's roles that pass the fetch gate, 2,086 reach fit 0.6, 1,580 reach CV closeness 0.6, and 2,437 reach either, so for agent 2 the counts decide.

Decided: affinity is a bonus on the level, as today in spirit. A role's level on both lists rises by the company's affinity. Today the affinity term is weighted 0.2 out of 1.25, which puts a liked company about 0.05 above a neutral one and a favourite about 0.1 above. The cut keeps those sizes. The four company settings are hidden (-1), neutral (0), liked (1) and favourite (2). A hidden company is not scored at all, so it never reaches the cut or the feed. There is no level that pushes a company down without hiding it, and none is planned. Affinity remains a nudge, not a pass: a favourite's role that matches nothing stays out of the cut. Favourite only raises the level. Today it also skips the Where lines; that moves to All roles (section 5).

Decided: a role you like or save always makes the cut. This is part of the analysis score, not a separate manual path: a like or a save passes the cut whatever the role's levels, so the role gets fetch text, extract and judge. It goes through on the next pull or agent save, not the moment you click.

Decided: the cut only takes roles that pass the fetch gate, location included. For agent 2 on 2026-10-04 (24,711 open roles):

Rules applied before the cut Roles passing Fit ≥ 0.6 Inside today's Where, of the top 500 by fit Inside today's Where, of the top 150
Today: Europe, Where lines, titles 7,263 2,086 500 150
Europe and titles, no Where lines 14,704 4,855 118 28
Titles only 22,297 9,161 66 18

Without the Where lines, about four in five model reads go to roles outside DK, SE, the Skåne towns and remote, and keeping today's coverage would take a budget about four times larger.

3 Fetch text

Today. The enrich stage reads the role's own page for open roles without text, on hosts that list without it. It uses its own bar from the config (enrich.top 500, enrich.min_score 0.5, at most 200 per board and 1,000 per run). Embed and score run again for the roles it filled.

Decided. Fetch text reads the whole analysis cut, so every role the model will read has its text first. The stage is renamed from enrich to fetch text so it is not mistaken for a model call.

4 Model pass

Today. Extract and judge run on Haiku (summarise is in section 6). Each has a count and a minimum score on the blended score; for agent 2 the count decides every stage.

  • Extract reads at most 500 roles scoring at least 0.55. It fills location, remote area, seniority, languages and deadline. A rescore follows so the gates see these columns.
  • Judge reads at most 150 roles scoring at least 0.55, plus the 50 closest to the CV that those leave out. It scores skills, experience and career 0 to 100, plus a verdict on each must-have. These are weighted 30/25/30 into cv_score.

Decided. One depth: the cut is the only selection, and both scoring model stages read all of it. A second score inside the cut would defeat its purpose. The order is:

  1. Extract reads the whole cut. It is the small, fast call (about 1.5 s a role on the live backend).
  2. The fetch gate and the cut are applied again with what extract filled, now knowing which roles are really remote and where the rest are. Some roles drop out; some enter, such as roles that turn out to be remote. A role that enters is extracted on the next pull.
  3. Judge reads the cut as recomputed (about 6 s a role).

Decided: extract and judge stay two calls. Extract needs no CV and its location and remote answers feed the fetch gate before judge reads anything. Judge works per agent and role, with the CV in the system prompt.

Cost. Every model read happens once per role (judge once per role and CV), so a larger cut costs mostly a one-off catch-up. With a cut of about 650 roles, that catch-up is about 16 minutes of extract and about an hour of judge on the live backend (claude_code, the subscription, so no per-token bill). After it, each pull reads only the roles new to the cut. The recurring cost is a re-judge: changing the CV, a prompt, or the like or dislike text changes every judgement's input, so the whole cut is judged again, about an hour for 650 roles. A changed posting text re-judges that role only. Accepted: the re-judge takes a while but is correct, since the old judgements were made against the old text.

5 Feed

Today. There are two orders. Score uses the same score as section 2. CV match shows the judged roles first by cv_score, then the unjudged ones by CV closeness. Roles that fail the gates are hidden unless Show gated is ticked, so the front-end place filter can only narrow inside the Where lines. A favourite company (affinity 2) skips the Where lines but also gains in the score; that is why OpenAI's London roles rank. All countries and All roles are separate company flags that only change what is fetched.

Decided.

  • The feed shows every role that passes the fetch gate, loaded as you scroll, not only the cut. Newest first stays inside the fetch gate as well, so no order shows roles outside the agent's places.
  • There are two orders: "what I'm looking for" (final score) and "what fits me" (CV match). The final score includes newness and likes. It is not one blended number.
  • Front-end filters only narrow further. The Roles page says that the feed is already filtered by the agent's fetch gate, location included.
  • All countries and All roles stay two company switches, now defined as overrides of the fetch gate: All countries bypasses the place part (Where lines) and All roles the role part (fetch terms and titles to leave out); the region (Europe) always applies. Neither changes any score: favourite is what pushes a company's roles up. Both apply to every agent, as today. Use case: small companies worth a speculative approach ("have you thought about hiring in Denmark?").
  • The company view says plainly what is filtered: for an All roles company, "every role in Europe, not filtered by country or title"; for any other company, that it shows the roles the agent's fetch gate passes. Looking up a company must not silently hide roles in other countries.
  • Saved filters, per agent. The current filter settings can be saved under a name, loaded with one click and deleted. There is no editing: to change one, save a new one and delete the old.

Decided: the final score is today's formula without the prefer gates: 0.6 fit to the prompts, 0.2 newness, 0.2 company affinity, 0.2 roles you liked, over the sum of the weights. Everything else, such as seniority, location and pay, is a filter, not a score term, so a role is never judged twice by the score and by a filter.

6 After the ranking

Today. Summarise reads at most 30 roles scoring at least 0.65, plus the roles you saved or liked. It runs inside the model pass, on the same score.

Decided. Summarise is not part of the preparation or the scoring. It runs once the final scores are set, on the top of the feed by the final score, with its own size on the form (50 to start, as a slider). Roles you saved or liked are summarised too, as today.

Planner choices

Two points the architecture leaves open, settled by the planner so the build can start:

  • On a narrow read (the agents' places), when fetch terms times the place series exceed MAX_SERIES, the terms are dropped and the places kept, because the places are what the agent's fetch gate needs. A wide read keeps today's rule: the place part is dropped and the terms kept.
  • Stored agent limits are reset to the new defaults by the migration. The old bars were measured on the blended score and mean nothing on plain fit.

Contract between the packages

  1. Limits keys: cut_fit_top 500, cut_fit_min 0.6, cut_cv_top 150, cut_cv_min 0.6, summarise_top 50. extract_*, judge_* and summarise_min are removed.
  2. Gate: modes require, any, exclude. prefer and the weight field are removed. Weights is unchanged.
  3. scores gets fit REAL (the prompt fit level, before the bonus) and cut TEXT NOT NULL DEFAULT '', one of fit, cv, kept or empty. score is the final score; cv_fit stays.
  4. Analysis levels: bonus = w.affinity * affinity / 3 / (w.fit + w.recency + w.affinity + w.taste); the fit level is fit + bonus and the CV level cv_fit + bonus. Per agent, over roles that pass the fetch gate: the best cut_fit_top by fit level at or above cut_fit_min are fit. Then the best cut_cv_top by CV level at or above cut_cv_min, among the roles not already in, are cv. Roles this agent liked, and saved roles (applications.role_uid), not already in, are kept. Ties break on posted_at DESC, then uid.
  5. Fetch gate in scoring: require gates always apply. The any group is skipped for a company with companies.all_countries = 1 and no longer for a favourite. exclude gates on title are skipped for a company with companies.all_roles = 1. A hidden company is not scored.
  6. Final score: today's score_one without prefer terms.
  7. Stage selection:
    • fetch text: open roles without text, in the cut of any enabled agent, within the existing per-board, per-run and retry budgets.
    • extract: waiting roles in the cut of any enabled agent.
    • judge: (agent, role) for roles in that agent's cut, for an enabled agent with a CV and a prompt.
    • summarise: roles without a summary that have text, and are either in the best summarise_top by final score of an enabled agent with a prompt, or saved, or liked.
  8. Cascades:
    • pull: pull, prep, embed, score, fetch_text, prep, embed, score, extract, score, judge, summarise.
    • fetch_text (after an agent save or an affinity change): fetch_text, prep, embed, score, extract, score, judge, summarise.
    • The skip past prep, embed and score when fetch text filled nothing stays.
  9. Rename enrich to fetch text:
    • Python: module app/jobs/fetch_text.py; job kinds fetch_text and fetch_text_batch; enrich_after becomes fetch_text_after.
    • Stage switch: name fetch_text, state key stage:fetch_text, and the /api/stages names extract, summarise, judge, fetch_text.
    • Config: section fetch_text: with enabled, per_board, per_run and retry_days; top and min_score are removed.
    • Job label: "Fetching the text of the roles in the cut".
  10. Feed: when the request names an agent, every order, newest included, joins scores and hides roles that fail the fetch gate unless gated=1.
  11. Saved filters:
    • Table saved_filters(id INTEGER PRIMARY KEY, agent_id INTEGER NOT NULL REFERENCES agents(id) ON DELETE CASCADE, name TEXT NOT NULL, query TEXT NOT NULL, created_at TEXT NOT NULL), with names unique per agent regardless of case.
    • GET /api/agents/{id}/filters returns [{id, name, query, created_at}] ordered by name.
    • POST /api/agents/{id}/filters {name, query} returns 201 with the row; 409 for a duplicate name, 404 for an unknown agent, 422 for an empty name.
    • DELETE /api/filters/{id} returns 204, or 404.
    • query is the feed's query string as roles.js addressFor writes it, without the agent.
  12. Migration 32, in one step:
    • rewrite every agents.limits to the new defaults;
    • drop prefer gates and every gate's weight from agents.gates;
    • add the two scores columns and the saved_filters table;
    • rename the state key stage:enrich to stage:fetch_text;
    • delete queued enrich and enrich_batch jobs.

Work packages

Wave 1 runs in parallel. Wave 2 starts once wave 1 is merged into main. Each package works in its own worktree, made from main by the planner, and does not merge or rebase.

WP1: ranking core, in Python (Opus), wave 1

Owns app/models.py, app/storage/schema.py, app/storage/store.py, app/config.py, app/cli.py, app/web/api.py, app/jobs/{score,extract,judge,summarise,enrich,cascade,stages,posting,prep}.py, config.example.yaml, plugins/ats.py, plugins/readers/__init__.py and plugins/README.md (the enrich wording only), README.md, and every Python test except those WP2 owns. It builds contract points 1 to 12 with tests, including a test that loading an agent stored before migration 32 works.

WP2: fetch gate at the host (Sonnet), wave 1

Owns app/jobs/reach.py, plugins/readers/_runner.py, app/jobs/limits.py, the Limits options in app/templates/settings/sources.html, and tests/test_{fetch_terms,overrides,reach,runner,web_reach,limits,readers_filters}.py.

  • Fetch terms go to every board with 300 or more roles (why is agents or countries) whose host takes query="one", unless the company is set to All roles. ask_for no longer requires wide.
  • Apply the planner's choice on series.
  • Update the reach line, the docstrings and the limit words: "text per role" now says fetch text reads each role's page for the roles in the cut.

WP3: settings pages (Sonnet), wave 2

Owns app/templates/settings/agents.html, app/static/agents.js, app/static/remote.js, tests/js/remote.test.mjs, the Runs and Agent match wording in app/templates/settings/sources.html, app/static/jobs.js, app/static/companies.js, the overrides text in app/static/app.js, and the form assertions in tests/test_web.py.

  • The agent form follows Settings in architecture.md:
    • a fetch gate block in three parts: Region (scope), Place (countries, towns, remote), Role (fetch terms, titles to leave out);
    • the feed order weights and the CV match weights;
    • the cut: the best N by fit, each at least X; the best M by CV closeness not already in, each at least Y; liked and saved roles always in;
    • summarise the top N of the feed.
  • Hybrid and prefer leave the form. The card lines follow the new limits.
  • The tooltips say what All countries (place part) and All roles (role part) bypass.
  • The Runs page names fetch text.

WP4: Roles feed (Opus), wave 2

Owns app/templates/roles.html, app/static/roles.js, app/static/app.css, app/static/chips.js.

  • A rail note under Viewing as says the feed is already filtered by the agent's fetch gate, places included.
  • Show gated is offered in every order when an agent is chosen.
  • Saved filters in the rail, per agent: save the current filters under a name, apply with one click, delete. Patterns come from ../news (the #active pills).
  • The company strip says which parts of the fetch gate apply to the company, and no longer says that a favourite skips the places.

Done when

uv run pytest and node --test tests/js/*.test.mjs pass on main, every statement in architecture.md holds, the reasoning is folded into decisions.md, and this plan is deleted. The live server picks the change up on Wolf's restart. Migration 32 runs then, and the first pull judges the whole cut (about an hour).