The Roles tab takes newsdesk's shell, ported from its index.html and
app.css rather than rewritten: a filter rail (Show read, Best first or
Newest first, first seen, tag chips with counts, the old filters behind a
disclosure), the feed in newsdesk's row layout (score gutter, title,
company, place, badges, age, tags, the summary as excerpt, judge buttons)
and the reader as a third column that replaces the list on a phone. The
router pushes /roles/{uid}, which the server also renders with the reader
open, so the separate read page and role.js go. j/k, u/n/d, S, Enter,
Escape, o, /, f and c work as in newsdesk. The phone rules come with their
comments.
Two things the row lacked: roles.read_at, stamped when the reader opens a
role or a vote is cast, behind include_read on /api/roles; and role_tags,
written by summarise alongside the summary and served with counts by
/api/filters, filtered in SQL before LIMIT. Migration 5.
PyCharm run configurations in .run/ for serve, pull, seed and pytest.
core.autocrlf=input is set locally so a Windows-side checkout commits LF.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
134 lines
19 KiB
Markdown
134 lines
19 KiB
Markdown
# Key decisions
|
|
|
|
What was decided for roledesk and the one reason for each. The longer reasoning is in `../news/docs/JobBoardPlan/roles-build-plan.md` and `jobhunt-hq-assessment.md`.
|
|
|
|
## Shape
|
|
|
|
**Built on the boilerplate, with `desk` copied from news.** `pipeline/` (queue and scheduler, pipeline 2.0), `auth/` and `deploy/` come from `../boilerplate` and `desk/` from `../news`, all four as downstream copies refreshed with `git archive`. A fix lands upstream first, or the next refresh reverts it. The package is `app`, per the boilerplate rule, since the deploy kit and the `APP_*` variables assume it.
|
|
|
|
**The domain noun is `role`.** `jobs/` keeps meaning the work the app queues, so "job" never has two senses in one sentence.
|
|
|
|
**Own database, queue and port.** `roledesk.db`, `config.yaml`, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently.
|
|
|
|
**Storage is written, not imported.** `desk` has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`.
|
|
|
|
**Layers are `models, config, storage, jobs, web, main, cli`.** `tests/test_architecture.py` holds the direction and the doors within `app`; `tests/test_coupling.py` holds that `pipeline`, `auth` and `desk` import nothing from `app` or each other, except `desk` from `pipeline`, and that a plugin imports `desk` only.
|
|
|
|
## Data
|
|
|
|
**Clustering key is `(company_key, title_key, place_key)`.** newsdesk's `location_key OR title_key` collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.
|
|
|
|
**`place`, `country` and `remote` are separate columns.** `location` already means where an `Item`'s original lives. `remote` alone over-matches, because boards write "UK | Remote", so a gate on remote also reads `country`.
|
|
|
|
**`raw` is transport, columns are storage.** No `Item.facets`. A plugin puts structured fields in `raw` under the column's name and `upsert_role` unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what `extract` found.
|
|
|
|
**`extract` fills, the board wins.** `set_extracted` writes a model's value only into an empty column. A board's structured field is first-party, and `upsert_role` would put it back on the next pull anyway, so letting the model replace it would make a column flip between two answers. `place` brings `place_key` with it, which only ever moves a role from the empty place key to a real one.
|
|
|
|
**`extract` reads the board's location too.** The model gets title, the board's `place` text, and the first 3000 characters of the description. Greenhouse's location line ("Berlin, Germany") is often the only place a country is stated, and it is part of the posting, so the "only what the posting states" contract still holds.
|
|
|
|
**`country` is filled from `place` by a lookup, before `extract`.** `country_of` maps the folded location line to an ISO code from a table of European countries and cities, plus the non-European places that fill the seed boards, and returns nothing when a line names two countries. It resolves about 90% of the seed boards' roles for free, where `extract` would be a paid call per role for the same answer. `EU`, ISO's reserved code, stands for "Remote - Europe" and "EMEA", so a Europe gate is a list of codes that includes it. A US state after a comma counts only when no name matched, because DE and IN are countries too.
|
|
|
|
**`permit` and `languages` are columns, and so is `extract_tries`.** `permit` answers the plan's open item 1 (visa sponsorship as a gate field) while nothing depends on it yet. newsdesk keeps give-up counts in `raw`; here `upsert_role` rewrites `raw` on every pull, which would reset the count, so the counter is its own column.
|
|
|
|
**Postings are persisted.** Storing every posting is what gives `first_seen`, which is the only honest posted date since Greenhouse resets `updated_at` on every edit, and `closed_at`, set by `mark_closed` when a complete fetch of a board no longer lists a role. A role listed again is open again.
|
|
|
|
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with status in `saved, applied, screen, technical, onsite, offer, rejected, withdrawn`, `next_action`, `next_date`, and an append-only `application_events` written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.
|
|
|
|
**`subscriptions` names its labels `company` and `board`.** newsdesk's `outlet` and `section` are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.
|
|
|
|
## Ingestion
|
|
|
|
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** The `ats` plugin stamps each board it parses without raising with the run's marker and the ids it saw, and the pull's settle closes a board only when its stamp matches the marker written before any batch ran. So a batch that raises part way still closes the boards it finished, and a board no batch reached stays open instead of being closed on an empty id list.
|
|
|
|
**Boards follow companies; searches find them.** `ats` only reads companies someone listed, so it cannot surface a role at a company nobody knew of, and the seed list came from a US-centred project: of 29,000 open roles, 31 were in Denmark. The `search` plugin runs saved queries on boards that span companies, as `search` subscriptions whose `params` are the query, and pulls them in the same task as the boards.
|
|
|
|
**Jobindex and Adzuna, not Jobnet or Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jobnet's search endpoint answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
|
|
|
|
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. A search whose results ran out before its page limit did list everything, and closes like a board; the rest close once no search has listed them for 21 days. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists.
|
|
|
|
**Jobindex ads are read from the feed, not followed to the employer.** Each item links to Jobindex's own page, and its `/c?t=` redirect leads to the employer's site or ATS. Following it per role would count as a click in Jobindex's statistics and costs a request per role, so discovering a company's ATS board from a search result is left for later.
|
|
|
|
**`app companies prune` cuts the list by where it hires.** A board none of whose open roles is in the region (`--in`, Europe by default, or codes such as `DK,SE,EU`) is switched off. A board whose roles name no country is kept, because it says nothing about where the company hires.
|
|
|
|
**Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch already in flight when the switch happens reopens them again, so the pull's settle closes every switched-off target once more.
|
|
|
|
**The Sources tab and the CLI share `app.jobs.sources`.** Adding a board or a search, switching one, and pruning are one function each, called by `app companies`, `app searches` and `/api/sources*`, so the two cannot disagree about what "off" does. A search or board is run once before it is saved, and refused if it does not answer or finds nothing.
|
|
|
|
**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department; the job text is only on the per-posting endpoint, and one extra request per role would multiply a board's request count by its role count. `description` stays empty for this host.
|
|
|
|
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
|
|
|
|
**Workable's shape is confirmed, not a populated example.** Every slug tried during the build returned the documented empty `{"total": 0, "results": []}` before Cloudflare rate-limited the session, so the parser follows Workable's published job object. Recheck it against a live board with open roles the next time the plugin is touched.
|
|
|
|
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit on the Schedules tab overrides for good. `pull` is not also a task, which would draw two buttons on the Jobs tab.
|
|
|
|
## Ranking
|
|
|
|
**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the gates and floors the score, `-1` mutes.
|
|
|
|
**`fit` is the cosine between a role and the agent's intent, min-max scaled across the roles that agent scores in the run.** `(cos + 1) / 2` spanned only about 0.55 to 0.79 in the real run, so a prefer gate at 0.2 outweighed all of fit. A single role gets 1, gated roles stay out of the range, and the starter agents' prefer weights are 0.05.
|
|
|
|
**The embedding backend is effectively required.** The intent is a CV or a paragraph per agent, and TF-IDF cannot relate "Kubernetes" to "container orchestration".
|
|
|
|
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty.
|
|
|
|
**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 and the 0.9 floor for a favourite means the same thing whatever the weights are.
|
|
|
|
**Gates match loosely where the column is free text.** `place` is a case-insensitive substring, so `Berlin` matches "Berlin, Germany"; `company` compares company keys, `languages` passes when every required language is a value, `salary_min` is at least the smallest value, and the rest are case-insensitive equality. Each gate is one SQL `CASE` column, so a require gate filters while the gated row keeps every failed gate as its reason.
|
|
|
|
**A role with no vector is scored without `fit`.** The fit weight leaves the denominator, so a role `embed` has not reached yet ranks on recency, affinity and prefer gates instead of sinking to the bottom. It is gated as "no vector" only when fit is the only weighted term, and as "no intent vector" when the intent could not be embedded.
|
|
|
|
**Each run replaces an agent's scores.** `replace_scores` deletes and inserts in one transaction, so a role closed or muted since the last run drops out rather than keeping a stale score, and a reader sees one list or the other.
|
|
|
|
**Starter agents are imported once, by a state marker.** newsdesk seeds whenever its table is empty. Here `agents:seeded` in `state` is set after the import, so deleting every agent survives a restart.
|
|
|
|
**`embed` and `score` run in the server process.** newsdesk isolates `embed` in a child process with a timeout. roledesk's run is one bounded pass over open roles, and in-process keeps the embedder replaceable in tests. The cost is torch's memory staying in the server after the first run; isolate it if that becomes the problem.
|
|
|
|
**The pull cascade is `pull → extract → embed → score`.** Every stage is always in the chain and a stage that is off finds nothing, as in newsdesk. Extract precedes score so the gates read what it filled, and reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly.
|
|
|
|
## The pages
|
|
|
|
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models` and `plugins.ats.FUNCTIONS`, so they cannot drift from what the store accepts). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change.
|
|
|
|
**`roles(company=, place=)` match a case-insensitive substring of what the board wrote.** The API's company and place fields are free text a person types, not a slug, so "Trade Republic" finds `trade-republic-bank` without knowing the key.
|
|
|
|
**A muted company is excluded in SQL, before `LIMIT`.** Filtering after `LIMIT` shipped first and showed nothing once Databricks' 884 freshly pulled roles filled the newest 200 rows and Databricks was muted.
|
|
|
|
## The phone
|
|
|
|
**Two tabs and a cog.** `TABS` in `app/web/pages.py` becomes Roles and Applications; the other six pages (Agents, Companies, Sources, Jobs, Schedules, Messages, Account) move behind a settings cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and six of them were setup or system pages. The old paths answer 301 to their new place, query string kept, so a bookmark or a `?agent=` link still lands.
|
|
|
|
**One ranking in the UI, while the `agents` table keeps every row.** The owner read "search agent" as something that searches a board, which is what a saved search does, so `/settings/ranking` edits only the first enabled agent and creates one when the table is empty. Code, API, CLI and config keep `agent`, and nothing is migrated.
|
|
|
|
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest.
|
|
|
|
**A vote moves the rank at read time and never hides.** `Store.roles` multiplies the served score by the vote (`0.5` passed, `1.2` liked, capped at 1) instead of triggering a rescore, so a vote's effect on Best match shows at once and needs no job. A vote never removes a role from the list either, because a stray tap must stay visible and a single further tap must be able to undo it.
|
|
|
|
**Read state is a column on `roles`, and a vote stamps it too.** `read_at` sits beside `vote`, the same shape as newsdesk's `items.read_at`, so the feed filters and counts by it directly rather than joining a side table on every page load. `set_vote` stamps it only when it is still NULL, since voting is reading in newsdesk as well; clearing a vote leaves it alone, because taking back an opinion is not un-reading the posting.
|
|
|
|
**Tags are a table, written only by `summarise`.** `role_tags(uid, tag)` follows newsdesk's `tags` table rather than a JSON array on `roles`, so `tag_counts` and the any/all filter are ordinary SQL (`GROUP BY`, `EXISTS`) instead of scanning every row's array. `summarise` already reads the whole posting once per role, so writing 3 to 8 tags alongside the summary costs no extra model call.
|
|
|
|
**Tag filtering runs in SQL before `LIMIT`, the same as `hide_muted`.** An `EXISTS` or `COUNT(DISTINCT tag)` clause in `_roles_from` keeps the guarantee the muted-company exclusion already established: a filter narrows the rows a page is built from, not the page after it is already built, so a tag matching only older roles is not silently empty on page one.
|
|
|
|
**"Hidden" is the word the UI uses for affinity -1; the code keeps `mute` and `hide_muted`.** Renaming the columns would touch every reader of `agents` and `companies` for no behavioural gain, so only the toggle's title, the filter label and the dimmed row say "hidden" in the UI.
|
|
|
|
**Companies, following a board and pruning boards by region merge into one settings page.** A company's affinity used to live on Companies while following its board and pruning boards by region lived on Sources, so acting on one company took two tabs; now Companies is one list with the follow form on top and the prune behind a details below it.
|
|
|
|
**A company affinity change enqueues `score`.** `POST /api/companies/{key}/affinity` queues the `score` job right after the update, deduped to a live run of the same kind so ten clicks are one pass. Otherwise an affinity change is invisible in Best match until the next 6-hourly pull, and an unmuted company's roles have no score rows at all until then, so they are missing from that order entirely.
|
|
|
|
**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` scrapes `original_url` only when the read page asks for it, because following Jobindex's redirect for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair.
|
|
|
|
**Summarising is bounded to the top `per_agent` roles of each agent, and off by default.** The pull's `summarise` stage only considers roles with no summary yet that are in the top `summarise.per_agent` (default 30) of an enabled agent's scores, so the cost stays flat whatever the backlog of open roles is. `summarise.enabled: false` matches `extract`'s own default; the read page's Summarise button asks for one role's summary either way.
|
|
|
|
**Jobs and Schedules merge into one Pulls sub-tab.** Both named the same thing from two angles, what is running now and when it runs next, so splitting them was two tabs for one job kind.
|
|
|
|
**The schedules element's narrow layout is fixed upstream, in `../boilerplate`.** `<desk-schedules>` is a shadow root the app cannot lay out from outside, and `pipeline/` is a downstream copy refreshed with `git archive`, so a fix landing only in roledesk would be reverted by the next refresh.
|
|
|
|
**The phone CSS is copied from health and news, not written from scratch.** Both apps have already been through several rounds of phone fixes, and the comments in their `app.css` say what each rule avoids, so adapting their fragments keeps those fixes rather than rediscovering the same edge cases, such as iOS zooming on a 16px input or a card losing width to a side gutter.
|
|
|
|
## Outside the app
|
|
|
|
**Unposted roles and the network stay outside.** Their state is inside: an application row with no `role_uid`, a `contact`, `next_action` and `next_date`, and a company `status` such as researching or contacted.
|
|
|
|
**LinkedIn enters only as alert mail over IMAP.** Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.
|