seed/companies.json: 179 companies from jobhunt-hq's seed.json (MIT), reduced to key/name/board/careers_url, plus a hand-added DACH set. `roledesk companies seed|add|check` import it, add one company from its board with a live check first, and report dead boards. roledesk/jobs/pull.py is now a pipeline task: ten boards per pull_batch job on the http resource, upserting every role the ats plugin yields. Its final job's settle closes every board this run answered in full and no longer lists a role from, reading the plugin's own self.state stamps rather than each batch job's return value, so one bad board in a batch does not cost the other nine their closure. `roledesk job pull` runs it, and it registers itself in pipeline.TASKS so <desk-jobs> can start it too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
54 lines
5.9 KiB
Markdown
54 lines
5.9 KiB
Markdown
# Key decisions
|
|
|
|
What was decided for roledesk and the one reason for each. The longer reasoning is in `../news/docs/JobBoardPlan/roles-build-plan.md` and `jobhunt-hq-assessment.md`.
|
|
|
|
## Shape
|
|
|
|
**Own repo, depending on news for `desk` and `pipeline`.** `pyproject.toml` takes `newsdesk` as an editable path dependency on `../news`. `desk` imports no app and news's architecture test enforces it, so it works as a library. Copying it would fork the plugin contract; `pipeline` alone may be copied later if depending on a repo named news becomes the problem.
|
|
|
|
**`desk` and `pipeline` are never edited here.** A change roles needs lands in news, and roledesk picks it up through the editable install.
|
|
|
|
**The domain noun is `role`.** `jobs/` keeps meaning the work the app queues, so "job" never has two senses in one sentence.
|
|
|
|
**Own database, queue and port.** `roledesk.db`, `roles.yaml`, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently now that `Jobs` takes a path.
|
|
|
|
**Storage is written, not imported.** `desk` has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`.
|
|
|
|
**Layers are `models, config, storage, jobs, web, cli`.** `tests/test_architecture.py` holds the direction, the doors, and that a plugin imports `desk` only, since a plugin is app-agnostic.
|
|
|
|
## Data
|
|
|
|
**Clustering key is `(company_key, title_key, place_key)`.** newsdesk's `location_key OR title_key` collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.
|
|
|
|
**`place`, `country` and `remote` are separate columns.** `location` already means where an `Item`'s original lives. `remote` alone over-matches, because boards write "UK | Remote", so a gate on remote also reads `country`.
|
|
|
|
**`raw` is transport, columns are storage.** No `Item.facets`. A plugin puts structured fields in `raw` under the column's name and `upsert_role` unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what `extract` found.
|
|
|
|
**Postings are persisted.** Storing every posting is what gives `first_seen`, which is the only honest posted date since Greenhouse resets `updated_at` on every edit, and `closed_at`, set by `mark_closed` when a complete fetch of a board no longer lists a role. A role listed again is open again.
|
|
|
|
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with status in `saved, applied, screen, technical, onsite, offer, rejected, withdrawn`, `next_action`, `next_date`, and an append-only `application_events` written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.
|
|
|
|
**`subscriptions` names its labels `company` and `board`.** newsdesk's `outlet` and `section` are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.
|
|
|
|
## Ingestion
|
|
|
|
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** The `ats` plugin stamps `ats:ok:<target>` with the run's own marker and `ats:ids:<target>` with the ids it saw on every board it parses without raising. The `pull` task's settle then closes a board only when its stamp matches the marker the finder wrote before any batch ran. A batch job that raises part way through a ten-board chunk still leaves the boards it had already finished stamped, so one bad board does not cost its batch-mates their closure — and a board a batch never got to (the whole job crashed, or was never claimed) stays open rather than being closed on an empty id list.
|
|
|
|
**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department but not the job text; that is only on the per-posting detail endpoint, and fetching one extra request per role would multiply a board's request count by however many roles it lists. `description` stays empty for this host; title, location, department and the derived `function` are still filled.
|
|
|
|
**Workable's shape is confirmed, not a populated example.** Every slug tried against `apply.workable.com/api/v3/accounts/{slug}/jobs` during this build returned the documented `{"total": 0, "results": []}` before Cloudflare rate-limited further requests from this session, so the parser is written from Workable's long-published job object rather than from a fetched row. Worth a fixture-level recheck against a live board with open roles the next time this plugin is touched.
|
|
|
|
## Ranking
|
|
|
|
**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the gates and floors the score, `-1` mutes.
|
|
|
|
**`fit` is cosine between the role and the agent's intent text.** The intent is a CV or a paragraph per agent. This is why the embedding backend is effectively required: TF-IDF cannot relate "Kubernetes" to "container orchestration".
|
|
|
|
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and skips rows a plugin already filled from structured fields.
|
|
|
|
## Outside the app
|
|
|
|
**Unposted roles and the network stay outside.** Their state is inside: an application row with no `role_uid`, a `contact`, `next_action` and `next_date`, and a company `status` such as researching or contacted.
|
|
|
|
**LinkedIn enters only as alert mail over IMAP.** Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.
|