Fold the hosts plan into the decisions, and plan the readers that are left
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
b3f9bfcd9d
commit
7785e62b0d
9 files changed
+74
-949
No files matched your search
@@ -96,6 +96,20 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**A board that fails on every fetch for 21 days closes its roles.** A board whose fetch raised was only logged, so a company that moved ATS kept its roles open for good behind a 404. The `ats` plugin now records `ats:failing:<target>` in `state` with the time of the first failure in a row and the latest reason, and a fetch that answers removes it. Once the first failure is `STALE_DAYS` old, the same window a search role gets, the settle closes the board's roles; the next answer reopens what it lists through `upsert_role`, as for a board switched back on. The record is plugin state rather than a column because the plugin only sees `self.state`. `/api/companies` returns it as `failing_since` and `failing_reason`, the Companies list shows "not answering since", and `app companies check` prints it beside its live result.
|
||||
|
||||
**A host is a module in `plugins/atshosts/`, and registers by existing.** `plugins/ats.py` held eight hosts in six parallel tables and `app/jobs/discover.py` three more, so two people adding a host edited the same lines. Each host is now one file that states its name, label, URL patterns, slug check, allowed host names and either a feed and a parser or a pager; `ats.py` and `discover.py` build their tables from the package, ordered by the module's `RANK` so the select and the probes kept their order. The package sits beside `ats.py` and is not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module. The contract is the docstring of `plugins/atshosts/__init__.py`, and a module that breaks it fails the import by name.
|
||||
|
||||
**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
|
||||
|
||||
**A description stated only on the role's own page is read once, 200 roles per board per pull.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. A module's `detail` is called for roles not yet read, those never tried before those that failed, and the ids read are kept in `ats:detailed:<target>`; a known role is sent without a description on later pulls, which `upsert_role` ignores. `detail` may fill `DETAIL_FIELDS` only (description, location, remote, currency, salary_min, salary_max, posted), the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing, since the listing rewrites them every pull. Completeness is decided by the listing alone, so a board of 800 roles closes its roles from the first pull and has all its text after four.
|
||||
|
||||
**robots.txt is honoured, by RFC 9309.** A module names the page a person opens, and a board whose site disallows that page for `*` or for the app's own token fails with "robots.txt disallows it" and closes its roles after 21 days like any failing board. The matcher takes the longest rule, an Allow on a tie, `*` and `$`, and case-sensitive paths, or paths without case for a host that serves any case, as Workday does. A 4xx means no rules; a 5xx, a 429 or a timeout fails the board for that pull. `urllib.robotparser` was replaced because it takes the first rule and knows no wildcard, which refused Infineon and let `/*company` through. This keeps Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy and Umbraco off the automatic route. The SuccessFactors reader uses `/search/` and `/job/` only, never `/services/` or a `career*.successfactors.eu` back end.
|
||||
|
||||
**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key the page's JavaScript carries, which can change without notice and marks an endpoint that is not open. They are on the Watchlist with reason `key`.
|
||||
|
||||
**A company has one board, and a company inside its owner's board is read by hand.** DFDS has eleven Teamtailor boards by country and is read from the Danish one. Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio and Native Instruments list their roles in Entrust's, Visa's, Delivery Hero's, NICE's, Scrive's, SAP's and inMusic's boards, where no field separates them reliably; they carry reason `group`.
|
||||
|
||||
**The seed's gap was mostly discovery, not parsers.** A survey of the 155 companies without a board (2026-09-29, every careers site opened) found 49 on a host the app already read: the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain or on Lever's EU instance (`lever:eu/<site>`), or the careers page refuses a plain request. 43 were attached by hand from the survey, 46 more with the fourteen hosts added that day, and 286 of 362 companies have a board. Discovery keeps a slug's case for a module with `KEEP_CASE`, since Workday's site names, Gem's, Deel's and Rippling's slugs are case-sensitive; SmartRecruiters and Ashby are not, checked live.
|
||||
|
||||
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs.
|
||||
|
||||
## Ranking
|
||||
@@ -154,6 +168,8 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**Watchlist is a third tab: the companies the app cannot read, rated yes or no.** The owner goes through the careers pages of such companies by hand and wanted that list one click from Roles rather than behind the cog, so `TABS` has three entries and the phone bar's tabs are tighter. A company is on it when it has no board subscription and is not known from a search only, or when its board is failing: the rule of the Company boards filter "Not readable yet", read from `GET /api/companies`. The rating is `companies.watch` (migration 15: NULL unrated, 1 on the list, 0 not), global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. A row rated under Unrated leaves the list at once. On a phone the search field of Roles is left a few letters wide by the third tab, so while it has the focus the tabs and the queue chip step aside.
|
||||
|
||||
**A company on the Watchlist says why it is there, and can be added, edited and removed on the tab.** `companies.hand_reason` (migration 16) is one of `unread` (readable, no module yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone` and `added` (by hand or by an import, not looked at); attaching a board clears it. `POST`, `PATCH` and `DELETE /api/companies` and `app companies add`, `edit`, `checked`, `remove` and `by-hand` are what the LinkedIn import will write through. A company with a reason is never "known from a search only", so one added with a name alone is listed. No second table and no `checked_at`: a company has one careers link, and `careers_opened_at` already says when somebody last looked. A careers link somebody typed is kept: discovery and the seed fill it only when empty. A name is changed only while the company has no board, since the subscription is found by the key the name folds to.
|
||||
|
||||
**Settings are three sub-tabs: Search agents, Sources, Account.** Six sub-tabs were three setup pages and three system pages for two things, what the app ranks by and where roles come from, so each is one page now and Account stays its own.
|
||||
|
||||
**The old paths' 301s are gone.** `MOVED` answered 301 for the twelve paths above, kept for the few days between the move and this cut, on the assumption that a bookmark should still land. A browser caches a 301 indefinitely, so a path a future page reused would be unreachable through no fault of that page; a one-person app with no outstanding bookmarks does not need to carry that risk.
|
||||
|
||||
Reference in new issue
Block a user