Write the plugins ADR: what a plugin delivers, the five cases and who handles each, and the API
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8f37f51254
commit
9b70f64a20
1 file changed
+256
@@ -0,0 +1,256 @@
|
||||
# ADR Plugins
|
||||
|
||||
What a plugin has to deliver so that the flow in `ADR Fetch prep and score` gets every role a source lists, and what is deliberately left to the rest of the app. A new plugin is written, and an existing one tested, against this document. The API at the bottom is enough to write one.
|
||||
|
||||
## Terms
|
||||
|
||||
- **Source**: a site that lists roles.
|
||||
- **Plugin** (reader): the code for one kind of source. There are two kinds:
|
||||
- a **host**, which reads a company's own list on an ATS (Workday, Greenhouse, ...). That list is a **board**, attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
|
||||
- a **provider**, which runs a **search**, a saved query on a site that lists roles across companies (Jobindex, Jobnet, ...).
|
||||
- **Runner**: the shared code that calls a plugin. It asks, pages, retries, guards and stamps, so no plugin repeats that.
|
||||
- **Row**: one role as a plugin hands it over, in the host's own words.
|
||||
- **Declaration**: what a plugin states it can do (`READS`). The runner acts on the declaration, never on guesses.
|
||||
|
||||
## 1 What a plugin delivers
|
||||
|
||||
A plugin turns one source into rows. It:
|
||||
|
||||
- hands over every role the source lists for what it was asked, as one row per role the host lists;
|
||||
- asks the host only for the filters it was handed, by building them into the host's query;
|
||||
- keeps the host's own words for places, remote, seniority, employment and country;
|
||||
- declares truthfully what the source can do: how the list arrives, where the text is, how completeness is known, what it can filter and which fields it states.
|
||||
|
||||
A plugin never:
|
||||
|
||||
- filters, drops or reorders rows itself;
|
||||
- maps a word to the app's vocabulary;
|
||||
- merges, splits or multiplies rows;
|
||||
- decides closing, retries or failure;
|
||||
- reads robots.txt at run time.
|
||||
|
||||
## 2 The five cases
|
||||
|
||||
A source can present roles in five ways that need handling. Three are the plugin's job and two are handled upstream.
|
||||
|
||||
| # | Case | Who handles it |
|
||||
|---|---|---|
|
||||
| 1 | The list arrives in pages | Plugin, with the runner |
|
||||
| 2 | The list has no text; each role's text is on its own page | Plugin, with the fetch text stage |
|
||||
| 3 | One role states several places | Plugin |
|
||||
| 4 | One job is listed several times | Upstream |
|
||||
| 5 | Facts such as remote are only in the free text | Upstream |
|
||||
|
||||
### 2.1 Pagination (plugin, with the runner)
|
||||
|
||||
The plugin says how to ask for one page and what the page holds. The runner does the rest.
|
||||
|
||||
- The plugin declares `listing="paged"`, the page size and the host's **ceiling**, the offset past which the host answers nothing whatever is asked (Workday 2,000).
|
||||
- `page` asks for one page from a cursor and returns the page's rows, the raw results with their ids, the total the host states if any, and the next cursor, or none after the last page.
|
||||
- The runner reads the pages one after another, 0.5 s apart, at most 100 pages a pull. A board above 100 pages is **swept**: each pull continues from the saved cursor.
|
||||
- **Completeness**: a read is complete when the distinct ids read equal the stated total (`complete="stated"`), or when it reaches an empty page (`complete="end"`). A read that reaches the ceiling is incomplete, is not swept and closes nothing; only a filter brings such a board under the ceiling.
|
||||
- A role of a swept board closes only when two finished sweeps in a row missed it, since roles closing between pulls shift later roles back past the cursor.
|
||||
- A 429 after the first page ends the read. What was read is kept, and the read is incomplete.
|
||||
|
||||
### 2.2 List without text (plugin, with the fetch text stage)
|
||||
|
||||
Some hosts list title, place and date only, and state the text on each role's own page.
|
||||
|
||||
- The plugin declares `text="detail"` and supplies `detail`, which reads one role's page and returns the fields it found. A list that carries a short excerpt hands it over as `teaser` and declares `teaser=True`.
|
||||
- The plugin does not decide which roles are read. The fetch text stage calls `detail` after scoring, only for roles in the cut (`ADR Fetch prep and score`, section 3), at most 200 per board and 1,000 per run.
|
||||
- `detail` may fill only description, location, places, teaser, remote, country, employment, pay figures, currency and the posting date. Title, link, department and the pay line come from the list, which states them on every pull.
|
||||
- A role read in detail is sent by later pulls without the list's place and remote, so what its own page said stays.
|
||||
- A 404 or 410 counts as read. Another failure is retried on a later run. A read that gave no text is retried after 7 days. A 429 ends the reads of that board for the run.
|
||||
- Completeness, and so closing, is decided by the list alone.
|
||||
|
||||
### 2.3 One role, several places (plugin)
|
||||
|
||||
- A row carries `places`: every place the host states for the role, in the host's order and the host's words, each marked remote, office or unsaid.
|
||||
- A plugin never makes one row per place, never drops a place and never joins places into one text.
|
||||
- A host that states no place declares `places="none"` and hands over an empty list.
|
||||
- Resolving a place text to a town, region or country is the app's job.
|
||||
|
||||
### 2.4 One job listed several times (upstream)
|
||||
|
||||
One job can come as several rows: one per office, each with its own id; several requisition ids for the same job; or a search provider and the company's board listing the same role.
|
||||
|
||||
- The plugin hands every listing over as its own row, with its own id and link. A host that publishes one posting per place declares `places="copies"`.
|
||||
- Folding them into one role, and choosing the copy that is kept, happens upstream (`ADR Fetch prep and score`, section 1.4).
|
||||
|
||||
### 2.5 Facts in the free text (upstream)
|
||||
|
||||
- The plugin passes on only what the host states in a field: remote, country, seniority, employment, pay and date, in the host's words, and declares each as `stated` or `none`.
|
||||
- Mapping the host's words to the app's vocabulary happens upstream, through a table per host.
|
||||
- Facts found only in the prose, e.g. a Copenhagen role that is actually remote, are read upstream: from the place words first, then by extract (`ADR Fetch prep and score`, section 4.1).
|
||||
|
||||
## 3 Further rules
|
||||
|
||||
### 3.1 Filtering at the host
|
||||
|
||||
- The plugin declares per dimension (country, place, remote, query) whether the host filters it: `none`, `one` (one value a request), `many` (a list in one request) or `ids` (the host's own ids, looked up with `facets`).
|
||||
- The runner hands the plugin an `Ask` that holds only declared dimensions. With `one`, it runs one series of requests per value and unions the rows by id. A read costs at most 8 series. Above that the runner narrows less: a read for the location gate keeps the places and drops the fetch terms; a read of an All locations company keeps the terms and drops the place part; more than 8 terms alone are dropped.
|
||||
- "In these countries, or remote" cannot be asked in one request, because two facets intersect. The runner makes remote a series of its own, so a plugin never gets remote and a country in the same `Ask`.
|
||||
- A host with `query="one"` also takes the fetch terms, one term a series, handed over as `query`.
|
||||
- A host's facet ids are looked up at most once a day per board.
|
||||
- Which boards are filtered at all, and by what, is decided upstream (`ADR Fetch prep and score`, section 1.2).
|
||||
|
||||
### 3.2 Completeness and closing
|
||||
|
||||
The plugin declares how completeness is known (`stated`, `end`, `whole` for a one-answer list, `never`). The runner stamps each read as complete or not. Closing follows `ADR Fetch prep and score`, section 1.3. A search is never a company's whole list, so it declares `never` or `stated`.
|
||||
|
||||
### 3.3 Failures
|
||||
|
||||
- A plugin lets every error through: HTTP errors, timeouts, parse errors. The runner logs, records and decides.
|
||||
- A read that lists under 10% of what the same read listed last time, once that was at least 10 roles, counts as failed. This catches a host that answers an empty list for a mistyped board; such a host declares `empty="suspect"`.
|
||||
- A board that fails on every read for 21 days closes its roles, and a suspect answer that has stood that long is accepted.
|
||||
|
||||
### 3.4 Fetching
|
||||
|
||||
Every request goes through the runner's `get`:
|
||||
|
||||
- https on port 443 only, never to an IP address or a local name;
|
||||
- only to the host names the plugin allows for that slug;
|
||||
- at most 3 redirects, each checked against the same allow-list;
|
||||
- 10 s to connect, 30 s between bytes, 60 s for the whole request;
|
||||
- at most 96 MiB a response. A plugin may name a smaller fallback feed, read once when the main one is over the cap.
|
||||
|
||||
### 3.5 Searches
|
||||
|
||||
A provider has saved params instead of a board. It declares which filter dimension each param fills, cleans a param it would not accept with a reason a person can read, and receives the saved params in `Ask.params`. A search row carries `company`, the employer's name. A provider has no slug patterns, probes or careers page, and takes no fetch terms: its query replaces them. A provider that needs an API key declares it, and the runner hands it the key.
|
||||
|
||||
### 3.6 Discovery
|
||||
|
||||
A host can be found from a careers page link: its slug patterns are regexes over a URL, with the slug in group 1, or several groups joined by `/`. A host may offer a cheap probe URL that answers 2xx only for a real board, and a check of the probe's body where 2xx is not enough. The slug is lowercased unless the host's paths are case-sensitive.
|
||||
|
||||
### 3.7 robots.txt
|
||||
|
||||
Whoever builds a plugin reads the robots.txt of every host it requests, for the API paths it requests, and the owner decides whether the host is read. The declaration records what the file says, the date it was read and the decision. A record without a date is refused. Nothing reads robots.txt at run time.
|
||||
|
||||
### 3.8 Rows
|
||||
|
||||
- `source_id` is the host's id, unique on the board or search. `title` and `url` (http or https) are required.
|
||||
- `description` and `teaser` are plain text, never HTML.
|
||||
- `posted` is an aware datetime or none. `posted="trusted"` declares it the posting date; `listed` declares a date the host moves, e.g. on an edit.
|
||||
- Pay figures are yearly integers with an ISO 4217 currency. A monthly figure is multiplied by 12 where the host states the period. An hourly, daily or weekly figure is kept as a pay line only.
|
||||
- `country` is ISO 3166 alpha-2, and only where the host states it.
|
||||
- Any other key the host states is kept as raw, so a new field needs no change in the runner.
|
||||
|
||||
---
|
||||
|
||||
## API
|
||||
|
||||
A plugin is one module, named after its source, importing only the shared plugin helpers, the standard library, `httpx` and `pydantic`. It registers by existing: a module that breaks the contract fails at import, naming every problem.
|
||||
|
||||
### Module attributes
|
||||
|
||||
```python
|
||||
NAME = "workday" # the host of a target `workday:<slug>`; equals the module name
|
||||
LABEL = "Workday" # shown in the source select
|
||||
PATTERNS = (r"...",) # host only: discovery regexes over a URL, IGNORECASE
|
||||
RANK = 100 # optional: order in the select and in probes, lower first
|
||||
KEEP_CASE = False # optional: True keeps a discovered slug's case
|
||||
PARAMS = {"q": "query"} # provider only: saved param -> filter dimension it fills
|
||||
|
||||
READS = Reads(
|
||||
scope="board", # "board" | "search"
|
||||
listing="paged", # "whole" | "paged"
|
||||
page_size=20, # paged only
|
||||
ceiling=2000, # paged only: offset past which the host answers nothing
|
||||
text="detail", # "listing" | "excerpt" | "detail" | "none"
|
||||
teaser=False, # with "detail": the list also carries a teaser
|
||||
complete="stated", # "stated" | "end" | "whole" | "never"
|
||||
places="many", # "one" | "many" | "copies" | "none"
|
||||
filters=Filters(country="ids", place="none", remote="ids", query="one"),
|
||||
# each "none" | "one" | "many" | "ids"
|
||||
fields=Fields(pay="none", posted="trusted", remote="stated",
|
||||
country="none", seniority="none", employment="stated"),
|
||||
# pay "figures" | "line" | "none"; posted "trusted" | "listed" | "none";
|
||||
# the rest "stated" | "none"
|
||||
key=None, # Key(name, fields, env) when an API key is needed
|
||||
empty=None, # "suspect" when an unknown slug answers an empty list
|
||||
robots="read: ... (checked 2026-09-30)", # required, with the date
|
||||
)
|
||||
```
|
||||
|
||||
### Functions
|
||||
|
||||
```python
|
||||
# Every host
|
||||
def slug_ok(slug: str) -> bool # full match; refuses anything that adds a path,
|
||||
# query, fragment, port or user to a URL
|
||||
def allowed(slug: str) -> Callable[[str], bool] # the host names a request may reach
|
||||
def careers_url(slug: str) -> str # the page a person opens
|
||||
|
||||
# listing="whole"
|
||||
def feed_url(slug: str, ask: Ask) -> str # the one request
|
||||
def parse(body: bytes, slug: str) -> list[Row]
|
||||
def fallback_url(slug: str) -> str # optional: a smaller feed when over 96 MiB
|
||||
|
||||
# listing="paged"
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page # first call: cursor None
|
||||
def rows(decoded: Any, slug: str) -> list[Row]
|
||||
|
||||
# text="detail"
|
||||
def detail(get: Get, slug: str, row: Row) -> dict[str, Any] # fields from the role's own page
|
||||
|
||||
# any filter declared "ids"
|
||||
def facets(get: Get, slug: str) -> dict[str, dict[str, str]]
|
||||
# {"country": {"DE": "<host id>"}, "place": {...},
|
||||
# "remote": {...}}; a missing dimension is read
|
||||
# unfiltered
|
||||
|
||||
# Every provider
|
||||
def clean_params(params: dict[str, Any]) -> dict[str, Any] # raises ValueError with a reason
|
||||
|
||||
# Discovery, optional
|
||||
def probe_url(slug: str) -> str # 2xx only for a real board
|
||||
def probe_ok(body: bytes) -> bool # when 2xx is not enough
|
||||
def probe_slugs(guess: str) -> tuple[str, ...] # slugs a guessed company name is tried as
|
||||
```
|
||||
|
||||
### Types
|
||||
|
||||
```python
|
||||
get(url, *, method="GET", payload=None, headers=None) -> bytes
|
||||
# POST sends payload as JSON. Raises httpx.HTTPStatusError on non-2xx,
|
||||
# httpx.HTTPError on timeout or connection failure, ValueError on a refused
|
||||
# host, an oversized body or a broken redirect. A plugin lets these through.
|
||||
|
||||
Ask(countries=(), places=(), remote=None, remote_ids=(), query=None, terms=(), params={})
|
||||
# only declared dimensions are set; with "one" a series carries one value;
|
||||
# with "ids" countries and places hold host ids; never remote together with
|
||||
# a country or place; terms are handed over one a series as query, so a
|
||||
# plugin reads query only; params holds a search's saved params and a key under "key"
|
||||
|
||||
Page(decoded, results, total, cursor)
|
||||
# decoded: handed to rows; results: raw dicts with an "id", counted by
|
||||
# distinct id; total: stated by the host, or None; cursor: next page, or None
|
||||
|
||||
Place = {"text": str, "remote": bool | None}
|
||||
# the host's words; True remote, False office, None unsaid
|
||||
|
||||
Row = {
|
||||
"source_id": str, "title": str, "url": str, "places": list[Place], # required
|
||||
"location": str, "posted": datetime | None, "department": str,
|
||||
"description": str, "teaser": str, "remote": bool | str | None,
|
||||
"salary_text": str, "salary_min": int, "salary_max": int, "currency": str,
|
||||
"country": str, "seniority": str, "employment": str,
|
||||
"company": str, # search rows only
|
||||
}
|
||||
```
|
||||
|
||||
### Helpers the plugins share
|
||||
|
||||
- `place(text, remote=None)` makes a `Place`, remote when the text says remote; `add_place(found, entry)` appends one unless it is empty or already there.
|
||||
- `html_text(html)` turns HTML into plain text; `parse_dt(value)` makes an aware datetime; `yearly_amount(...)` makes a yearly figure; `is_web_url(url)` checks a link.
|
||||
- `path_slug_ok(slug)` and `label_slug_ok(slug)` check a slug that is one URL path segment or one DNS label; `only(host)` is the usual `allowed`.
|
||||
- A table of the country names hosts write.
|
||||
|
||||
### Adding a plugin
|
||||
|
||||
1. Write the module with its declaration and functions.
|
||||
2. Add a table of the host's words for seniority, employment and remote, when the host states them in its own words.
|
||||
3. Add fixtures from real answers, and tests. A contract test checks each declared field against the fixtures: a field declared `stated` must show on some row, a field declared `none` on none.
|
||||
4. Record the robots.txt decision in the declaration.
|
||||
|
||||
Nothing else changes: the list of plugins, the source select, discovery and the filters a board is read with all follow from the module.
|
||||
Reference in new issue
Block a user