Files
jobboard/documentation/ADR Plugins.md
T

16 KiB

ADR Plugins

What a plugin has to deliver so that the flow in ADR Fetch prep and score gets every role a source lists, and what is deliberately left to the rest of the app. A new plugin is written, and an existing one tested, against this document. The API at the bottom is enough to write one.

Terms

  • Source: a site that lists roles.
  • Plugin (reader): the code for one kind of source. There are two kinds:
    • a host, which reads a company's own list on an ATS (Workday, Greenhouse, ...). That list is a board, attached as a target host:slug, e.g. workday:lego.wd103/LEGO_External.
    • a provider, which runs a search, a saved query on a site that lists roles across companies (Jobindex, Jobnet, ...).
  • Runner: the shared code that calls a plugin. It asks, pages, retries, guards and stamps, so no plugin repeats that.
  • Row: one role as a plugin hands it over, in the host's own words.
  • Declaration: what a plugin states it can do (READS). The runner acts on the declaration, never on guesses.

1 What a plugin delivers

A plugin turns one source into rows. It:

  • hands over every role the source lists for what it was asked, as one row per role the host lists;
  • asks the host only for the filters it was handed, by building them into the host's query;
  • keeps the host's own words for places, remote, seniority, employment and country;
  • declares truthfully what the source can do: how the list arrives, where the text is, how completeness is known, what it can filter and which fields it states.

A plugin never:

  • filters, drops or reorders rows itself;
  • maps a word to the app's vocabulary;
  • merges, splits or multiplies rows;
  • decides closing, retries or failure;
  • reads robots.txt at run time.

2 The five cases

A source can present roles in five ways that need handling. Three are the plugin's job and two are handled upstream.

# Case Who handles it
1 The list arrives in pages Plugin, with the runner
2 The list has no text; each role's text is on its own page Plugin, with the fetch text stage
3 One role states several places Plugin
4 One job is listed several times Upstream
5 Facts such as remote are only in the free text Upstream

2.1 Pagination (plugin, with the runner)

The plugin says how to ask for one page and what the page holds. The runner does the rest.

  • The plugin declares listing="paged", the page size and the host's ceiling, the offset past which the host answers nothing whatever is asked (Workday 2,000).
  • page asks for one page from a cursor and returns the page's rows, the raw results with their ids, the total the host states if any, and the next cursor, or none after the last page.
  • The runner reads the pages one after another, 0.5 s apart, at most 100 pages a pull. A board above 100 pages is swept: each pull continues from the saved cursor.
  • Completeness: a read is complete when the distinct ids read equal the stated total (complete="stated"), or when it reaches an empty page (complete="end"). A read that reaches the ceiling is incomplete, is not swept and closes nothing; only a filter brings such a board under the ceiling.
  • A role of a swept board closes only when two finished sweeps in a row missed it, since roles closing between pulls shift later roles back past the cursor.
  • A 429 after the first page ends the read. What was read is kept, and the read is incomplete.

2.2 List without text (plugin, with the fetch text stage)

Some hosts list title, place and date only, and state the text on each role's own page.

  • The plugin declares text="detail" and supplies detail, which reads one role's page and returns the fields it found. A list that carries a short excerpt hands it over as teaser and declares teaser=True.
  • The plugin does not decide which roles are read. The fetch text stage calls detail after scoring, only for roles in the cut (ADR Fetch prep and score, section 3), at most 200 per board and 1,000 per run.
  • detail may fill only description, location, places, teaser, remote, country, employment, pay figures, currency and the posting date. Title, link, department and the pay line come from the list, which states them on every pull.
  • A role read in detail is sent by later pulls without the list's place and remote, so what its own page said stays.
  • A 404 or 410 counts as read. Another failure is retried on a later run. A read that gave no text is retried after 7 days. A 429 ends the reads of that board for the run.
  • Completeness, and so closing, is decided by the list alone.

2.3 One role, several places (plugin)

  • A row carries places: every place the host states for the role, in the host's order and the host's words, each marked remote, office or unsaid.
  • A plugin never makes one row per place, never drops a place and never joins places into one text.
  • A host that states no place declares places="none" and hands over an empty list.
  • Resolving a place text to a town, region or country is the app's job.

2.4 One job listed several times (upstream)

One job can come as several rows: one per office, each with its own id; several requisition ids for the same job; or a search provider and the company's board listing the same role.

  • The plugin hands every listing over as its own row, with its own id and link. A host that publishes one posting per place declares places="copies".
  • Folding them into one role, and choosing the copy that is kept, happens upstream (ADR Fetch prep and score, section 1.4).

2.5 Facts in the free text (upstream)

  • The plugin passes on only what the host states in a field: remote, country, seniority, employment, pay and date, in the host's words, and declares each as stated or none.
  • Mapping the host's words to the app's vocabulary happens upstream, through a table per host.
  • Facts found only in the prose, e.g. a Copenhagen role that is actually remote, are read upstream: from the place words first, then by extract (ADR Fetch prep and score, section 4.1).

3 Further rules

3.1 Filtering at the host

  • The plugin declares per dimension (country, place, remote, query) whether the host filters it: none, one (one value a request), many (a list in one request) or ids (the host's own ids, looked up with facets).
  • The runner hands the plugin an Ask that holds only declared dimensions. With one, it runs one series of requests per value and unions the rows by id. A read costs at most 8 series. Above that the runner narrows less: a read for the location gate keeps the places and drops the fetch terms; a read of an All locations company keeps the terms and drops the place part; more than 8 terms alone are dropped.
  • "In these countries, or remote" cannot be asked in one request, because two facets intersect. The runner makes remote a series of its own, so a plugin never gets remote and a country in the same Ask.
  • A host with query="one" also takes the fetch terms, one term a series, handed over as query.
  • A host's facet ids are looked up at most once a day per board.
  • Which boards are filtered at all, and by what, is decided upstream (ADR Fetch prep and score, section 1.2).

3.2 Completeness and closing

The plugin declares how completeness is known (stated, end, whole for a one-answer list, never). The runner stamps each read as complete or not. Closing follows ADR Fetch prep and score, section 1.3. A search is never a company's whole list, so it declares never or stated.

3.3 Failures

  • A plugin lets every error through: HTTP errors, timeouts, parse errors. The runner logs, records and decides.
  • A read that lists under 10% of what the same read listed last time, once that was at least 10 roles, counts as failed. This catches a host that answers an empty list for a mistyped board; such a host declares empty="suspect".
  • A board that fails on every read for 21 days closes its roles, and a suspect answer that has stood that long is accepted.

3.4 Fetching

Every request goes through the runner's get:

  • https on port 443 only, never to an IP address or a local name;
  • only to the host names the plugin allows for that slug;
  • at most 3 redirects, each checked against the same allow-list;
  • 10 s to connect, 30 s between bytes, 60 s for the whole request;
  • at most 96 MiB a response. A plugin may name a smaller fallback feed, read once when the main one is over the cap.

3.5 Searches

A provider has saved params instead of a board. It declares which filter dimension each param fills, cleans a param it would not accept with a reason a person can read, and receives the saved params in Ask.params. A search row carries company, the employer's name. A provider has no slug patterns, probes or careers page, and takes no fetch terms: its query replaces them. A provider that needs an API key declares it, and the runner hands it the key.

3.6 Discovery

A host can be found from a careers page link: its slug patterns are regexes over a URL, with the slug in group 1, or several groups joined by /. A host may offer a cheap probe URL that answers 2xx only for a real board, and a check of the probe's body where 2xx is not enough. The slug is lowercased unless the host's paths are case-sensitive.

3.7 robots.txt

Whoever builds a plugin reads the robots.txt of every host it requests, for the API paths it requests, and the owner decides whether the host is read. The declaration records what the file says, the date it was read and the decision. A record without a date is refused. Nothing reads robots.txt at run time.

3.8 Rows

  • source_id is the host's id, unique on the board or search. title and url (http or https) are required.
  • description and teaser are plain text, never HTML.
  • posted is an aware datetime or none. posted="trusted" declares it the posting date; listed declares a date the host moves, e.g. on an edit.
  • Pay figures are yearly integers with an ISO 4217 currency. A monthly figure is multiplied by 12 where the host states the period. An hourly, daily or weekly figure is kept as a pay line only.
  • country is ISO 3166 alpha-2, and only where the host states it.
  • Any other key the host states is kept as raw, so a new field needs no change in the runner.

API

A plugin is one module, named after its source, importing only the shared plugin helpers, the standard library, httpx and pydantic. It registers by existing: a module that breaks the contract fails at import, naming every problem.

Module attributes

NAME = "workday"          # the host of a target `workday:<slug>`; equals the module name
LABEL = "Workday"         # shown in the source select
PATTERNS = (r"...",)      # host only: discovery regexes over a URL, IGNORECASE
RANK = 100                # optional: order in the select and in probes, lower first
KEEP_CASE = False         # optional: True keeps a discovered slug's case
PARAMS = {"q": "query"}   # provider only: saved param -> filter dimension it fills

READS = Reads(
    scope="board",            # "board" | "search"
    listing="paged",          # "whole" | "paged"
    page_size=20,             # paged only
    ceiling=2000,             # paged only: offset past which the host answers nothing
    text="detail",            # "listing" | "excerpt" | "detail" | "none"
    teaser=False,             # with "detail": the list also carries a teaser
    complete="stated",        # "stated" | "end" | "whole" | "never"
    places="many",            # "one" | "many" | "copies" | "none"
    filters=Filters(country="ids", place="none", remote="ids", query="one"),
                              # each "none" | "one" | "many" | "ids"
    fields=Fields(pay="none", posted="trusted", remote="stated",
                  country="none", seniority="none", employment="stated"),
                              # pay "figures" | "line" | "none"; posted "trusted" | "listed" | "none";
                              # the rest "stated" | "none"
    key=None,                 # Key(name, fields, env) when an API key is needed
    empty=None,               # "suspect" when an unknown slug answers an empty list
    robots="read: ... (checked 2026-09-30)",   # required, with the date
)

Functions

# Every host
def slug_ok(slug: str) -> bool                   # full match; refuses anything that adds a path,
                                                 # query, fragment, port or user to a URL
def allowed(slug: str) -> Callable[[str], bool]  # the host names a request may reach
def careers_url(slug: str) -> str                # the page a person opens

# listing="whole"
def feed_url(slug: str, ask: Ask) -> str         # the one request
def parse(body: bytes, slug: str) -> list[Row]
def fallback_url(slug: str) -> str               # optional: a smaller feed when over 96 MiB

# listing="paged"
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page   # first call: cursor None
def rows(decoded: Any, slug: str) -> list[Row]

# text="detail"
def detail(get: Get, slug: str, row: Row) -> dict[str, Any]    # fields from the role's own page

# any filter declared "ids"
def facets(get: Get, slug: str) -> dict[str, dict[str, str]]
                                                 # {"country": {"DE": "<host id>"}, "place": {...},
                                                 #  "remote": {...}}; a missing dimension is read
                                                 # unfiltered

# Every provider
def clean_params(params: dict[str, Any]) -> dict[str, Any]     # raises ValueError with a reason

# Discovery, optional
def probe_url(slug: str) -> str                  # 2xx only for a real board
def probe_ok(body: bytes) -> bool                # when 2xx is not enough
def probe_slugs(guess: str) -> tuple[str, ...]   # slugs a guessed company name is tried as

Types

get(url, *, method="GET", payload=None, headers=None) -> bytes
    # POST sends payload as JSON. Raises httpx.HTTPStatusError on non-2xx,
    # httpx.HTTPError on timeout or connection failure, ValueError on a refused
    # host, an oversized body or a broken redirect. A plugin lets these through.

Ask(countries=(), places=(), remote=None, remote_ids=(), query=None, terms=(), params={})
    # only declared dimensions are set; with "one" a series carries one value;
    # with "ids" countries and places hold host ids; never remote together with
    # a country or place; terms are handed over one a series as query, so a
    # plugin reads query only; params holds a search's saved params and a key under "key"

Page(decoded, results, total, cursor)
    # decoded: handed to rows; results: raw dicts with an "id", counted by
    # distinct id; total: stated by the host, or None; cursor: next page, or None

Place = {"text": str, "remote": bool | None}
    # the host's words; True remote, False office, None unsaid

Row = {
    "source_id": str, "title": str, "url": str, "places": list[Place],  # required
    "location": str, "posted": datetime | None, "department": str,
    "description": str, "teaser": str, "remote": bool | str | None,
    "salary_text": str, "salary_min": int, "salary_max": int, "currency": str,
    "country": str, "seniority": str, "employment": str,
    "company": str,                                # search rows only
}

Helpers the plugins share

  • place(text, remote=None) makes a Place, remote when the text says remote; add_place(found, entry) appends one unless it is empty or already there.
  • html_text(html) turns HTML into plain text; parse_dt(value) makes an aware datetime; yearly_amount(...) makes a yearly figure; is_web_url(url) checks a link.
  • path_slug_ok(slug) and label_slug_ok(slug) check a slug that is one URL path segment or one DNS label; only(host) is the usual allowed.
  • A table of the country names hosts write.

Adding a plugin

  1. Write the module with its declaration and functions.
  2. Add a table of the host's words for seniority, employment and remote, when the host states them in its own words.
  3. Add fixtures from real answers, and tests. A contract test checks each declared field against the fixtures: a field declared stated must show on some row, a field declared none on none.
  4. Record the robots.txt decision in the declaration.

Nothing else changes: the list of plugins, the source select, discovery and the filters a board is read with all follow from the module.