Plugins#
The router runs custom logic against each request or response through plugins. A plugin can rewrite a request before routing, transform it after a provider is selected, act on the response, or expose its own HTTP endpoints — without editing the core API layer.
This page explains how the plugin system works, lists the plugins currently implemented, and shows how to add and register a new one.
Concepts#
Plugin type (
node). APluginBaseNodesubclass, identified by itsplugin_type()key. This key is thenodeyou write in the configuration and the API.Plugin instance (
name). Each entry underpluginsinconfig.yamlis one instance of a type. A type can have many instances with different settings;nameis the unique instance identifier.Stage (
trigger). Stages in the request lifecycle:prerouting— When the plugin instance runs before provider selection, i.e., before the routing decision is made.postrouting— When the plugin instance runs after provider selection and before the backend call.postresponse— When the plugin instance runs on the response after backend inference.standalone— A non-hook phase: the instance is built (its settings are validated) and its routes are mounted, but it is never run in any request/response chain. Use it for route-only plugins whose work is driven by their own HTTP endpoints rather than the per-request pipeline (e.g.provider_management). Placing such a plugin in a real stage would add a no-op to that chain on every request.
Auto-discovery. Every module under
src/plugins/is imported at startup, so any class decorated with@register_pluginself-registers. There is no central registry to edit when adding a plugin.
Within a stage, instances run sequentially in the order they appear in
config.yaml; each one receives the output of the previous. standalone plugins
are exempt — they do not run in any stage.
Attaching Plugins to Providers#
By default a postrouting or postresponse instance runs for every request. To
run it only for some providers, list its name under those providers’ plugins:
providers:
- name: "local"
# ...
plugins: ["compressor_harness"]
- name: "cloud"
# ...
plugins: ["compressor_harness", "audit_logger"]
- name: "edge" # no `plugins`: runs only unattached plugins
# ...
plugins:
postrouting:
- name: "compressor_harness"
node: "compressor"
# ...
An instance listed by at least one provider runs only for requests routed to those providers. An instance listed by no provider runs for every provider.
Providers listing the same name share one instance (and its state, such as compressor caches and metrics).
The stage order in
config.yamlstill decides the run order; a provider’spluginslist only filters it.Only
postroutingandpostresponseinstances can be attached.preroutinginstances always run for every request, because the provider is not chosen yet; attaching one (or astandaloneone) is rejected at load time, as is an unknown name. A disabled instance may be listed; it simply does not run.A provider stays attached to its plugins while disabled, so disabling it does not make those plugins run for every provider.
The providers API reads and replaces the list through the
pluginsfield. A plugin that a provider lists cannot be deleted or moved topreroutinguntil it is removed from that list.
To attach plugins at runtime, post a provider’s plugins list to the providers
API. The list replaces the provider’s existing one; the other provider fields are
left unchanged, and the change is written to config.yaml:
# attach compressor_harness to "local" and "cloud" (they share one instance)
curl -s -XPOST localhost:8000/v1/providers/local -H 'Content-Type: application/json' \
-d '{"plugins": ["compressor_harness"]}'
curl -s -XPOST localhost:8000/v1/providers/cloud -H 'Content-Type: application/json' \
-d '{"plugins": ["compressor_harness", "audit_logger"]}'
# check what a provider has attached
curl -s localhost:8000/v1/providers/local | jq .plugins
# ["compressor_harness"]
# detach everything from "cloud"
curl -s -XPOST localhost:8000/v1/providers/cloud -H 'Content-Type: application/json' \
-d '{"plugins": []}'
A bad name is rejected with 400 Bad Request and nothing changes:
curl -s -XPOST localhost:8000/v1/providers/local -H 'Content-Type: application/json' \
-d '{"plugins": ["compressor_tool"]}' # a prerouting plugin
# {"detail":"Provider 'local' cannot attach plugin 'compressor_tool' (trigger: prerouting); only postresponse, postrouting plugins can be attached to a provider"}
The Plugin Contract#
The plugin subclass PluginBaseNode in src/plugins/base.py
implements the plugin contract in Inference Router. Only the following two methods are
required; everything else has a safe default, so you override only what you need.
Required:
plugin_type() -> str— the uniquenodekey.settings_model() -> Type[BaseModel]— a Pydantic schema for the instance’ssettings. Settings are validated at construction; invalid configuration is rejected with aPluginSchemaError.
Optional hooks (defaults in parentheses):
init()— set up after the plugin instance’ssettingsare validated; used to build clients, register with shared managers, and etc. (no-op).process_request(request, **kwargs)— acts on the request; returns the possibly modified request (passthrough). Overrides must keep**kwargs: at thepostroutingstage the router passesprovider_name(the provider chosen for the request) andmetrics_values={"provider": provider_name}.process_response(response, **kwargs)— acts on the response (passthrough). At thepostresponsestage the router passes the sameprovider_nameandmetrics_valueskeyword arguments.describe()— theGET /v1/plugins/{node}/{name}payload; folds in per-instance runtime info, typically{**super().describe(), "metrics": {...}}(instance metadata).describe_node()— theGET /v1/plugins/{node}payload; exposes type-wide aggregates spanning all instances (the node metadata).reset()andreset_node()— implementPOST .../resetfor the plugin-instance state and plugin-node state (report “unsupported”, HTTP 400).health_check()— probes backing dependencies (reports “unavailable”).routes()— returns a FastAPIAPIRouterobject to mount under/v1, letting the plugin expose its own HTTP API without modifying the central API layer. This method is called once per plugin type, regardless of instance count. Define plugin endpoints under the/plugins/{node}/...path namespace to avoid collisions (None).
Currently Implemented Plugins#
|
Stage(s) |
What It Does |
|---|---|---|
|
|
Compresses prompts before the backend call to cut token usage. |
|
any (route-only) |
Starts or stops a provider via an external Local Provider Manager; loads/unloads vLLM LoRA adapters. |
|
any |
Logs the stage where the plugin was invoked and serves as a plugin contract reference implementation. |
compressor Plugin#
Source: src/plugins/compressor.py.
Reduces prompt tokens before the request reaches the backend, using the
adaptive-token-compressor
library (bundled in the router image). One compressor node covers every
compressor type; the type is chosen per instance through settings.type:
tool— filters the requesttoolsschema down to a relevant subset using a tool predictor (an OpenAI-compatible LLM endpoint).harness— compresses system or developer messages using a compress server.context— compresses the conversation context.
Caution: The context compressor is experimental and is disabled by default.
How it works:
Compressors act upon requests only, so configure them in the
preroutingorpostroutingstage. At thepostresponsestage, a compressor is a no-op (and logs a warning).All instances share caching and metrics aggregation. Compression runs in a worker thread to keep the event loop responsive.
On any compression error, the request is returned unmodified — a failing compressor degrades gracefully rather than dropping the request.
Metrics:
describe()returns instance metrics;describe_node()also includesoverall.*totals across compressors. Instance responses never includeoverall.*. See Metrics Checking for details.settings.metrics_keys: ["provider"]additionally splits apostroutingcompressor’s metrics by the provider that served the request, reported undermetrics_by_keys.provideris the only supported key, and the setting is rejected at any other stage (atpreroutingthe provider is not chosen yet). See Splitting metrics by provider.A
postroutingcompressor can be attached to providers, so it compresses only requests routed to them. Providers that list the same compressor share one instance; withmetrics_keys: ["provider"], itsmetrics_by_keys.providersplits the numbers per provider.
Configuration example — a tool compressor at the prerouting stage, and
harness and context compressors at the postrouting stage.
node is always compressor; settings.type selects the compressor type:
plugins:
prerouting:
- name: "compressor_tool"
node: "compressor"
enabled: true
settings:
type: "tool"
predictor_url: "http://localhost:8088/v1/chat/completions"
predictor_model: "Qwen/Qwen3.6-35B-A3B"
predictor_backend: "generative"
score_threshold: 2.0
placement: "schema"
accumulate: true
postrouting:
- name: "compressor_harness"
node: "compressor"
enabled: true
settings:
type: "harness"
profile: "openclaw"
compress_server_url: "http://localhost:8001/compress"
compress_rate: 0.5
compress_min_chars: 200
timeout: 60.0
enable_quantum_lock: false
metrics_keys: ["provider"] # split this compressor's metrics by provider
- name: "compressor_context"
node: "compressor"
enabled: false
settings:
type: "context"
protect_recent_tool_turns: 1
large_result_stub: true
large_result_stub_guidance: "legacy"
large_result_min_tokens: 4000
large_result_keep_tool_turns: 1
compression_mode: "none"
postresponse: []
The tool example above uses the large-model predictor (Qwen/Qwen3.6-35B-A3B).
To use the small-model predictor instead, point the tool compressor at a
reranker endpoint and select tools with top_k:
plugins:
prerouting:
- name: "compressor_tool"
node: "compressor"
enabled: true
settings:
type: "tool"
predictor_url: "http://localhost:8091/v1/rerank"
predictor_model: "Qwen/Qwen3-Reranker-0.6B"
predictor_backend: "reranker"
top_k: 10
placement: "schema"
accumulate: true
The two configurations trade off differently: the small model is faster but
saves fewer tool tokens. top_k should be sized to the number of tools per
request. See the
adaptive-token-compressor
ToolCompressor configuration guide for measured results and how to choose K.
The backing services, which are the compress server and tool predictor, are not part of the router — deploy them separately. See the adaptive-token-compressor repository for deployment and per-compressor behavior.
provider_management Plugin#
Source: src/plugins/provider_management.py.
This plugin lets the router drive an external Local Provider Manager — a
service that starts and stops serving backends (containers) on demand and exposes
an HTTP API (POST /providers, DELETE /providers/{name}). It contributes HTTP
routes rather than a request hook (its process_request hook is a passthrough).
The plugin controls the provider services only — it never reads or changes
the router’s provider config. Whether the router routes to a service is up to you,
through the providers API
(POST / DELETE /v1/providers/{name}). The two are independent: starting a
service does not register or enable a provider, and stopping one does not disable
or remove it.
The manager is a single service; give the plugin its base URL in a plugins:
entry under the standalone (non-hook) stage — it is route-only and must not run in
any request/response chain:
plugins:
standalone: # non-hook: routes only
- name: "provmgr"
node: "provider_management"
settings:
manager_url: "http://localhost:9900" # base URL of the Local Provider Manager
# timeout: 1200 # optional; seconds, for slow cold starts
Start — POST /v1/providers/manage with the manager’s start-request body
(docker params: name / image / entrypoint / env / volumes /
docker_args / command). The service name travels in the body (there is no
{name} in the path — the manager keys the container on it). Optionally add
health_endpoint — a URL (reachable from the router) that answers 2xx once the
backend is ready, e.g. http://localhost:24680/health (vLLM) or
http://localhost:24690/v2/health/ready (OVMS).
The plugin forwards the docker params to POST {manager_url}/providers, which
runs docker run -d and records the result, and returns the manager’s response
(e.g. a 500 if docker run fails). With health_endpoint set the plugin first
polls it until it answers 2xx — failing fast with 502 if the manager reports the
container stopped, or 504 after management_timeout — and on failure removes the
container through the manager.
# vllm backend
curl -s -XPOST localhost:8000/v1/providers/manage -H 'Content-Type: application/json' -d '{
"name":"vllm-qwen35-9b",
"image":"intel/llm-scaler-vllm:0.26.0-b2",
"entrypoint":"vllm",
"env":{"VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT":"1","ZE_AFFINITY_MASK":"0"},
"volumes":{"/path/to/Qwen3.5-9B":"/llm/models/qwen3-5-9b"},
"docker_args":["--privileged","--shm-size","32g","--device","/dev/dri",
"--group-add","992","--group-add","44","-p","24680:24680"],
"command":["serve","/llm/models/qwen3-5-9b","--served-model-name","qwen3-5-9b",
"--port","24680","--host","0.0.0.0", "--gpu-memory-util", "0.7",
"--enforce-eager", "--max-model-len", "32768",
"--enable-auto-tool-choice", "--enable_prefix_caching",
"--reasoning-parser","qwen3","--tool-call-parser","qwen3_coder"],
"health_endpoint":"http://localhost:24680/health"
}'
# ovms backend
curl -s -XPOST localhost:8000/v1/providers/manage -H 'Content-Type: application/json' -d '{
"name":"ovms-qwen36-35b",
"image":"openvino/model_server:latest-gpu",
"volumes":{"/path/to/Qwen3.6-35B-A3B-INT4":"/models/qwen3-6-35b-a3b-int4"},
"docker_args":["--user","1000:1000","--device","/dev/dri",
"--group-add","992","--group-add","44","-p","24690:24690"],
"command":["--rest_port","24690","--model_repository_path","/models",
"--source_model","qwen3-6-35b-a3b-int4","--task","text_generation",
"--target_device","GPU","--reasoning_parser","qwen3","--tool_parser","qwen3coder"],
"health_endpoint":"http://localhost:24690/v2/health/ready"
}'
Route to the service — once it is up, add (or enable) its provider through
the providers API; model and settings.endpoint must match what the container
serves:
# register + enable the vLLM service as a provider
curl -s -XPOST localhost:8000/v1/providers/vllm-qwen35-9b -H 'Content-Type: application/json' -d '{
"type":"hosted_vllm",
"model":"qwen3-5-9b",
"enabled":true,
"settings":{"endpoint":"http://localhost:24680/v1"}
}'
# an already-declared provider (e.g. enabled: false in config.yaml) only needs
curl -s -XPOST localhost:8000/v1/providers/ovms-qwen36-35b -H 'Content-Type: application/json' \
-d '{"enabled":true}'
Stop — DELETE /v1/providers/manage?name=<name>. The plugin forwards
DELETE {manager_url}/providers/{name} and returns the manager’s response. It
does not touch the router config, so disable ({"enabled":false}) or delete the
provider through the providers API first — otherwise the router keeps routing to
a backend that is gone. Conversely, DELETE /v1/providers/{name} only edits
config and does not stop the container.
Inspect — GET /v1/providers/manage proxies the manager’s GET /providers
and returns its live view of every started backend (the recorded docker run
command and result, plus the current container state); add ?name=<name> for a
single one. This reflects what is actually running at the manager, as opposed to
GET /v1/providers, which reports the router’s provider configuration.
curl -s localhost:8000/v1/providers/manage # all started backends
curl -s "localhost:8000/v1/providers/manage?name=qwen3" # one
LoRA adapters — POST /v1/providers/manage/load_lora_adapter and
.../unload_lora_adapter hot-swap LoRA adapters on a running hosted_vllm
provider (one present in the router config). Unlike the routes above they go
straight to the provider’s vLLM server
(its settings.endpoint, with its settings.auth), not the manager: the body is
vLLM’s (lora_name / lora_path) plus provider, which is stripped before
forwarding to vLLM’s /v1/load_lora_adapter / /v1/unload_lora_adapter. The
backend must run with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and --enable-lora.
curl -s -XPOST localhost:8000/v1/providers/manage/load_lora_adapter -H 'Content-Type: application/json' \
-d '{"provider":"vllm-qwen35-9b","lora_name":"sql_adapter","lora_path":"/models/loras/sql"}'
curl -s -XPOST localhost:8000/v1/providers/manage/unload_lora_adapter -H 'Content-Type: application/json' \
-d '{"provider":"vllm-qwen35-9b","lora_name":"sql_adapter"}'
Per-request overrides (start body only, stripped before forwarding together with
health_endpoint): manager_url overrides the configured base URL, and
management_timeout the timeout (seconds) for the manager call and for the
readiness wait.
The /v1/providers/manage routes are always mounted for the registered type; a
plugins: entry configures the manager_url they use. Start can also take it per
request, but stop and inspect read it only from the plugin settings.
dummy_logger Plugin#
Source: src/plugins/dummy.py.
A plugin example that prints the stage that invoked it, and passes the
request or response through, unchanged. It also serves as a reference implementation
for the runtime contract: it counts invocations per stage, exposes the counts through
describe(), resets them through reset(), and exposes a routes() endpoint
(GET /v1/plugins/dummy_logger/ping) to demonstrate a plugin contributing its
own HTTP API. Use it to verify that the plugin works end to end.
Configuration example — the same instance can be placed in any stage; add it to whichever stage(s) you want to trace:
plugins:
prerouting:
- name: "logger_pre"
node: "dummy_logger"
enabled: true
settings:
label: "pre"
postresponse:
- name: "logger_post"
node: "dummy_logger"
enabled: true
settings:
label: "post"
Register a New Plugin#
A plugin type is a PluginBaseNode subclass identified by its node key;
each entry in the configuration is one instance (name) of a type. Adding
a type needs three steps — no central registry edit is needed, because every
module under src/plugins/ is auto-discovered at startup.
1. Create a module in src/plugins/ (e.g. src/plugins/word_count.py) that
defines a settings schema and a plugin class decorated with @register_plugin:
from typing import Any, Dict, Type
from pydantic import BaseModel
from src.models import ChatCompletionRequest
from src.plugins.base import PluginBaseNode
from src.plugins.manager import register_plugin
class WordCountSettings(BaseModel):
prefix: str = "words"
@register_plugin
class WordCountPlugin(PluginBaseNode):
"""Counts words in the latest user message (example plugin)."""
@classmethod
def plugin_type(cls) -> str: # the `node` key, must be unique
return "word_count"
@classmethod
def settings_model(cls) -> Type[BaseModel]:
return WordCountSettings
def init(self) -> None: # optional: setup with validated settings
self._count = 0
async def process_request(self, request: ChatCompletionRequest, **kwargs):
text = request.messages[-1].content if request.messages else ""
self._count += len(str(text).split())
return request # return the (possibly modified) request
def describe(self) -> Dict[str, Any]: # optional: fold info into the GET
return {**super().describe(), "metrics": {"words_seen": self._count}}
def reset(self) -> bool: # optional: back POST .../reset
self._count = 0
return True
Only plugin_type() and settings_model() are required. Everything else has a
safe default: process_request or process_response pass through, describe() or
describe_node() return metadata, reset() or reset_node() report “unsupported”
(HTTP 400), and health_check() reports healthy. Override just what you need.
2. Configure an instance under plugins in workspace/config.yaml. Choose
the stage (prerouting, postrouting, postresponse, or standalone for a
route-only plugin) and set node to the type’s plugin_type():
plugins:
prerouting:
- name: "counter"
node: "word_count"
enabled: true
settings:
prefix: "words"
3. Verify that the plugin has registered and is serving:
curl http://localhost:8000/v1/plugins/nodes # lists word_count + its schema
curl http://localhost:8000/v1/plugins/word_count/counter # instance view + metrics
Manage Plugins at Runtime#
Plugins are ordinary configuration entries, therefore they can be listed, inspected,
created or updated, reset, and deleted at runtime through the /v1/plugins API —
changes are persisted to the on-disk configuration and take effect immediately. See the
API Reference for the full contract:
GET /v1/plugins— to list configured plugin instances.GET /v1/plugins/nodes— to list plugin types registered in code.GET /v1/plugins/{node}andGET /v1/plugins/{node}/{name}— to get node- and instance-level views.POST /v1/plugins/{node}/{name}— to create or update an instance.DELETE /v1/plugins/{node}/{name}— to remove an instance.POST /v1/plugins/{node}/resetandPOST /v1/plugins/{node}/{name}/reset— to reset the node- or instance-level state.
Learn More#
The Get Started Guide section covers enabling the compressor plugins and reading compression metrics.
The API Reference documents every plugin endpoint.