Plugins#

The router runs custom logic against each request or response through plugins. A plugin can rewrite a request before routing, transform it after a provider is selected, act on the response, or expose its own HTTP endpoints — without editing the core API layer.

This page explains how the plugin system works, lists the plugins currently implemented, and shows how to add and register a new one.

Concepts#

  • Plugin type (node). A PluginBaseNode subclass, identified by its plugin_type() key. This key is the node you write in the configuration and the API.

  • Plugin instance (name). Each entry under plugins in config.yaml is one instance of a type. A type can have many instances with different settings; name is the unique instance identifier.

  • Stage (trigger). Stages in the request lifecycle:

    • prerouting — When the plugin instance runs before provider selection, i.e., before the routing decision is made.

    • postrouting — When the plugin instance runs after provider selection and before the backend call.

    • postresponse — When the plugin instance runs on the response after backend inference.

    • standalone — A non-hook phase: the instance is built (its settings are validated) and its routes are mounted, but it is never run in any request/response chain. Use it for route-only plugins whose work is driven by their own HTTP endpoints rather than the per-request pipeline (e.g. provider_management). Placing such a plugin in a real stage would add a no-op to that chain on every request.

  • Auto-discovery. Every module under src/plugins/ is imported at startup, so any class decorated with @register_plugin self-registers. There is no central registry to edit when adding a plugin.

Within a stage, instances run sequentially in the order they appear in config.yaml; each one receives the output of the previous. standalone plugins are exempt — they do not run in any stage.

Attaching Plugins to Providers#

By default a postrouting or postresponse instance runs for every request. To run it only for some providers, list its name under those providers’ plugins:

providers:
  - name: "local"
    # ...
    plugins: ["compressor_harness"]
  - name: "cloud"
    # ...
    plugins: ["compressor_harness", "audit_logger"]
  - name: "edge"        # no `plugins`: runs only unattached plugins
    # ...
plugins:
  postrouting:
    - name: "compressor_harness"
      node: "compressor"
      # ...
  • An instance listed by at least one provider runs only for requests routed to those providers. An instance listed by no provider runs for every provider.

  • Providers listing the same name share one instance (and its state, such as compressor caches and metrics).

  • The stage order in config.yaml still decides the run order; a provider’s plugins list only filters it.

  • Only postrouting and postresponse instances can be attached. prerouting instances always run for every request, because the provider is not chosen yet; attaching one (or a standalone one) is rejected at load time, as is an unknown name. A disabled instance may be listed; it simply does not run.

  • A provider stays attached to its plugins while disabled, so disabling it does not make those plugins run for every provider.

  • The providers API reads and replaces the list through the plugins field. A plugin that a provider lists cannot be deleted or moved to prerouting until it is removed from that list.

To attach plugins at runtime, post a provider’s plugins list to the providers API. The list replaces the provider’s existing one; the other provider fields are left unchanged, and the change is written to config.yaml:

# attach compressor_harness to "local" and "cloud" (they share one instance)
curl -s -XPOST localhost:8000/v1/providers/local -H 'Content-Type: application/json' \
  -d '{"plugins": ["compressor_harness"]}'
curl -s -XPOST localhost:8000/v1/providers/cloud -H 'Content-Type: application/json' \
  -d '{"plugins": ["compressor_harness", "audit_logger"]}'

# check what a provider has attached
curl -s localhost:8000/v1/providers/local | jq .plugins
# ["compressor_harness"]

# detach everything from "cloud"
curl -s -XPOST localhost:8000/v1/providers/cloud -H 'Content-Type: application/json' \
  -d '{"plugins": []}'

A bad name is rejected with 400 Bad Request and nothing changes:

curl -s -XPOST localhost:8000/v1/providers/local -H 'Content-Type: application/json' \
  -d '{"plugins": ["compressor_tool"]}'    # a prerouting plugin
# {"detail":"Provider 'local' cannot attach plugin 'compressor_tool' (trigger: prerouting); only postresponse, postrouting plugins can be attached to a provider"}

The Plugin Contract#

The plugin subclass PluginBaseNode in src/plugins/base.py implements the plugin contract in Inference Router. Only the following two methods are required; everything else has a safe default, so you override only what you need.

Required:

  • plugin_type() -> str — the unique node key.

  • settings_model() -> Type[BaseModel] — a Pydantic schema for the instance’s settings. Settings are validated at construction; invalid configuration is rejected with a PluginSchemaError.

Optional hooks (defaults in parentheses):

  • init() — set up after the plugin instance’s settings are validated; used to build clients, register with shared managers, and etc. (no-op).

  • process_request(request, **kwargs) — acts on the request; returns the possibly modified request (passthrough). Overrides must keep **kwargs: at the postrouting stage the router passes provider_name (the provider chosen for the request) and metrics_values={"provider": provider_name}.

  • process_response(response, **kwargs) — acts on the response (passthrough). At the postresponse stage the router passes the same provider_name and metrics_values keyword arguments.

  • describe() — the GET /v1/plugins/{node}/{name} payload; folds in per-instance runtime info, typically {**super().describe(), "metrics": {...}} (instance metadata).

  • describe_node() — the GET /v1/plugins/{node} payload; exposes type-wide aggregates spanning all instances (the node metadata).

  • reset() and reset_node() — implement POST .../reset for the plugin-instance state and plugin-node state (report “unsupported”, HTTP 400).

  • health_check() — probes backing dependencies (reports “unavailable”).

  • routes() — returns a FastAPI APIRouter object to mount under /v1, letting the plugin expose its own HTTP API without modifying the central API layer. This method is called once per plugin type, regardless of instance count. Define plugin endpoints under the /plugins/{node}/... path namespace to avoid collisions (None).

Currently Implemented Plugins#

node

Stage(s)

What It Does

compressor

prerouting and postrouting

Compresses prompts before the backend call to cut token usage.

provider_management

any (route-only)

Starts or stops a provider via an external Local Provider Manager; loads/unloads vLLM LoRA adapters.

dummy_logger

any

Logs the stage where the plugin was invoked and serves as a plugin contract reference implementation.

compressor Plugin#

Source: src/plugins/compressor.py.

Reduces prompt tokens before the request reaches the backend, using the adaptive-token-compressor library (bundled in the router image). One compressor node covers every compressor type; the type is chosen per instance through settings.type:

  • tool — filters the request tools schema down to a relevant subset using a tool predictor (an OpenAI-compatible LLM endpoint).

  • harness — compresses system or developer messages using a compress server.

  • context — compresses the conversation context.

Caution: The context compressor is experimental and is disabled by default.

How it works:

  • Compressors act upon requests only, so configure them in the prerouting or postrouting stage. At the postresponse stage, a compressor is a no-op (and logs a warning).

  • All instances share caching and metrics aggregation. Compression runs in a worker thread to keep the event loop responsive.

  • On any compression error, the request is returned unmodified — a failing compressor degrades gracefully rather than dropping the request.

  • Metrics: describe() returns instance metrics; describe_node() also includes overall.* totals across compressors. Instance responses never include overall.*. See Metrics Checking for details.

  • settings.metrics_keys: ["provider"] additionally splits a postrouting compressor’s metrics by the provider that served the request, reported under metrics_by_keys. provider is the only supported key, and the setting is rejected at any other stage (at prerouting the provider is not chosen yet). See Splitting metrics by provider.

  • A postrouting compressor can be attached to providers, so it compresses only requests routed to them. Providers that list the same compressor share one instance; with metrics_keys: ["provider"], its metrics_by_keys.provider splits the numbers per provider.

Configuration example — a tool compressor at the prerouting stage, and harness and context compressors at the postrouting stage. node is always compressor; settings.type selects the compressor type:

plugins:
  prerouting:
    - name: "compressor_tool"
      node: "compressor"
      enabled: true
      settings:
        type: "tool"
        predictor_url: "http://localhost:8088/v1/chat/completions"
        predictor_model: "Qwen/Qwen3.6-35B-A3B"
        predictor_backend: "generative"
        score_threshold: 2.0
        placement: "schema"
        accumulate: true
  postrouting:
    - name: "compressor_harness"
      node: "compressor"
      enabled: true
      settings:
        type: "harness"
        profile: "openclaw"
        compress_server_url: "http://localhost:8001/compress"
        compress_rate: 0.5
        compress_min_chars: 200
        timeout: 60.0
        enable_quantum_lock: false
        metrics_keys: ["provider"]   # split this compressor's metrics by provider
    - name: "compressor_context"
      node: "compressor"
      enabled: false
      settings:
        type: "context"
        protect_recent_tool_turns: 1
        large_result_stub: true
        large_result_stub_guidance: "legacy"
        large_result_min_tokens: 4000
        large_result_keep_tool_turns: 1
        compression_mode: "none"
  postresponse: []

The tool example above uses the large-model predictor (Qwen/Qwen3.6-35B-A3B). To use the small-model predictor instead, point the tool compressor at a reranker endpoint and select tools with top_k:

plugins:
  prerouting:
    - name: "compressor_tool"
      node: "compressor"
      enabled: true
      settings:
        type: "tool"
        predictor_url: "http://localhost:8091/v1/rerank"
        predictor_model: "Qwen/Qwen3-Reranker-0.6B"
        predictor_backend: "reranker"
        top_k: 10
        placement: "schema"
        accumulate: true

The two configurations trade off differently: the small model is faster but saves fewer tool tokens. top_k should be sized to the number of tools per request. See the adaptive-token-compressor ToolCompressor configuration guide for measured results and how to choose K.

The backing services, which are the compress server and tool predictor, are not part of the router — deploy them separately. See the adaptive-token-compressor repository for deployment and per-compressor behavior.

provider_management Plugin#

Source: src/plugins/provider_management.py.

This plugin lets the router drive an external Local Provider Manager — a service that starts and stops serving backends (containers) on demand and exposes an HTTP API (POST /providers, DELETE /providers/{name}). It contributes HTTP routes rather than a request hook (its process_request hook is a passthrough).

The plugin controls the provider services only — it never reads or changes the router’s provider config. Whether the router routes to a service is up to you, through the providers API (POST / DELETE /v1/providers/{name}). The two are independent: starting a service does not register or enable a provider, and stopping one does not disable or remove it.

The manager is a single service; give the plugin its base URL in a plugins: entry under the standalone (non-hook) stage — it is route-only and must not run in any request/response chain:

plugins:
  standalone:                                       # non-hook: routes only
    - name: "provmgr"
      node: "provider_management"
      settings:
        manager_url: "http://localhost:9900"   # base URL of the Local Provider Manager
        # timeout: 1200                          # optional; seconds, for slow cold starts

Start — POST /v1/providers/manage with the manager’s start-request body (docker params: name / image / entrypoint / env / volumes / docker_args / command). The service name travels in the body (there is no {name} in the path — the manager keys the container on it). Optionally add health_endpoint — a URL (reachable from the router) that answers 2xx once the backend is ready, e.g. http://localhost:24680/health (vLLM) or http://localhost:24690/v2/health/ready (OVMS).

The plugin forwards the docker params to POST {manager_url}/providers, which runs docker run -d and records the result, and returns the manager’s response (e.g. a 500 if docker run fails). With health_endpoint set the plugin first polls it until it answers 2xx — failing fast with 502 if the manager reports the container stopped, or 504 after management_timeout — and on failure removes the container through the manager.

# vllm backend
curl -s -XPOST localhost:8000/v1/providers/manage -H 'Content-Type: application/json' -d '{
  "name":"vllm-qwen35-9b",
  "image":"intel/llm-scaler-vllm:0.26.0-b2",
  "entrypoint":"vllm",
  "env":{"VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT":"1","ZE_AFFINITY_MASK":"0"},
  "volumes":{"/path/to/Qwen3.5-9B":"/llm/models/qwen3-5-9b"},
  "docker_args":["--privileged","--shm-size","32g","--device","/dev/dri",
                 "--group-add","992","--group-add","44","-p","24680:24680"],
  "command":["serve","/llm/models/qwen3-5-9b","--served-model-name","qwen3-5-9b",
             "--port","24680","--host","0.0.0.0", "--gpu-memory-util", "0.7",
             "--enforce-eager", "--max-model-len", "32768",
             "--enable-auto-tool-choice", "--enable_prefix_caching",
             "--reasoning-parser","qwen3","--tool-call-parser","qwen3_coder"],
  "health_endpoint":"http://localhost:24680/health"
}'

# ovms backend
curl -s -XPOST localhost:8000/v1/providers/manage -H 'Content-Type: application/json' -d '{
  "name":"ovms-qwen36-35b",
  "image":"openvino/model_server:latest-gpu",
  "volumes":{"/path/to/Qwen3.6-35B-A3B-INT4":"/models/qwen3-6-35b-a3b-int4"},
  "docker_args":["--user","1000:1000","--device","/dev/dri",
                 "--group-add","992","--group-add","44","-p","24690:24690"],
  "command":["--rest_port","24690","--model_repository_path","/models",
             "--source_model","qwen3-6-35b-a3b-int4","--task","text_generation",
             "--target_device","GPU","--reasoning_parser","qwen3","--tool_parser","qwen3coder"],
  "health_endpoint":"http://localhost:24690/v2/health/ready"
}'

Route to the service — once it is up, add (or enable) its provider through the providers API; model and settings.endpoint must match what the container serves:

# register + enable the vLLM service as a provider
curl -s -XPOST localhost:8000/v1/providers/vllm-qwen35-9b -H 'Content-Type: application/json' -d '{
  "type":"hosted_vllm",
  "model":"qwen3-5-9b",
  "enabled":true,
  "settings":{"endpoint":"http://localhost:24680/v1"}
}'

# an already-declared provider (e.g. enabled: false in config.yaml) only needs
curl -s -XPOST localhost:8000/v1/providers/ovms-qwen36-35b -H 'Content-Type: application/json' \
  -d '{"enabled":true}'

Stop — DELETE /v1/providers/manage?name=<name>. The plugin forwards DELETE {manager_url}/providers/{name} and returns the manager’s response. It does not touch the router config, so disable ({"enabled":false}) or delete the provider through the providers API first — otherwise the router keeps routing to a backend that is gone. Conversely, DELETE /v1/providers/{name} only edits config and does not stop the container.

Inspect — GET /v1/providers/manage proxies the manager’s GET /providers and returns its live view of every started backend (the recorded docker run command and result, plus the current container state); add ?name=<name> for a single one. This reflects what is actually running at the manager, as opposed to GET /v1/providers, which reports the router’s provider configuration.

curl -s localhost:8000/v1/providers/manage                 # all started backends
curl -s "localhost:8000/v1/providers/manage?name=qwen3"    # one

LoRA adapters — POST /v1/providers/manage/load_lora_adapter and .../unload_lora_adapter hot-swap LoRA adapters on a running hosted_vllm provider (one present in the router config). Unlike the routes above they go straight to the provider’s vLLM server (its settings.endpoint, with its settings.auth), not the manager: the body is vLLM’s (lora_name / lora_path) plus provider, which is stripped before forwarding to vLLM’s /v1/load_lora_adapter / /v1/unload_lora_adapter. The backend must run with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and --enable-lora.

curl -s -XPOST localhost:8000/v1/providers/manage/load_lora_adapter -H 'Content-Type: application/json' \
  -d '{"provider":"vllm-qwen35-9b","lora_name":"sql_adapter","lora_path":"/models/loras/sql"}'
curl -s -XPOST localhost:8000/v1/providers/manage/unload_lora_adapter -H 'Content-Type: application/json' \
  -d '{"provider":"vllm-qwen35-9b","lora_name":"sql_adapter"}'

Per-request overrides (start body only, stripped before forwarding together with health_endpoint): manager_url overrides the configured base URL, and management_timeout the timeout (seconds) for the manager call and for the readiness wait.

The /v1/providers/manage routes are always mounted for the registered type; a plugins: entry configures the manager_url they use. Start can also take it per request, but stop and inspect read it only from the plugin settings.

dummy_logger Plugin#

Source: src/plugins/dummy.py.

A plugin example that prints the stage that invoked it, and passes the request or response through, unchanged. It also serves as a reference implementation for the runtime contract: it counts invocations per stage, exposes the counts through describe(), resets them through reset(), and exposes a routes() endpoint (GET /v1/plugins/dummy_logger/ping) to demonstrate a plugin contributing its own HTTP API. Use it to verify that the plugin works end to end.

Configuration example — the same instance can be placed in any stage; add it to whichever stage(s) you want to trace:

plugins:
  prerouting:
    - name: "logger_pre"
      node: "dummy_logger"
      enabled: true
      settings:
        label: "pre"
  postresponse:
    - name: "logger_post"
      node: "dummy_logger"
      enabled: true
      settings:
        label: "post"

Register a New Plugin#

A plugin type is a PluginBaseNode subclass identified by its node key; each entry in the configuration is one instance (name) of a type. Adding a type needs three steps — no central registry edit is needed, because every module under src/plugins/ is auto-discovered at startup.

1. Create a module in src/plugins/ (e.g. src/plugins/word_count.py) that defines a settings schema and a plugin class decorated with @register_plugin:

from typing import Any, Dict, Type
from pydantic import BaseModel
from src.models import ChatCompletionRequest
from src.plugins.base import PluginBaseNode
from src.plugins.manager import register_plugin


class WordCountSettings(BaseModel):
    prefix: str = "words"


@register_plugin
class WordCountPlugin(PluginBaseNode):
    """Counts words in the latest user message (example plugin)."""

    @classmethod
    def plugin_type(cls) -> str:        # the `node` key, must be unique
        return "word_count"

    @classmethod
    def settings_model(cls) -> Type[BaseModel]:
        return WordCountSettings

    def init(self) -> None:             # optional: setup with validated settings
        self._count = 0

    async def process_request(self, request: ChatCompletionRequest, **kwargs):
        text = request.messages[-1].content if request.messages else ""
        self._count += len(str(text).split())
        return request                  # return the (possibly modified) request

    def describe(self) -> Dict[str, Any]:   # optional: fold info into the GET
        return {**super().describe(), "metrics": {"words_seen": self._count}}

    def reset(self) -> bool:                # optional: back POST .../reset
        self._count = 0
        return True

Only plugin_type() and settings_model() are required. Everything else has a safe default: process_request or process_response pass through, describe() or describe_node() return metadata, reset() or reset_node() report “unsupported” (HTTP 400), and health_check() reports healthy. Override just what you need.

2. Configure an instance under plugins in workspace/config.yaml. Choose the stage (prerouting, postrouting, postresponse, or standalone for a route-only plugin) and set node to the type’s plugin_type():

plugins:
  prerouting:
    - name: "counter"
      node: "word_count"
      enabled: true
      settings:
        prefix: "words"

3. Verify that the plugin has registered and is serving:

curl http://localhost:8000/v1/plugins/nodes          # lists word_count + its schema
curl http://localhost:8000/v1/plugins/word_count/counter   # instance view + metrics

Manage Plugins at Runtime#

Plugins are ordinary configuration entries, therefore they can be listed, inspected, created or updated, reset, and deleted at runtime through the /v1/plugins API — changes are persisted to the on-disk configuration and take effect immediately. See the API Reference for the full contract:

  • GET /v1/plugins — to list configured plugin instances.

  • GET /v1/plugins/nodes — to list plugin types registered in code.

  • GET /v1/plugins/{node} and GET /v1/plugins/{node}/{name} — to get node- and instance-level views.

  • POST /v1/plugins/{node}/{name} — to create or update an instance.

  • DELETE /v1/plugins/{node}/{name} — to remove an instance.

  • POST /v1/plugins/{node}/reset and POST /v1/plugins/{node}/{name}/reset — to reset the node- or instance-level state.

Learn More#

  • The Get Started Guide section covers enabling the compressor plugins and reading compression metrics.

  • The API Reference documents every plugin endpoint.