Provider Probes#

A probe is a per-provider background task that keeps a live signal about a backend fresh. On a fixed interval it runs one small pipeline:

tick → GET(endpoint) → extract numeric fields → aggregate (sliding window) → publish

Everything a probe does is declared in config — there is no per-backend code. The most common use is a performance probe: scrape a backend’s Prometheus /metrics endpoint, turn a few counters/gauges into numbers, and publish them into the provider’s routing metadata so the router can prefer less-loaded, faster providers. Values can also be appended to a JSONL file for history.


Probe configuration#

Probes live under a provider’s settings.probes list. Each entry:

Field

Required

Default

Meaning

name

yes

—

Unique within the provider.

enabled

no

true

Set false to stop it (see lifecycle).

endpoint

no

provider’s endpoint

URL to GET each tick. Set this to the /metrics URL — the provider’s own endpoint is the chat /v1 base.

interval

no

10

Seconds between ticks.

timeout

no

5

Per-request timeout (must not stall reporting).

method

no

GET

HTTP method.

extract

yes

—

One or more rules that turn the response body into {name: value} pairs.

aggregate

no

{window: 20, method: mean}

Sliding-window reduction per series. method ∈ latest | mean | median | p50 | p90 | p95 | p99 | ewma | min | max | stdev.

publish

no

metadata sink

Where each aggregate goes (see sinks).

max_series

no

200

Cardinality cap, so a broad regex can’t create unbounded series.

Extract rules#

Each rule is either a regex (for Prometheus/plain-text bodies) or a json dot-path (for JSON bodies):

  • regex — matched against the whole body. Per match, the last capture group is the value and the group before it is the field name. One pattern can therefore name and read many series at once (one per matching line). Non-numeric values are skipped.

    # vllm:num_requests_waiting{engine="0",model_name="..."} 3.0
    #      └──────── name ─────┘                             └ value ┘
    - regex: 'vllm:(num_requests_waiting)\{[^}]*\}\s+([0-9.eE+-]+)'
    
  • json — a dotted path into the parsed body; the field name is the last segment, the value is the numeric leaf.

    - json: stats.pending        # {"stats": {"pending": 7}} → {"pending": 7.0}
    

How groups are counted#

The name and value are picked by position: second-to-last and last capture group. Every ( … ) counts, so:

  • A rule needs at least two capture groups. A pattern with none (e.g. a metric line pasted verbatim) extracts nothing.

  • A group nested inside the name group becomes the name instead: vllm:(num_requests_(running|waiting))… yields running / waiting.

  • A group inside the value pattern becomes the “value”: ([0-9.]+(e[+-]?\d+)?) reads the exponent group, and the line is dropped.

Use non-capturing groups (?: … ) for anything that is neither the name nor the value: vllm:(num_requests_(?:running|waiting))… yields num_requests_running / num_requests_waiting.

Field names are normalized#

A captured name is made safe as a single dotted-path segment, because routing reads probe values by dotted path (performance.<name>) and a . inside a name — common in labels such as model_name="Qwen/Qwen3.6" — would split it:

  1. Prometheus label blocks {…} are dropped.

  2. ., : and whitespace become _.

  3. Leading/trailing _ are trimmed (an empty result becomes value).

captured name

published name

num_requests_waiting

num_requests_waiting

vllm:kv_cache_usage_perc

vllm_kv_cache_usage_perc

vllm:request_success_total{engine="0",finished_reason="stop",…}

vllm_request_success_total

The normalized name is used everywhere — metadata, the file sink’s aggregates, and the window — so reference it (not the raw capture) in strategy.yaml. Capture the bare metric name (as in the examples above) to keep names readable.

Several matches with the same name: the last one wins#

When more than one match produces the same (normalized) field name in one response, only the last is kept. Rules apply in the order listed and matches in response order, so a later rule overrides an earlier one. This bites on any metric with several label sets:

vllm:request_success_total{engine="0",finished_reason="stop",model_name="…"} 911.0
vllm:request_success_total{engine="0",finished_reason="length",model_name="…"} 66.0
vllm:request_success_total{engine="0",finished_reason="repetition",model_name="…"} 0.0

vllm:(request_success_total)\{[^}]*\}\s+([0-9.eE+-]+) matches all three and publishes request_success_total: 0.0 — the repetition line. The same happens with histogram buckets (one line per le) and with gauges on a multi-engine backend (one line per engine). Labels captured inside the name group don’t help: normalization drops them, so the series still collide.

The probe logs a warning once per field name when this happens — either more than one match produced field … or, for names that only became equal after normalization, … both publish as field … after name normalization.

To read a specific series, pin its label in the regex so only one line matches:

# finished_reason="stop" only → request_success_total: 911.0
- regex: 'vllm:(request_success_total)\{[^}]*finished_reason="stop"[^}]*\}\s+([0-9.eE+-]+)'

To read several series of one metric at once, capture the distinguishing label value as the name; each value becomes its own field:

# → stop: 911.0, length: 66.0, repetition: 0.0
- regex: 'vllm:request_success_total\{[^}]*finished_reason="(\w+)"[^}]*\}\s+([0-9.eE+-]+)'

Two separate rules for the same metric (one per label) still publish the same name and collide; use the pattern above instead.

Counters. *_total metrics only ever grow, so a mean over the window lags the current count. Use aggregate: { method: latest } for counters.


Example 1 — add a performance probe in config.yaml#

This probes a vLLM backend’s /metrics every 5 s, extracts two load signals (both lower-is-better), and publishes them under the provider’s performance.* namespace — the exact field the PerformanceBased strategy reads — plus a JSONL history file.

providers:
- name: local
  type: hosted_vllm
  model: Qwen/Qwen3.5-9B
  enabled: true
  metadata:
    labels: [local]
    cost: 0
  settings:
    endpoint: http://localhost:8088/v1        # chat base (/v1)
    timeout: 300.0
    auth:
      scheme: none
      api_key: null
      custom_headers: {}
    probes:
    - name: perf
      enabled: true
      endpoint: http://localhost:8088/metrics # scrape target (NOT /v1)
      interval: 5
      timeout: 5
      extract:
      - regex: 'vllm:(num_requests_waiting)\{[^}]*\}\s+([0-9.eE+-]+)'
      - regex: 'vllm:(kv_cache_usage_perc)\{[^}]*\}\s+([0-9.eE+-]+)'
      aggregate: { window: 12, method: mean }
      publish:
      - { to: metadata, prefix: performance }                       # → routing (live)
      - { to: file, path: probes/local.perf.jsonl }                 # → history

File paths are workspace-relative. A relative path resolves against the runtime workspace directory (the config.yaml directory, i.e. $GATEWAY_WORKSPACE or the dir of $GATEWAY_CONFIG), not the process CWD. So probes/local.perf.jsonl lands in <workspace>/probes/ wherever the server is launched from. Use an absolute path to write elsewhere.

The metadata sink writes each aggregate to performance.<name>, e.g. performance.num_requests_waiting. A PerformanceBased strategy can then filter and rank on it (a bare number is an upper bound; [lower, upper] is an inclusive range with null for an open side — see Performance-Based Routing):

# strategy.yaml
strategies:
- name: performance_based
  provider_selector:
    performance:
      num_requests_waiting: 5      # drop providers with a longer queue
      kv_cache_usage_perc: [null, 0.9]  # drop providers whose KV cache is saturated
  sort:
  - { provider_attribute: performance.num_requests_waiting, descending: false }

Prefixes. Use prefix: performance to feed the routing performance.* field directly. Use prefix: attributes.probe (the default) for generic live metadata that surfaces as probe.<name> in metadata.to_dict().

Latency metrics. vLLM exposes latencies as histograms (vllm:time_to_first_token_seconds_sum / _count). A probe reads single values, so it cannot divide sum by count to get a mean TTFT in one rule — prefer the load gauges above, or publish the raw _sum/_count and compute downstream.


Example 2 — add the same probe at runtime via the providers API#

No restart needed: POST the provider with the probe in settings.probes. The providers API rebuilds the runtime on every mutation, so the probe starts immediately. settings is replaced wholesale, so resend the full settings block, not just the probe.

curl -X POST http://localhost:8000/v1/providers/local \
  -H 'content-type: application/json' \
  -d '{
    "settings": {
      "endpoint": "http://localhost:8088/v1",
      "timeout": 300.0,
      "auth": {"scheme": "none", "api_key": null, "custom_headers": {}},
      "probes": [
        {
          "name": "perf",
          "enabled": true,
          "endpoint": "http://localhost:8088/metrics",
          "interval": 5,
          "extract": [
            {"regex": "vllm:(num_requests_waiting)\\{[^}]*\\}\\s+([0-9.eE+-]+)"}
          ],
          "publish": [
            {"to": "metadata", "prefix": "performance"},
            {"to": "file", "path": "probes/local.perf.jsonl"}
          ]
        }
      ]
    }
  }'

Confirm it was persisted:

curl -s http://localhost:8000/v1/providers/local | jq '.settings.probes'

Optional — edit in place: GET, append the probe, POST back#

Instead of hand-writing the whole settings block, you can fetch the provider’s current config, splice the probe into settings.probes, and POST the result. Because a GET response round-trips as a POST payload, jq can do the merge. (Posting the GET’s metadata back is also fine: live probe values in it are not persisted, see Reading live values.)

# 1. Define the probe to add.
read -r -d '' PROBE <<'JSON'
{
  "name": "perf",
  "enabled": true,
  "endpoint": "http://localhost:8088/metrics",
  "interval": 5,
  "extract": [{"regex": "vllm:(num_requests_waiting)\\{[^}]*\\}\\s+([0-9.eE+-]+)"}],
  "publish": [
    {"to": "metadata", "prefix": "performance"},
    {"to": "file", "path": "probes/local.perf.jsonl"}
  ]
}
JSON

# 2. GET current config, append the probe (replacing any probe of the same
#    name), and POST just the settings back.
curl -s http://localhost:8000/v1/providers/local \
  | jq --argjson probe "$PROBE" '{
      settings: (.settings + {
        probes: (((.settings.probes // []) | map(select(.name != $probe.name))) + [$probe])
      })
    }' \
  | curl -X POST http://localhost:8000/v1/providers/local \
      -H 'content-type: application/json' -d @-

Secrets caveat. A GET redacts secrets (api_key: "***REDACTED***"). Posting that mask back would persist the literal string, breaking auth. This in-place edit is therefore safe for providers with no secret (e.g. auth.scheme: none, as above); for a provider that carries an api_key, re-supply the real value (or a ${VAR} placeholder) in the merged payload before POSTing.

Starting / stopping#

Lifecycle is just the enabled flag, driven through the same providers API:

  • start / add — POST with the probe present and enabled: true.

  • stop / remove — POST again with that probe’s enabled: false (or omit it).

  • all probes start on boot (initialize()) and stop on shutdown.


Publishing & reading values#

A probe publishes through one or more sinks:

  • metadata — writes each aggregate into the live ProviderMetadata at <prefix>.<name>. The strategy layer rebuilds {**metadata.to_dict()} on every route, so the next routing decision sees fresh values with no cache. This is how a probe influences routing.

  • file — appends one JSONL record per tick: {ts, provider, probe, aggregates}. This is the durable/offline read path. Because a probe runs forever, the history is capped: the sink keeps the most recent max_entries records (default 10000) and drops older ones. Set max_entries on the sink to change the cap, or max_entries: 0 to disable it (unbounded growth).

    - { to: file, path: probes/local.perf.jsonl, max_entries: 50000 }
    
tail -1 <workspace>/probes/local.perf.jsonl
# {"ts": 1790136107.6, "provider": "local", "probe": "perf",
#  "aggregates": {"num_requests_waiting": 0.0, "kv_cache_usage_perc": 0.0}}

Reading live values over HTTP — the providers API, not /v1/config#

Read live probe values from the providers API. GET /v1/providers and GET /v1/providers/{name} return each running provider’s metadata as routing sees it right now: the static metadata from config.yaml with the values the metadata sinks have published merged in.

curl -s http://localhost:8000/v1/providers/local | jq '.metadata.performance'
# {"num_requests_waiting": 0.0, "kv_cache_usage_perc": 0.0}

GET /v1/config returns the configuration only: provider metadata there is exactly what config.yaml declares, and never includes probe output. Changing a probe’s extract and restarting therefore leaves /v1/config’s metadata.performance unchanged; the probe’s configuration change is visible under settings.probes, and its new values under the providers API.

A few details of the live view:

  • Values appear after the probe’s first successful tick (up to interval seconds after start or after any config API change, which rebuilds the runtime). Until then metadata shows only the static values.

  • A disabled provider has no running probes, so the providers API returns its static metadata.

  • When the probe and config.yaml set the same key (e.g. a static performance.ttft_ms seed that a probe also publishes), the live view shows the probe’s value.

Probe output is never written to config.yaml. Posting a GET response back to POST /v1/providers/{name} — whole, or edited, as the UI’s provider edit dialog does — is safe: values the provider’s running probes published are dropped from the posted metadata before it is saved (a key that config.yaml also defines keeps its config.yaml value). So you can edit a probe’s settings from the UI without freezing its current readings into the config. Hand-written metadata keys the probes don’t publish are saved as sent.


Tips#

  • Point endpoint at the /metrics URL; the bare provider endpoint is the chat /v1 base and won’t yield metrics.

  • Anchor a regex name group ((num_requests_running)) for one metric; use a wildcard ((num_requests_\w+)) to discover several from one pattern. If the metric has several label sets, pin the label too — otherwise the last line wins (see same-name matches).

  • Check the router log for only the last match is kept warnings after adding a probe: they mean a rule is silently discarding series.

  • A failed scrape never kills the probe — it increments a fail counter and the window keeps its last good values until the next successful tick.

  • Keep interval sane: it is scrape load on the backend, not free.