Provider Probes#
A probe is a per-provider background task that keeps a live signal about a backend fresh. On a fixed interval it runs one small pipeline:
tick → GET(endpoint) → extract numeric fields → aggregate (sliding window) → publish
Everything a probe does is declared in config — there is no per-backend code.
The most common use is a performance probe: scrape a backend’s Prometheus
/metrics endpoint, turn a few counters/gauges into numbers, and publish them
into the provider’s routing metadata so the router can prefer less-loaded, faster
providers. Values can also be appended to a JSONL file for history.
Probe configuration#
Probes live under a provider’s settings.probes list. Each entry:
Field |
Required |
Default |
Meaning |
|---|---|---|---|
|
yes |
— |
Unique within the provider. |
|
no |
|
Set |
|
no |
provider’s |
URL to GET each tick. Set this to the |
|
no |
|
Seconds between ticks. |
|
no |
|
Per-request timeout (must not stall reporting). |
|
no |
|
HTTP method. |
|
yes |
— |
One or more rules that turn the response body into |
|
no |
|
Sliding-window reduction per series. |
|
no |
metadata sink |
Where each aggregate goes (see sinks). |
|
no |
|
Cardinality cap, so a broad regex can’t create unbounded series. |
Extract rules#
Each rule is either a regex (for Prometheus/plain-text bodies) or a
json dot-path (for JSON bodies):
regex— matched against the whole body. Per match, the last capture group is the value and the group before it is the field name. One pattern can therefore name and read many series at once (one per matching line). Non-numeric values are skipped.# vllm:num_requests_waiting{engine="0",model_name="..."} 3.0 # └──────── name ─────┘ └ value ┘ - regex: 'vllm:(num_requests_waiting)\{[^}]*\}\s+([0-9.eE+-]+)'
json— a dotted path into the parsed body; the field name is the last segment, the value is the numeric leaf.- json: stats.pending # {"stats": {"pending": 7}} → {"pending": 7.0}
How groups are counted#
The name and value are picked by position: second-to-last and last capture
group. Every ( … ) counts, so:
A rule needs at least two capture groups. A pattern with none (e.g. a metric line pasted verbatim) extracts nothing.
A group nested inside the name group becomes the name instead:
vllm:(num_requests_(running|waiting))…yieldsrunning/waiting.A group inside the value pattern becomes the “value”:
([0-9.]+(e[+-]?\d+)?)reads the exponent group, and the line is dropped.
Use non-capturing groups (?: … ) for anything that is neither the name nor
the value: vllm:(num_requests_(?:running|waiting))… yields
num_requests_running / num_requests_waiting.
Field names are normalized#
A captured name is made safe as a single dotted-path segment, because routing
reads probe values by dotted path (performance.<name>) and a . inside a
name — common in labels such as model_name="Qwen/Qwen3.6" — would split it:
Prometheus label blocks
{…}are dropped..,:and whitespace become_.Leading/trailing
_are trimmed (an empty result becomesvalue).
captured name |
published name |
|---|---|
|
|
|
|
|
|
The normalized name is used everywhere — metadata, the file sink’s
aggregates, and the window — so reference it (not the raw capture) in
strategy.yaml. Capture the bare metric name (as in the examples above) to
keep names readable.
Several matches with the same name: the last one wins#
When more than one match produces the same (normalized) field name in one response, only the last is kept. Rules apply in the order listed and matches in response order, so a later rule overrides an earlier one. This bites on any metric with several label sets:
vllm:request_success_total{engine="0",finished_reason="stop",model_name="…"} 911.0
vllm:request_success_total{engine="0",finished_reason="length",model_name="…"} 66.0
vllm:request_success_total{engine="0",finished_reason="repetition",model_name="…"} 0.0
vllm:(request_success_total)\{[^}]*\}\s+([0-9.eE+-]+) matches all three and
publishes request_success_total: 0.0 — the repetition line. The same
happens with histogram buckets (one line per le) and with gauges on a
multi-engine backend (one line per engine). Labels captured inside the name
group don’t help: normalization drops them, so the series still collide.
The probe logs a warning once per field name when this happens — either
more than one match produced field … or, for names that only became equal
after normalization, … both publish as field … after name normalization.
To read a specific series, pin its label in the regex so only one line matches:
# finished_reason="stop" only → request_success_total: 911.0
- regex: 'vllm:(request_success_total)\{[^}]*finished_reason="stop"[^}]*\}\s+([0-9.eE+-]+)'
To read several series of one metric at once, capture the distinguishing label value as the name; each value becomes its own field:
# → stop: 911.0, length: 66.0, repetition: 0.0
- regex: 'vllm:request_success_total\{[^}]*finished_reason="(\w+)"[^}]*\}\s+([0-9.eE+-]+)'
Two separate rules for the same metric (one per label) still publish the same name and collide; use the pattern above instead.
Counters.
*_totalmetrics only ever grow, so ameanover the window lags the current count. Useaggregate: { method: latest }for counters.
Example 1 — add a performance probe in config.yaml#
This probes a vLLM backend’s /metrics every 5 s, extracts two load signals
(both lower-is-better), and publishes them under the provider’s
performance.* namespace — the exact field the PerformanceBased strategy
reads — plus a JSONL history file.
providers:
- name: local
type: hosted_vllm
model: Qwen/Qwen3.5-9B
enabled: true
metadata:
labels: [local]
cost: 0
settings:
endpoint: http://localhost:8088/v1 # chat base (/v1)
timeout: 300.0
auth:
scheme: none
api_key: null
custom_headers: {}
probes:
- name: perf
enabled: true
endpoint: http://localhost:8088/metrics # scrape target (NOT /v1)
interval: 5
timeout: 5
extract:
- regex: 'vllm:(num_requests_waiting)\{[^}]*\}\s+([0-9.eE+-]+)'
- regex: 'vllm:(kv_cache_usage_perc)\{[^}]*\}\s+([0-9.eE+-]+)'
aggregate: { window: 12, method: mean }
publish:
- { to: metadata, prefix: performance } # → routing (live)
- { to: file, path: probes/local.perf.jsonl } # → history
File paths are workspace-relative. A relative
pathresolves against the runtime workspace directory (theconfig.yamldirectory, i.e.$GATEWAY_WORKSPACEor the dir of$GATEWAY_CONFIG), not the process CWD. Soprobes/local.perf.jsonllands in<workspace>/probes/wherever the server is launched from. Use an absolute path to write elsewhere.
The metadata sink writes each aggregate to performance.<name>, e.g.
performance.num_requests_waiting. A PerformanceBased strategy can then filter
and rank on it (a bare number is an upper bound; [lower, upper] is an inclusive
range with null for an open side — see
Performance-Based Routing):
# strategy.yaml
strategies:
- name: performance_based
provider_selector:
performance:
num_requests_waiting: 5 # drop providers with a longer queue
kv_cache_usage_perc: [null, 0.9] # drop providers whose KV cache is saturated
sort:
- { provider_attribute: performance.num_requests_waiting, descending: false }
Prefixes. Use
prefix: performanceto feed the routingperformance.*field directly. Useprefix: attributes.probe(the default) for generic live metadata that surfaces asprobe.<name>inmetadata.to_dict().
Latency metrics. vLLM exposes latencies as histograms (
vllm:time_to_first_token_seconds_sum/_count). A probe reads single values, so it cannot divide sum by count to get a mean TTFT in one rule — prefer the load gauges above, or publish the raw_sum/_countand compute downstream.
Example 2 — add the same probe at runtime via the providers API#
No restart needed: POST the provider with the probe in settings.probes. The
providers API rebuilds the runtime on every mutation, so the probe starts
immediately. settings is replaced wholesale, so resend the full settings
block, not just the probe.
curl -X POST http://localhost:8000/v1/providers/local \
-H 'content-type: application/json' \
-d '{
"settings": {
"endpoint": "http://localhost:8088/v1",
"timeout": 300.0,
"auth": {"scheme": "none", "api_key": null, "custom_headers": {}},
"probes": [
{
"name": "perf",
"enabled": true,
"endpoint": "http://localhost:8088/metrics",
"interval": 5,
"extract": [
{"regex": "vllm:(num_requests_waiting)\\{[^}]*\\}\\s+([0-9.eE+-]+)"}
],
"publish": [
{"to": "metadata", "prefix": "performance"},
{"to": "file", "path": "probes/local.perf.jsonl"}
]
}
]
}
}'
Confirm it was persisted:
curl -s http://localhost:8000/v1/providers/local | jq '.settings.probes'
Optional — edit in place: GET, append the probe, POST back#
Instead of hand-writing the whole settings block, you can fetch the provider’s
current config, splice the probe into settings.probes, and POST the result.
Because a GET response round-trips as a POST payload, jq can do the merge.
(Posting the GET’s metadata back is also fine: live probe values in it are not
persisted, see Reading live values.)
# 1. Define the probe to add.
read -r -d '' PROBE <<'JSON'
{
"name": "perf",
"enabled": true,
"endpoint": "http://localhost:8088/metrics",
"interval": 5,
"extract": [{"regex": "vllm:(num_requests_waiting)\\{[^}]*\\}\\s+([0-9.eE+-]+)"}],
"publish": [
{"to": "metadata", "prefix": "performance"},
{"to": "file", "path": "probes/local.perf.jsonl"}
]
}
JSON
# 2. GET current config, append the probe (replacing any probe of the same
# name), and POST just the settings back.
curl -s http://localhost:8000/v1/providers/local \
| jq --argjson probe "$PROBE" '{
settings: (.settings + {
probes: (((.settings.probes // []) | map(select(.name != $probe.name))) + [$probe])
})
}' \
| curl -X POST http://localhost:8000/v1/providers/local \
-H 'content-type: application/json' -d @-
Secrets caveat. A GET redacts secrets (
api_key: "***REDACTED***"). Posting that mask back would persist the literal string, breaking auth. This in-place edit is therefore safe for providers with no secret (e.g.auth.scheme: none, as above); for a provider that carries anapi_key, re-supply the real value (or a${VAR}placeholder) in the merged payload before POSTing.
Starting / stopping#
Lifecycle is just the enabled flag, driven through the same providers API:
start / add — POST with the probe present and
enabled: true.stop / remove — POST again with that probe’s
enabled: false(or omit it).all probes start on boot (
initialize()) and stop on shutdown.
Publishing & reading values#
A probe publishes through one or more sinks:
metadata — writes each aggregate into the live
ProviderMetadataat<prefix>.<name>. The strategy layer rebuilds{**metadata.to_dict()}on every route, so the next routing decision sees fresh values with no cache. This is how a probe influences routing.file — appends one JSONL record per tick:
{ts, provider, probe, aggregates}. This is the durable/offline read path. Because a probe runs forever, the history is capped: the sink keeps the most recentmax_entriesrecords (default10000) and drops older ones. Setmax_entrieson the sink to change the cap, ormax_entries: 0to disable it (unbounded growth).- { to: file, path: probes/local.perf.jsonl, max_entries: 50000 }
tail -1 <workspace>/probes/local.perf.jsonl
# {"ts": 1790136107.6, "provider": "local", "probe": "perf",
# "aggregates": {"num_requests_waiting": 0.0, "kv_cache_usage_perc": 0.0}}
Reading live values over HTTP — the providers API, not /v1/config#
Read live probe values from the providers API. GET /v1/providers and
GET /v1/providers/{name} return each running provider’s metadata as routing
sees it right now: the static metadata from config.yaml with the values the
metadata sinks have published merged in.
curl -s http://localhost:8000/v1/providers/local | jq '.metadata.performance'
# {"num_requests_waiting": 0.0, "kv_cache_usage_perc": 0.0}
GET /v1/config returns the configuration only: provider metadata there is
exactly what config.yaml declares, and never includes probe output. Changing
a probe’s extract and restarting therefore leaves /v1/config’s
metadata.performance unchanged; the probe’s configuration change is visible
under settings.probes, and its new values under the providers API.
A few details of the live view:
Values appear after the probe’s first successful tick (up to
intervalseconds after start or after any config API change, which rebuilds the runtime). Until thenmetadatashows only the static values.A disabled provider has no running probes, so the providers API returns its static metadata.
When the probe and
config.yamlset the same key (e.g. a staticperformance.ttft_msseed that a probe also publishes), the live view shows the probe’s value.
Probe output is never written to config.yaml. Posting a GET response back to
POST /v1/providers/{name} — whole, or edited, as the UI’s provider edit dialog
does — is safe: values the provider’s running probes published are dropped from
the posted metadata before it is saved (a key that config.yaml also defines
keeps its config.yaml value). So you can edit a probe’s settings from the UI
without freezing its current readings into the config. Hand-written metadata
keys the probes don’t publish are saved as sent.
Tips#
Point
endpointat the/metricsURL; the bare providerendpointis the chat/v1base and won’t yield metrics.Anchor a regex name group (
(num_requests_running)) for one metric; use a wildcard ((num_requests_\w+)) to discover several from one pattern. If the metric has several label sets, pin the label too — otherwise the last line wins (see same-name matches).Check the router log for
only the last match is keptwarnings after adding a probe: they mean a rule is silently discarding series.A failed scrape never kills the probe — it increments a fail counter and the window keeps its last good values until the next successful tick.
Keep
intervalsane: it is scrape load on the backend, not free.