Tool Predictor — Bring Your Own vLLM Docker Compose#
The ToolCompressor in adaptive_token_compressor.tool uses an external model
to score tool relevance. Two predictor backends are supported, each needing a different endpoint:
|
Validated model |
Endpoint |
|---|---|---|
|
|
OpenAI-compatible |
|
|
Jina/Cohere-style |
See Predictor configurations for how to choose between them.
This library does not provide or maintain a vLLM deployment yaml anymore. Please start your own model service with Docker Compose and expose the endpoint for the backend you chose.
Experiment Baseline#
Internal validation was run with:
|
Model |
Image |
Endpoint |
|---|---|---|---|
|
|
|
|
|
|
|
|
You may use different models/images as long as the endpoint matches the contract below.
Setup (Bring Your Own Compose)#
Prepare your own Docker Compose yaml for vLLM.
Mount your model weights path in that yaml.
Start your service, for example:
docker compose -f /path/to/your-vllm-compose.yaml up -d
Confirm the service endpoint is reachable.
If needed, adjust your system GPU group GIDs (typical Debian/Ubuntu values are 44 for video
and 992 for render):
getent group video | cut -d: -f3
getent group render | cut -d: -f3
Endpoint#
Use the host/port exposed by your own Docker Compose service.
Generative backend#
OpenAI-compatible chat completions:
http://<your-host>:<your-port>/v1/chat/completions
Wire into the tool compressor:
from adaptive_token_compressor.tool import ToolCompressor
compressor = ToolCompressor(
predictor_url="http://<your-host>:<your-port>/v1/chat/completions",
predictor_model="Qwen/Qwen3.6-35B-A3B",
predictor_backend="generative",
score_threshold=2.0,
)
Reranker backend#
Jina/Cohere-style rerank API, as served by vLLM for reranker models:
http://<your-host>:<your-port>/v1/rerank
The compressor sends one request per turn that scores every tool:
{"model": "Qwen/Qwen3-Reranker-0.6B", "query": "<task and agent context>", "documents": ["<tool 1>", "..."], "top_n": <number of tools>}
The response must contain a results array with one entry per document, each
with its index and a relevance_score in [0, 1]. A missing or out-of-range
score fails the prediction, and the request is passed through uncompressed.
Health checks use /v1/models on the same host/port, which must list the
model name.
Wire into the tool compressor:
from adaptive_token_compressor.tool import ToolCompressor
compressor = ToolCompressor(
predictor_url="http://<your-host>:<your-port>/v1/rerank",
predictor_model="Qwen/Qwen3-Reranker-0.6B",
predictor_backend="reranker",
top_k=20,
)
Size top_k to the number of tools per request; see
Predictor configurations.
Override the port#
Use any host port you prefer by changing the port mapping in your own Docker Compose yaml, then restart the service.
docker compose -f /path/to/your-vllm-compose.yaml up -d
Stop#
docker compose -f /path/to/your-vllm-compose.yaml down
Alternative#
You’re not locked into one exact deployment. Any endpoint that matches the contract above for your chosen backend works.
See also#
Library usage and configuration: get-started.md
Companion compress server: compress-server-deployment.md