Lingua Server — Docker Compose Deployment#

Reference docker compose recipe for the Lingua FastAPI server. The PyTorch service uses XPU by default. Select the backend by starting the corresponding service; use LINGUA_DEVICE to choose the hardware.

The deployment uses two backend-specific Dockerfiles:

  • Dockerfile.pytorch for PyTorch / IPEX / XPU

  • Dockerfile.ov for OpenVINO

These are alternative backends. Start only the server that matches your requirements; you do not need to run both services.

The compose file defines two services:

  • lingua-pytorch on host port 8001

  • lingua-ov on host port 8002 (start explicitly; override with LINGUA_OV_PORT)

The server uses the llmlingua2 (LLMLingua-2) compression mode.

Each image carries only the standalone server file and its runtime deps. The PyTorch image installs torch + IPEX + llmlingua + fastapi + uvicorn. The OpenVINO image installs CPU torch + OpenVINO + optimum[openvino]. Neither image installs the adaptive-token-compressor library — clients reach the running container over HTTP via LinguaHTTPBackend(lingua_url=...).

Quick start#

cd deployment/lingua
docker compose up -d --build lingua-pytorch

Default: --device xpu, --port 8001. The library client default (LinguaHTTPBackend(lingua_url="http://localhost:8001/compress")) matches out of the box — no config change required.

To start the OpenVINO service:

docker compose up -d --build lingua-ov

Current backend/mode support status:

  • PyTorch + llmlingua2: supported

  • OpenVINO + llmlingua2: supported

Startup mode: llmlingua2.

Override variables (no .env file needed)#

All variables have :-default fallbacks in docker-compose.yaml. Pass inline on the command line:

# CPU fallback
LINGUA_DEVICE=cpu docker compose up -d --build lingua-pytorch

# OpenVINO backend on XPU (maps to OV GPU)
LINGUA_DEVICE=xpu docker compose up -d --build lingua-ov

# Select XPU index (PyTorch xpu:<index>; OpenVINO prefers GPU.<index>)
LINGUA_DEVICE=xpu LINGUA_XPU_INDEX=1 docker compose up -d --build lingua-pytorch

# Different port
LINGUA_PORT=9000 docker compose up -d --build lingua-pytorch

# Pin model independently
LINGUA_MODEL_NAME_ID=microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank \
  docker compose up -d --build lingua-pytorch

# Combine
LINGUA_DEVICE=cpu LINGUA_PORT=9000 docker compose up -d --build lingua-pytorch

Variable

Default

Notes

LINGUA_BIND_HOST

127.0.0.1

Host bind address for the published port. Setting it to 0.0.0.0 exposes the HTTP service to the network and is generally not recommended without appropriate network and TLS controls.

LINGUA_PORT

8001

Container always listens on 8001; this maps host port.

LINGUA_OV_PORT

8002

Host port for the lingua-ov service.

LINGUA_DEVICE

xpu

xpu / cpu / cuda. xpu requires /dev/dri on host.

LINGUA_XPU_INDEX

0

XPU index when LINGUA_DEVICE=xpu. PyTorch uses xpu:<index>; OpenVINO prefers GPU.<index> and accepts generic GPU as fallback for index 0.

LINGUA_MODEL_NAME_ID

(empty)

HF model ID. If empty, defaults to microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank.

HF_HUB_OFFLINE

0

First-run downloads allowed. Set 1 for strict offline. Default is 0 because the image ships no model — a fresh machine must pull it on first start. Once the model is cached (and, for OV, the IR is persisted), prefer HF_HUB_OFFLINE=1: with 0, hf_hub still issues metadata/revalidation calls on every start, 1 trusts the local cache and makes zero network calls, so it is faster.

HF_ENDPOINT

https://hf-mirror.com

Mainland China mirror; unset/override for upstream HF.

http_proxy/https_proxy/no_proxy

(unset)

Build-time + runtime proxies.

VIDEO_GID/RENDER_GID

44/992

GPU passthrough; detect on host with getent group.

Verify#

curl http://localhost:8001/health
# → {
#     "status":"ok",
#     "mode":"llmlingua2",
#     "supports_request_mode_override":true,
#     "supported_modes":["llmlingua2"],
#     "initialized_modes":{
#       "llmlingua2":{
#         "model_name_id":"...",
#         "device":"xpu:0",
#         "execution_devices":"n/a"
#       }
#     }
#   }

curl -X POST http://localhost:8001/compress \
  -H 'Content-Type: application/json' \
  -d '{"text":"...","rate":0.5}'
# → {"compressed_prompt":"...","compression_time_ms":12.3,...}

For OV backend, the startup logs include explicit device mapping and resolved runtime info, e.g.:

  • Backend=ov  Requested device=xpu

  • OpenVINO requested device=xpu mapped device=GPU.0 (or GPU fallback for index 0)

  • OV[GPU.0] name: Intel(R) ... (device name may vary by runtime)

  • OpenVINO execution devices: ['GPU'] / GPU.0 (runtime-specific)

  • Model runtime device: ov:GPU.0 (or ov:GPU)

Test that the digit_neighbor_radius patch is active:

curl -X POST http://localhost:8001/compress \
  -H 'Content-Type: application/json' \
  -d '{"text":"price is 99.5 USD nearby","rate":0.3,"force_reserve_digit":true,"digit_neighbor_radius":3}'
# → compressed_prompt should retain "99.5" and surrounding words

Stop & cleanup#

docker compose --profile pytorch down  # stop / remove the PyTorch service/container
docker compose --profile ov down       # stop / remove the OpenVINO service/container
docker compose --profile pytorch --profile ov down -v  # remove both services and the HF model cache volume

See also#

  • Bare-metal install (without docker): src/adaptive_token_compressor/model_servers/lingua/README.md uses pip install adaptive-token-compressor[lingua-server-xpu|-cpu|-ov] + python -m adaptive_token_compressor.model_servers.lingua.apply_patch + python -m adaptive_token_compressor.model_servers.lingua

  • Companion vLLM tool predictor: tool-predictor-deployment.md