Get Started#
This guide provides step-by-step instructions to quickly deploy and test the Multimodal Embedding Serving microservice.
Prerequisites#
Before you begin, confirm the following:
System Requirements: Your system meets the minimum requirements.
Docker Installed: Install Docker if needed. See Get Docker.
This guide assumes basic familiarity with Docker commands and terminal usage.
Environment Variables Reference#
Model Configuration#
EMBEDDING_MODEL_NAME - The model to use (e.g., “CLIP/clip-vit-b-16”). Refer to the Supported Models list for additional choices.
EMBEDDING_DEVICE - Device for inference (CPU/GPU/NPU, default: CPU)
EMBEDDING_USE_OV - Enable OpenVINO optimization (true/false, default: false;
setup.shsets it totrueunless it is already set). GME models (GME/*) run only on OpenVINO and requireEMBEDDING_USE_OV=trueon every device; withfalse, the service fails to start.EMBEDDING_OV_MODELS_DIR - OpenVINO model root. With
setup.shand Docker Compose, this is a container path, defaulting to/app/ov_models, not a host directory. See Use a prepared GME OpenVINO model.GME_PRECISION - Precision of GME models:
int8(default),int4orfp32. Each precision is converted and stored separately; see Use a prepared GME OpenVINO model.GME_QUERY_INSTRUCTION - Task instruction in the prompt of GME queries (default:
Find an image or text that matches the given query.). Documents are not affected.
Model Handler Performance#
INFER_BATCH_SIZE - Batch size for inference (default: 64;
setup.shoverrides this to 16 on CPU/NPU and 32 on GPU). Compiles model to accept fixed batch input. Padding or split is done to accommodate dynamic input sizes.PREPROCESS_WORKERS - Number of parallel preprocessing workers (default: min(16, cpu_count * 2)). Higher is better but yields diminishing returns if > number of CPU cores.
QwenText Models on NPU#
QwenText models require static-shape compilation on NPU and expose two
additional variables (EMBEDDING_STATIC_SEQ_LEN and
EMBEDDING_CHUNK_LONG_TEXT) that control the token budget and how longer text
is handled. These apply only to the QwenText family on NPU and need no
configuration by default.
Video Frame Extraction#
These variables control the video frame extraction pipeline performance and memory usage.
Extraction Performance#
VIDEO_FRAME_BATCH_SIZE - Batch size for video frame extraction (default: 64;
setup.shoverrides this to 64 on CPU/NPU and 256 on GPU)VIDEO_FRAME_DECODER_WORKERS - Number of workers for video frame decoding (default: 8)
VIDEO_FRAME_QUEUE_SIZE - Queue size for frame extraction pipeline (default: 32)
The defaults above are the application’s built-in values, which apply when the
service runs without setup.sh (for example the SDK/wheel or a bare container).
When you source setup.sh, it selects the batch sizes from EMBEDDING_DEVICE.
Exporting either variable before sourcing setup.sh always takes precedence.
Logging#
VIDEO_FRAME_LOG_LEVEL - Logging level for video frame extraction (DEBUG/INFO/WARNING/ERROR/CRITICAL, default: INFO)
Set Environment Values#
Basic Setup#
Set the required environment variables before launching the service.
export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32
Refer to the Supported Models list for additional choices.
NOTE: You can change the model, OpenVINO conversion, device, or tokenization parameters by editing
setup.sh.
Configure the Registry#
export REGISTRY_URL=intel
export TAG=2026.3.0-rc1
Configuration Examples#
Basic CPU setup (default):
export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32
GPU acceleration with OpenVINO:
export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32
export EMBEDDING_DEVICE=GPU
export EMBEDDING_USE_OV=true
High Performance Video Processing:
export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32
export VIDEO_FRAME_BATCH_SIZE=256
export VIDEO_FRAME_DECODER_WORKERS=8
export VIDEO_FRAME_SHM_POOL_BLOCK_SIZE=$((1920 * 1080 * 3)) # 6MB for 1080p
export VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER=2
export INFER_BATCH_SIZE=64
export PREPROCESS_WORKERS=16
Memory-Constrained Environment:
export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32
export VIDEO_FRAME_BATCH_SIZE=64
export VIDEO_FRAME_DECODER_WORKERS=4
export VIDEO_FRAME_SHM_POOL_BLOCK_SIZE=$((1280 * 720 * 3)) # 2.8MB for 720p
export VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER=2
With OpenVINO Optimization on GPU:
export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-16
export EMBEDDING_USE_OV=true
export EMBEDDING_DEVICE=GPU
export PREPROCESS_WORKERS=16
EMBEDDING_DEVICE=GPU already selects the tuned GPU batch sizes
(INFER_BATCH_SIZE=32, VIDEO_FRAME_BATCH_SIZE=256), so set those two only to
override the defaults.
Debug Mode with Detailed Logging:
export VIDEO_FRAME_LOG_LEVEL=INFO
export EMBEDDING_DEVICE=CPU
Performance Tuning Guide#
For Video Processing Bottlenecks#
Increase
VIDEO_FRAME_BATCH_SIZE(trades memory for throughput)Increase
VIDEO_FRAME_DECODER_WORKERS(limited by CPU cores)Increase
VIDEO_FRAME_QUEUE_SIZEif frames are being dropped
For Memory Constraints#
Decrease
VIDEO_FRAME_BATCH_SIZEDecrease
VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIERReduce
VIDEO_FRAME_SHM_POOL_BLOCK_SIZEif processing lower resolutions
For Inference Performance#
Increase
INFER_BATCH_SIZEandPREPROCESS_WORKERSEnable OpenVINO:
EMBEDDING_USE_OV=trueUse GPU or NPU if available:
EMBEDDING_DEVICE=GPUorEMBEDDING_DEVICE=NPU
Set the environment variables#
Set the environment with default values by running the below command. Note that this needs to be run anytime the environment variables are changed. For example: if running on GPU, additional environment variables will need to be set.
source setup.sh
Quick Start with Docker#
You can build the Docker image or pull a prebuilt image from the configured registry and tag. For prebuilt image, the setup script will configure the necessary variables to pull the right version of the image.
Use a prepared GME OpenVINO model#
On its first start, the service downloads the original GME checkpoint from Hugging Face and converts it to OpenVINO. To avoid that, for example on a host without Hugging Face access, convert the model beforehand and copy it into the service’s model volume.
GME_PRECISION selects the precision, for the conversion and for the service:
int8 (default), int4 (fastest, lower accuracy) or fp32 (original
accuracy, slowest).
Run the following from the microservice root, in one Bash terminal:
Select the precision and build the image:
export EMBEDDING_MODEL_NAME=GME/gme-qwen2-vl-2b-instruct export GME_PRECISION=int8 # or int4, fp32 unset REGISTRY_URL PROJECT_NAME export TAG=gme-local source setup.sh docker compose -f docker/compose.yaml build
Download the original checkpoint (about 9 GB). The download tool goes into a temporary virtual environment, which is deleted once the checkpoint is on disk. On Ubuntu,
python3 -m venvneeds thepython3-venvpackage.python3 -m venv /tmp/hf-venv /tmp/hf-venv/bin/pip install -U huggingface_hub /tmp/hf-venv/bin/hf download Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \ --include "*.json" --include "*.safetensors" --include "merges.txt" \ --local-dir ./gme-Qwen2-VL-2B-Instruct rm -rf /tmp/hf-venv
Convert it to OpenVINO with
scripts/export_gme_ov.py, which uses the precision fromGME_PRECISIONand writes the model under./gme-ov:mkdir -p ./gme-ov docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp -e GME_PRECISION \ -v "$PWD/gme-Qwen2-VL-2B-Instruct:/checkpoint:ro" -v "$PWD/gme-ov:/output" \ "multimodal-embedding-serving:$TAG" \ python scripts/export_gme_ov.py --checkpoint /checkpoint --output-dir /output
If you already have a GME OpenVINO model, skip steps 2 and 3 and place it at
./gme-ov/gme-qwen2-vl-2b-instruct-openvinoinstead; for INT4 or FP32, append-int4or-fp32to the directory name.Stop a running service and copy the model into the volume. The copy never overwrites a model already in the volume; it fails with
File existsinstead:docker compose -f docker/compose.yaml stop multimodal-embedding-serving set -o pipefail tar -C ./gme-ov -cf - . | docker compose -f docker/compose.yaml run --rm --no-deps -T \ --user root --entrypoint sh multimodal-embedding-serving -c ' set -eu tar -xkf - -C /app/ov_models chown -R appuser:appuser /app/ov_models '
Start the service, with the same
GME_PRECISION, and check it:docker compose -f docker/compose.yaml up -d --force-recreate docker logs -f --tail=100 multimodal-embedding-serving # wait for "GME loaded", then Ctrl+C curl -fsS http://localhost:9777/health
The log shows
Reusing existing GME OpenVINO packageand the precision.
The model stays in the ov-models volume across container restarts and
recreation; do not delete the volume.
Running the Server with CPU#
docker compose -f docker/compose.yaml up -d
Verify the deployment by running the below command. The user should see a healthy status printed on the console.
curl --location --request GET 'http://localhost:9777/health'
Running the Server with GPU#
1. Configure GPU Device#
# Automatic GPU selection
export EMBEDDING_DEVICE=GPU
# Specific GPU index (if applicable)
export EMBEDDING_DEVICE=GPU.0
2. Run Setup Script#
source setup.sh
Note
When EMBEDDING_DEVICE=GPU is set, setup.sh applies GPU-friendly defaults, including setting EMBEDDING_USE_OV=true.
Important
To run a GME model on the integrated GPU of an Intel Core Ultra Series 3
(Panther Lake) processor, set OV_PERFORMANCE_MODE=LATENCY. setup.sh sets
THROUGHPUT for GPU, so export LATENCY again after source setup.sh:
export OV_PERFORMANCE_MODE=LATENCY
In THROUGHPUT mode on this GPU, GME image and image+text requests can hang,
while the host kernel log reports a reset of the xe driver’s BCS engine.
Other GPUs are not affected.
3. Start the Service#
docker compose -f docker/compose.yaml up -d
4. Verify GPU Configuration#
# Check service health
curl --location --request GET 'http://localhost:9777/health'
# Inspect active model capabilities
curl --location --request GET 'http://localhost:9777/model/capabilities'
5. (Optional) Optimize GME on GPU#
For lower GME latency on an Intel GPU, use INT4 GME (GME_PRECISION=int4)
together with an image whose OpenVINO
GPU plugin is rebuilt with GME-specific optimization patches. See
Optimize GME on Intel GPU.
Running the Server with NPU#
1. Configure NPU Device#
export EMBEDDING_DEVICE=NPU
Note
NPU support is model-dependent. Verify that the selected model is supported on NPU by checking the OpenVINO Supported Models.
2. Run the Setup Script#
source setup.sh
Note
When EMBEDDING_DEVICE=NPU is set, setup.sh automatically enables EMBEDDING_USE_OV=true.
3. Start the Service with docker compose#
docker compose -f docker/compose.yaml up -d
4. Verify NPU Configuration#
# Verify /dev/accel/accel0 is available on the host
ls -l /dev/accel/accel0
# Check service health
curl --location --request GET 'http://localhost:9777/health'
# Inspect active model capabilities
curl --location --request GET 'http://localhost:9777/model/capabilities'
Stop the Multimodal Embedding microservice#
docker compose -f docker/compose.yaml down
Sample CURL Commands#
The following samples mirror the accompanying Postman collection. All requests target http://localhost:9777.
encoding_format is optional for /embeddings; if omitted, the service defaults it to float.
Text Embedding#
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"input": {
"type": "text",
"text": "Sample input text1"
},
"model": "CLIP/clip-vit-b-32"
}'
Document Embedding (multiple texts)#
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"input": {
"type": "text",
"text": ["Sample input text1", "Sample input text2"]
},
"model": "CLIP/clip-vit-b-32",
"encoding_format": "float"
}'
Image URL Embedding#
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"input": {
"type": "image_url",
"image_url": "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg"
},
"model": "CLIP/clip-vit-b-32",
"encoding_format": "float"
}'
Image Base64 Embedding#
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"model": "CLIP/clip-vit-b-32",
"encoding_format": "float",
"input": {
"type": "image_base64",
"image_base64": "<image base64 value here>"
}
}'
Video Frames Embedding#
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"model": "CLIP/clip-vit-b-32",
"encoding_format": "float",
"input": {
"type": "video_frames",
"video_frames": [
{
"type": "image_url",
"image_url": "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg"
},
{
"type": "image_base64",
"image_base64": "<image base64 value here>"
}
]
}
}'
Video URL Embedding (with segment config)#
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"model": "CLIP/clip-vit-b-32",
"encoding_format": "float",
"input": {
"type": "video_url",
"video_url": "https://raw.githubusercontent.com/intel-iot-devkit/sample-videos/refs/heads/master/car-detection.mp4",
"segment_config": {
"startOffsetSec": 0,
"clip_duration": -1,
"num_frames": 64,
"frame_indexes": [1, 10, 20]
}
}
}'
Video Base64 Embedding#
set num_frames: 0 to process all the frames.
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"model": "CLIP/clip-vit-b-32",
"encoding_format": "float",
"input": {
"type": "video_base64",
"segment_config": {
"startOffsetSec": 0,
"clip_duration": -1,
"num_frames": 64
},
"video_base64": "<video base64 value here>"
}
}'
Fused Image + Text Embedding#
Each item’s image and text are embedded jointly into one vector. Only models whose /model/capabilities reports supports_fused: true (GME) accept this input.
curl --location 'http://localhost:9777/embeddings' \
--header 'Content-Type: application/json' \
--data '{
"model": "GME/gme-qwen2-vl-2b-instruct",
"encoding_format": "float",
"input": {
"type": "fused",
"items": [
{
"text": "Figure 3: throughput under different loads",
"image": {"type": "image_base64", "image_base64": "<image base64 value here>"}
}
]
}
}'
Models, Current Model, and Capabilities#
# List all available models
curl --location --request GET 'http://localhost:9777/models'
# Inspect the currently loaded model
curl --location --request GET 'http://localhost:9777/model/current'
# View modality support for the active model
curl --location --request GET 'http://localhost:9777/model/capabilities'
Troubleshooting#
Docker container fails to start
Run
docker logs multimodal-embedding-servingto inspect failures.Ensure required ports (default
9777) are available.
Cannot access the microservice
Confirm the containers are running:
docker psVerify
EMBEDDING_MODEL_NAMEpoints to a supported entry and rerunsource setup.shif you make changes.
GPU runtime errors
Check Intel GPU device nodes:
ls -la /dev/dri
Confirm
EMBEDDING_USE_OV=truefor best performance with OpenVINO on GPU.
NPU acceleration inactive
Confirm
/dev/accel/accel0is available on the host:ls -l /dev/accel/accel0
Ensure
EMBEDDING_DEVICE=NPUis set andEMBEDDING_USE_OV=true.Verify the selected model supports NPU inference via the OpenVINO Supported Models page.
Known Issues and NPU Notes#
First NPU run is slow (one-time model compilation): The first time a model is loaded on NPU, OpenVINO compiles the model to an NPU-specific blob, which can take noticeably longer than CPU/GPU startup. This is expected and happens only once per model/configuration.
Model caching speeds up subsequent runs: The service caches the compiled OpenVINO blob on disk, so later starts reuse it and come up quickly. The cache location is controlled by
OV_CACHE_DIR(orEMBEDDING_OV_CACHE_DIR); when unset it defaults to anov_cache/directory next to the OpenVINO IR. Persist this directory (for example, via a mounted volume) to retain the cache across container restarts.