# Get Started This guide provides step-by-step instructions to quickly deploy and test the **Multimodal Embedding Serving microservice**. ## Prerequisites Before you begin, confirm the following: - **System Requirements**: Your system meets the [minimum requirements](./get-started/system-requirements.md). - **Docker Installed**: Install Docker if needed. See [Get Docker](https://docs.docker.com/get-docker/). This guide assumes basic familiarity with Docker commands and terminal usage. ## Environment Variables Reference ### Model Configuration - **EMBEDDING_MODEL_NAME** - The model to use (e.g., "CLIP/clip-vit-b-16"). Refer to the [Supported Models](./supported-models.md) list for additional choices. - **EMBEDDING_DEVICE** - Device for inference (CPU/GPU/NPU, default: CPU) - **EMBEDDING_USE_OV** - Enable OpenVINO optimization (true/false, default: false; `setup.sh` sets it to `true` unless it is already set). GME models (`GME/*`) run only on OpenVINO and require `EMBEDDING_USE_OV=true` on every device; with `false`, the service fails to start. - **EMBEDDING_OV_MODELS_DIR** - OpenVINO model root. With `setup.sh` and Docker Compose, this is a **container path**, defaulting to `/app/ov_models`, not a host directory. See [Use a prepared GME OpenVINO model](#use-a-prepared-gme-openvino-model). - **GME_PRECISION** - Precision of GME models: `int8` (default), `int4` or `fp32`. Each precision is converted and stored separately; see [Use a prepared GME OpenVINO model](#use-a-prepared-gme-openvino-model). - **GME_QUERY_INSTRUCTION** - Task instruction in the prompt of GME queries (default: `Find an image or text that matches the given query.`). Documents are not affected. ### Model Handler Performance - **INFER_BATCH_SIZE** - Batch size for inference (default: 64; `setup.sh` overrides this to 16 on CPU/NPU and 32 on GPU). Compiles model to accept fixed batch input. Padding or split is done to accommodate dynamic input sizes. - **PREPROCESS_WORKERS** - Number of parallel preprocessing workers (default: min(16, cpu_count * 2)). Higher is better but yields diminishing returns if > number of CPU cores. ### QwenText Models on NPU QwenText models require static-shape compilation on NPU and expose two additional variables (`EMBEDDING_STATIC_SEQ_LEN` and `EMBEDDING_CHUNK_LONG_TEXT`) that control the token budget and how longer text is handled. These apply only to the QwenText family on NPU and need no configuration by default. See [QwenText Models on NPU](./qwentext-on-npu.md). ### Video Frame Extraction These variables control the video frame extraction pipeline performance and memory usage. #### Extraction Performance - **VIDEO_FRAME_BATCH_SIZE** - Batch size for video frame extraction (default: 64; `setup.sh` overrides this to 64 on CPU/NPU and 256 on GPU) - **VIDEO_FRAME_DECODER_WORKERS** - Number of workers for video frame decoding (default: 8) - **VIDEO_FRAME_QUEUE_SIZE** - Queue size for frame extraction pipeline (default: 32) The defaults above are the application's built-in values, which apply when the service runs without `setup.sh` (for example the SDK/wheel or a bare container). When you source `setup.sh`, it selects the batch sizes from `EMBEDDING_DEVICE`. Exporting either variable before sourcing `setup.sh` always takes precedence. #### Shared Memory Configuration - **VIDEO_FRAME_SHM_POOL_BLOCK_SIZE** - Shared memory block size in bytes (default: 1920*1080*3 = 6,220,800 bytes for 1080p RGB) - **VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER** - Multiplier for total shared memory blocks (default: 2) - Total blocks = VIDEO_FRAME_BATCH_SIZE × VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER #### Logging - **VIDEO_FRAME_LOG_LEVEL** - Logging level for video frame extraction (DEBUG/INFO/WARNING/ERROR/CRITICAL, default: INFO) ## Set Environment Values ### Basic Setup Set the required environment variables before launching the service. ```bash export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32 ``` Refer to the [Supported Models](./supported-models.md) list for additional choices. > ***NOTE:*** You can change the model, OpenVINO conversion, device, or tokenization parameters by editing `setup.sh`. ### Configure the Registry ```bash export REGISTRY_URL=intel export TAG=2026.3.0-rc1 ``` ### Configuration Examples **Basic CPU setup (default)**: ```bash export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32 ``` **GPU acceleration with OpenVINO**: ```bash export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32 export EMBEDDING_DEVICE=GPU export EMBEDDING_USE_OV=true ``` **High Performance Video Processing**: ```bash export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32 export VIDEO_FRAME_BATCH_SIZE=256 export VIDEO_FRAME_DECODER_WORKERS=8 export VIDEO_FRAME_SHM_POOL_BLOCK_SIZE=$((1920 * 1080 * 3)) # 6MB for 1080p export VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER=2 export INFER_BATCH_SIZE=64 export PREPROCESS_WORKERS=16 ``` **Memory-Constrained Environment**: ```bash export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-32 export VIDEO_FRAME_BATCH_SIZE=64 export VIDEO_FRAME_DECODER_WORKERS=4 export VIDEO_FRAME_SHM_POOL_BLOCK_SIZE=$((1280 * 720 * 3)) # 2.8MB for 720p export VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER=2 ``` **With OpenVINO Optimization on GPU**: ```bash export EMBEDDING_MODEL_NAME=CLIP/clip-vit-b-16 export EMBEDDING_USE_OV=true export EMBEDDING_DEVICE=GPU export PREPROCESS_WORKERS=16 ``` `EMBEDDING_DEVICE=GPU` already selects the tuned GPU batch sizes (`INFER_BATCH_SIZE=32`, `VIDEO_FRAME_BATCH_SIZE=256`), so set those two only to override the defaults. **Debug Mode with Detailed Logging**: ```bash export VIDEO_FRAME_LOG_LEVEL=INFO export EMBEDDING_DEVICE=CPU ``` ### Performance Tuning Guide #### For Video Processing Bottlenecks 1. Increase `VIDEO_FRAME_BATCH_SIZE` (trades memory for throughput) 2. Increase `VIDEO_FRAME_DECODER_WORKERS` (limited by CPU cores) 3. Increase `VIDEO_FRAME_QUEUE_SIZE` if frames are being dropped #### For Memory Constraints 1. Decrease `VIDEO_FRAME_BATCH_SIZE` 2. Decrease `VIDEO_FRAME_SHM_POOL_BLOCKS_MULTIPLIER` 3. Reduce `VIDEO_FRAME_SHM_POOL_BLOCK_SIZE` if processing lower resolutions #### For Inference Performance 1. Increase `INFER_BATCH_SIZE` and `PREPROCESS_WORKERS` 2. Enable OpenVINO: `EMBEDDING_USE_OV=true` 3. Use GPU or NPU if available: `EMBEDDING_DEVICE=GPU` or `EMBEDDING_DEVICE=NPU` ### Set the environment variables Set the environment with default values by running the below command. Note that this needs to be run anytime the environment variables are changed. For example: if running on GPU, additional environment variables will need to be set. ```bash source setup.sh ``` ## Quick Start with Docker You can [build the Docker image](./get-started/build-from-source.md#steps-to-build) or pull a prebuilt image from the configured registry and tag. For prebuilt image, the `setup` script will configure the necessary variables to pull the right version of the image. ### Use a prepared GME OpenVINO model On its first start, the service downloads the original GME checkpoint from Hugging Face and converts it to OpenVINO. To avoid that, for example on a host without Hugging Face access, convert the model beforehand and copy it into the service's model volume. `GME_PRECISION` selects the precision, for the conversion and for the service: `int8` (default), `int4` (fastest, lower accuracy) or `fp32` (original accuracy, slowest). Run the following from the microservice root, in one Bash terminal: 1. Select the precision and build the image: ```bash export EMBEDDING_MODEL_NAME=GME/gme-qwen2-vl-2b-instruct export GME_PRECISION=int8 # or int4, fp32 unset REGISTRY_URL PROJECT_NAME export TAG=gme-local source setup.sh docker compose -f docker/compose.yaml build ``` 2. Download the original checkpoint (about 9 GB). The download tool goes into a temporary virtual environment, which is deleted once the checkpoint is on disk. On Ubuntu, `python3 -m venv` needs the `python3-venv` package. ```bash python3 -m venv /tmp/hf-venv /tmp/hf-venv/bin/pip install -U huggingface_hub /tmp/hf-venv/bin/hf download Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \ --include "*.json" --include "*.safetensors" --include "merges.txt" \ --local-dir ./gme-Qwen2-VL-2B-Instruct rm -rf /tmp/hf-venv ``` 3. Convert it to OpenVINO with [`scripts/export_gme_ov.py`](../../scripts/export_gme_ov.py), which uses the precision from `GME_PRECISION` and writes the model under `./gme-ov`: ```bash mkdir -p ./gme-ov docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp -e GME_PRECISION \ -v "$PWD/gme-Qwen2-VL-2B-Instruct:/checkpoint:ro" -v "$PWD/gme-ov:/output" \ "multimodal-embedding-serving:$TAG" \ python scripts/export_gme_ov.py --checkpoint /checkpoint --output-dir /output ``` If you already have a GME OpenVINO model, skip steps 2 and 3 and place it at `./gme-ov/gme-qwen2-vl-2b-instruct-openvino` instead; for INT4 or FP32, append `-int4` or `-fp32` to the directory name. 4. Stop a running service and copy the model into the volume. The copy never overwrites a model already in the volume; it fails with `File exists` instead: ```bash docker compose -f docker/compose.yaml stop multimodal-embedding-serving set -o pipefail tar -C ./gme-ov -cf - . | docker compose -f docker/compose.yaml run --rm --no-deps -T \ --user root --entrypoint sh multimodal-embedding-serving -c ' set -eu tar -xkf - -C /app/ov_models chown -R appuser:appuser /app/ov_models ' ``` 5. Start the service, with the same `GME_PRECISION`, and check it: ```bash docker compose -f docker/compose.yaml up -d --force-recreate docker logs -f --tail=100 multimodal-embedding-serving # wait for "GME loaded", then Ctrl+C curl -fsS http://localhost:9777/health ``` The log shows `Reusing existing GME OpenVINO package` and the precision. The model stays in the `ov-models` volume across container restarts and recreation; do not delete the volume. ## Running the Server with CPU ```bash docker compose -f docker/compose.yaml up -d ``` Verify the deployment by running the below command. The user should see a `healthy` status printed on the console. ```bash curl --location --request GET 'http://localhost:9777/health' ``` ## Running the Server with GPU ### 1. Configure GPU Device ```bash # Automatic GPU selection export EMBEDDING_DEVICE=GPU # Specific GPU index (if applicable) export EMBEDDING_DEVICE=GPU.0 ``` ### 2. Run Setup Script ```bash source setup.sh ``` > [!NOTE] > When `EMBEDDING_DEVICE=GPU` is set, `setup.sh` applies GPU-friendly defaults, including setting `EMBEDDING_USE_OV=true`. > [!IMPORTANT] > To run a GME model on the integrated GPU of an Intel Core Ultra Series 3 > (Panther Lake) processor, set `OV_PERFORMANCE_MODE=LATENCY`. `setup.sh` sets > `THROUGHPUT` for GPU, so export `LATENCY` again after `source setup.sh`: > > ```bash > export OV_PERFORMANCE_MODE=LATENCY > ``` > > In `THROUGHPUT` mode on this GPU, GME image and image+text requests can hang, > while the host kernel log reports a reset of the `xe` driver's BCS engine. > Other GPUs are not affected. ### 3. Start the Service ```bash docker compose -f docker/compose.yaml up -d ``` ### 4. Verify GPU Configuration ```bash # Check service health curl --location --request GET 'http://localhost:9777/health' # Inspect active model capabilities curl --location --request GET 'http://localhost:9777/model/capabilities' ``` ### 5. (Optional) Optimize GME on GPU For lower GME latency on an Intel GPU, use INT4 GME (`GME_PRECISION=int4`) together with an image whose OpenVINO GPU plugin is rebuilt with GME-specific optimization patches. See [Optimize GME on Intel GPU](./gme-embedding-optimization.md). ## Running the Server with NPU ### 1. Configure NPU Device ```bash export EMBEDDING_DEVICE=NPU ``` > [!NOTE] > NPU support is model-dependent. Verify that the selected model is supported on NPU by checking the [OpenVINO Supported Models](https://docs.openvino.ai/2026/documentation/compatibility-and-support/supported-models.html). ### 2. Run the Setup Script ```bash source setup.sh ``` > [!NOTE] > When `EMBEDDING_DEVICE=NPU` is set, `setup.sh` automatically enables `EMBEDDING_USE_OV=true`. ### 3. Start the Service with docker compose ```bash docker compose -f docker/compose.yaml up -d ``` ### 4. Verify NPU Configuration ```bash # Verify /dev/accel/accel0 is available on the host ls -l /dev/accel/accel0 # Check service health curl --location --request GET 'http://localhost:9777/health' # Inspect active model capabilities curl --location --request GET 'http://localhost:9777/model/capabilities' ``` ## Stop the Multimodal Embedding microservice ```bash docker compose -f docker/compose.yaml down ``` ## Sample CURL Commands The following samples mirror the accompanying Postman collection. All requests target `http://localhost:9777`. `encoding_format` is optional for `/embeddings`; if omitted, the service defaults it to `float`. ### Text Embedding ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "input": { "type": "text", "text": "Sample input text1" }, "model": "CLIP/clip-vit-b-32" }' ``` ### Document Embedding (multiple texts) ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "input": { "type": "text", "text": ["Sample input text1", "Sample input text2"] }, "model": "CLIP/clip-vit-b-32", "encoding_format": "float" }' ``` ### Image URL Embedding ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "input": { "type": "image_url", "image_url": "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg" }, "model": "CLIP/clip-vit-b-32", "encoding_format": "float" }' ``` ### Image Base64 Embedding ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "model": "CLIP/clip-vit-b-32", "encoding_format": "float", "input": { "type": "image_base64", "image_base64": "" } }' ``` ### Video Frames Embedding ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "model": "CLIP/clip-vit-b-32", "encoding_format": "float", "input": { "type": "video_frames", "video_frames": [ { "type": "image_url", "image_url": "https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_bike.jpg" }, { "type": "image_base64", "image_base64": "" } ] } }' ``` ### Video URL Embedding (with segment config) ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "model": "CLIP/clip-vit-b-32", "encoding_format": "float", "input": { "type": "video_url", "video_url": "https://raw.githubusercontent.com/intel-iot-devkit/sample-videos/refs/heads/master/car-detection.mp4", "segment_config": { "startOffsetSec": 0, "clip_duration": -1, "num_frames": 64, "frame_indexes": [1, 10, 20] } } }' ``` ### Video Base64 Embedding set `num_frames: 0` to process all the frames. ```bash curl --location 'http://localhost:9777/embeddings' \ --header 'Content-Type: application/json' \ --data '{ "model": "CLIP/clip-vit-b-32", "encoding_format": "float", "input": { "type": "video_base64", "segment_config": { "startOffsetSec": 0, "clip_duration": -1, "num_frames": 64 }, "video_base64": "