# Environment Variables Configuration is managed via environment variables. All variables map directly to Pydantic Settings field names (case-insensitive). The `.env` file is loaded automatically by `docker compose up`. ## Core Settings | Variable | Default | Description | |----------|---------|-------------| | `HOST` | `0.0.0.0` | API server bind address | | `METRICS_PORT` | `9090` | Metrics Manager API port | | `SERVICE_NAME` | `metrics-manager` | Service name used in logs and health checks | | `SERVICE_VERSION` | `2026.3.0` | Service version reported in health endpoints | | `ENVIRONMENT` | `production` | Deployment environment: `development`, `staging`, `production`. Production disables `/docs` and `/redoc` Swagger endpoints | | `LOG_LEVEL` | `INFO` | Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` | | `LOG_FORMAT` | `json` | Log format: `json` (structured, for log aggregators) or `text` (human-readable) | | `LOG_INCLUDE_TIMESTAMP` | `true` | Include timestamp field in log entries | | `CORS_ORIGINS` | `*` | Allowed CORS origins (comma-separated or JSON array of strings). Malformed JSON arrays or non-string entries fail startup. Set to `http://localhost:3000,http://my-dashboard:3000` to restrict | | `CORS_ALLOW_CREDENTIALS` | `false` | Allow credentials in CORS requests (cookies, etc.) | | `METRICS_MANAGER_HOSTNAME` | _(unset)_ | Override the `host=` tag stamped on every metric (Telegraf and the `mm-plugin-*` collectors). Unset = use kernel hostname. Set to a stable value (e.g., `lab-node-42`) to keep Grafana dashboards stable across reboots | ## Metrics Storage | Variable | Default | Description | |----------|---------|-------------| | `METRICS_RETENTION_SECONDS` | `300` | How long to keep metrics in memory (seconds). After this duration, metrics are expired and removed on next store access | | `MAX_METRICS_BATCH_SIZE` | `1000` | Maximum metrics per single batch request (`POST /api/v1/metrics`) | | `MAX_METRICS_IN_MEMORY` | `100000` | Maximum metrics in memory. When exceeded, oldest entries are evicted automatically | | `CUSTOM_METRICS_DIR` | `/app/custom-metrics` | Directory where custom metric scripts are executed (via Telegraf `inputs.exec`). Do not change inside container | | `FILE_PERSIST_DEBOUNCE_MS` | `100` | Debounce interval for Telegraf HTTP persistence (milliseconds). Higher values reduce HTTP calls but increase latency. Range: 10–5000 ms | ## Rate Limiting | Variable | Default | Description | |----------|---------|-------------| | `RATE_LIMIT_ENABLED` | `true` | Enable rate limiting by client IP | | `RATE_LIMIT_REQUESTS_PER_MINUTE` | `1000` | Maximum requests per minute per client IP | | `RATE_LIMIT_BURST` | `100` | Burst allowance (tokens available before rate limit kicks in) | **Exempt paths**: `/health`, `/api/v1/stats`, and SSE endpoints are NOT rate limited. ## Performance | Variable | Default | Description | |----------|---------|-------------| | `ENABLE_GZIP_COMPRESSION` | `true` | Enable gzip compression for HTTP responses >1 KB. Reduces bandwidth but increases CPU usage | ## Security | Variable | Default | Description | |----------|---------|-------------| | `TRUST_FORWARDED_HEADERS` | `false` | Honor `X-Forwarded-For` / `X-Real-IP` headers for client IP detection. Set to `true` ONLY when running behind a trusted reverse proxy (Nginx, Traefik, etc.) | > **Warning:** Setting this to `true` without a reverse proxy allows clients to spoof their IP and bypass rate limiting. ## Telegraf Settings (Application Configuration) These variables configure the Telegraf locations for the application to use: | Variable | Default | Description | |----------|---------|-------------| | `TELEGRAF_CONFIG_PATH` | `/etc/telegraf/telegraf.conf` | Main Telegraf config inside the container, read by the entrypoint and passed to Telegraf | | `TELEGRAF_PORT` | `9273` | Telegraf Prometheus endpoint port (where SSE poller fetches system metrics) | | `TELEGRAF_HTTP_PORT` | `8186` | Docker-only Telegraf HTTP listener port; used by the shipped configuration | | `TELEGRAF_HTTP_ENDPOINT` | `http://localhost:8186/write` | Telegraf HTTP listener endpoint used to persist custom metrics in InfluxDB Line Protocol. Must be accessible from the app container | ## SSE Poller Settings The SSE endpoint (`/metrics/stream`) polls the Telegraf Prometheus endpoint for each connected client independently. There is no shared queue. | Variable | Default | Description | |----------|---------|-------------| | `PROMETHEUS_POLLER_INTERVAL_MS` | `500` | Polling interval in milliseconds (100–5000). Lower values = more frequent SSE events but higher CPU usage per client. Recommended: 500 ms | | `PROMETHEUS_TELEGRAF_ENDPOINT` | `http://localhost:9273` | Telegraf Prometheus endpoint polled by SSE clients. Must be accessible from the app container | ## Docker Compose Variables These variables are used ONLY by `compose.yaml` and are NOT read by the application: | Variable | Default | Description | |----------|---------|-------------| | `TELEGRAF_CONFIG` | `./telegraf.conf` | Host path to Telegraf config file (mounted into container) | | `TELEGRAF_CONFIG_DIR` | `./telegraf.d` | Host directory mounted at `/etc/telegraf/telegraf.d`; `.conf` files are snapshotted and loaded at startup | | `HOST_METRICS_PORT` | `9090` | API host mapping in bridge mode; actual API listener in host-NIC mode | | `HOST_TELEGRAF_PORT` | `9273` | Prometheus host mapping in bridge mode; actual listener in host-NIC mode | | `HOST_TELEGRAF_HTTP_PORT` | `8186` | HTTP ingest host mapping in bridge mode; actual listener in host-NIC mode | The default `metrics-manager` service uses bridge networking. `metrics-manager-host-nic` is opt-in via `host-nic-telemetry` and shares the host network. Stop the other variant and name the desired service explicitly: activating the profile alone selects both. See [switching variants and exposure](./native-docker-collectors.md#compose-deployment-variants). The existing `test` profile is only a development test runner. Compose supplies matching listener ports and local scrape/ingestion URLs for each variant; internal API/Prometheus/HTTP ports must be distinct integers from 1 through 65535. The entrypoint derives local URLs from port values for plain Docker unless explicitly overridden. If running Telegraf directly with the Docker configuration, export `TELEGRAF_PORT` and `TELEGRAF_HTTP_PORT` first. Custom configurations must match these listeners themselves; they are never rewritten. ## Optional Docker Collectors These settings are read by the Docker entrypoint, not Pydantic. Compose passes them into the container; use `-e` with plain `docker run`. Optional flags accept only lowercase `true`/`false` (including rejecting empty values and `auto`). They control bundled additions, not the baseline or user-supplied inputs. | Variable | Default | Description | |----------|---------|-------------| | `ENABLE_RAPL` | `false` | Native package/platform RAPL power | | `ENABLE_POWERSTAT_EXTRA` | `false` | Per-CPU frequency/temperature, without duplicating base package/C-state inputs | | `ENABLE_TURBOSTAT` | `false` | Turbostat diagnostics with a bounded executable/MSR preflight | | `ENABLE_TEMPERATURE` | `false` | High-resolution package temperature and min/max/mean | | `ENABLE_INTERRUPTS` | `false` | Numbered device IRQs with per-CPU series | | `ENABLE_NETWORK` | `false` (bridge), `true` (host-NIC) | Network traffic and NIC driver counters; explicit `false` overrides either default | | `ENABLE_NETWORK_VIRTUAL` | `false` | Separate veth/lxc traffic and driver counters | | `ENABLE_STORAGE` | `false` | Filesystem usage and disk I/O; bundled serial/WWID export disabled | | `TURBOSTAT_INTERVAL` | `5` | Positive integer seconds fitting a Telegraf duration | | `TURBOSTAT_BIN` | `/usr/local/bin/turbostat` | Absolute container executable path, without spaces/shell expansion | | `HOST_TURBOSTAT_BIN` | _(unset)_ | Host binary source for the explicit read-only mount in the optional Compose override | Invalid flags/interval/path syntax fail startup. Unavailable or incompatible Turbostat binaries and unreadable MSRs produce a warning and skip the automatic Turbostat input, not the whole stack. Docker bind failures occur before that check and can still prevent container creation. See the [collectors guide](./native-docker-collectors.md) for the mount, conflicts with custom inputs and hardware/namespace scope. Recreate containers after changing settings; there is no hot reload. Bare-metal `auto` semantics are unchanged. --- ## Example Configurations ### Development Setup ```bash # .env ENVIRONMENT=development LOG_LEVEL=DEBUG LOG_FORMAT=text CORS_ORIGINS=* RATE_LIMIT_ENABLED=false ``` ### High-Throughput Scenario ```bash # .env RATE_LIMIT_REQUESTS_PER_MINUTE=5000 RATE_LIMIT_BURST=500 MAX_METRICS_IN_MEMORY=500000 PROMETHEUS_POLLER_INTERVAL_MS=200 ENABLE_GZIP_COMPRESSION=true ``` ### Production with Reverse Proxy ```bash # .env ENVIRONMENT=production LOG_LEVEL=WARNING LOG_FORMAT=json CORS_ORIGINS=https://my-dashboard.example.com,https://grafana.example.com TRUST_FORWARDED_HEADERS=true METRICS_MANAGER_HOSTNAME=production-node-01 ``` ### Behind Corporate Proxy ```bash # .env http_proxy=http://proxy.example.com:8080 HTTP_PROXY=http://proxy.example.com:8080 https_proxy=http://proxy.example.com:8080 HTTPS_PROXY=http://proxy.example.com:8080 no_proxy=localhost,127.0.0.1 NO_PROXY=localhost,127.0.0.1 ``` --- ## Custom Metrics Scripts The image ships with a Telegraf [`inputs.exec`](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/exec/README.md) block that, every 10 seconds, runs **every executable `*.sh` and `*.py` file** it finds in `/app/custom-metrics/` and feeds the stdout straight into the Prometheus endpoint on `:9273`. For details, see [Custom Metrics Scripts](./custom-metrics.md). Simply dropping a script into the directory is the easiest way to publish a metric the service does not collect by default. ### How the Directory is Wired - The directory is created by the Dockerfile (`/app/custom-metrics`) - In `compose.yaml` it is mounted as a named volume `custom-metrics:`, so scripts survive container restarts - `telegraf.conf` ships this block (do not edit unless you know what you are doing): ```toml [[inputs.exec]] commands = ["/bin/sh -c 'for f in /app/custom-metrics/*.sh /app/custom-metrics/*.py; do [ -f \"$f\" ] && [ -x \"$f\" ] && \"$f\"; done 2>/dev/null; true'"] timeout = "5s" data_format = "influx" interval = "10s" ``` ### Script Requirements Each script must: - Be **executable** (`chmod +x`) - Print **InfluxDB Line Protocol** on stdout, one metric per line - Finish **within 5 seconds** (Telegraf kills longer runs) - Produce clean output (no debug prints, banners, or stderr) - Handle errors gracefully (non-zero exit codes do not crash Telegraf) ### Example: Fan RPM Metric See [Custom Metrics Scripts](./custom-metrics.md#end-to-end-example-fan-rpm-metric) for a complete end-to-end example. --- ## Optional Components The Metrics Manager image includes optional components that are bundled but not active by default: ### qmmd (Prometheus GPU Exporter) **What it is:** A lightweight Prometheus exporter for Intel Arc GPUs. It reads GPU metrics from sysfs and exposes them in Prometheus format. **Current Status:** Bundled in the image but **NOT started by default**. **Why not enabled:** The default Metrics Manager already collects GPU metrics via `mm-plugin-gpu` and Telegraf's `inputs.execd`. Using both would be redundant. **When to enable:** If you want a dedicated GPU metrics exporter that outputs to a separate Prometheus port (typically `:9100` or similar) without going through Telegraf. **How to enable:** Edit `supervisord.conf` or extend it in your downstream image: ```ini [program:qmmd] command=/usr/local/bin/qmmd autostart=true autorestart=true stdout_logfile=/dev/stdout stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 priority=40 ``` **License:** Prometheus exporter for Intel® GPUs, published on crates.io under the MIT license. See . --- ## Optional Services & Extending supervisord The container's bundled `supervisord.conf` contains an `[include]` section: ```ini [include] files=/etc/supervisor/conf.d/*.conf ``` Any `*.conf` file dropped into `/etc/supervisor/conf.d/` is picked up automatically at supervisord start, so you do not need to fork or edit `supervisord.conf` to add your own programs. ### Pattern: Add a Program in Your Downstream Image ```dockerfile FROM intel/metrics-manager:2026.3.0 # Drop additional supervisord program units into the include directory. COPY my-extra-service.conf /etc/supervisor/conf.d/my-extra-service.conf ``` `my-extra-service.conf`: ```ini [program:my-extra-service] command=/usr/local/bin/my-extra-service --flag autostart=true autorestart=true stdout_logfile=/dev/stdout stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 priority=40 ; >30 = starts after metrics-manager (priority 20) ``` ### Verify the Extra Unit is Running ```bash docker exec metrics-manager supervisorctl -c /etc/supervisor/supervisord.conf status # my-extra-service RUNNING pid 47, uptime 0:01:23 ``` --- ## Custom Telegraf Configuration ### Mount Custom Config ```yaml # compose.yaml services: metrics-manager: volumes: - ./my-telegraf.conf:/etc/telegraf/telegraf.conf:ro ``` Or via environment: ```bash TELEGRAF_CONFIG=./my-telegraf.conf docker compose up ``` ### Additional Config Directory The native-enabled container loads the main configuration plus a private snapshot of mounted `telegraf.d/**/*.conf` files and enabled bundled additions. It passes that snapshot using `--config-directory`; mounted source files are not modified. Review old drop-ins before upgrading: earlier native-enabled images mounted this directory but did not load it. Keep only one agent, output and HTTP listener configuration, and avoid duplicate inputs. See [Additional collectors in the native-enabled Docker image](./native-docker-collectors.md) for optional RAPL/Turbostat setup, customization and metric verification. Use the Docker flags listed above; Python-reader flag names are not aliases. `native/metrics-options.conf` applies to installed bare-metal services only. ### Example: Disable GPU or NPU Metrics The default `telegraf.conf` registers both GPU (qmassa) and NPU readers as `[[inputs.execd]]` blocks. To disable: ```toml # my-telegraf.conf (omit GPU/NPU inputs) [agent] interval = "1s" [[outputs.prometheus_client]] listen = ":9273" [[inputs.cpu]] [[inputs.mem]] # GPU and NPU inputs omitted ``` --- ## Structured Logging Logs are output in JSON format by default for easy parsing by log aggregators: ```json { "timestamp": "2026-03-04T10:15:30.123456Z", "level": "INFO", "logger": "app.routes", "message": "Accepted metrics via batch", "correlation_id": "abc-123-def", "extra": {"count": 10} } ``` Switch to human-readable format for development: ```bash LOG_FORMAT=text docker compose up ``` ## Correlation IDs Every request is assigned a correlation ID for distributed tracing. Pass your own via header or receive an auto-generated UUID: ```bash curl -H "X-Correlation-ID: my-trace-123" http://localhost:9090/api/v1/metrics # Response header: X-Correlation-ID: my-trace-123 ``` Correlation IDs appear in all log entries for request tracing. ## Supporting Resources - [Get Started Guide](../get-started.md) - [System Requirements](./system-requirements.md) - [Custom Metrics Scripts](./custom-metrics.md) - [Helm Deployment](./deploy-with-helm.md) - [Troubleshooting](../troubleshooting.md) ## License Copyright (C) 2025-2026 Intel Corporation SPDX-License-Identifier: Apache-2.0