Troubleshooting#

Containers Started but Detections Do Not Appear#

Symptom: The dashboard shows no detection events and GET /detections/summary returns zero totals.

Steps to diagnose:

Check that the DL Streamer container is running and processing the video:
```
docker logs -f apm-dlstreamer
```
Look for log lines showing inference results. If the container exited, the video file may be missing.
Confirm that datastream.mp4 exists in the expected location:
```
ls apps/pipeline-defect-detection/resources/videos/
```
If the file is missing, run the data preparation script described in Get Started.
Check that MQTT messages are flowing from DL Streamer to the broker:
```
docker exec apm-mqtt-broker mosquitto_sub -t 'dlstreamer/detections'
```
If no messages appear, the DL Streamer pipeline may not have been triggered. Start it manually:
```
docker exec apm-dlstreamer curl -X POST \
  http://localhost:8554/pipelines/user_defined_pipelines/pipeline_defect_detection \
  -H "Content-Type: application/json" -d '{}'
```
The pipeline server port is available only inside the Compose network and is not published on the host.

Agent Run Stays in `in_progress` and Never Completes#

Symptom: A run triggered via POST /api/agents/runs shows status: in_progress indefinitely.

Steps to diagnose:

Check the agent service logs:
```
docker logs apm-agent
```
If LLM_MODE=llm, check that the LLM service served via the OpenVINO Model Server is healthy:
```
docker exec apm-ui python3 -c \
  "import urllib.request; print(urllib.request.urlopen('http://apm-llm:8000/v1/config').status)"
```
If the OpenVINO model server service is unhealthy or is still loading the model, wait for it to finish. The first startup can take several minutes while it loads the model.

To test the pipeline without the LLM service, switch to the fallback mode:

./setup.sh --stop
LLM_MODE=fallback ./setup.sh --use-case pipeline-defect-detection

Dashboard Shows No Runs or Returns an Error#

Symptom: The UI displays no run history, or GET /api/agents/runs returns an error.

Check: Verify that the NGINX proxy is routing requests correctly:

curl http://localhost:8080/api/agents/runs

If that fails, check the NGINX container:

docker logs apm-nginx

Also confirm that the agent service itself is healthy:

curl http://localhost:8080/api/agents/health

Ask & Analyze Is Unavailable or Returns an Error#

Symptom: The chat page reports that LLM-backed analysis is disabled, cannot reach the model, or cannot gather supporting data.

Confirm the deployment mode:
```
docker inspect apm-ui --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -E '^(LLM_BASE_URL|LLM_MODEL_NAME)='
```
LLM_MODE=fallback omits apm-llm, leaving Ask & Analyze unavailable while preserving the dashboard and rule-based detect-then-reason workflow.

In LLM mode, verify that the shared OVMS service is healthy:

docker inspect apm-llm --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'
docker logs apm-llm

Verify internal DNS/connectivity from the UI container:
```
docker exec apm-ui python3 -c \
  "import urllib.request; print(urllib.request.urlopen('http://apm-llm:8000/v1/config').status)"
```
apm-llm must remain in the UI’s no_proxy list. Do not replace the internal URL with localhost; inside apm-ui, localhost refers to the UI container itself.
If an answer cannot be grounded, check the relevant source:
```
curl http://localhost:8080/api/agents/runs
curl http://localhost:8080/api/storage/detections/summary
```
Analysis mode needs completed agent output. Detection mode needs stored detections. A specified run ID must exist, have completed, and include a valid detection ID window.
If the services are healthy but chat repeatedly returns an LLM or invalid-query error, try a larger instruction model. Smaller models can be less reliable at following the structured query schema used by Detection and Combined modes. In apps/pipeline-defect-detection/.env_pipeline-defect-detection, set LLM_MODEL_NAME to the Hugging Face model ID of a larger model supported by your OpenVINO model server release, such as a supported Qwen3 instruction model.

Ensure the target system has enough memory for the larger model. Keep LLM_WEIGHT_FORMAT=int4 on memory-constrained systems, then download, convert, and redeploy it:
```
source ./scripts/download_llm_model.sh --use-case pipeline-defect-detection
source ./setup.sh --use-case pipeline-defect-detection
```
Wait for apm-llm to become healthy before retrying chat. The agent pipeline and Ask & Analyze share this model, so the change applies to Analysis, Detection, and Combined modes.

Questions longer than 4,000 characters, malformed run IDs, unsupported modes/control characters, extra request fields, and invalid structured-query plans are rejected. General upstream requests have a 15-second timeout; LLM generation requests allow 60 seconds by default. On constrained hardware, wait until apm-llm is healthy, shorten the question, and retry.

Application APIs are exposed only through NGINX at http://localhost:8080. The model-download service is the only other service with a published host port (http://localhost:8200 by default). Use container logs, health status, or docker exec for services that are internal to the Compose network.

OpenVINO Model Server Service is Unhealthy after Startup#

Symptom: apm-llm shows as unhealthy in docker ps even after several minutes.

Cause: Model loading on the first startup can take several minutes depending on model size and hardware.

Steps:

Check the progress in the OpenVINO model server logs:
```
docker logs -f apm-llm
```

If you need to run quickly without waiting, use the fallback mode:

./setup.sh --stop
LLM_MODE=fallback ./setup.sh --use-case pipeline-defect-detection

If you see permission errors related to /model, remove the model cache volume and restart:

./setup.sh --stop
docker volume rm apm_model_cache
./setup.sh --use-case pipeline-defect-detection

Storage Service is Unhealthy#

Symptom: apm-storage shows as unhealthy and detections are not being persisted.

Check the storage service logs:

docker logs apm-storage

Common causes:

Port 5001 is already in use on the host. Change the STORAGE_PORT in the .env file.
The apm_sqlite_data volume has a permission issue. Remove the volume and restart:
```
./setup.sh --clean-data
./setup.sh --use-case pipeline-defect-detection
```

Quick Verification Checklist#

Run these commands in order after startup to verify each stage of the pipeline:

# 1. All containers healthy?
docker ps --format "table {{.Names}}\t{{.Status}}"

# 2. Detections stored?
curl http://localhost:8080/api/storage/detections/summary

# 3. Agent service reachable?
curl http://localhost:8080/api/agents/runs

# 4. Trigger one agent run manually
RUN_ID=$(curl -s -X POST http://localhost:8080/api/agents/runs \
  -H "Content-Type: application/json" -d '{}' | python3 -c "import sys,json; print(json.load(sys.stdin)['run_id'])")
echo "Run ID: $RUN_ID"

# 5. Wait for completion and check the result
sleep 15
curl http://localhost:8080/api/agents/runs/$RUN_ID | python3 -m json.tool

Common Error Summary#

Symptom	Likely Cause	Action
No detections in storage	`datastream.mp4` is missing or pipeline is not triggered	Prepare data and trigger the DL Streamer pipeline
Agent run is stuck in `in_progress`	OpenVINO model server service is unhealthy or is still loading	Check `docker logs apm-llm` or switch to the fallback mode
UI shows no runs	NGINX proxy issue or agent service is down	Check `docker logs apm-nginx` and `docker logs apm-agent`
Ask & Analyze is disabled	`LLM_MODE=fallback`	Restart in LLM mode after preparing the configured model
Ask & Analyze cannot reach the model	`apm-llm` is unhealthy, still loading, or incorrectly proxied	Check `apm-llm`, `LLM_BASE_URL`, and the UI `no_proxy` value
Ask & Analyze has no grounding data	Requested run is incomplete/missing or storage has no detections	Check agent runs and the detection summary
`apm-storage` is unhealthy	Container startup or volume permission issue	Check `docker logs apm-storage`, query `http://localhost:8080/api/storage/health`, or run `./setup.sh --clean-data`
OpenVINO model server container restarts repeatedly	GPU out of memory or the model is not supported	Switch to CPU inference or use a smaller model