AMT Issues#

This section covers troubleshooting AMT issues for Edge Orchestrator.

External Power Off on vPro Edge Nodes#

Important: External power operations should NOT be performed on vPro-initiated edge nodes.

Background: When an external power off is performed (e.g., physically pressing the power button or using APC/KVM/BMC to power off the device), it completely shuts down the edge node at the CSME level. From the Device Management Toolkit (DMT) perspective, this results in MPS losing its CIRA connection with the edge node.

Expected Behavior:

  • When a vPro edge node is powered off externally, the CIRA connection to MPS is immediately lost

  • MPS will attempt to communicate with the device and receive timeout errors

  • After multiple timeout failures, MPS will disconnect the CIRA connection and update the device status in the database

  • Once CIRA is disconnected, MPS returns 404 “Device not found/connected” errors for subsequent power operation requests

  • The orchestrator UI may continue showing the last known power status (e.g., “Powered on”) even though the device is disconnected.

  • Power operations initiated from the orchestrator UI will fail with “Device not found/connected” errors

Recommendation: Always use the orchestrator UI or orch-cli to perform power operations (Power On, Power Off, Restart) rather than external methods to maintain proper device state management and CIRA connectivity.

Reference:

Soft off - Shutdown Auto-Recovery#

Important: When a vPro edge node is shut down via shutdown (e.g., sudo shutdown now), the device automatically powers back on within ~48 seconds due to the EMF orchestrator desired-state reconciliation mechanism.

Background: The orchestrator maintains a desired power state (desiredPowerState: POWER_STATE_ON) for managed edge nodes. The dm-manager service continuously reconciles the actual power state with the desired state. When it detects a mismatch (device powered off but desired state is ON), it automatically triggers a power-on operation through the MPS API.

Expected Behavior:

During Shutdown:

  • Edge Node OS executes shutdown command successfully

  • System enters S5 (soft-off) state

  • CIRA connection remains active (AMT stays powered in S5 when properly configured)

  • UI displays power status error: "mismatch between desired and current power state"

Auto-Recovery Process:

  • Within next reconciliation cycle (~60 seconds), dm-manager detects power state mismatch

  • MPS sends power-on command via AMT within 14-21 seconds of detection

  • Device powers back on and returns to “Powered on” state in orchestrator UI

  • Total recovery time: approximately 48 seconds

Example from actual logs:

  • dm-manager detects POWER_STATE_OFF (mismatch detected)

  • MPS power action sent successfully (21 seconds elapsed)

  • Power state confirmed POWER_STATE_ON (48 seconds total)

BIOS Configuration Requirement:

For CIRA connection to persist during S5 (soft-off) state and enable remote power-on, the following BIOS setting must be configured:

Setting: Advanced → Power → ME Configuration → Power Control → ME on in Host Sleep Status

Required Value: Mobile ON in S0, ME wake in S3, S4-S5 (AC only)

What This Configuration Enables:

  • CIRA connection remains active during shutdown (S5 state)

  • Remote power-on capability via AMT during soft-off

Note: Without this setting, CIRA disconnects on shutdown, and the device cannot be remotely powered on

BIOS Verification Steps:

  1. Access BIOS setup during boot (typically F2 key)

  2. Navigate to: Advanced → Power → ME Configuration → Power Control → ME on in Host Sleep Status

  3. Verify setting is: Mobile ON in S0, ME wake in S3, S4-S5 (AC only)

  4. Save and exit if changes are needed

SMBIOS UUID and ME UUID Mismatch on ASRock Platforms#

Important: On ASRock platforms, UUID mismatch between SMBIOS and AMT firmware causes power operations to fail with 404 “Device not found” errors.

Background: ASRock devices may have different UUIDs stored in SMBIOS (read by PMA during initialization) versus AMT firmware (used by CIRA/MPS connections). This mismatch prevents the orchestrator from executing power operations on the device.

Root Cause:

During device initialization:

  • PMA reads SMBIOS UUID: 5b006b9c-09bf-0000-0000-000000000000

  • dm-manager stores this UUID as hostID in inventory

  • AMT activation configures CIRA with firmware UUID: 03000200-0400-0500-0006-000700080009

When power operations are attempted:

  • Orchestrator UI sends power command for SMBIOS UUID (5b006b9c-09bf-0000-0000-000000000000)

  • MPS only recognizes devices by their AMT firmware UUID (03000200-0400-0500-0006-000700080009)

  • Result: 404 “Device not found” error

Expected Behavior:

  • PMA initialization fails to register device with correct UUID

  • Device appears in orchestrator UI but power operations (Power On, Power Off, Restart) fail with “Device not found” errors

  • CIRA connection may establish but operations fail due to UUID mismatch

Solution:

ME firmware must be re-flashed in manufacturing mode on ASRock platforms to synchronize SMBIOS UUID with AMT firmware UUID.

Steps to Resolve UUID Mismatch:

  1. Verify UUID Mismatch:

    # Check SMBIOS UUID
sudo dmidecode -s system-uuid

# Check AMT firmware UUID from MPS CIRA connection logs
kubectl logs -n <namespace> <mps-pod> | grep -i "cira\|uuid"

    Compare SMBIOS UUID with the UUID shown in MPS CIRA connection logs. If they differ, proceed to next step.

  2. Contact ASRock Support:

    Request ME firmware update tools and procedures for your specific platform model.

  3. Update ME Firmware:

    • Boot to manufacturing/service mode (ASRock-specific procedure)

    • Use ASRock ME tools to update firmware UUID to match SMBIOS UUID

    • Verify flash operation completed successfully

  4. Re-provision and Test:

    • Clear AMT configuration and re-provision

    • Re-register device with orchestrator

    • Test power operations to confirm resolution

Reference:

  • Intel® Endpoint Management Assistant (EMA) - Remotely manage Intel® AMT devices beyond the firewall via cloud - Cloud-based platform with in-band and out-of-band management - Agent-based for Microsoft Windows 10 and 11 platforms

  • Contact ASRock technical support for platform-specific ME firmware reflashing procedures

  • SMBIOS UUID and AMT firmware UUID must match for proper device management

CIRA Connection and Edge Node Reuse Between EMF Orchestrator#

Important: Moving an edge node from one EMF orchestrator to another without proper deauthorization causes stale CIRA connections

Background: When an edge node is activated for vPro to an EMF orchestrator, it establishes a CIRA connection with that edge node. If the node is moved to a different EMF orchestrator without proper cleanup, the original CIRA connection persists with the first EMF orchestrator with RPS status “connected”

Issue:

  • Edge node moves from EMF orchestrator1 to EMF orchestrator2 without proper vPro deactivation or host deauthorization

  • Stale CIRA connection remains active on EMF orchestrator 1

  • New CIRA connection on EMF orchestrator 2 may fail

  • Power operations and management commands fail due to connection state mismatch

Recommended Clean Flow:

Before moving an edge node to a new EMF orchestrator:

  1. Deactivate vPro from Host Action or Deauthorize Host in the current EMF orchestrator

  2. Delete the Host from the EMF orchestrator inventory

Note: Skipping deauthorization/deletion steps will result in orphaned CIRA connections and require manual deactivation using rpc commands

vPro Enabled Without BIOS Configuration Option on OnLogic K804#

Important: OnLogic K804 systems do not provide BIOS options to enable or disable Intel® vPro/AMT.

Background: OnLogic K804 ships with vPro/AMT permanently enabled at the firmware level. The BIOS does not expose any controls to manage vPro/AMT features.

Observed Behavior:

  • No vPro or AMT configuration options in BIOS

  • vPro/AMT features works correctly

  • Users cannot disable vPro/AMT through BIOS settings

Recommendation:

If BIOS control over vPro enablement is required for your security policies, consider alternative hardware platforms. For OnLogic K804, document this as a platform-specific exception in security compliance reports.

Reference:

Power Off Queued During Active KVM or SOL Session#

Issue: When a power-off command is initiated via the orch-cli while a KVM or SOL session is active, the system does not power off immediately. The power-off is queued by Intel® AMT and is triggered automatically as soon as the active KVM or SOL session is deactivated.

Expected Behavior:

  • A power-off command sent via orch-cli during an active KVM/SOL session is accepted but not executed immediately.

  • The system remains powered on for the duration of the active session.

  • Once the KVM or SOL session is stopped or disconnected, the queued power-off is executed automatically and the edge node shuts down.

Recommendation:

If an immediate power-off is required, stop the active KVM or SOL session first using the orch-cli, then issue the power-off command:

# Stop the active KVM or SOL session
./orch-cli set host <HOST_ID or HOST_NAME> --session-type kvm --session-state stop

# Then power off
./orch-cli set host <HOST_ID or HOST_NAME> --power off

Reference:

KVM Session WebSocket Disconnected When Idle or During MPS/RPS Restart#

Issue: A KVM session WebSocket connection may be disconnected in two scenarios:

  1. Idle session — the KVM session is left idle (no user interaction) for an extended period. The WebSocket is disconnected automatically by the DMT (Device Management Toolkit).

  2. MPS/RPS token refresh — MPS and RPS pods restart approximately every 50 seconds during token refreshment. During this restart window, the DMT disconnects the active KVM session.

Expected Behavior:

  • An idle KVM session will be disconnected automatically. The session does not remain open indefinitely without user activity.

  • During each MPS/RPS token refresh cycle (~50 seconds), the pods restart briefly. Any active KVM session is disconnected by the DMT during this period.

Recommendation:

  • Avoid leaving the KVM session idle. Keep the session active by performing operations on the KVM viewer browser application to prevent automatic disconnection. If the session is disconnected due to inactivity, restart it using the orch-cli and continue performing operations in the KVM viewer — do not leave the browser window idle:

    ./orch-cli set host <HOST_ID or HOST_NAME> --session-type kvm --session-state start --orch-ca orch-ca.crt

  • MPS/RPS restarts are expected. If the KVM session disconnects unexpectedly, check whether an MPS or RPS pod restart has occurred. Once the pods have restarted and stabilized, start a new KVM session using the orch-cli:

    ./orch-cli set host <HOST_ID or HOST_NAME> --session-type kvm --session-state start --orch-ca orch-ca.crt

Note

MPS and RPS pod restarts during token refresh are a known behaviour of the Edge Out-of-Band Manageability orchestrator. KVM session disconnection during these restarts is expected.

Issue: Sol session did not got os prompt after successful connection established.

Solution:

Enable the serial-getty service on the edge node to ensure the SOL session can access the OS prompt. You can do this by running the following commands on the edge node:

TTY=$(sudo dmesg | grep -oP 'ttyS\d+' | head -n1) && \
echo "Using $TTY" && \
sudo systemctl start serial-getty@$TTY.service && \
sudo systemctl enable serial-getty@$TTY.service