Trusted Compute Issues#

This section covers troubleshooting of Trusted Compute issues for EMF.

Failure of the Trusted Compute plugin installation due to security pre-requisites compatibility. trusted-compute-compatible is not true on EMF UI
Host Attestation status is Unknown and all the pods are up and running
Verifier and Attestation Manager PODs have cordoned the node, and attestation PODs failed to transition to the ready state
Attestation failed and overall trust is showing as failed on the UI
trusted-workload POD status is not running

Symptom: Failure of the Trusted Compute plugin installation due to security pre-requisites compatibility. trusted-compute-compatible is not true on EMF UI

Solution: To resolve this issue, ensure that the prerequisites are met. Full Disk Encryption (FDE) and Secure Boot (SB) must be enabled on the platform. Once these features are activated on the edge-node platform, the onboarding profile should also enable FDE and SB. The fix needs to be implemented in two areas: first, enable the features on the platform (FDE, SB); second, ensure that the profile features are aligned. To recover from this condition, re-onboard your edge-node with the prerequisites enabled. For details on the pre-requisites see Prerequisites.

Symptom: Host Attestation status is Unknown and all the pods are up and running.

Solution: This issue may arise if the PCR banks on the TPM are not configured correctly. To resolve it, set the PCR bank of the TPM to SHA256 by changing the BIOS settings on the edge node.

Symptom: Verifier and Attestation Manager PODs have cordoned the node, and attestation PODs failed to transition to the ready state.

Solution: When verification of the system fails, the node where attestation failed will be cordoned to protect the system for security analysis. To recover from this state, you must uncordon the node from the cluster controller.

Sample commands:

# Check the status of nodes
kubectl get nodes

# Uncordon the node
kubectl uncordon <node_name>

Symptom: Attestation failed and overall trust is showing as failed on the UI.

Solution: This issue could be caused by a failure in updated components or a change in the bill of material of the system. To mitigate and audit the full attestation report, the user can check the verifier POD log and details. To recover from this state, you can analyze the attestation report and update the BOM accordingly or revert the faulty component from the Bill Of Material (BOM) list. To update the BOM, you can either delete and re-install the deployment package in question from the EMF UI or re-provision the edge node with the desired OS profile.

Symptom: trusted-workload POD status is not running.

Solution: This issue could be caused by a failure in the following components. Please follow the steps below to debug it.

Verify CRD installation: The operator creates the ccruntime CRD as can be observed in the following command:
```
# Check if the CRD is installed
kubectl get crd | grep ccruntime
```

Verify Runtime class.

# verify runtime class
kubectl get runtimeclass | grep kata

Verify TC VM launch after pod deployment: Once the pod gets deployed with the kata-qemu runtimeclass, verify the VM launch using the command below.
```
ps -aux | grep /opt/kata/bin/qemu-system-x86_64
```
Enable debug console for trusted VM

To enable full debug logs for QEMU hypervisor, Kata runtime, and Kata agent, use the following steps:
1. Open the containerd configuration file located at:
```
sudo vim /etc/containerd/config.toml
```
  1. Edit the below section and enable debug in containerd:
    [debug] level = "debug"
  2. Restart containerd to apply the changes:
    sudo systemctl restart containerd
2. Modify Kata Containers Configuration
  1. Navigate to the Kata Containers default configuration directory:
    sudo su - cd /opt/kata/share/defaults/kata-containers
  2. Open the appropriate .toml file for the VM you want to launch and uncomment the following lines:
    debug_console_enabled = true enable_debug = true
  3. Save the changes and exit the editor.
3. Identify the Sandbox ID
  1. Find the sandbox ID by running the following command:
    ps aux | grep qemu
  2. Copy the sandbox ID from the output.
4. Execute Kata Runtime.
```
sudo /opt/kata/bin/kata-runtime exec <sandbox_id>
```
5. Follow the below mentioned steps:
  1. Edit the file:
    
    /opt/kata/share/defaults/kata-containers/configuration-qemu.toml
  2. In the respective sections (“[hypervisor.qemu]”, “[runtime]”, and “[agent]”), enable the debug configuration parameters.
  3. Check the journalctl logs of containerd:
    # Check containerd logs sudo journalctl -xeu containerd