Trusted Compute Issues
======================

This section covers troubleshooting of Trusted Compute issues for EMF.

* :ref:`Failure of the Trusted Compute plugin installation due to security pre-requisites compatibility. trusted-compute-compatible is not true on EMF UI <tc_plugin_installation_failure>`
* :ref:`Host Attestation status is Unknown and all the pods are up and running <tc_attestation_status_unknown>`
* :ref:`Verifier and Attestation Manager PODs have cordoned the node, and attestation PODs failed to transition to the ready state <tc_pods_cordoned>`
* :ref:`Attestation failed and overall trust is showing as failed on the UI <tc_attestation_failed>`
* :ref:`trusted-workload POD status is not running <tc_workload_pod_not_running>`


.. _tc_plugin_installation_failure:

**Symptom:** Failure of the Trusted Compute plugin installation due to security pre-requisites compatibility. trusted-compute-compatible is not true on EMF UI


**Solution:** To resolve this issue, ensure that the prerequisites are met. Full Disk Encryption (FDE) and Secure Boot (SB)
must be enabled on the platform. Once these features are activated on the edge-node platform, the onboarding profile
should also enable FDE and SB.
The fix needs to be implemented in two areas: first, enable the features on the platform (FDE, SB); second, ensure that
the profile features are aligned. To recover from this condition, re-onboard your edge-node with the prerequisites enabled.
For details on the pre-requisites see :doc:`/user_guide/set_up_edge_infra/edge_node_onboard/prerequisites/index`.

.. _tc_attestation_status_unknown:

**Symptom:** Host Attestation status is Unknown and all the pods are up and running.

.. figure:: images/Attestation_status_unknown.png
   :alt: Host Attestation Unknown
   :width: 750px


**Solution:**
This issue may arise if the PCR banks on the TPM are not configured correctly. To resolve it, set the PCR bank of the TPM to SHA256 by changing the BIOS settings on the edge node.

.. figure:: images/SHA256_sample_BIOS_Settings.png
   :alt: TPM PCR bank settings
   :width: 750px

.. _tc_pods_cordoned:

**Symptom:** Verifier and Attestation Manager PODs have cordoned the node, and attestation PODs failed to transition to the ready state.


**Solution:** When verification of the system fails, the node where attestation failed will be cordoned to protect the system for security analysis.
To recover from this state, you must uncordon the node from the cluster controller.

Sample commands:

.. code-block:: bash

   # Check the status of nodes
   kubectl get nodes

   # Uncordon the node
   kubectl uncordon <node_name>

.. _tc_attestation_failed:

**Symptom:** Attestation failed and overall trust is showing as failed on the UI.

**Solution:** This issue could be caused by a failure in updated components or a change in the bill of material of the system.
To mitigate and audit the full attestation report, the user can check the verifier POD log and details. To recover from this state,
you can analyze the attestation report and update the BOM accordingly or revert the faulty component from the Bill Of Material (BOM) list.
To update the BOM, you can either delete and re-install the deployment package in question from the EMF UI or re-provision the
edge node with the desired OS profile.

.. _tc_workload_pod_not_running:

**Symptom:** trusted-workload POD status is not running.

**Solution:** This issue could be caused by a failure in the following components. Please follow the steps below to debug it.

1. Verify CRD installation: The operator creates the ccruntime CRD as can be observed in the following command:

   .. code-block:: bash

      # Check if the CRD is installed
      kubectl get crd | grep ccruntime

   .. figure:: images/tc-wl-runtime-class-installation.png
      :align: left
      :width: 750px
      :alt: verify CRD installation

2. Verify Runtime class.

   .. code-block:: bash

      # verify runtime class
      kubectl get runtimeclass | grep kata

   .. figure:: images/tc-wl-crd-installation.png
      :align: left
      :width: 750px
      :alt: verify Runtime class


3. Verify TC VM launch after pod deployment: Once the pod gets deployed with the kata-qemu runtimeclass, verify the VM launch using the command below.

   .. code-block:: bash

      ps -aux | grep /opt/kata/bin/qemu-system-x86_64

   .. figure:: images/tc-vm-debug.png
      :align: left
      :width: 750px
      :alt: tc vm debug

4. Enable debug console for trusted VM

   To enable full debug logs for QEMU hypervisor, Kata runtime, and Kata agent, use the following steps:

   1. Open the containerd configuration file located at:

      .. code-block:: bash

         sudo vim /etc/containerd/config.toml

      a. Edit the below section and enable debug in containerd:

         .. code-block:: toml

            [debug]
              level = "debug"

      b. Restart containerd to apply the changes:

         .. code-block:: bash

            sudo systemctl restart containerd

   2. Modify Kata Containers Configuration

      a. Navigate to the Kata Containers default configuration directory:

         .. code-block:: bash

            sudo su -
            cd /opt/kata/share/defaults/kata-containers

      b. Open the appropriate .toml file for the VM you want to launch and uncomment the following lines:

         .. code-block:: toml

            debug_console_enabled = true
            enable_debug = true

      c. Save the changes and exit the editor.

   3. Identify the Sandbox ID

      a. Find the sandbox ID by running the following command:

         .. code-block:: bash

            ps aux | grep qemu

      b. Copy the sandbox ID from the output.

   4. Execute Kata Runtime.

      .. code-block:: bash

         sudo /opt/kata/bin/kata-runtime exec <sandbox_id>

   5. Follow the below mentioned steps:

      a. Edit the file:

         /opt/kata/share/defaults/kata-containers/configuration-qemu.toml

      b. In the respective sections ("[hypervisor.qemu]", "[runtime]", and "[agent]"), enable the debug configuration parameters.

      c. Check the journalctl logs of containerd:

         .. code-block:: bash

            # Check containerd logs
            sudo journalctl -xeu containerd