Developer Troubleshooting

This section contains developer-oriented troubleshooting and complements the User Guide Troubleshooting doc.

• Amazon* EKS Node Issues
  • Symptom
  • Cause
  • Solution
• Capturing Logs
  • Orchestrator Logs
    • Audit Logs
    • Orchestrator Pod Log Search
  • Edge Node Logs
    • Pod Logs
    • Agent Logs
    • System Logs
  • Explore Feature
• Edge Orchestrator Performance
  • Cluster Overview (“Orchestrator Cluster”)
  • Node Overview (“Orchestrator Node”)
  • Pods Resources Usage (“Pods Resources”)
  • Pods Overview (“Orchestrator Pod List”)
  • Pods Log Search (“Orchestrator Pods Log Search”)
  • Exploring Metrics
• o11y Access Issues
  • Grafana* Dashboard login page shows Internal Error
  • Grafana Dashboard login page shows Login error
  • Grafana dashboard shows “user token not found” error
  • Grafana dashboard shows no projects found error
  • User does not have Explore permissions while being an admin
  • Error sending telemetry to Edge Orchestrator - rejected by Auth Service
  • Grafana Dashboard shows no log or metric data for edge node
• Argo* CD Issues
  • Application out of sync
  • Application stuck in progressing
  • Application stuck in deleting
  • Application refresh deadline exceeded
• Vault* System Issues
  • Health Check
  • Common Failures
    • Vault is sealed
      • Symptom
      • Recovery method for environment without auto unseal
      • Recovery method for environment with auto unseal
  • Catastrophic Failures
    • Vault is unrecoverable
      • Recovery Method
  • Seal Migration
    • Step 1: Update Vault configuration
    • Step 2: Kill all standby instances and let Kubernetes restart them
    • Step 3: Unseal all restarted standby instances with migration flag
    • Step 4: Step down the active instance
    • Step 5: Finally, restart the original active instance
    • Reference
• AWS* Load Balancer
  • AWS Load Balancer Architecture
  • Health Check
  • Common Failures
    • DNS is not resolved - (1)
    • Target group does not have any entry - (2)
    • Target group has entries but showing unhealthy - (3)(4)
  • Troubleshooting Tips
    • Kubernetes Service Port (5)
    • Traefik ingressroute (6)
• Inconsistent State Between Infrastructure and Deployment
  • Symptom
  • Cause
  • Preconditions
  • Steps
• Virtual Edge Node Issues
  • User Access Setup
    • Add the User to Groups
    • Edit the Sudoers File
  • Troubleshooting Bridge Network Issues
    • Verifying Network Configuration
    • Check Network Status
    • AppArmor Issue Resolution
    • Reinstall Packages
    • Reload AppArmor Module
    • Restart libvirt
    • Comprehensive Commands
      • Remove Boot Certificates
      • Delete OVMF Firmware Files
      • Clear VM Disk Images
      • Move and Set Permissions for Logs
  • Docker* Rate Limit Exceeded
  • Permission Denied Errors [VMs Creation with Ansible* Scripts]
  • Variable Not Found in Inventory or Secret File
  • Vagrant File and Network Configuration [VMs Creation with Ansible Scripts]
  • Resource Limitations [VMs Creation with Ansible Scripts]
  • Additional General Troubleshooting Steps [VMs Creation with Ansible Scripts]
• Edge Cluster Orchestrator Troubleshooting
  • Prerequisite Knowledge
  • Edge Orchestrator
  • Edge Node Logs
    • Prerequisite Knowledge to Access the Edge Node Logs
  • Troubleshooting Edge Node Clusters Extensions Deployment
    • Edge Node
  • Testing Extension Changes
• App Orchestration Services
  • Prerequisites
  • Troubleshooting Deployment States
    • Deployment is stuck in Deploying or Updating state
    • Deployment is in No Target Clusters state
    • Deployment is in Down state
    • Deployment is in Unknown state
    • Deployment is in Error or InternalError state
      • Deployment Process and Troubleshooting Checklist
  • Checking List of App Services
  • Checking Logs for App Orchestration Services
    • App Deployment Manager (ADM)
    • App Resource Manager
    • Application Orchestrator Catalog
    • App Interconnect Manager
    • App Service Proxy
  • App Deployment Manager Custom Resources
• Keycloak* Solution Issues
  • Backup and Restore Identity Provider (IdP) (Keycloak Solution) Configuration INCLUDING USERS
    • Cause
    • Preconditions
    • Steps
  • Keycloak Solution - Backup and Restore IdP (Keycloak Solution) Configuration WITHOUT USERS
  • Import Realm Config from JSON
  • Keycloak Solution - Enable Local Password for SSO User
  • Keycloak Solution - Enable SSO Sign-on
  • Keycloak solution - temporarily enable 30 day access tokens and session spans
  • Keycloak Solution - Verify IdP (Keycloak Solution) basic functionality
• Hook OS Logs
  • Docker Logs
• User Interface
  • How to check the API calls that the UI is making and the corresponding responses
  • Known issues and solutions
    • Application registries cannot be fetched
    • User has no roles assigned
• SRE Exporter Issues
  • Metrics not received at the destination
  • SRE Exporter application is not healthy
  • Troubleshooting Tips
    • Health Check
    • Checking metric export status by viewing logs
    • Checking destination endpoint configuration
    • Update destination endpoint configuration on a running cluster
    • Checking metric collection status by querying the sre-exporter endpoint