AO / CO / Observability Composability#

This document provides end-to-end guidance on:

  1. Deploying orchestration with Application Orchestrator (AO), Cluster Orchestrator (CO), and Observability (O11Y) profiles, including how to enable or disable them using composability flags.

  2. Onboarding edge nodes in NIO mode using orch-cli.

1. AO / CO / Observability Composability Overview#

During orchestration deployment, the following profiles are enabled by default:

  • Application Orchestrator (AO) — handles application orchestration at the edge.

  • Cluster Orchestrator (CO) — manages cluster-level orchestration, scaling, and coordination.

  • Observability (O11Y) — integrates telemetry, metrics, and monitoring.

Composability provides flexibility and control, allowing you to include or exclude specific profiles as needed. These profiles are controlled through environment flags set before starting the orchestration deployment.

Important

The flags must be defined before orchestration deployment begins. For upgrades, ensure the same flags are used to maintain consistent orchestration state and avoid unexpected composability changes.

2. Configuration Flags#

By default, all profiles are enabled (flags unset or set to false).

To modify which components are deployed, export the following environment variables before starting orchestration deployment or upgrade:

export DISABLE_AO_PROFILE=true      # Disable Application Orchestrator
export DISABLE_CO_PROFILE=true      # Disable Cluster Orchestrator
export DISABLE_O11Y_PROFILE=true    # Disable Observability

Note

These flags must be exported to your environment prior to both deployment and upgrade to ensure consistent composability across lifecycle operations.

3. Verification After Deployment or Upgrade#

After orchestration deployment or upgrade, you can verify which profiles are enabled or disabled using the following one-liner command:

root_app_ns=$(kubectl get application -A | grep root-app | awk '{print $1}')
VALUE_FILES=$(kubectl get application root-app -n $root_app_ns -o yaml)
echo "$VALUE_FILES" | grep -q "enable-cluster-orch.yaml" && echo "✅ CO enabled" || echo "⛔ CO disabled"
echo "$VALUE_FILES" | grep -q "enable-app-orch.yaml" && echo "✅ AO enabled" || echo "⛔ AO disabled"
echo "$VALUE_FILES" | grep -qE "(enable-o11y)" && echo "✅ O11Y enabled" || echo "⛔ O11Y disabled"

Example Output:

⛔ CO disabled     --> if CO disabled
⛔ AO disabled     --> if AO disabled
✅ O11Y enabled    --> if observability enabled

You can also confirm the same from the ArgoCD root-app application view. For pre-deployment verification (before cluster creation), review the orchestration clustername.yaml file.

4. Edge Node Onboarding in NIO Mode#

Edge nodes can be onboarded in NIO mode using the orch-cli tool.

This section walks through the complete process of onboarding edge nodes in NIO mode using the orch-cli tool.

4.1 Prerequisites#

  • orch-cli installed on your system

  • Access to the cluster API and Keycloak

Note

The orch-cli binary and dependencies are managed through the orch-cli repository. Ensure orch-cli is available and configured correctly in your environment.

4.2 Configure Environment Variables#

Set the following environment variables and authenticate with Keycloak:

export CLUSTER_FQDN=<cluster.example.com>
export EP=https://api.${CLUSTER_FQDN}
export PROJECT_NAME=itep
export ORCH_DEFAULT_USER=itep-user
export ORCH_DEFAULT_PASSWORD=itep-user-password

# Login to orch-cli using Keycloak
orch-cli logout
orch-cli login $ORCH_DEFAULT_USER $ORCH_DEFAULT_PASSWORD --keycloak https://keycloak.${CLUSTER_FQDN}/realms/master

# Set default project and API endpoint
orch-cli config set project $PROJECT_NAME
orch-cli config set api-endpoint $EP

Note

Replace CLUSTER_FQDN, PROJECT_NAME, ORCH_DEFAULT_USER, and ORCH_DEFAULT_PASSWORD with valid credentials for your deployment.

4.3 Create Region and Site (if not available)#

Before registering hosts, ensure the required region and site are created.

# Create a region
orch-cli create region <region-name> --type <region-type>
orch-cli list region

# Create a site under the region
orch-cli create site <site-name> --region <region-id>
orch-cli list site

4.4 View Available OS Profiles#

To list all available OS profiles:

orch-cli list osprofile

4.5 Register Host in NIO Mode#

Hosts can be registered individually or in bulk using a CSV file.

orch-cli create host -i host-config.csv
orch-cli list host
orch-cli get host host-868ee99c

Sample Host CSV (``host-config.csv``):

Serial,UUID,OSProfile,Site,Secure,RemoteUser,Metadata,LVMSize,K8sEnable,K8sClusterTemplate,K8sConfig,Error - do not fill
aocotest0001,,Edge Microvisor Toolkit 3.0.20251204,site-bcbbbbf8,FALSE,,,,,,

Note

Update values as needed for your specific environment.

4.6 Collecting Edge Node Logs When Observability is Disabled#

When Observability (O11Y) is disabled, the centralized log collection and monitoring capabilities are not available. In this scenario, you must collect logs directly from each edge node by connecting via SSH.

Important

To access edge nodes when observability is disabled, the edge node must be onboarded with SSH enabled in the configuration.

For details on how to add SSH users, refer to the Local SSH Account Management section in the orch-cli guide.

4.6.1 SSH Login to Edge Node#

Once the edge node is onboarded successfully with SSH enabled, you can log in to collect logs.

Get the edge node IP address after onboarding:

orch-cli list host
orch-cli get host <host-id> | grep NIC

Login to the edge node:

ssh <ssh-enabled-username>@<edge-node-ip>

4.6.2 Collect Edge Node Logs#

After logging in to the edge node, you can collect logs from various edge services.

Check Node Agent Logs:

sudo journalctl -u node-agent -f
sudo systemctl status node-agent

Check Cluster Agent Logs (if CO is enabled):

sudo journalctl -u cluster-agent -f
sudo systemctl status cluster-agent

Check Other Service Logs:

sudo journalctl -u <service-name> -f

4.7 Create a Cluster (if CO is enabled)#

If the Cluster Orchestrator is enabled, create and verify the cluster using the commands below:

orch-cli create cluster cli-cluster --nodes <EDGENODE-UUID>:all
orch-cli list cluster