Troubleshooting Vault Server#

Edge Orchestrator uses Hashicorp Vault* to secure and manage secrets.

Check Vault Server Health#

Verify pod readiness to test that the Vault server for Edge Orchestrator is in a healthy state. All Vault server instances should be ready, with 3/3 pods.

  1. Run the code below to verify Vault server health.

    $ kubectl -n orch-platform get pods | grep -E 'vault-\d+'

  2. Verify the output shows 3/3 pods up.

    vault-0  3/3  Running   0   23h

Possible Vault Server Failures#

Vault Server is Sealed#

The Vault server seals automatically after a pod or node restart.

Vault Server is Sealed, But Some Pods are Not Ready#

The Vault server is sealed, but the pod shows 2/3 readiness.

Example of 2/3 pods ready#
$ kubectl -n orch-platform get pods | grep -E 'vault-\d+'
vault-0  2/3  Running 0 23h

To check the Vault server status, edit the variable [vault-0] to match the name of the Vault pod.

$ kubectl -n orch-platform exec -it [vault-0] -- vault status | grep Sealed
Sealed false

Unseal Vault Server#

Edge Orchestrator stores the unseal key in a Kubernetes* secret.

Note

Each Vault server must be unsealed separately.

Unseal a Vault server using the following command. Edit the variable [vault-0] to match the name of the Vault pod.

kubectl -n orch-platform get secret vault-keys -o yaml | yq .data.vault-keys  | base64 -d | jq -r '.keys[]' | xargs -n1 kubectl  -n orch-platform exec -it [vault-0] -- vault operator unseal

Back Up and Restore Vault Keys#

Vault seal keys must be backed up outside of the cluster. If the keys are lost, there is no way to recover the Vault server.

Vault Server is Unrecoverable#

If the Vault unseal keys are not recoverable, clear the Vault tables and recreate them.

Clear Vault Tables#

Vault server uses two tables: vault_ha_locks and vault_kv_store.

  1. Access the Postgres shell.

    ./psql.sh orch-platform vault

  2. From the Postgres shell, run the following commands to drop the tables.

    orch-platform-vault=> drop table vault_ha_locks, vault_kv_store;
DROP TABLE

Recreate the Vault Server#

Run the following command to delete the old keys and secrets-config job.

kubectl -n orch-platform delete secret vault-keys
kubectl -n orch-platform delete job secrets-config

When the old keys and secrets-config job are removed, ArgoCD executes secrets-config which reinitializes the Vault server and stores the new unseal keys and root token in a Kubernetes secret named vault-keys.