Gitea ===== Gitea is currently deployed as part of the pre installer. Gitea is a local hosted git repository in which the EMF tarballs are expanded and committed. These tarballs come from the release service after been built by the CI. **Summary:** * **onprem**: It's installed as a .deb file after kubernetes install but before argo install * **aws**: gitea is deployed in the cloud environment before argo Database ======== * **onprem**: Postgres is deployed as a helm chart through ArgoCD. It's deployed within the kubernetes network boundary. * **coder**: Postgres is deployed as a helm chart through ArgoCD. It's deployed within the kubernetes network boundary. * **aws**: Aurora is deployed using the terraform at preinstall time Load Balancers ============== * **onprem**: MetalLB is used. It's deployed using ArgoCD. However its configuration is done at preinstall time. It takes the host network interfaces and assigns 3 IP addresses one for argo:80/443, traefik:443/4433 and nginx:443. Theese interfaces should be separate from your main kubernetes management network * **coder**: Docker traefik container is used. Mage is used to deploy it. ``mage router stop ; mage router start``. It's brought up using docker compose with host networking. Binds to the management interface :4433 and :443 while using SNI routing rules that are autogenerated from the template ``edge-manageability-framework/tools/router/traefik.template`` it forwards to the kubernetes IP address for the 3 different load balancers. * **aws**: * NLB for raw TCP traffic nginx port 443 and mps port 4433 * ALB for HTTPS traffic :443 reroutes to 8433 in eks Kubernetes ========== * **onprem**: Rancher installed as a debian package (created by Intel) with CNI configured. Rancher listens on 0.0.0.0 for all interfaces have kubernetes services listening. * **aws**: 3 node cluster (EC2) deployed in AWS with jumphost (EC2) instance to connect * **coder**: Docker kind container pulled and deployed. NodePorts are not used for AWS but are for OnPrem =========================================================== You need to kubectl describe to see the LB IP in AWS: .. code-block:: bash orchestrator-admin:~$ kubectl get svc -A | grep -e "NodePort" argocd argocd-server NodePort 80:32080/TCP,443:32443/TCP 2d18h orch-boots ingress-nginx-controller NodePort 443:31443/TCP 2d18h orch-gateway traefik NodePort 4433:32299/TCP,443:30443/TCP 2d18h AWS EIPs usage ============== * 10 EIPs used in total * 1 for jumphost * 6 for the 2 NLBs which use 3 each, one for each sub region * 3 for NAT gateways, one ip each AWS EC2 Instances ================= .. code-block:: bash ~$ aws ec2 describe-instances --query 'Reservations[].Instances[].[Tags[?Key==`Name`]| [0].Value,InstanceId,InstanceType,PublicIpAddress,PrivateIpAddress]' --output table ---------------------------------------------------------------------------------------------------------------- | DescribeInstances | +-----------------------------------+----------------------+-------------+-----------------+-------------------+ | eks-nodegroup-user-1 | | t3.2xlarge | None | 192.168.255.122 | | eks-nodegroup-user-1 | | t3.2xlarge | None | 192.168.252.58 | | user-jump | | t3.medium | | 192.168.248.39 | | eks-nodegroup-user-observability | | t3.2xlarge | None | 192.168.250.164 | | eks-nodegroup-user-1 | | t3.2xlarge | None | 192.168.250.206 | +-----------------------------------+----------------------+-------------+-----------------+-------------------+ Load Balancer Architecture Diagram AWS ====================================== :: Edge Node | ┌───────────────────────────┼────────────────────────┐ │ │ │ │ ▼ ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ (NLB) │ │ (NLB) │ │ (ALB) │ │ (ALB) │ │Load Balancer│ │Load Balancer│ │Load Balancer│ │Load Balancer│ │port: 443 │ │port: 4433 │ │port: 443 │ │port: 443 │ │DstPort: 443 │ │DstPort: 4433│ │DstPort: 8443│ │dstPort: 443 │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ | \ \ / / \ \ / / ┌───────────────│─────────────────│─────│──────────────────│────────────┐ │ │ EKS Cluster │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │K8s Service │ │K8s Service │ │K8s Service │ │ │ │Nginx │ │Traefik │ │Argocd │ │ │ │port:443│ │port:8433│ │port:443 | │ │ │ │ │port:4433│ │port:80 | │ │ └─────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ │ │ │ └───────────────┼───────────────┘ │ │ │ │ │ ┌─────────┴─────────┐ │ │ │ │ │ │ ▼ ▼ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │ │ │ │ │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Pod 4 │ │ Pod 5 │ │ Pod 6 │ │ │ │ │ │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ │ └───────────────────────────────────────────────────────────────────────┘ Load Balancer Architecture Diagram Coder ======================================== :: Edge Node | │ ************************************************************************************ * │ * * ▼ * * ┌────────────────────────────┐ * * │(Docker Traefik) │ * * │Load Balancer │ * * │port: 443,4433 │ * * │Dport: 443,4433,443 │ * * │SNI rules with * for nginx │ * * └────────────────────────────┘ * * │ * * ┌──────────────────────┴───────────────────┐ * * │ │ │ │ * * ┌───────────────│─────────────────│─────│──────────────────│────────────┐ * * │ │ Docker Kind Cluster │ │ * * │ │ │ │ │ │ * * │ │ │ │ │ │ * * │ ▼ ▼ ▼ ▼ │ * * │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ * * │ │K8s Service │ │K8s Service │ │K8s Service │ │ * * │ │Nginx │ │Traefik │ │Argocd │ │ * * │ │port:443│ │port:433 │ │port:443 | │ * * │ │ │ │port:4433│ │port:80 | │ * * │ └─────────────┘ └──────────────┘ └──────────────┘ │ * * │ │ │ │ │ * * │ └───────────────┼───────────────┘ │ * * │ │ │ * * │ ┌─────────┴─────────┐ │ * * │ │ │ │ * * │ ▼ ▼ │ * * │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ * * │ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │ * * │ │ │ │ │ │ │ │ * * │ └─────────┘ └─────────┘ └─────────┘ │ * * │ │ * * │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ * * │ │ Pod 4 │ │ Pod 5 │ │ Pod 6 │ │ * * │ │ │ │ │ │ │ │ * * │ └─────────┘ └─────────┘ └─────────┘ │ * * └───────────────────────────────────────────────────────────────────────┘ * * * ************************************************************************************ Load Balancer Architecture Diagram OnPrem ========================================= :: Edge Node | │ │ ┌──────────────────────┴───────────────────┐ │ │ │ │ ************************************************************************************ * │ │ │ │ * * ┌───────────────│─────────────────│─────│──────────────────│────────────┐ * * │ │ Rancher Cluster │ │ * * │ │ │ │ │ │ * * │ │ │ │ │ │ * * │ ▼ ▼ ▼ ▼ │ * * │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ * * │ │K8s Service │ │K8s Service │ │K8s Service │ │ * * │ │Nginx │ │Traefik │ │Argocd │ │ * * │ │port:443│ │port:433 │ │port:443 | │ * * │ │ │ │port:4433│ │port:80 | │ * * │ └─────────────┘ └──────────────┘ └──────────────┘ │ * * │ │ │ │ │ * * │ └───────────────┼───────────────┘ │ * * │ │ │ * * │ ┌─────────┴─────────┐ │ * * │ │ │ │ * * │ ▼ ▼ │ * * │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ * * │ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │ * * │ │ │ │ │ │ │ │ * * │ └─────────┘ └─────────┘ └─────────┘ │ * * │ │ * * │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ * * │ │ Pod 4 │ │ Pod 5 │ │ Pod 6 │ │ * * │ │ │ │ │ │ │ │ * * │ └─────────┘ └─────────┘ └─────────┘ │ * * └───────────────────────────────────────────────────────────────────────┘ * * * ************************************************************************************