Gitea#
Gitea is currently deployed as part of the pre installer. Gitea is a local hosted git repository in which the EMF tarballs are expanded and committed. These tarballs come from the release service after been built by the CI.
Summary:
onprem: It’s installed as a .deb file after kubernetes install but before argo install
aws: gitea is deployed in the cloud environment before argo
Database#
onprem: Postgres is deployed as a helm chart through ArgoCD. It’s deployed within the kubernetes network boundary.
coder: Postgres is deployed as a helm chart through ArgoCD. It’s deployed within the kubernetes network boundary.
aws: Aurora is deployed using the terraform at preinstall time
Load Balancers#
onprem: MetalLB is used. It’s deployed using ArgoCD. However its configuration is done at preinstall time. It takes the host network interfaces and assigns 3 IP addresses one for argo:80/443, traefik:443/4433 and nginx:443. Theese interfaces should be separate from your main kubernetes management network
coder: Docker traefik container is used. Mage is used to deploy it.
mage router stop ; mage router start. It’s brought up using docker compose with host networking. Binds to the management interface :4433 and :443 while using SNI routing rules that are autogenerated from the templateedge-manageability-framework/tools/router/traefik.templateit forwards to the kubernetes IP address for the 3 different load balancers.aws:
NLB for raw TCP traffic nginx port 443 and mps port 4433
ALB for HTTPS traffic :443 reroutes to 8433 in eks
Kubernetes#
onprem: Rancher installed as a debian package (created by Intel) with CNI configured. Rancher listens on 0.0.0.0 for all interfaces have kubernetes services listening.
aws: 3 node cluster (EC2) deployed in AWS with jumphost (EC2) instance to connect
coder: Docker kind container pulled and deployed.
NodePorts are not used for AWS but are for OnPrem#
You need to kubectl describe to see the LB IP in AWS:
orchestrator-admin:~$ kubectl get svc -A | grep -e "NodePort"
argocd argocd-server NodePort <redacted> <none> 80:32080/TCP,443:32443/TCP 2d18h
orch-boots ingress-nginx-controller NodePort <redacted> <none> 443:31443/TCP 2d18h
orch-gateway traefik NodePort <redacted> <none> 4433:32299/TCP,443:30443/TCP 2d18h
AWS EIPs usage#
10 EIPs used in total
1 for jumphost
6 for the 2 NLBs which use 3 each, one for each sub region
3 for NAT gateways, one ip each
AWS EC2 Instances#
~$ aws ec2 describe-instances --query 'Reservations[].Instances[].[Tags[?Key==`Name`]| [0].Value,InstanceId,InstanceType,PublicIpAddress,PrivateIpAddress]' --output table
----------------------------------------------------------------------------------------------------------------
| DescribeInstances |
+-----------------------------------+----------------------+-------------+-----------------+-------------------+
| eks-nodegroup-user-1 | <redacted> | t3.2xlarge | None | 192.168.255.122 |
| eks-nodegroup-user-1 | <redacted> | t3.2xlarge | None | 192.168.252.58 |
| user-jump | <redacted> | t3.medium | <redacted> | 192.168.248.39 |
| eks-nodegroup-user-observability | <redacted> | t3.2xlarge | None | 192.168.250.164 |
| eks-nodegroup-user-1 | <redacted> | t3.2xlarge | None | 192.168.250.206 |
+-----------------------------------+----------------------+-------------+-----------------+-------------------+
Load Balancer Architecture Diagram AWS#
Edge Node
|
┌───────────────────────────┼────────────────────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ (NLB) │ │ (NLB) │ │ (ALB) │ │ (ALB) │
│Load Balancer│ │Load Balancer│ │Load Balancer│ │Load Balancer│
│port: 443 │ │port: 4433 │ │port: 443 │ │port: 443 │
│DstPort: 443 │ │DstPort: 4433│ │DstPort: 8443│ │dstPort: 443 │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
│ │ │ |
\ \ / /
\ \ / /
┌───────────────│─────────────────│─────│──────────────────│────────────┐
│ │ EKS Cluster │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │K8s Service │ │K8s Service │ │K8s Service │ │
│ │Nginx │ │Traefik │ │Argocd │ │
│ │port:<IP1>443│ │port:<IP2>8433│ │port:<IP3>443 | │
│ │ │ │port:<IP2>4433│ │port:<IP3>80 | │
│ └─────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ └───────────────┼───────────────┘ │
│ │ │
│ ┌─────────┴─────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │
│ │ │ │ │ │ │ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Pod 4 │ │ Pod 5 │ │ Pod 6 │ │
│ │ │ │ │ │ │ │
│ └─────────┘ └─────────┘ └─────────┘ │
└───────────────────────────────────────────────────────────────────────┘
Load Balancer Architecture Diagram Coder#
Edge Node
|
│
************************************************************************************
* │ *
* ▼ *
* ┌────────────────────────────┐ *
* │(Docker Traefik) │ *
* │Load Balancer │ *
* │port: 443,4433 │ *
* │Dport: 443,4433,443 │ *
* │SNI rules with * for nginx │ *
* └────────────────────────────┘ *
* │ *
* ┌──────────────────────┴───────────────────┐ *
* │ │ │ │ *
* ┌───────────────│─────────────────│─────│──────────────────│────────────┐ *
* │ │ Docker Kind Cluster │ │ *
* │ │ │ │ │ │ *
* │ │ │ │ │ │ *
* │ ▼ ▼ ▼ ▼ │ *
* │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ *
* │ │K8s Service │ │K8s Service │ │K8s Service │ │ *
* │ │Nginx │ │Traefik │ │Argocd │ │ *
* │ │port:<ip1>443│ │port:<ip2>433 │ │port:<ip3>443 | │ *
* │ │ │ │port:<ip2>4433│ │port:<ip3>80 | │ *
* │ └─────────────┘ └──────────────┘ └──────────────┘ │ *
* │ │ │ │ │ *
* │ └───────────────┼───────────────┘ │ *
* │ │ │ *
* │ ┌─────────┴─────────┐ │ *
* │ │ │ │ *
* │ ▼ ▼ │ *
* │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ *
* │ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │ *
* │ │ │ │ │ │ │ │ *
* │ └─────────┘ └─────────┘ └─────────┘ │ *
* │ │ *
* │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ *
* │ │ Pod 4 │ │ Pod 5 │ │ Pod 6 │ │ *
* │ │ │ │ │ │ │ │ *
* │ └─────────┘ └─────────┘ └─────────┘ │ *
* └───────────────────────────────────────────────────────────────────────┘ *
* *
************************************************************************************
Load Balancer Architecture Diagram OnPrem#
Edge Node
|
│
│
┌──────────────────────┴───────────────────┐
│ │ │ │
************************************************************************************
* │ │ │ │ *
* ┌───────────────│─────────────────│─────│──────────────────│────────────┐ *
* │ │ Rancher Cluster │ │ *
* │ │ │ │ │ │ *
* │ │ │ │ │ │ *
* │ ▼ ▼ ▼ ▼ │ *
* │ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ *
* │ │K8s Service │ │K8s Service │ │K8s Service │ │ *
* │ │Nginx │ │Traefik │ │Argocd │ │ *
* │ │port:<IP1>443│ │port:<IP2>433 │ │port:<IP3>443 | │ *
* │ │ │ │port:<IP2>4433│ │port:<IP3>80 | │ *
* │ └─────────────┘ └──────────────┘ └──────────────┘ │ *
* │ │ │ │ │ *
* │ └───────────────┼───────────────┘ │ *
* │ │ │ *
* │ ┌─────────┴─────────┐ │ *
* │ │ │ │ *
* │ ▼ ▼ │ *
* │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ *
* │ │ Pod 1 │ │ Pod 2 │ │ Pod 3 │ │ *
* │ │ │ │ │ │ │ │ *
* │ └─────────┘ └─────────┘ └─────────┘ │ *
* │ │ *
* │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ *
* │ │ Pod 4 │ │ Pod 5 │ │ Pod 6 │ │ *
* │ │ │ │ │ │ │ │ *
* │ └─────────┘ └─────────┘ └─────────┘ │ *
* └───────────────────────────────────────────────────────────────────────┘ *
* *
************************************************************************************