aws load-balancer is not registered with instances

Question

I use kubeadm to launch cluster on AWS. I can successfully create a load balancer on AWS by using kubectl, but the load balancer is not registered with any EC2 instances. That causes problem that the service cannot be accessed from public.

From the observation, when the ELB is created, it cannot find any healthy instances under all subnets. I am pretty sure I tag all my instances correctly.

Updated: I am reading the log from k8s-controller-manager, it shows my node does not have ProviderID set. And according to Github comment, ELB will ignore nodes where instance ID cannot be determined from provider. Could this cause the issue? How Should I set the providerID?

load balancer configuration

apiVersion: v1                                                                                                                                                                                                                                                                        
kind: Service                                                                                                                                                                                                                                                                         
metadata:                                                                                                                                                                                                                                                                             
  name: load-balancer                                                                                                                                                                                                                                                           
  annotations:                                                                                                                                                                                                                                                                        
    service.beta.kubernetes.io/aws-load-balancer-type: "elb"                                                                                                                                                                                                                          
spec:                                                                                                                                                                                                                                                                                 
  ports:                                                                                                                                                                                                                                                                              
  - name: http                                                                                                                                                                                                                                                                        
    port: 80                                                                                                                                                                                                                                                                          
    protocol: TCP                                                                                                                                                                                                                                                                     
    targetPort: 80                                                                                                                                                                                                                                                                    
  - name: https                                                                                                                                                                                                                                                                       
    port: 443                                                                                                                                                                                                                                                                         
    protocol: TCP                                                                                                                                                                                                                                                                     
    targetPort: 443                                                                                                                                                                                                                                                                   
  selector:                                                                                                                                                                                                                                                                           
    app: replica                                                                                                                                                                                                                                                                
  type: LoadBalancer

deployment configuration

apiVersion: apps/v1                                                                                                                                                                                                                                                                   
kind: Deployment                                                                                                                                                                                                                                                                      
metadata:                                                                                                                                                                                                                                                                             
  name: replica-deployment                                                                                                                                                                                                                                                      
  labels:                                                                                                                                                                                                                                                                             
    app: replica                                                                                                                                                                                                                                                                
spec:                                                                                                                                                                                                                                                                                 
  replicas: 1                                                                                                                                                                                                                                                                         
  selector:                                                                                                                                                                                                                                                                           
    matchLabels:                                                                                                                                                                                                                                                                      
      app: replica                                                                                                                                                                                                                                                              
  template:                                                                                                                                                                                                                                                                           
    metadata:                                                                                                                                                                                                                                                                         
      labels:                                                                                                                                                                                                                                                                         
        app: replica                                                                                                                                                                                                                                                            
    spec:                                                                                                                                                                                                                                                                             
      containers:                                                                                                                                                                                                                                                                     
        - name: web                                                                                                                                                                                                                                                   
          image: web                                                                                                                                                                                                                                                                  
          imagePullPolicy: IfNotPresent                                                                                                                                                                                                                                         
          ports:                                                                                                                                                                                                                                                                      
          - containerPort: 80                                                                                                                                                                                                                                                         
          - containerPort: 443                                                                                                                                                                                                                                                        
          command: ["/bin/bash"]                                                                                                                                                                                                                                                      
          args: ["-c", "script_to_start_server.sh"]

node output status section

status:                                                                                                                                                                                                                                                                               
  addresses:                                                                                                                                                                                                                                                                          
  - address: 172.31.35.209                                                                                                                                                                                                                                                            
    type: InternalIP                                                                                                                                                                                                                                                                  
  - address: k8s                                                                                                                                                                                                                                                                      
    type: Hostname                                                                                                                                                                                                                                                                    
  allocatable:                                                                                                                                                                                                                                                                        
    cpu: "4"                                                                                                                                                                                                                                                                          
    ephemeral-storage: "119850776788"                                                                                                                                                                                                                                                 
    hugepages-1Gi: "0"                                                                                                                                                                                                                                                                
    hugepages-2Mi: "0"                                                                                                                                                                                                                                                                
    memory: 16328856Ki                                                                                                                                                                                                                                                                
    pods: "110"                                                                                                                                                                                                                                                                       
  capacity:                                                                                                                                                                                                                                                                           
    cpu: "4"                                                                                                                                                                                                                                                                          
    ephemeral-storage: 130046416Ki                                                                                                                                                                                                                                                    
    hugepages-1Gi: "0"                                                                                                                                                                                                                                                                
    hugepages-2Mi: "0"                                                                                                                                                                                                                                                                
    memory: 16431256Ki                                                                                                                                                                                                                                                                
    pods: "110"                                                                                                                                                                                                                                                                       
  conditions:                                                                                                                                                                                                                                                                         
  - lastHeartbeatTime: 2018-07-12T04:01:54Z                                                                                                                                                                                                                                           
    lastTransitionTime: 2018-07-11T15:45:06Z                                                                                                                                                                                                                                          
    message: kubelet has sufficient disk space available                                                                                                                                                                                                                              
    reason: KubeletHasSufficientDisk                                                                                                                                                                                                                                                  
    status: "False"                                                                                                                                                                                                                                             
    type: OutOfDisk                                                                                                                                                                                                                                                                   
  - lastHeartbeatTime: 2018-07-12T04:01:54Z                                                                                                                                                                                                                                           
    lastTransitionTime: 2018-07-11T15:45:06Z                                                                                                                                                                                                                                          
    message: kubelet has sufficient memory available                                                                                                                                                                                                                                  
    reason: KubeletHasSufficientMemory                                                                                                                                                                                                                                                
    status: "False"                                                                                                                                                                                                                                                                   
    type: MemoryPressure
  - lastHeartbeatTime: 2018-07-12T04:01:54Z
    lastTransitionTime: 2018-07-11T15:45:06Z
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: 2018-07-12T04:01:54Z
    lastTransitionTime: 2018-07-11T15:45:06Z
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: 2018-07-12T04:01:54Z
    lastTransitionTime: 2018-07-11T15:45:06Z
    message: kubelet is posting ready status. AppArmor enabled
    reason: KubeletReady
    status: "True"
    type: Ready

How can I fix the issue?

Thanks!

Answer 1

In My case the issue was with the worker nodes not getting the providerId assigned properly.

I managed to patch the node like - kubectl patch node ip-xxxxx.ap-southeast-2.compute.internal -p '{"spec":{"providerID":"aws:///ap-southeast-2a/i-0xxxxx"}}'

to add the ProviderID. And then when i deployed the service . The ELB got created. the node group got added and end to end it worked. This is not a straight forward answer . But until i find a better solution let remain here

Answer 2

In my case - the problem was missing the option --cloud-provider=aws

After I placed the following in /etc/default/kubelet (via terraform in my case ) and redeployed my nodes, everything worked

/etc/default/kubelet

KUBELET_EXTRA_ARGS='--cloud-provider=aws'