Apache Zeppelin on K8S Interpreters crash due to UnknownHostException

Question

Hi, help needed!

I've deployed apache/zeppelin:0.9.0 on my k8s cluster, under zeppelin namespace, following the documentation, here is my zeppeling-server.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zeppelin-notebook
  namespace: zeppelin
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
status: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zeppelin-server-conf-map
data:
  # 'serviceDomain' is a Domain name to use for accessing Zeppelin UI.
  # Should point IP address of 'zeppelin-server' service.
  #
  # Wildcard subdomain need to be point the same IP address to access service inside of Pod (such as SparkUI).
  # i.e. if service domain is 'local.zeppelin-project.org', DNS configuration should make 'local.zeppelin-project.org' and '*.local.zeppelin-project.org' point the same address.
  #
  # Default value is 'local.zeppelin-project.org' while it points 127.0.0.1 and `kubectl port-forward zeppelin-server` will give localhost to connects.
  # If you have your ingress controller configured to connect to `zeppelin-server` service and have a domain name for it (with wildcard subdomain point the same address), you can replace serviceDomain field with your own domain.
  #SERVICE_DOMAIN: local.zeppelin-project.org:8080
  SERVICE_DOMAIN: https://zeppelin.karstecsa.com.ar/
  ZEPPELIN_K8S_SPARK_CONTAINER_IMAGE: spark:2.4.5
  ZEPPELIN_K8S_CONTAINER_IMAGE: apache/zeppelin:0.9.0-SNAPSHOT
  ZEPPELIN_HOME: /zeppelin
  ZEPPELIN_SERVER_RPC_PORTRANGE: 12320:12322
  # default value of 'master' property for spark interpreter.
  SPARK_MASTER: k8s://https://kubernetes.zeppelin.svc
  # default value of 'SPARK_HOME' property for spark interpreter.
  SPARK_HOME: /spark
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zeppelin-server-conf
data:
  nginx.conf: |
    daemon off;
    worker_processes auto;
    events {
      worker_connections 1024;
    }
    http {
      map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
      }

      # first server block will be default. Proxy zeppelin server.
      server {
        listen 80;
        location / {
          proxy_pass http://localhost:8080;
          proxy_set_header Host $host;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_redirect http://localhost $scheme://SERVICE_DOMAIN;
        }
      }

      # match request domain [port]-[service].[serviceDomain]
      # proxy extra service such as spark-ui
      server {
        listen 80;
        server_name "~(?<svc_port>[0-9]+)-(?<svc_name>[^.]*)\.(.*)";
        location / {
          resolver 127.0.0.1:53 ipv6=off;
          proxy_pass http://$svc_name.NAMESPACE.svc:$svc_port;
          proxy_set_header Host $host;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_redirect http://localhost $scheme://SERVICE_DOMAIN;

          # redirect rule for spark ui. 302 redirect response misses port number of service domain
          proxy_redirect ~(http:[/]+[0-9]+[-][^-]+[-][^.]+)[^/]+(\/jobs.*) $1.SERVICE_DOMAIN$2;
        }
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zeppelin-server
  labels:
    app.kubernetes.io/name: zeppelin-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: zeppelin-server
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: zeppelin-server
    spec:
      serviceAccountName: zeppelin-server
      volumes:
      - name: nginx-conf
        configMap:
          name: zeppelin-server-conf
          items:
          - key: nginx.conf
            path: nginx.conf
      - name: zeppelin-server-notebook-volume
        persistentVolumeClaim:
          claimName: zeppelin-notebook
#      - name: zeppelin-server-conf
#        persistentVolumeClaim:
#          claimName: zeppelin-conf
#      - name: zeppelin-server-custom-k8s
#        persistentVolumeClaim:
#          claimName: zeppelin-k8s
      nodeSelector:
        kubernetes.io/hostname: worker01convergente
      containers:
      - name: zeppelin-server
        image: apache/zeppelin:0.9.0-SNAPSHOT
        command: ["sh", "-c", "$(ZEPPELIN_HOME)/bin/zeppelin.sh"]
        lifecycle:
          preStop:
            exec:
              # SIGTERM triggers a quick exit; gracefully terminate instead
              command: ["sh", "-c", "ps -ef | grep org.apache.zeppelin.server.ZeppelinServer | grep -v grep | awk '{print $2}' | xargs kill"]
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        - name: rpc
          containerPort: 12320
        env:
        - name: POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        envFrom:
        - configMapRef:
            name: zeppelin-server-conf-map
        volumeMounts:
        - name: zeppelin-server-notebook-volume     # configure this to persist notebook
          mountPath: /zeppelin/notebook
#        - name: zeppelin-server-conf                # configure this to persist Zeppelin configuration
#          mountPath: /zeppelin/conf
#        - name: zeppelin-server-custom-k8s          # configure this to mount customized Kubernetes spec for interpreter
#          mountPath: /zeppelin/k8s
      - name: zeppelin-server-gateway
        image: nginx:1.14.0
        command: ["/bin/sh", "-c"]
        env:
        - name: SERVICE_DOMAIN
          valueFrom:
            configMapKeyRef:
              name: zeppelin-server-conf-map
              key: SERVICE_DOMAIN
        args:
          - cp -f /tmp/conf/nginx.conf /etc/nginx/nginx.conf;
            sed -i -e "s/SERVICE_DOMAIN/$SERVICE_DOMAIN/g" /etc/nginx/nginx.conf;
            sed -i -e "s/NAMESPACE/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/g" /etc/nginx/nginx.conf;
            cat /etc/nginx/nginx.conf;
            /usr/sbin/nginx
        volumeMounts:
          - name: nginx-conf
            mountPath: /tmp/conf
        lifecycle:
          preStop:
            exec:
              # SIGTERM triggers a quick exit; gracefully terminate instead
              command: ["/usr/sbin/nginx", "-s", "quit"]
      - name: dnsmasq  # nginx requires dns resolver for dynamic dns resolution
        image: "janeczku/go-dnsmasq:release-1.0.5"
        args:
          - --listen
          - "127.0.0.1:53"
          - --default-resolver
          - --append-search-domains
          - --hostsfile=/etc/hosts
          - --verbose
---
kind: Service
apiVersion: v1
metadata:
  name: zeppelin-server
spec:
  ports:
    - name: http
      port: 80
    - name: rpc            # port name is referenced in the code. So it shouldn't be changed.
      port: 12320
  selector:
    app.kubernetes.io/name: zeppelin-server
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: zeppelin-server
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["create", "get", "update", "patch", "list", "delete", "watch"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["roles", "rolebindings"]
  verbs: ["bind", "create", "get", "update", "patch", "list", "delete", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role-binding
subjects:
- kind: ServiceAccount
  name: zeppelin-server
roleRef:
  kind: Role
  name: zeppelin-server-role
  apiGroup: rbac.authorization.k8s.io

What I have tried

• At first, trying to run a "helloworld" with python interpreter was impossible, due to there was no kubectl installed on zeppelin:0.9.0 docker image, so I built the 0.9.0-SNAPSHOT image adding kubectl installation.
• Now python interpreter pod and service appears on my dashboard, but container crashes with the following error:

 INFO [2020-08-29 22:04:39,544] ({main} RemoteInterpreterServer.java[<init>]:161) - Starting remote interpreter server on port 0, intpEventServerAddress: zeppelin-server-6d7f46d8d8-m9jlv.zeppelin.svc:12320
Exception in thread "main" org.apache.zeppelin.shaded.org.apache.thrift.transport.TTransportException: 
java.net.UnknownHostException: zeppelin-server-6d7f46d8d8-m9jlv.zeppelin.svc
    at org.apache.zeppelin.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:226)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.<init>(RemoteInterpreterServer.java:167)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.<init>(RemoteInterpreterServer.java:152)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:321)
Caused by: java.net.UnknownHostException: zeppelin-server-6d7f46d8d8-m9jlv.zeppelin.svc
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at org.apache.zeppelin.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:221)
    ... 3 more

• Ok! So my zeppelin-server pod's name its the one that appears in python's interpreter log, zeppelin-server-6d7f46d8d8-m9jlv, as shown here!.

I don't know what else to do to make it work! Any help, hint or suggestion will be much appreciated! Also, if further information is needed, tell me and of course I'll provide it.

Answer 1

I was able to run this stack by changing Deployment to Pod. Here is full yaml I'm using:

Please note I'm using custom zeppelin:0.9.0 image instead of apache/zeppelin:0.9.0 in Pod specs as it has a lack of kubectl binary

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
apiVersion: v1
kind: ConfigMap
metadata:
  name: zeppelin-server-conf-map
data:
  # 'serviceDomain' is a Domain name to use for accessing Zeppelin UI.
  # Should point IP address of 'zeppelin-server' service.
  #
  # Wildcard subdomain need to be point the same IP address to access service inside of Pod (such as SparkUI).
  # i.e. if service domain is 'local.zeppelin-project.org', DNS configuration should make 'local.zeppelin-project.org' and '*.local.zeppelin-project.org' point the same address.
  #
  # Default value is 'local.zeppelin-project.org' while it points 127.0.0.1 and `kubectl port-forward zeppelin-server` will give localhost to connects.
  # If you have your ingress controller configured to connect to `zeppelin-server` service and have a domain name for it (with wildcard subdomain point the same address), you can replace serviceDomain field with your own domain.
  SERVICE_DOMAIN: local.zeppelin-project.org:8080
  ZEPPELIN_K8S_SPARK_CONTAINER_IMAGE: spark:2.4.7-bin-hadoop-3.2.0-cloud-scala-2.11
  ZEPPELIN_K8S_CONTAINER_IMAGE: apache/zeppelin:0.9.0
  ZEPPELIN_HOME: /zeppelin
  ZEPPELIN_SERVER_RPC_PORTRANGE: 12320:12320
  # default value of 'master' property for spark interpreter.
  SPARK_MASTER: k8s://https://kubernetes.default.svc
  # default value of 'SPARK_HOME' property for spark interpreter.
  SPARK_HOME: /spark
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zeppelin-server-conf
data:
  sparkContainerImage: spark:2.4.7-bin-hadoop-3.2.0-cloud-scala-2.11
  nginx.conf: |
    daemon off;
    worker_processes auto;
    events {
      worker_connections 1024;
    }
    http {
      map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
      }
      # first server block will be default. Proxy zeppelin server.
      server {
        listen 80;
        location / {
          proxy_pass http://localhost:8080;
          proxy_set_header Host $host;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_redirect http://localhost $scheme://SERVICE_DOMAIN;
        }
      }
      # match request domain [port]-[service].[serviceDomain]
      # proxy extra service such as spark-ui
      server {
        listen 80;
        server_name "~(?<svc_port>[0-9]+)-(?<svc_name>[^.]*)\.(.*)";
        location / {
          resolver 127.0.0.1:53 ipv6=off;
          proxy_pass http://$svc_name.NAMESPACE.svc:$svc_port;
          proxy_set_header Host $host;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_redirect http://localhost $scheme://SERVICE_DOMAIN;
          # redirect rule for spark ui. 302 redirect response misses port number of service domain
          proxy_redirect ~(http:[/]+[0-9]+[-][^-]+[-][^.]+)[^/]+(\/jobs.*) $1.SERVICE_DOMAIN$2;
        }
      }
    }
---
apiVersion: v1
kind: Pod
metadata:
  name: zeppelin-server
  labels:
    app.kubernetes.io/name: zeppelin-server
spec:
  serviceAccountName: zeppelin-server
  volumes:
  - name: nginx-conf
    configMap:
      name: zeppelin-server-conf
      items:
      - key: nginx.conf
        path: nginx.conf
  containers:
  - name: zeppelin-server
    image: zeppelin:0.9.0
    command: ["sh", "-c", "$(ZEPPELIN_HOME)/bin/zeppelin.sh"]
    lifecycle:
      preStop:
        exec:
          # SIGTERM triggers a quick exit; gracefully terminate instead
          command: ["sh", "-c", "ps -ef | grep org.apache.zeppelin.server.ZeppelinServer | grep -v grep | awk '{print $2}' | xargs kill"]
    ports:
    - name: http
      containerPort: 8080
    - name: https
      containerPort: 8443
    - name: rpc
      containerPort: 12320
    env:
    - name: POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    envFrom:
    - configMapRef:
        name: zeppelin-server-conf-map
  # volumeMounts:
  #  - name: zeppelin-server-notebook-volume     # configure this to persist notebook
  #    mountPath: /zeppelin/notebook
  #  - name: zeppelin-server-conf                # configure this to persist Zeppelin configuration
  #    mountPath: /zeppelin/conf
  #  - name: zeppelin-server-custom-k8s          # configure this to mount customized Kubernetes spec for interpreter
  #    mountPath: /zeppelin/k8s
  - name: zeppelin-server-gateway
    image: nginx:1.14.0
    command: ["/bin/sh", "-c"]
    env:
    - name: SERVICE_DOMAIN
      valueFrom:
        configMapKeyRef:
          name: zeppelin-server-conf-map
          key: SERVICE_DOMAIN
    args:
      - cp -f /tmp/conf/nginx.conf /etc/nginx/nginx.conf;
        sed -i -e "s/SERVICE_DOMAIN/$SERVICE_DOMAIN/g" /etc/nginx/nginx.conf;
        sed -i -e "s/NAMESPACE/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/g" /etc/nginx/nginx.conf;
        cat /etc/nginx/nginx.conf;
        /usr/sbin/nginx
    volumeMounts:
      - name: nginx-conf
        mountPath: /tmp/conf
    lifecycle:
      preStop:
        exec:
          # SIGTERM triggers a quick exit; gracefully terminate instead
          command: ["/usr/sbin/nginx", "-s", "quit"]
  - name: dnsmasq  # nginx requires dns resolver for dynamic dns resolution
    image: "janeczku/go-dnsmasq:release-1.0.5"
    args:
      - --listen
      - "127.0.0.1:53"
      - --default-resolver
      - --append-search-domains
      - --hostsfile=/etc/hosts
      - --verbose
---
kind: Service
apiVersion: v1
metadata:
  name: zeppelin-server
spec:
  ports:
    - name: http
      port: 80
    - name: rpc            # port name is referenced in the code. So it shouldn't be changed.
      port: 12320
  selector:
    app.kubernetes.io/name: zeppelin-server
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: zeppelin-server
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["create", "get", "update", "patch", "list", "delete", "watch"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["roles", "rolebindings"]
  verbs: ["bind", "create", "get", "update", "patch", "list", "delete", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: zeppelin-server-role-binding
subjects:
- kind: ServiceAccount
  name: zeppelin-server
roleRef:
  kind: Role
  name: zeppelin-server-role
  apiGroup: rbac.authorization.k8s.io

Also, we could try to override HOSTNAME env var within the zeppelin container, but it could cause issues. Didn't try it though.