Kubernetes Multi master setup failing

depam · 01-17-2017, 10:00 AM

I now have setup Kubernetes cluster with a master and 6 nodes. I've been trying to make the kube-controller-manager and kube-scheduler redundant without any success. I've been following the instruction from the Kubernetes Cookbook as the documentation from Kubernetes itself is not that detailed.

Here's the config

Code:

KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://127.0.0.1:8080"

Code:

# cat kubelet
###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=127.0.0.1"

# location of the api-server
KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080"

# pod infrastructure container
#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

# Add your own!
KUBELET_ARGS="--register-node=false --allow-privileged=true --config /etc/kubernetes/manifests"

In the book, it mentioned the following yaml files need to be there

Code:

[root@ip-172-16-225-42 manifests]# cat podmaster.yaml
apiVersion: v1
kind: Pod
metadata:
  name: podmaster
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: scheduler-elector
    image: gcr.io/google_containers/podmaster:1.1
    command:
    - /podmaster
    - --etcd-servers=http://172.16.225.42:2379
    - --key=scheduler
    - --source-file=/kubernetes/kube-scheduler.yaml
    - --dest-file=/manifests/kube-scheduler.yaml
    volumeMounts:
    - mountPath: /kubernetes
      name: k8s
      readOnly: true
    - mountPath: /manifests
      name: manifests
  - name: controller-manager-elector
    image: gcr.io/google_containers/podmaster:1.1
    command:
    - /podmaster
    - --etcd-servers=http://172.16.225.42:2379
    - --key=controller
    - --source-file=/kubernetes/kube-controller-manager.yaml
    - --dest-file=/manifests/kube-controller-manager.yaml
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /kubernetes
      name: k8s
      readOnly: true
    - mountPath: /manifests
      name: manifests
  volumes:
  - hostPath:
      path: /srv/kubernetes
    name: k8s
  - hostPath:
      path: /etc/kubernetes/manifests
    name: manifests

and

Code:

[root@ip-172-16-225-42 kubernetes]# cat kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-controller-manager --master=127.0.0.1:8080
      --cluster-cidr=10.254.0.0/16 --allocate-node-cidrs=true
      --v=2 1>>/var/log/kube-controller-manager.log 2>&1
    image: gcr.io/google_containers/kube-controller-manager:fda24638d51a48baa13c35337fcd4793
#    image: depam08/test:k8s_cluster
    livenessProbe:
      httpGet:
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    name: kube-controller-manager
    volumeMounts:
    - mountPath: /srv/kubernetes
      name: srvkube
      readOnly: true
    - mountPath: /var/log/kube-controller-manager.log
      name: logfile
    - mountPath: /usr/local/bin/kube-controller-manager
      name: binfile
  volumes:
  - hostPath:
      path: /srv/kubernetes
    name: srvkube
  - hostPath:
      path: /var/log/kube-controller-manager.log
    name: logfile
  - hostPath:
      path: /usr/local/bin/kube-controller-manager
    name: binfile

[root@ip-172-16-225-42 kubernetes]#

and

Code:

[root@ip-172-16-225-42 kubernetes]# cat kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: gcr.io/google_containers/kube-scheduler:34d0b8f8b31e27937327961528739bc9
#    image: depam08/test:k8s_cluster
    command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-scheduler --master=127.0.0.1:8080 --v=2 1>>/var/log/kube-scheduler.log
      2>&1
    livenessProbe:
      httpGet:
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /var/log/kube-scheduler.log
      name: logfile
    - mountPath: /usr/local/bin/kube-scheduler
      name: binfile
  volumes:
  - hostPath:
      path: /var/log/kube-scheduler.log
    name: logfile
  - hostPath:
      path: /usr/local/bin/kube-scheduler
        name: binfile

When running this, I do get a CrashLoopback. The error is see from /var/log/kube-controller-manager is:

Code:

[root@ip-172-16-225-42 kubernetes]# tail -f /var/log/kube*
==> /var/log/kube-apiserver.log <==

==> /var/log/kube-controller-manager.log <==
/bin/sh: /usr/local/bin/kube-controller-manager: not found
/bin/sh: /usr/local/bin/kube-controller-manager: not found
/bin/sh: /usr/local/bin/kube-controller-manager: not found
/bin/sh: /usr/local/bin/kube-controller-manager: not found
/bin/sh: /usr/local/bin/kube-controller-manager: not found
/bin/sh: /usr/local/bin/kube-controller-manager: not found

==> /var/log/kube-scheduler.log <==

I have tried to check this by running the image and overriding the entrypoint to bash. It appears that even if I docker-cp the kube-controller-manager from my end and run it manually, it yields the same error. Is there something I am doing wrong here or the image could have been bad? Thanks.