Research on Ubernetes (High availability zone) and federation clusters for Kubernetes

Posted on | In |

Abstract

  1. Multiple zone minion nodes share the same master node (api server, etcd and etc). Need to seriously considering the high availability of master node. Nodes is labeled by different zone and failure-domain to allow distribute workload across zones.

  2. PV Affinity is used for zone awareness. But PV affinity is not ready. Most cloud providers (at least GCE and AWS) cannot attach their persistent volumes across zones. Thus when a pod is being scheduled, if there is a volume
    attached, that will dictate the zone. This will be implemented using a new
    scheduler predicate (a hard constraint): VolumeZonePredicate.
    The PV is also labeled with the zone & region it was created in. For version 1.3.4, dynamic persistent volumes are always created in the zone of the cluster master (here us-central1-a / us-west-2a); this will be improved in a future version (issue #23330.).

  3. Current kube-up.sh scripts depends on kube-up/kube-down/kube-push implementation and SDK of cloudprovider. gce and aws are supported well, but not for other cloudprovider today. (v1.3.4)

  4. Alternative solution to setup an multiple availability zone with simple cloudprovider ubuntu/centos.

    4.1 Key spec

    • share etcd
    • share flannel network
    • share docker registry

    • share apiserver. MASTER_INTERNAL_IP=172.20.0.9

      4.2 Solution

      Create VM firslty, then kube-up.sh an ubuntu (centos) worker only to reuse the same master.

Prerequisite

pip install --upgrade aws-cli

Init install

curl -sS https://get.k8s.io | MULTIZONE=true KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2 NUM_NODES=1 bash

Setup an multizone cluster on aws

  1. Setup 1 node from zone us-west-2

    MULTIZONE=true KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2a NUM_NODES=1 ./cluster/kube-up.sh

  2. Add 1 node cluster from zone us-west-2a

    KUBE_USE_EXISTING_MASTER=true MULTIZONE=true KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2b NUM_NODES=1 KUBE_SUBNET_CIDR=172.20.1.0/24 MASTER_INTERNAL_IP=172.20.0.9 kubernetes/cluster/kube-up.sh

  3. Nodes labels for availability zone

1
2
3
4
kubectl get nodes --show-labels
NAME                                         STATUS     AGE       LABELS
ip-172-20-0-31.us-west-2.compute.internal    NotReady   2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=t2.micro,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2a,kubernetes.io/hostname=ip-172-20-0-31.us-west-2.compute.internal
ip-172-20-1-138.us-west-2.compute.internal   Ready      2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=t2.micro,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2b,kubernetes.io/hostname=ip-172-20-1-138.us-west-2.compute.internal

Create PV with zone affinity, provisioned at zone of master node

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
kubectl create -f - <<EOF
{
  "kind": "PersistentVolumeClaim",
  "apiVersion": "v1",
  "metadata": {
    "name": "claim1",
    "annotations": {
        "volume.alpha.kubernetes.io/storage-class": "foo"
    }
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "1Gi"
      }
    }
  }
}
EOF
1
2
NAME                                       CAPACITY   ACCESSMODES   STATUS    CLAIM            REASON    AGE       LABELS
pvc-feb20d75-593a-11e6-afe8-06d12fd78863   1Gi        RWO           Bound     default/claim1             42s       failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2a

Destroy multiple-zone cluster

1
2
KUBERNETES_PROVIDER=aws KUBE_USE_EXISTING_MASTER=true KUBE_AWS_ZONE=us-west-2b kubernetes/cluster/kube-down.sh
KUBERNETES_PROVIDER=aws KUBE_AWS_ZONE=us-west-2a kubernetes/cluster/kube-down.sh

Multiple cluster management

Federation panel

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#Mandatory to build your own federation images with source code 
FEDERATION=true KUBE_RELEASE_RUN_TESTS=n make quick-release
FEDERATION=true FEDERATION_PUSH_REPO_BASE=<Your private registry> ./build/push-federation-images.sh
#Setup federation container, dns, router, lb, kubeconfig and federation namespace
GOROOT=/root/go KUBE_ROOT=/root/k8s/src/k8s.io/kubernetes KUBERNETES_PROVIDER=aws DNS_ZONE_NAME=myfederation.aws FEDERATION_NAME=myfederation FEDERATION_PUSH_REPO_BASE=<your private registry> /root/k8s/src/k8s.io/kubernetes/federation/cluster/federation-up.sh
GOROOT=/root/go KUBE_ROOT=/root/k8s/src/k8s.io/kubernetes KUBERNETES_PROVIDER=aws DNS_ZONE_NAME=myfederation.aws FEDERATION_NAME=myfederation FEDERATION_PUSH_REPO_BASE=<your private registry> /root/k8s/src/k8s.io/kubernetes/federation/cluster/federation-down.sh
service "federation-apiserver" deleted
secret "default-token-k6v8j" deleted
secret "federation-apiserver-kubeconfig" deleted
namespace "federation" deleted