Autoscale workload over Span-VLAN k8s cluster

Posted on | In , , , |

Span-VLAN k8s cluster Env

1
2
3
4
5
6
7
8
9
10
11
12
13
:..........:........................:...............:................:............:........:
:    id    :        hostname        :   primary_ip  :   backend_ip   : datacenter : action :  public vlan/private vlan
:..........:........................:...............:................:............:........:
: 27407789 : hydra-calico-dal09-m01 : 169.46.186.87 :  10.173.49.4   :   dal09    :   -    :   1213/1319
: 27407791 : hydra-calico-dal09-w01 : 169.45.171.24 : 10.155.230.151 :   dal09    :   -    :  996/968
: 27407793 : hydra-calico-dal09-w02 :  169.45.131.2 :  10.153.86.9   :   dal09    :   -    :  959/1211
: 27407873 : hydra-calico-dal10-w03 : 169.47.195.84 :  10.171.90.69  :   dal10    :   -    :  1136/1197
: 27407875 : hydra-calico-dal10-w04 : 169.47.195.88 :  10.171.90.94  :   dal10    :   -    :  1136/1197
:..........:........................:...............:................:............:........:
root@hydra-calico-dal09-m01:~# kubectl get nodes --show-labels|grep zone
10.153.86.9      Ready                         26d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,beta.kubernetes.io/zone=dal09,kubernetes.io/hostname=10.153.86.9
10.171.90.94     Ready                         26d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,beta.kubernetes.io/zone=dal10,kubernetes.io/hostname=10.171.90.94

10.171.90.94: zone=dal10
10.153.86.9: zone=dal9

deploy an nginx deployment

1
2
3
kubectl run nginx-hpa --image=nginx --requests=cpu=100m,memory=50M --expose --port=80
kubectl get deployment

run an test against apache server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl run -i --tty service-test --image=busybox /bin/sh
$ wget -q -O- http://nginx-hpa.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

manually scale out application cross zone dal9 and dal10

1
2
3
4
5
6
 $ kubectl scale deployment nginx-hpa --replicas=2
deployment "nginx-hpa" scaled
kubectl get pod -o wide|grep hpa
nginx-hpa-2266641329-901jn        1/1       Running   0          19m       10.171.29.98    10.171.90.94 <= dal10
nginx-hpa-2266641329-zmnrg        1/1       Running   0          8m        10.98.23.204    10.153.86.9 <=dal9

autoscale, set target cpu usage with 30%

1
2
3
4
5
6
7
kubectl autoscale deployment nginx-hpa --cpu-percent=30 --min=2 --max=10
kubectl get hpa
NAME          REFERENCE                TARGET    CURRENT   MINPODS   MAXPODS   AGE
nginx-hpa     Deployment/nginx-hpa     30%       0%        2         10        4s

raising workload with 10 concurrent query threads to reach threshold of CPU usage

1
2
3
kubectl run -i --tty load-generator --image=busybox /bin/sh
while true; do wget -q -O- http://nginx-hpa.default.svc.cluster.local &>/dev/null;done &

result: pod scale out from 2 to 3

1
2
3
4
5
6
7
8
9
$ kubectl get hpa
NAME          REFERENCE                TARGET    CURRENT   MINPODS   MAXPODS   AGE
nginx-hpa     Deployment/nginx-hpa     30%       67%       2         10        5m
$ kubectl get pod -o wide|grep hpa
nginx-hpa-2266641329-2l98k        1/1       Running   0          14s       10.98.23.206    10.153.86.9 <= **newly created pod**
nginx-hpa-2266641329-901jn        1/1       Running   0          38m       10.171.29.98    10.171.90.94
nginx-hpa-2266641329-zmnrg        1/1       Running   0          27m       10.98.23.204    10.153.86.9

drain application from availability zone dal9 to zone dal10 online

1
2
3
4
5
6
7
8
9
10
11
$kubectl drain 10.153.86.9
node "10.153.86.9" cordoned
pod "nginx-hpa-2266641329-zmnrg" evicted
pod "nginx-hpa-2266641329-2l98k" evicted
node "10.153.86.9" drained
$kubectl get pod -o wide|grep hpa
nginx-hpa-2266641329-5w5jt        1/1       Running   0          1m        10.171.29.104   10.171.90.94
nginx-hpa-2266641329-901jn        1/1       Running   0          47m       10.171.29.98    10.171.90.94
nginx-hpa-2266641329-wzwhm        1/1       Running   0          1m        10.171.29.107   10.171.90.94

limitation. local data could not be reserved after drain app from one available zone to another one.

1
error: pods with local storage (use --delete-local-data to override): monitoring-grafana-3730655072-g667m, monitoring-influxdb-957705310-s144d