Kubernetes Pod Vertical Auto-Scaling (VPA)

Kubernetes Pod Vertical Auto-Scaling (VPA)

This article has participated in the Haowen Convening Order Activity, click to view: Back-end, big front-end dual-track submissions, 20,000 yuan prize pool waiting for you to challenge

1 Introduction to VPA

The full name of VPA is Vertical Pod Autoscaler, which means vertical Pod autoscaler. It automatically sets CPU and memory requests according to the resource usage of the container, thus allowing proper scheduling on the node to provide appropriate resources for each Pod. It can not only shrink the container that excessively requests resources, but also increase the capacity of insufficient resources at any time according to its usage. PS: VPA will not change the resource limits of Pod.

Don't talk nonsense, just go to the picture and see the VPA workflow

2 Deploy metrics-server

2.1 Download the deployment manifest file

[the root @ ~ the VM-10-48-CentOS] # wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml copy the code

2.2 Modify the components.yaml file

  • The mirror address is modified as:
    scofield/metrics-server:v0.3.7
  • Modified the metrics-server startup parameter args
- name: metrics-Server Image: Scofield/metrics-Server: v0.3.7 imagePullPolicy: IfNotPresent args: - --cert-the dir =/tmp - --secure-Port = 4443 - /Server-metrics - --kubelet-in the insecure - TLS - --kubelet-Preferred-address-types = InternalIP duplicated code

2.3 Perform deployment

[the root @ ~ the VM-10-48-CentOS] # kubectl Apply -f components.yaml duplicated code

2.4 Verification

[root@VM-10-48-centos ~] # kubectl get po -n kube-system | grep metrics-server metrics-server-5b58f4df77-f7nks 1/1 Running 0 35d # Can get the top information as success [root@VM-10-48-centos ~] # kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 10.1.2.15 138m 3% 4207Mi 29% 10.1.2.16 159m 4% 3138Mi 45% 10.1.2.17 147m 3% 4118Mi 59% 10.1.50.2 82m 4% 1839Mi 55% Copy code

3 Deploy vertical-pod-autoscaler

3.1 Clone the autoscaler project

[the root @ ~ the VM-10-48-CentOS] # Git clone https://github.com/kubernetes/autoscaler.git duplicated code

3.2 Modify the deployment file

[root@VM-10-48-centos ~] # cd autoscaler/vertical-pod-autoscaler/deploy admission-controller-deployment.yaml us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-admission-controller:0.8.0 To scofield/vpa-admission-controller:0.8.0 recommender-deployment.yaml us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-recommender:0.8.0 To image: scofield/vpa-recommender:0.8.0 updater-deployment.yaml us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-updater:0.8.0 To scofield/vpa-updater:0.8.0 Copy code

3.3 deployment

[root@VM-10-48-centos ~] # cd autoscaler/vertical-pod-autoscaler [root@VM-10-48-centos ~] # ./hack/vpa-up.sh customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created clusterrole.rbac.authorization.k8s.io/system:metrics-reader created clusterrole.rbac.authorization.k8s.io/system:vpa-actor created clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created clusterrole.rbac.authorization.k8s.io/system:evictioner created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created serviceaccount/vpa-admission-controller created clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created serviceaccount/vpa-updater created deployment.apps/vpa-updater created serviceaccount/vpa-recommender created deployment.apps/vpa-recommender created Generating certs for the VPA Admission Controller in/tmp/vpa-certs. Generating RSA private key, 2048 bit long modulus (2 primes) .................................................. ..........................+++++ .+++++ e is 65537 (0x010001) Generating RSA private key, 2048 bit long modulus (2 primes) ............+++++ .................................................. .........................+++++ e is 65537 (0x010001) Signature ok subject=CN = vpa-webhook.kube-system.svc Getting CA Private Key Uploading certs to the cluster. secret/vpa-tls-certs created Deleting/tmp/vpa-certs. deployment.apps/vpa-admission-controller created service/vpa-webhook created Copy code

If there is an error here: ERROR: Failed to create CA certificate for self-signing. If the error is "unknown option -addext", update your openssl version or deploy VPA from the vpa-release-0.8 branch

Need to upgrade

openssl
The version resolved

[root@VM-10-48-centos ~] # yum install gcc gcc-c++ -y [root@VM-10-48-centos ~] # openssl version -a [root@VM-10-48-centos ~] # wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz && tar zxf openssl-1.1.1k.tar.gz && cd openssl-1.1.1k [root@VM-10-48 -centos ~] # ./config [root@VM-10-48-centos ~] # make && make install [root@VM-10-48-centos ~] # mv/usr/local/bin/openssl/usr/local/bin/openssl.bak [root@VM-10-48-centos ~] # mv apps/openssl/usr/local/bin [root@VM-10-48-centos ~] # openssl version -a OpenSSL 1.1.1k 25 Mar 2021 (Library: OpenSSL 1.1.1g FIPS 21 Apr 2020) built on: Mon Mar 29 23:48:12 2021 UTC platform: linux-x86_64 options: bn(64,64) rc4(16x,int) des(int) idea(int) blowfish(ptr) compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer- size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_SMMONT_N -DOPENSSL_SMMON5 -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM DOPENSSL_BN_ASM_GF2m -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM = "\"/dev/urandom/"" OPENSSLDIR: "/etc/pki/TLS " ENGINESDIR: "/usr/lib64/engines-1.1 " Seeding Source : os-specificCopy code

Execute again

vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh

3.4 View results

You can see that metrics-server and vpa are already running normally

[root@VM-10-48-centos ~] # kubectl get po -n kube-system | grep -E "metrics-server|vpa" metrics-server-5b58f4df77-f7nks 1/1 Running 0 35d vpa-admission-controller-7ff888c959-tvtmk 1/1 Running 0 104m vpa-recommender-74f69c56cb-zmzwg 1/1 Running 0 104m vpa-updater-79b88f9c55-m4xx5 1/1 Running 0 103m Copy code

4 example

4.1 updateMode: Off

1 First we deploy one

nginx
Service, deploy to
namespace: vpa
in

apiVersion: Apps/V1 kind: the Deployment Metadata: Labels: App: Nginx name: Nginx namespace: VPA spec: Replicas: 2 Selector: matchLabels: App: Nginx Template: Metadata: Labels: App: Nginx spec: Containers: - Image: Nginx name: nginx Resources: Requests: the CPU: 100m Memory: 250mi copy the code

Look at the results, 2 pods ran normally

[root@VM-10-48-centos ~] # kubectl get po -n vpa NAME READY STATUS RESTARTS AGE nginx-59fdffd754-cb5dn 1/1 Running 0 8s nginx-59fdffd754-cw8d7 1/1 Running 0 9s Copy code

2 Create a NodePort type service

[root@VM-10-48-centos ~] # cat svc.yaml apiVersion: v1 kind: Service metadata: name: nginx namespace: vpa spec: type : NodePort ports: -port: 80 targetPort: 80 selector: app: nginx [root@VM-10-48-centos ~] # kubectl get svc -n vpa | grep nginx nginx NodePort 10.255.253.166 <none> 80:30895/TCP 54s [root@VM-2-16-centos ~] # curl -I 10.1.2.16:30895 HTTP/1.1 200 OK Server: nginx/1.21.1 Date: Fri, 09 Jul 2021 09:54:58 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 06 Jul 2021 14:59:17 GMT Connection: keep-alive ETag: "60e46fc5-264" Accept-Ranges: bytes Copy code

3 Create a VPA and use it first

updateMode: "Off"
Mode, this mode only obtains resource recommendations , but does not update the Pod

[root@VM-10-48-centos ~] # cat nginx-vpa-demo.yaml apiVersion: autoscaling.k8s.io/v1beta2 kind: VerticalPodAutoscaler metadata: name: nginx-vpa namespace: vpa spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: nginx updatePolicy: updateMode: "Off" resourcePolicy: containerPolicies: -containerName: "nginx" minAllowed: cpu: "250m" memory: "100Mi" maxAllowed: the CPU: "2000m" Memory: "2048Mi" Copy the code

4 View deployment results

[root@VM-10-48-centos ~] # kubectl get vpa -n vpa NAME MODE CPU MEM PROVIDED AGE nginx-vpa Off 7s Copy code

5 Use describe to view vpa details, mainly focusing on Container Recommendations

[root@VM-10-48-centos ~] # kubectl describe vpa nginx-vpa -n vpa Name: nginx-vpa Namespace: vpa Spec: Resource Policy: Container Policies: Container Name: nginx Max Allowed: Cpu: 2000m Memory: 2048Mi Min Allowed: Cpu: 250m Memory: 100Mi Target Ref: API Version: apps/v1 Kind: Deployment Name: nginx Update Policy: Update Mode: Off Status: Conditions: Last Transition Time: 2021-07-09T09:59:50Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 250m Memory: 262144k Uncapped Target: Cpu: 25m Memory: 262144k Upper Bound: Cpu: 670m Memory: 700542995 Copy code

among them

Lower Bound: Lower limit Target: Recommended value Upper Bound: Upper limit Uncapped Target: If there is no minimum or maximum boundary for the VPA, it means the target utilization The above results show that the recommended Pod's CPU request is 25 m, and the recommended memory request is 262144 k bytes. Copy code

6 Right now

nginx
Perform pressure test and execute pressure test command

[root@VM-10-48-centos ~]# ab -c 100 -n 10000000 http://10.1.2.16:30895/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.2.16 (be patient) Completed 1000000 requests Completed 2000000 requests Completed 3000000 requests Copy code

7 Watch the VPA Recommendation changes in a few minutes

[root@VM-10-48-centos ~] # kubectl describe vpa -n vpa nginx-vpa | tail -n 20 Conditions: Last Transition Time: 2021-07-09T09:59:50Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 1643m Memory: 262144k Uncapped Target: Cpu: 1643m Memory: 262144k Upper Bound: Cpu: 2 Memory: 562581530 Events: <none> Copy code

It can be seen from the output information that VPA gives recommended values for Pod:

Cpu: 1643m
Because we set up here
updateMode: "Off"
, So the Pod will not be updated

4.2 updateMode: Auto

1 Change updateMode: "Auto" to see what the VPA will do. Here change the resources to:

memory: 50Mi, cpu: 100m

[root@VM-10-48-centos ~] # kubectl get po -n vpa NAME READY STATUS RESTARTS AGE nginx-5594c66dc6-lzs67 1/1 Running 0 26s nginx-5594c66dc6-zk6h9 1/1 Running 0 21s Copy code

2 Deploy vpa again, here VPA deployment file

nginx-vpa-demo.yaml
Only changed
updateMode: "Auto"

[root@k8s-node001 examples]# cat nginx-vpa-demo.yaml apiVersion: autoscaling.k8s.io/v1beta2 kind: VerticalPodAutoscaler metadata: name: nginx-vpa-2 namespace: vpa spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: nginx updatePolicy: updateMode: "Auto" resourcePolicy: containerPolicies: -containerName: "nginx" minAllowed: cpu: "250m" memory: "100Mi" maxAllowed: cpu: "2000m" memory: "2048Mi" Copy code

3 Pressure test again

[root@VM-10-48-centos ~] # ab -c 100 -n 10000000 http://10.1.2.16:30895/Copy code

4 After a few minutes, use

describe
View the details of vpa, also only pay attention to
Container Recommendations

[root@VM-10-48-centos ~] # kubectl describe vpa nginx-vpa -n vpa | tail -n 20 Conditions: Last Transition Time: 2021-07-09T09:59:50Z Status: True Type: RecommendationProvided Recommendation: Container Recommendations: Container Name: nginx Lower Bound: Cpu: 250m Memory: 262144k Target: Cpu: 1643m Memory: 262144k Uncapped Target: Cpu: 1643m Memory: 262144k Upper Bound: Cpu: 2 Memory: 511550327 Events: <none> Copy code

Target becomes

Cpu: 1643m, Memory: 262144k

5. Look at the event event

[root@VM-10-48-centos ~] # kubectl get event -n vpa LAST SEEN TYPE REASON OBJECT MESSAGE 38s Normal Scheduled pod/nginx-5594c66dc6-d8d6h Successfully assigned vpa/nginx-5594c66dc6-d8d6h to 10.1.2.16 38s Normal Pulling pod/nginx-5594c66dc6-d8d6h Pulling image "nginx" 37s Normal Pulled pod/nginx-5594c66dc6-d8d6h Successfully pulled image "nginx" 37s Normal Created pod/nginx-5594c66dc6-d8d6h Created container nginx 37s Normal Started pod/nginx-5594c66dc6-d8d6h Started container nginx 3m10s Normal Scheduled pod/nginx-5594c66dc6-lzs67 Successfully assigned vpa/nginx-5594c66dc6-lzs67 to 10.1.2.15 3m9s Normal Pulling pod/nginx-5594c66dc6-lzs67 Pulling image "nginx" 3m5s Normal Pulled pod/nginx-5594c66dc6-lzs67 Successfully pulled image "nginx" 3m5s Normal Created pod/nginx-5594c66dc6-lzs67 Created container nginx 3m5s Normal Started pod/nginx-5594c66dc6-lzs67 Started container nginx 99s Normal EvictedByVPA pod/nginx-5594c66dc6-lzs67 Pod was evicted by VPA Updater to apply resource recommendation. 99s Normal Killing pod/nginx-5594c66dc6-lzs67 Stopping container nginx 98s Normal Scheduled pod/nginx-5594c66dc6-tdmnh Successfully assigned vpa/nginx-5594c66dc6-tdmnh to 10.1.2.15 98s Normal Pulling pod/nginx-5594c66dc6-tdmnh Pulling image "nginx" 97s Normal Pulled pod/nginx-5594c66dc6-tdmnh Successfully pulled image "nginx" 97s Normal Created pod/nginx-5594c66dc6-tdmnh Created container nginx 97s Normal Started pod/nginx-5594c66dc6-tdmnh Started container nginx 3m5s Normal Scheduled pod/nginx-5594c66dc6-zk6h9 Successfully assigned vpa/nginx-5594c66dc6-zk6h9 to 10.1.2.17 3m4s Normal Pulling pod/nginx-5594c66dc6-zk6h9 Pulling image "nginx" 3m Normal Pulled pod/nginx-5594c66dc6-zk6h9 Successfully pulled image "nginx" 2m59s Normal Created pod/nginx-5594c66dc6-zk6h9 Created container nginx 2m59s Normal Started pod/nginx-5594c66dc6-zk6h9 Started container nginx 39s Normal EvictedByVPA pod/nginx-5594c66dc6-zk6h9 Pod was evicted by VPA Updater to apply resource recommendation. 39s Normal Killing pod/nginx-5594c66dc6-zk6h9 Stopping container nginx 3m10s Normal SuccessfulCreate replicaset/nginx-5594c66dc6 Created pod: nginx-5594c66dc6-lzs67 3m5s Normal SuccessfulCreate replicaset/nginx-5594c66dc6 Created pod: nginx-5594c66dc6-zk6h9 99s Normal SuccessfulCreate replicaset/nginx-5594c66dc6 Created pod: nginx-5594c66dc6-tdmnh 38s Normal SuccessfulCreate replicaset/nginx-5594c66dc6 Created pod: nginx-5594c66dc6-d8d6h 35m Normal Scheduled pod/nginx-59fdffd754-cb5dn Successfully assigned vpa/nginx-59fdffd754-cb5dn to 10.1.2.16 35m Normal Pulling pod/nginx-59fdffd754-cb5dn Pulling image "nginx" 35m Normal Pulled pod/nginx-59fdffd754-cb5dn Successfully pulled image "nginx" 35m Normal Created pod/nginx-59fdffd754-cb5dn Created container nginx 35m Normal Started pod/nginx-59fdffd754-cb5dn Started container nginx 3m5s Normal Killing pod/nginx-59fdffd754-cb5dn Stopping container nginx 35m Normal Scheduled pod/nginx-59fdffd754-cw8d7 Successfully assigned vpa/nginx-59fdffd754-cw8d7 to 10.1.2.16 35m Normal Pulling pod/nginx-59fdffd754-cw8d7 Pulling image "nginx" 35m Normal Pulled pod/nginx-59fdffd754-cw8d7 Successfully pulled image "nginx" 35m Normal Created pod/nginx-59fdffd754-cw8d7 Created container nginx 35m Normal Started pod/nginx-59fdffd754-cw8d7 Started container nginx 2m58s Normal Killing pod/nginx-59fdffd754-cw8d7 Stopping container nginx 35m Normal SuccessfulCreate replicaset/nginx-59fdffd754 Created pod: nginx-59fdffd754-cw8d7 35m Normal SuccessfulCreate replicaset/nginx-59fdffd754 Created pod: nginx-59fdffd754-cb5dn 3m5s Normal SuccessfulDelete replicaset/nginx-59fdffd754 Deleted pod: nginx-59fdffd754-cb5dn 2m58s Normal SuccessfulDelete replicaset/nginx-59fdffd754 Deleted pod: nginx-59fdffd754-cw8d7 35m Normal ScalingReplicaSet deployment/nginx Scaled up replica set nginx-59fdffd754 to 2 34m Normal EnsuringService service/nginx Deleted Loadbalancer 34m Normal EnsureServiceSuccess service/nginx Service Sync Success. RetrunCode: S2000 3m10s Normal ScalingReplicaSet deployment/nginx Scaled up replica set nginx-5594c66dc6 to 1 3m5s Normal ScalingReplicaSet deployment/nginx Scaled down replica set nginx-59fdffd754 to 1 3m5s Normal ScalingReplicaSet deployment/nginx Scaled up replica set nginx-5594c66dc6 to 2 Normal ScalingReplicaSet Deployment 2m58s/Nginx as Scaled Down Replica SET Nginx-59fdffd754 0 to copy the code

It can be understood from the output information that vpa executed EvictedByVPA, automatically stopped nginx, and then started a new nginx using the resources recommended by VPA, we can check the nginx pod to get confirmation

[root@VM-10-48-centos ~] # kubectl describe po -n vpa nginx-5594c66dc6-d8d6h Name: nginx-5594c66dc6-d8d6h Namespace: vpa Priority: 0 Node: 10.1.2.16/10.1.2.16 Start Time: Fri, 09 Jul 2021 18:09:26 +0800 Labels: app=nginx pod-template-hash=5594c66dc6 Annotations: tke.cloud.tencent.com/networks-status: [{ "name" : "tke-bridge" , "interface" : "eth0" , "ips" : [ "10.252.1.50" ], "mac" : "e6:38:26:0b:c5:97" , "default" : true , "dns" : {} }] vpaObservedContainers: nginx vpaUpdates: Pod resources updated by nginx-vpa: container 0: cpu request, memory request Status: Running IP: 10.252.1.50 IPs: IP: 10.252.1.50 Controlled By: ReplicaSet/nginx-5594c66dc6 Containers: nginx: Container ID: docker://42e45f5f122ba658e293395d78a073cfe51534c773f9419a179830fd6d1698ea Image: nginx Image ID: docker-pullable://nginx@sha256:8df46d7414eda82c2a8c9c50926545293811ae59f977825845dda7d558b4125b Port: <none> Host Port: <none> State: Running Started: Fri, 09 Jul 2021 18:09:27 +0800 Ready: True Restart Count: 0 Requests: cpu: 1643m memory: 262144k Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-m2j2z (ro)

Requests

cpu: 1643m memory: 262144k

requests: cpu: 100m memory: 50Mi

VPA VPA pod VPA pod

4.3 VPA

  • HPA Horizontal Pod Autoscaler
  • Pod Deployment StatefulSet

4.4 VPA

  • Pod
  • Pod
  • CPU
  • VPA CPU