技术学堂
Kubernetes(k8s)是Google开源的一个容器编排引擎，它支持自动化部署、大规模可伸缩、应用容器化管理，具备可移植、可扩展、自动化等特点。
本系列教程讲述在CentOS7系统中k8s集群的部署过程，本节内容重点讲述如何部署master节点。
核心组件介绍
| 名称 | 介绍 |
| kube-apiserver | Kubernetes API，集群的统一稿，各组件协调者，以 HTTP API 提供接口服务，所有对象资源的增删改查和监听操作都交给APIServer处理后再提交给etcd存储。 |
| kube-controller-manager | 处理集群中常规后台任务，一个资源对应一个控制器，而ControllerManager就是负责管理这些控制器的。 |
| kube-scheduler | 根据调度算法为新创建的Pod选择一个Node节点。 |
配置Mater环境
0x01 安装应用包
在Master上安装以下应用包：
|
1 2 3 4 |
# yum -y install ebtables ethtool yum -y install docker-ce kubelet kubeadm kubectl systemctl enable docker kubelet systemctl start docker |
0x02 检查cgroup
以下三个方法中任选一个，不管用systemd还是cgroupfs，统一即可。
方法1
在/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf加入参数：
|
1 |
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs" |
方法2(推荐)
在/usr/lib/systemd/system/docker.service的ExecStart=/usr/bin/dockerd行末添加参数：
|
1 |
--exec-opt native.cgroupdriver=systemd |
方法3
编辑文件/etc/docker/daemon.json：
|
1 |
"exec-opts": ["native.cgroupdriver=systemd"] |
统一cgroup后重启服务：
|
1 2 |
systemctl daemon-reload systemctl restart docker |
配置Master节点
1x01 配置高可用
安装haproxy和keepalived：
|
1 |
yum install -y haproxy keepalived |
修改haproxy配置/etc/haproxy/haproxy.cfg：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
# /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global retries 1 timeout http-request 10s timeout queue 20s timeout connect 5s timeout client 20s timeout server 20s timeout http-keep-alive 10s timeout check 10s listen admin_stats mode http bind 0.0.0.0:1080 log 127.0.0.1 local0 err stats refresh 30s stats uri /haproxy-status stats realm Haproxy\ Statistics stats auth admin:admin stats hide-version stats admin if TRUE #--------------------------------------------------------------------- # apiserver frontend which proxys to the masters #--------------------------------------------------------------------- frontend apiserver bind *:8443 mode tcp option tcplog default_backend apiserver #--------------------------------------------------------------------- # round robin balancing for apiserver #--------------------------------------------------------------------- backend apiserver option httpchk GET /healthz http-check expect status 200 mode tcp option ssl-hello-chk balance roundrobin server k8sm01 10.10.200.201:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3 server k8sm02 10.10.200.202:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3 server k8sm03 10.10.200.203:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3 |
编辑守护脚本/etc/keepalived/check_apiserver.sh：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
#!/bin/bash errorExit() { echo "*** $*" 1>&2 exit 1 } curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null || errorExit "Error GET https://localhost:8443/" if ip addr | grep -q 10.10.200.200 then curl --silent --max-time 2 --insecure https://10.10.200.200:8443/ -o /dev/null || errorExit "Error GET https://10.26.25.23:8443/" fi |
|
1 |
chmod +x /etc/keepalived/check_apiserver.sh |
修改keepalived(MASTER)配置/etc/keepalived/keepalived.conf：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
! /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_K8SM } vrrp_script check_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 100 authentication { auth_type PASS auth_pass kubernetes } virtual_ipaddress { 10.10.200.200 } track_script { check_apiserver } } |
修改keepalived(BACKUP)配置/etc/keepalived/keepalived.conf：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
! /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_K8SM } vrrp_script check_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 priority 90 authentication { auth_type PASS auth_pass kubernetes } virtual_ipaddress { 10.10.200.200 } track_script { check_apiserver } } |
启动haproxy和keepalived：
|
1 2 |
systemctl start keepalived haproxy systemctl enable keepalived haproxy |
1x02 导入配置镜像文件
使用“kubeadm config images list”命令查看需要拉取的镜像：
- k8s.gcr.io/kube-apiserver:v1.19.2
- k8s.gcr.io/kube-controller-manager:v1.19.2
- k8s.gcr.io/kube-scheduler:v1.19.2
- k8s.gcr.io/kube-proxy:v1.19.2
- k8s.gcr.io/pause:3.2
- k8s.gcr.io/etcd:3.4.13-0
- k8s.gcr.io/coredns:1.7.0
|
1 2 |
# 开始拉取镜像 # kubeadm config images pull |
因为某些原因，国内无法访问国外的某些资源，这里我们改用阿里云的镜像服务器，使用以下脚本拉取镜像并贴上标签（根据需要拉取的镜像修改）：
|
1 2 3 4 5 6 7 8 9 |
#!/bin/bash images=(kube-apiserver:v1.19.2 kube-controller-manager:v1.19.2 kube-scheduler:v1.19.2 kube-proxy:v1.19.2 pause:3.2 etcd:3.4.13-0 coredns:1.7.0) for image in ${images[@]} do docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${image} docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image} done |
1x03 初始化master节点
切记！仅首master节点需要初始化，备用master节点跳过此步骤：
|
1 2 3 4 |
kubeadm init \ --pod-network-cidr=10.232.0.0/16 \ --service-cidr=10.196.0.0/16 \ --control-plane-endpoint=10.10.200.200:8443 |
如果初始化顺利，则会显示如下日志信息，提示如何添加master/worker节点：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
... mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... You can now join any number of control-plane nodes by copying certificate authorities and service account keys on each node and then running the following as root: kubeadm join 10.10.200.200:8443 --token bf1bni.w11z5kymeaomqddr \ --discovery-token-ca-cert-hash sha256:26e4de3f6e72ed2b143627cf10068c84e094c2fa30c0aa2fee32f53a2b0254f2 \ --control-plane Then you can join any number of worker nodes by running the following on each as root: kubeadm join 10.10.200.200:8443 --token bf1bni.w11z5kymeaomqddr \ --discovery-token-ca-cert-hash sha256:26e4de3f6e72ed2b143627cf10068c84e094c2fa30c0aa2fee32f53a2b0254f2 |
如果 kubeadm init ... 执行失败，必须通过以下命令重置，然后重新操作：
|
1 |
kubeadm reset |
附：kubeadm ini 参数说明
| 参数 | 说明 |
| --apiserver-advertise-address | API Server 将要广播的监听地址 |
| --apiserver-bind-port | API Server 绑定的端口，默认“6443” |
| --apiserver-cert-extra-sans | 可选的额外提供的证书主题别名（SANs）用于指定API Server的服务器证书 |
| --cert-dir | 证书的存储路径，默认“/etc/kubernetes/pki” |
| --config | kubeadm配置文件的路径 |
| --cri-socket | 要连接的 CRI socket 文件，默认“/var/run/dockershim.sock” |
| --dry-run | 只输出将要执行的操作，不应用任何改变 |
| --feature-gates | 键值对的集合，用来控制各种功能的开关，默认“Auditing=false, CoreDNS=true, DynamicKubeletConfig=false” |
| -h, --help | 获取init命令的帮助信息 |
| --ignore-preflight-errors | 忽视检查项错误列表，列表中的每一个检查项如发生错误将被展示输出为警告，而非错误 |
| --kubernetes-version | 为 control plane 选择一个特定的Kubernetes版本，默认“stable-1” |
| --node-name | 指定节点的名称 |
| --pod-network-cidr | 指明pod网络可以使用的IP地址段 |
| --service-cidr | 为service的虚拟IP地址另外指定IP地址段，默认“10.96.0.0/12” |
| --service-dns-domain | 为services另外指定域名，默认“cluster.local” |
| --skip-token-print | 不打印出由 kubeadm init 命令生成的默认令牌 |
| --token | 这个令牌用于建立主从节点间的双向受信链接 |
| --token-ttl | 令牌被自动删除前的可用时长，设置为 '0', 令牌将永不过期，默认“24h0m0s” |
1x04 配置kubectl
备用master节点跳过此步骤。
|
1 2 3 4 5 |
mkdir ~/.kube cp -i /etc/kubernetes/admin.conf ~/.kube/config # sudo chown $(id -u):$(id -g) ~/.kube/config echo "export KUBECONFIG=~/.kube/config" >> ~/.bash_profile source ~/.bash_profile |
1x05 配置网络
备用master节点加入集群前无法操作。
方案1：采用flannel (推荐)
|
1 |
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml |
方案2：采用calico
|
1 2 |
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml |
1x06 验证master
备用master节点加入集群前无法操作。
|
1 |
kubectl get componentstatus |
此时发现controller-manager和scheduler的状态是“Unhealthy”，修改它们的配置文件，去掉配置中的“--port=0”，然后重启kubelet：
|
1 2 3 |
sed -i '/port=0/d' /etc/kubernetes/manifests/kube-scheduler.yaml sed -i '/port=0/d' /etc/kubernetes/manifests/kube-controller-manager.yaml systemctl restart kubelet |
继续检验master健康状态：
|
1 2 3 |
kubectl get nodes kubectl describe node k8sm01 kubectl get pod --all-namespaces |
使Master参与Pod调度
备用master节点加入集群前无法操作。
|
1 2 3 4 5 6 |
# 参与POD负载 kubectl taint nodes --all node-role.kubernetes.io/master- # 不参与POD负载 kubectl taint nodes <node-name> node-role.kubernetes.io/master=:NoSchedule # 不参与POD负载并驱逐Node上已经存在的Pod kubectl taint nodes <node-name> node-role.kubernetes.io/master=:NoExecute |
配置Master集群
2x01 在首master节点操作
在首master节点验证证书有效时间(notBefore为生效时间/notAfter为失效时间)：
|
1 |
for crt in $(find /etc/kubernetes/pki/ -name "*.crt"); do openssl x509 -in $crt -noout -dates; done |
先做免密：
|
1 2 3 4 |
ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys for n in 2 3; do ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8sm0$n; done |
编辑脚本，将master证书拷贝到备用节点：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
#!/bin/bash USER=root BAK_MASTERS="10.10.200.202 10.10.200.203" for host in ${BAK_MASTERS} do scp /etc/kubernetes/pki/ca.crt "${USER}"@$host: scp /etc/kubernetes/pki/ca.key "${USER}"@$host: scp /etc/kubernetes/pki/sa.key "${USER}"@$host: scp /etc/kubernetes/pki/sa.pub "${USER}"@$host: scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host: scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host: scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key scp /etc/kubernetes/admin.conf "${USER}"@$host: done |
2x02 在备用master节点操作
创建证书目录：
|
1 2 |
mkdir -p /etc/kubernetes/pki/etcd mkdir ~/.kube |
依次将首master节点复制过来的证书移至对应的目录中，并使配置生效：
|
1 2 3 4 5 6 7 |
cd ~/ mv admin.conf ~/.kube/config mv etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt mv etcd-ca.key /etc/kubernetes/pki/etcd/ca.key mv ca.* front-proxy-ca.* sa.* /etc/kubernetes/pki/ echo "export KUBECONFIG=~/.kube/config" >> ~/.bash_profile source ~/.bash_profile |
然后执行以下命令加入集群：
|
1 2 3 |
kubeadm join 10.10.200.200:8443 --token bf1bni.w11z5kymeaomqddr \ --discovery-token-ca-cert-hash sha256:26e4de3f6e72ed2b143627cf10068c84e094c2fa30c0aa2fee32f53a2b0254f2 \ --control-plane |
原创文章禁止转载：技术学堂 » Kubernetes集群部署教程二·Master节点
