Kubeadm默认在由master节点上的kubelet管理的静态pod中运行单个成员etcd集群。这不是高可用性设置,因为etcd集群只包含一个成员,并且无法维持任何成员变得不可用。

@[TOC]

本节将介绍创建由三个成员组成的高可用性etcd集群的过程,该集群可在使用kubeadm设置kubernetes集群时用作外部etcd。

Creating Highly Available Clusters with kubeadm

使用kubeadm设置高可用性Kubernetes集群主要是创建高可用的ETCD群集,有两种不同方法:

  1. etcd 与 master节点一起部署。只需较少的主机,etcd与master部署在同一位置。
  2. etcd 单独部署,使用外部独立的etcd集群。需要较多的主机,master与etcd分开部署。

etcd 与master 一起部署

第一台master节点主机上执行配置

  1. 创建 kubeadm-config.yaml 模版文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
clusterName: kubernetes
etcd:
local:
extraArgs:
listen-client-urls: "https://127.0.0.1:2379,https://CP0_IP:2379"
advertise-client-urls: "https://CP0_IP:2379"
listen-peer-urls: "https://CP0_IP:2380"
initial-advertise-peer-urls: "https://CP0_IP:2380"
initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380"
serverCertSANs:
- CP0_HOSTNAME
- CP0_IP
peerCertSANs:
- CP0_HOSTNAME
- CP0_IP

以上模版文件仅显示配置了etcd配置,其他配置没有显示

  1. 替换 kubernetesVersion: v1.11.x 中的x最近可用的版本,如kubernetesVersion: v1.11.3
  2. 替换模版中的变量

    • CP0_HOSTNAME
    • CP0_IP
  3. 运行初始化

    kubeadm 过程中可能会连接外网gcr.io查询当前稳定版本,由于无法连接gcr.io导致执行失败。
    添加代理export http_proxy=http://IP:PORT ,执行完以后取消代理 unset http_proxy

kubeadm init --config kubeadm-config.yaml

复制第一台生成的证书及配置文件到其他master主机相同位置

  • /etc/kubernetes/pki/ca.crt
  • /etc/kubernetes/pki/ca.key
  • /etc/kubernetes/pki/sa.key
  • /etc/kubernetes/pki/sa.pub
  • /etc/kubernetes/pki/front-proxy-ca.crt
  • /etc/kubernetes/pki/front-proxy-ca.key
  • /etc/kubernetes/pki/etcd/ca.crt
  • /etc/kubernetes/pki/etcd/ca.key
  • /etc/kubernetes/admin.conf

第二台master节点主机上执行配置

  1. 创建 kubeadm-config.yaml 模版文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
clusterName: kubernetes
etcd:
local:
extraArgs:
listen-client-urls: "https://127.0.0.1:2379,https://CP1_IP:2379"
advertise-client-urls: "https://CP1_IP:2379"
listen-peer-urls: "https://CP1_IP:2380"
initial-advertise-peer-urls: "https://CP1_IP:2380"
initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380,CP1_HOSTNAME=https://CP1_IP:2380"
initial-cluster-state: existing
serverCertSANs:
- CP1_HOSTNAME
- CP1_IP
peerCertSANs:
- CP1_HOSTNAME
- CP1_IP

以上模版文件仅显示配置了etcd配置,其他配置没有显示

  1. 替换 kubernetesVersion: v1.11.x 中的x最近可用的版本,如kubernetesVersion: v1.11.3
  2. 替换模版中的变量

    • CP0_HOSTNAME
    • CP0_IP
    • CP1_HOSTNAME
    • CP1_IP
  3. 确认复制第一台主机目录 /etc/kubernetes/pki/ 及文件 /etc/kubernetes/admin.conf 到本主机相同位置

  4. 运行kubeadm phase 命令引导kubelet:

1
2
3
4
5
kubeadm alpha phase certs all --config kubeadm-config.yaml
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml
kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml
kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml
systemctl start kubelet
  1. 运行命令,添加节点到etcd集群

    按实际修改以下IP 和 HOSTNAME

1
2
3
4
5
6
7
8
export CP0_IP=10.0.0.7
export CP0_HOSTNAME=cp0
export CP1_IP=10.0.0.8
export CP1_HOSTNAME=cp1

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-config.yaml

在将节点添加到正在运行的集群之后,以及在将新节点加入etcd集群之前,此命令会导致etcd集群在短时间内不可用

  1. 部署其他控制组件并将节点标记为master主节点:
1
2
3
kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml
kubeadm alpha phase controlplane all --config kubeadm-config.yaml
kubeadm alpha phase mark-master --config kubeadm-config.yaml

第三台master节点主机上执行配置

  1. 创建 kubeadm-config.yaml 模版文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.x
clusterName: kubernetes
etcd:
local:
extraArgs:
listen-client-urls: "https://127.0.0.1:2379,https://CP2_IP:2379"
advertise-client-urls: "https://CP2_IP:2379"
listen-peer-urls: "https://CP2_IP:2380"
initial-advertise-peer-urls: "https://CP2_IP:2380"
initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380,CP1_HOSTNAME=https://CP1_IP:2380,CP2_HOSTNAME=https://CP2_IP:2380"
initial-cluster-state: existing
serverCertSANs:
- CP2_HOSTNAME
- CP2_IP
peerCertSANs:
- CP2_HOSTNAME
- CP2_IP

以上模版文件仅显示配置了etcd配置,其他配置没有显示

  1. 替换 kubernetesVersion: v1.11.x 中的x最近可用的版本,如kubernetesVersion: v1.11.3
  2. 替换模版中的变量
  • CP0_HOSTNAME
  • CP0_IP
  • CP1_HOSTNAME
  • CP1_IP
  • CP2_HOSTNAME
  • CP2_IP
  1. 确认复制第一台master主机目录 /etc/kubernetes/pki/ 及文件 /etc/kubernetes/admin.conf 到本主机相同位置

  2. 运行kubeadm phase 命令引导kubelet:

1
2
3
4
5
kubeadm alpha phase certs all --config kubeadm-config.yaml
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml
kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml
kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml
systemctl start kubelet
  1. 运行命令,添加节点到etcd集群

    按实际修改以下IP 和 HOSTNAME

1
2
3
4
5
6
7
8
export CP0_IP=10.0.0.7
export CP0_HOSTNAME=cp0
export CP2_IP=10.0.0.8
export CP2_HOSTNAME=cp1

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP2_HOSTNAME} https://${CP2_IP}:2380
kubeadm alpha phase etcd local --config kubeadm-config.yaml

在将节点添加到正在运行的集群之后,以及在将新节点加入etcd集群之前,此命令会导致etcd集群在短时间内不可用

  1. 部署其他控制组件并将节点标记为master主节点:
1
2
3
kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml
kubeadm alpha phase controlplane all --config kubeadm-config.yaml
kubeadm alpha phase mark-master --config kubeadm-config.yaml

整个集群部署完成。

使用 kubeadm 部署单独的 etcd 集群,并使用kubelet管理

Set up a high availability etcd cluster with kubeadm

准备工作:

  1. 准备三个单独的主机,不能部署kubernetes ,仅部署 etcd
  2. 三个主机可以通过端口2379和2380相互通信。本文档假定这些默认端口。但是,它们可以通过kubeadm配置文件进行配置。
  3. 每个主机必须安装dockerkubeletkubeadm
  4. 一些在主机之间复制文件的基础结构 例如ssh,scp 可以满足这个要求

设置群集

一般方法是在一个节点上生成所有证书,并仅将必要的文件分发给其他节点。

1. 将kubelet配置为etcd的服务管理器。

(每台主机上执行)
运行etcd比运行kubernetes更简单,因此您必须通过创建一个具有更高优先级的新文件来覆盖kubeadm提供的kubelet单元文件。

cgroup-driver=systemd 要与 docker info中的Cgroup Driver 相匹配。

1
2
3
4
5
6
7
8
9
10
cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_CGROUP_ARGS --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true
Restart=always
EOF

systemctl daemon-reload
systemctl restart kubelet

2. 为kubeadm创建配置文件。

(仅在HOST0主机上执行)
使用以下脚本为每个将在其上运行etcd成员的主机生成一个kubeadm配置文件。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
export HOST0=10.0.0.6
export HOST1=10.0.0.7
export HOST2=10.0.0.8

# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("infra0" "infra1" "infra2")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1alpha2"
kind: MasterConfiguration
etcd:
local:
serverCertSANs:
- "${HOST}"
peerCertSANs:
- "${HOST}"
extraArgs:
initial-cluster: infra0=https://${ETCDHOSTS[0]}:2380,infra1=https://${ETCDHOSTS[1]}:2380,infra2=https://${ETCDHOSTS[2]}:2380
initial-cluster-state: new
name: ${NAME}
listen-peer-urls: https://${HOST}:2380
listen-client-urls: https://${HOST}:2379
advertise-client-urls: https://${HOST}:2379
initial-advertise-peer-urls: https://${HOST}:2380
EOF
done

3. 生成证书颁发机构

如果您已有CA,那么唯一的操作是将CA crt和 key文件复制到/etc/kubernetes/pki/etcd/ca.crt/etc/kubernetes/pki/etcd/ca.key。复制这些文件后,继续执行下一步“为每个成员创建证书”。

如果您还没有CA,则在$HOST0(生成kubeadm的配置文件的位置)上运行此命令。

kubeadm alpha phase certs etcd-ca

这会创建两个文件

  • /etc/kubernetes/pki/etcd/ca.crt
  • /etc/kubernetes/pki/etcd/ca.key

4. 为每个成员创建证书

(仅在HOST0主机上执行)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
# cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0

# clean up certs that should not be copied off this host
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete

5. 复制证书和kubeadm配置

已生成证书,现在必须将它们移动到各自的主机。
kubeadmcfg.yamlpki 目录拷贝到对应的主机上/etc/kubernetes/pki

1
2
3
4
5
6
7
USER=ubuntu
HOST=${HOST1}
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
ssh ${USER}@${HOST}
USER@HOST $ sudo -Es
root@HOST $ chown -R root:root pki
root@HOST $ mv pki /etc/kubernetes/

6. 确保存在所有预期文件

所需文件的完整列表$HOST0是:

7. 创建静态pod清单

现在证书和配置已到位,是时候创建清单了。在每个主机上运行kubeadm命令以生成etcd的静态清单。

1
2
3
root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
root@HOST1 $ kubeadm alpha phase etcd local --config=/tmp/${HOST1}/kubeadmcfg.yaml
root@HOST2 $ kubeadm alpha phase etcd local --config=/tmp/${HOST2}/kubeadmcfg.yaml

查看是否生成文件 /etc/kubernetes/manifests/etcd.yaml .

8. 过几分钟,查看 etcd pod 的状态

1
2
3
# docker ps | grep etcd
e29577f035f6 b8df3b177be2 "etcd --advertise-..." 24 seconds ago Up 23 seconds k8s_etcd_etcd-test01_kube-system_61e08b291cb3ea4992e74c5a47f0b8e0_12
c9312e7f67af k8s.gcr.io/pause:3.1 "/pause" 24 minutes ago Up 24 minutes k8s_POD_etcd-test01_kube-system_61e08b291cb3ea4992e74c5a47f0b8e0_0

有两个容器正在运行,如果 pod k8s_etcd_etcd启动失败,使用 docker logs -f k8s_etcd_etcd.. 查看启动错误日志。

9. 检查群集运行状况

1
2
3
4
5
6
7
8
9
10
docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.18 etcdctl \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST0}:2379 cluster-health

member 327e88294e54db55 is healthy: got healthy result from https://++:2379
cluster is healthy

使用 kubeadm 生成etcd证书,,并使用systemctl管理

使用 systemctl管理 etcd ,不占用 kubelet,可以在主机上同时作为kubernetes 节点。

1. 使用以下脚本为每个将在其上运行etcd成员的主机生成一个etcd.service服务启动文件。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
export HOST0=10.0.0.6
export HOST1=10.0.0.7
export HOST2=10.0.0.8

# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("infra0" "infra1" "infra2")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\
--name=${NAME} \\
--cert-file=/etc/kubernetes/pki/etcd/server.crt \\
--key-file=/etc/kubernetes/pki/etcd/server.key \\
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \\
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key \\
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \\
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \\
--initial-advertise-peer-urls=https://${HOST}:2380 \\
--listen-peer-urls=https://${HOST}:2380 \\
--listen-client-urls=https://${HOST}:2379,http://127.0.0.1:2379 \\
--advertise-client-urls=https://${HOST}:2379 \\
--initial-cluster=infra0=https://${ETCDHOSTS[0]}:2380,infra1=https://${ETCDHOSTS[1]}:2380,infra2=https://${ETCDHOSTS[2]}:2380 \\
--initial-cluster-state=new \\
--client-cert-auth=true \\
--peer-client-cert-auth=true \\
--snapshot-count=10000 \\
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
done
1
把三个 etcd.service 文件复制到对应主机的 /etc/systemd/system/ 目录下

2. 使用 kubeadm 生成etcd证书

仅生成一套证书,供etcd集群节点公用、

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

export HOST0=10.0.0.6
export HOST1=10.0.0.7
export HOST2=10.0.0.8

cat << EOF > /tmp/kubeadm-etcd-cfg.yaml
apiVersion: "kubeadm.k8s.io/v1alpha2"
kind: MasterConfiguration
etcd:
local:
serverCertSANs:
- "127.0.0.1"
- "${HOST0}"
- "${HOST1}"
- "${HOST2}"
peerCertSANs:
- "127.0.0.1"
- "${HOST0}"
- "${HOST1}"
- "${HOST2}"
EOF

创建根证书

kubeadm alpha phase certs etcd-ca

这会创建两个文件

  1. /etc/kubernetes/pki/etcd/ca.crt
  2. /etc/kubernetes/pki/etcd/ca.key

创建证书

1
2
3
kubeadm alpha phase certs etcd-server --config=/tmp/kubeadm-etcd-cfg.yaml
kubeadm alpha phase certs etcd-peer --config=/tmp/kubeadm-etcd-cfg.yaml
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/kubeadm-etcd-cfg.yaml

复制证书到其他两台主机对应目录

1
2
3
4
5
6
7
8
9
10
11
12
pki/
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
└── etcd
├── ca.crt
├── ca.key
├── peer.crt
├── peer.key
├── server.crt
└── server.key

1 directory, 8 files

3. 启动etcd服务,验证服务

启动 etcd 服务

1
2
3
4
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

验证服务

1
2
3
4
5
6
etcdctl \
--endpoints=https://${NODE_IP}:2379 \
--ca-file=/etc/kubernetes/pki/etcd/ca.crt \
--cert-file=/etc/kubernetes/pki/etcd/server.crt \
--key-file=/etc/kubernetes/pki/etcd/server.key \
cluster-health

Kubernetes 配置外部 etcd

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: kubeadm.k8s.io/v1alpha2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
etcd:
external:
endpoints:
- https://etcd_HOST0:2379
- https://etcd_HOST1:2379
- https://etcd_HOST2:2379
caFile: /etc/kubernetes/pki/etcd/ca.crt
certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key