Kubernetes Control Plane and ETCD

Kubernetes control planes are a critical component of both on-premises and cloud-hosted clusters. They hold all of the cluster's metadata and provide a consistent view of cluster state, so understanding how they work together is essential for building highly available services.

A control plane hosts several services, including the API server, DNS, and ETCD. ETCD is particularly important because it's what allows control planes to maintain cluster consistency during failovers, through quorum-based operations. That's also why it's recommended to run an odd number of control planes; each with its own ETCD instance.

Still, this setup can create complications in certain failure scenarios, for example, when more than half of the control planes become unreachable, or when multiple control planes crash at once (say, 2 out of 3).

Listing ETCD Members

To see the current members of the ETCD cluster, run the following from a healthy control plane:

READY_ETCD_MEMBER="etcd-kubemaster1"
ETCDCTL="ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379"
kubectl -n kube-system exec -it "$READY_ETCD_MEMBER" -- sh -c "$ETCDCTL member list"

Replacing a Crashed Control Plane

Once you've identified the ID of the ETCD member that belongs to the crashed control plane, remove it so a replacement can be added cleanly:

READY_ETCD_MEMBER="etcd-kubemaster1"
ETCD_ID_TO_REMOVE="729aac77d384f8bd"
ETCDCTL="ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt --endpoints=https://127.0.0.1:2379"
kubectl -n kube-system exec -it "$READY_ETCD_MEMBER" -- sh -c "$ETCDCTL member remove $ETCD_ID_TO_REMOVE"