February 01, 2018
These days I’m helping koko to build their Kubernetes infrastructure on AWS. Initially, we used kops to deploy the cluster but weren’t very happy. I’ve researched various option and blogged about them earlier.
tl;dr: There aren’t that many options that are secure, highly available and don’t require buying into a vendor ecosystem.
kubeadm is the official vendor independent solution to installer Kubernetes. While it lacks automated setup of multi master clusters, it’s possible.
To have a reliable and reproducable way to build clusters, we decided to create Cloudformation templates to fully automate this setup.
All this is opend sourced and available on https://github.com/itskoko/kubecfn
The cluster consists of multiple components required for the cluster.
The first step to create a new cluster is generating the TLS keys for etcd and kubeadm. These credentials get uploaded to S3 and the instances get a IAM instance profile with a policy allowing it to download them.
The controller instances are managed by an autoscaling group.
On initial cluster creation (
make create-cluster), the ClusterState flag is
new, which enables etcd bootstrapping. To find the other etcd peers,
we’re using SRV record based DNS
This option give us one way to bootstap, as well as dynamically add and remove
members. On the other hand, the ‘static’ and ‘etcd’ discovery can be only used
for initial cluster configuration.
Unfortunately AWS can’t automatically add DNS records for instances in an autoscaling group. To solve this, we’re using a lambda to update a route53 zone (see next section).
Once etcd is up,
kubeadm init creates the manifests and keys for the
controller components but use the pre-generated keys for the CA generated in the
first step. This makes sure all controller keys are signed with this trusted CA.
An ELB in front of the apiservers provide a stable endpoint for both the workers as well as users of the cluster.
The cluster creates it’s own Route53 zone, which needs to be a subdomain of the
ParentZoneID parameter. Beside a record pointing to the apiserver ELB,
this zone is updated by a
Each time a controller instances get added or removed by the autoscaling group, the lambda gets triggered and adds/removes DNS records for etcd discovery.
The worker setup is straight forward. It’s also managed by an autoscaling group. The ignition config downloads credentials for the kubelet from S3
kubecfn wasn’t build by a big team nor as an end in itself. We focused on making it easy to maintain, reliable and fail gracefully. That means there are some rough edges around automation. The security could be improved further and I’m not very happy with the whole concept of IAM instance profiles which effectively grants these privileges to every process running on a host. We mitigate this by installing kube2iam but this isn’t a robust as something like this should be. Unfortunately it’s current best practice and there isn’t an easy way to improve. Let me know if you know a better way.
There is also the nasy
of patching the kube-proxy configmap after
kubeadm init which requires
That being said, thanks to CloudFormation with primitives like WaitOnResourceSignals and features like Change Sets, I feel more comfortable operating this cluster than I was with any other installer I used in the past.