101 - How to use Terraform to deploy Managed Kubernetes cluster(s)

Implementing RPA, creating CI/CD pipelines, even automating processes through CRMs are now things where companies accept to spend some time to avoid losing more time later. However, in the IT world, this automation spirit took more time to conquer the infrastructure world. Hopefully, since the rise of the Cloud and more generally the art of managing infrastructure through APIs, automation is finally a thing for building infrastructure.

This article is the first piece of a series of 5 dedicated to the “good way” of deploying a Managed Kubernetes cluster thanks to different technologies and methodologies. In this one, we will see how we can use one of the best infrastructure-as-code software to create and deploy a cluster.

First, we will take a look at what Terraform is and how it can be helpful for our use cases. Then we will go directly to the practice, we will see how we can initialize a new terraform project, how to create a cluster and a configuration topology and some modules. And of course we will see how to use all of this to create and delete your cluster on-demand.

What is Terraform ?

Terraform, created by Hashicorp, is an “Infrastructure as Code” (IaC) tool. It allows users to deploy and manage infrastructure with developers' practices thanks to a high-level configuration language called HCL (Hashicorp Configuration Language). 

Infra-as-code allows engineers to treat infrastructure as a full piece of software. Codifying the infrastructure will help us to automate processes and to make them faster and easily reproducible. And of course, to codify infrastructure is one of the key components of the DevOps methodology, furthermore, it also allows infrastructure deployment to be version controllers and to be embedded in pipelines.

In other words, infrastructure as code will help you to improve speed, to have better reliability (avoiding human mistakes and oversights) and to improve the ability to experiment and test new features on-demand.

Also, Terraform is using providers. Providers are basically plugins that allows terraform to use the API of a specific service (e.g. AWS, Azure, Kubernetes..). It is possible to use well-known providers or to write our own. With that in mind, we can say that it is possible to manage every service which provides an API with Terraform.

With such a tool, a user will be able to create configuration files that describe declaratively the desired state of your infrastructure, and will call Terraform to proceed to the creation of this infrastructure through API calls then will generate and keep up-to-date a “state” of the deployed infrastructure.

This is just a basic overview of what terraform is capable of, but because Terraform uses such a simple syntax and thanks to a large number of providers, we can efficiently and safely provision, manage and destroy complex infrastructure and it inherently eases the management of a multi-cloud or hybrid (e.g. cloud/on-premises) environment.

Now that we all want to use Terraform..

Let’s practice !

Before starting working on our topology, let's take a look at the layout of our Git repository (prerequisites in the first article of this series). We will create the base folders and populate them throughout this article and the following ones.

$ mkdir config terraform


├── config
└── terraform


2 directories, 0 files.
        

2 main folders should be present at the root of your repository, config, where manifests that will be deployed through our GitOps pipelines will be stored. And terraform, which will host all our terraform code.

We will start our journey by creating a new folder in the terraform one, called cluster. This folder will be a terraform topology dedicated to the creation of the cluster. In the meantime, we will create all the files that we need.

$ mkdir terraform/cluster

$ cd terraform/cluster

$ touch main.tf outputs.tf provider.tf


├── config
└── terraform
    └── cluster
        ├── main.tf
        ├── outputs.tf
        └── provider.tf

3 directories, 3 files

        

We will start by taking a look at the provider.tf file. In this one, we will define: the minimum required version of terraform (e.g. >= 1.1), the required providers and if you need it, configuration of the remote storage that will host our terraform state.

During all this series, I am going to use Scaleway, young but promising French Public Cloud Provider (feel free to choose your favorite one). To do so, we will just need to go to the providers page of Terraform, and search for a scaleway provider. On this page, you will find some documentation on how to use it, and on every resource that you can create with this provider.

The backend block here is for the terraform state remote storage. All the information that are needed will be given to you by your Cloud provider when you will create an object storage. Of course, if you want to keep your terraform state locally you can get rid of this backend block !

Before going on, we can make sure that our configuration works by running a terraform init command that will initialize terraform in the working directory. If you chose to use a backend storage for the state, you may need to do some actions to permit terraform to access the back-end (e.g. in this case, exporting some AWS variables).

$ export AWS_ACCESS_KEY_ID=xxxx

$ export AWS_SECRET_ACCESS_KEY=xxxx

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding scaleway/scaleway versions matching "~> 2.2.1"...
- Installing scaleway/scaleway v2.2.3...
- Installed scaleway/scaleway v2.2.3 (signed by a HashiCorp partner, key ID F5BF26CADF6F9614)

...

Terraform has been successfully initialized!

...
        

Now, we can start working on the main.tf file, this file will host the code for creating a Managed Kubernetes Cluster on Scaleway (Kapsule), to do so, we will make a call to a module (small terraform topologies that allow resources to be re-used easily and allow us to use kind of constructors). Let's take a look at how are we calling the module:

In this block, we can see at first the source variable, which is the path (can be a local absolute path or an url) where terraform will be able to download the module during the initialization phase. Then, we can see there is a bunch of configurations, the name of the cluster, the region, some tags, and even the workers node pools configuration. All these configurations are available in the module that we are calling. Of course, the configuration can be different based on the module author and the targeted provider.

Let's now take a look at the outputs.tf file. This file is a list of links to variables that will be available once the topology will be applied.

As we can notice, all the outputs are available through the module that we called. You can see that some of them are defined as sensitive. This is to make sure that the data will never be stored or output unencrypted. We will see soon why this can be useful to output data that we don't want to see.

Since we add an new external resource (the module) we will need to run the init one more time. Then we can run some compliance commands (validate, fmt) that will validate our configuration and format the code if needed. (If you want to be more efficient on running all these commands, think about using a Makefile).

$ terraform init

Initializing modules...
Downloading git::ssh://git@github.com/vmarlier/terraform-modules.git?ref=main for kapsule...
- kapsule in .terraform/modules/kapsule/kapsule

...

Terraform has been successfully initialized!

...

$ terraform validate

Success! The configuration is valid.

$ terraform fmt

        

Now, let's try a plan (a preview of the changes that will be brought to the infrastructure). Also, don't forget that you will maybe need to configure your provider to be able to interact with the API (for scaleway, I can export some variables):

$ export SCW_ACCESS_KEY=xxxx

$ export SCW_SECRET_KEY=xxxx
 
$ terraform plan

Terraform will perform the following actions:

  # module.kapsule.scaleway_k8s_cluster.cluster will be created
  + resource "scaleway_k8s_cluster" "cluster" {
      ...
      + name                        = "fr-production"
      + type                        = "kapsule"
      + version                     = "1.22.9"
      ...
    }

  # module.kapsule.scaleway_k8s_pool.node_pool["default"] will be created
  + resource "scaleway_k8s_pool" "node_pool" {
      ...
      + name                = "default"
      + node_type           = "DEV1-M"
      ...
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cluster_ca_certificate = (sensitive value)
  + cluster_host           = (sensitive value)
  + cluster_id             = (known after apply)
  + cluster_kubeconfig     = (sensitive value)
  + cluster_token          = (sensitive value)
        

The code seems to be working as expected. It is time to apply the topology.

Let's create the cluster !

In order to deploy your stack, you will just need to run the apply command, it will create a new plan and prompt a message to ask if you want to deploy your changes, let's type "yes".

$ terraform apply

...

Plan: 2 to add, 0 to change, 0 to destroy.

...

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.kapsule.scaleway_k8s_cluster.cluster: Creating...
module.kapsule.scaleway_k8s_cluster.cluster: Creation complete after 6s [id=fr-par/eccd3532-75b7-4ef2-9a7a-adaa854d4175]
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Creating...
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Still creating... [10s elapsed]
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Still creating... [20s elapsed]
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Still creating... [30s elapsed]
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Still creating... [40s elapsed]
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Still creating... [50s elapsed]
module.kapsule.scaleway_k8s_pool.node_pool["default"]: Creation complete after 56s [id=fr-par/d807679c-5aa4-44fc-a71b-341ca690567a]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

cluster_ca_certificate = <sensitive>
cluster_host = <sensitive>
cluster_id = "fr-par/eccd3532-75b7-4ef2-9a7a-adaa854d4175"
cluster_kubeconfig = <sensitive>
cluster_token = <sensitive>
        

If we keep an eye on the command line during the apply, you will see some logs keeping us up-to-date with the process. In the example above, we can see that the creation of the cluster took 6s and the node pool has been created in less than a minute ! (These performances will depend on many factors and mostly on the Cloud provider).

We can also notice the Outputs we defined in the outputs.tf file are diplayed here but the one with a sensitive attribute do not show the data.

If we want to make sure that our cluster has been properly created, we can take a look at the Cloud Provider UI:

Seems ok ! Now, let's try to get a kubeconfig and run a basic command on the cluster. The way of getting a Kubeconfig depends once again on your Cloud Provider. I suggest to create a Makefile with a cli command.

$ make get-kubeconfig

Enter the desired cluser: eccd3532-75b7-4ef2-9a7a-adaa854d4175
Enter the scaleway profile: miniature-fiesta

$ export KUBECONFIG=~/.kube/scaleway_fr-production

$ kubectl cluster-info

Kubernetes control plane is running at https://eccd3532-75b7-4ef2-9a7a-adaa854d4175.api.k8s.fr-par.scw.cloud:6443
CoreDNS is running at https://eccd3532-75b7-4ef2-9a7a-adaa854d4175.api.k8s.fr-par.scw.cloud:6443/api/v1/namespaces/kube-system/services/coredns:dns/proxy

$ kubectl get pods -n kube-system

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6f9fc8d66d-h7ddh   1/1     Running   0          13m
calico-node-mzpn5                          1/1     Running   0          9m
coredns-56bf5f7789-th2sl                   1/1     Running   0          13m
csi-node-blrfl                             2/2     Running   0          9m
konnectivity-agent-sm2n6                   1/1     Running   0          9m
kube-proxy-hcf7v                           1/1     Running   0          9m
metrics-server-74cf4b5cf9-kx8vx            1/1     Running   0          13m
node-problem-detector-hqhw8                1/1     Running   0          9m
        

Everything seems to be working as expected. Of course, don't forget that you can destroy, update and recreate the cluster at any moment.

Conclusion

During this first part of the series, we saw that it is very easy to create a cluster with terraform and to make sure that it is standardized thanks to a Terraform Module. Now that we are able to create a cluster and to interact with it, we can go ahead and start to implement our GitOps solution.

References:

My Terraform Modules

The Miniature Fiesta Repository