A few days ago I wrote a walkthrough of setting up Azure Container Service (AKS) in Norwegian. Someone asked me for an English version of that, and here it is.

Kubernetes(K8s) is becoming the de-facto standard for deploying container-based applications and workloads. Microsoft is currently in preview of their managed Kubernetes offering (Azure Kubernetes Service, AKS) which makes it easy to create a Kubernetes cluster and deploy workloads without the skill and time required to manage day-to-day operations of a Kubernetes-cluster, which today can be complex and time consuming.

In this post we will set up a Kubernetes cluster from scratch using Azure CLI.

Table of contents

• Background
  • Docker containers
  • Container orchestration
• Getting started with Azure Kubernetes - AKS
  • Caveats
  • Preparations
  • Azure login
  • Activate ContainerService
  • Create a resource group
  • Create a Kubernetes cluster
  • Install kubectl
  • Inspect cluster
  • Start some nginx containere
  • Making nginx available with a service
  • Scale cluster
  • Delete cluster
• Bonus material
  • Deploying services with Helm
    • Deploy MineCraft with Helm
  • Kubernetes Dashboard
• Conclusion

Microsoft Azure

If you don’t have a Azure subscription already you can try services for $200 for 30 days. The VM size Standard_B2s is Burstable, has 2vCPU, 4GB RAM, 8GB temp storage and costs roughly $38 / month. For $200 you can have a cluster of 3-4 B2s nodes plus traffic, loadbalancers and other additional costs.

We have no affiliation with Microsoft Azure except their sponsorship of our startup DataDynamics with cloud services for 24 months in their BizSpark program.

Background

Docker containers

We will not do a deep dive on Docker containers in this post, but here is a summary for those who are not familiar with it.

Docker is a way to package software so that it can run on the most popular platforms without worrying about installation, dependencies and to a certain degree, configuration.

In addition, a Docker container uses the operating system of the host machine when it runs. Because of this it’s possible to run many more containers on the same host machine compared to running virtual machines.

Here is a incomplete and rough comparison between a Docker container and a virtual machine:

PS The numbers for virtual machines is taken from memory. I tried starting a MySQL virtual appliance on my laptop but VMware Player refuses to run because of Windows Hyper-V incompatibility. VMware Workstation refuses to run because of license issues and Oracle VirtualBox repeatedly gives me a nasty bluescreen. Hooray!

Protip The smallest and fastest Docker images are built on Alpine Linux. For the webserver Nginx the Alpine-based image is 15MB compared to 108MB for the normal Debian-based image. PostgreSQL:Alpine is 38MB compared to 287MB with “full” OS. Last version of MySQL is 343MB but will in version 8 support Alpine Linux as well.

To recap, some of the advantages of Docker containers are:

• Compatibility across platforms, Linux, Windows, MacOS.
• 10-100x smaller size. Faster to download, build and upload.
• Memory usage only for application and not base OS.
  • Advantage when developing. Ability to run 10-20-30 containers on a development laptop.
  • Advantage in production. Can reduce hardware/cloud costs considerably.
• Near instant startup. Makes dynamic scaling of applications easier.

Download Docker for Windows here.

To start a MySQL database container from Windows CMD or Powershell:

1

docker run --name mysql -p 3306:3306 -e MYSQL_RANDOM_ROOT_PASSWORD=true mysql

Stop the container with:

1

docker kill mysql

You can search for already built Docker images on Docker Hub. It’s also possible to create private Docker repositories for your own software that you don’t want to be publicly available.

Container orchestration

Now that Docker container images has become the preferred way to package and distribute software on the Linux platform, there has emerged a need for systems to coordinate running and deploying these containers. Similar to the ecosystem of products VMware has built up around development and operation of virtual machines.

Container orchestration systems have the responsibility for:

• Load balancing.
• Service discovery.
• Health checks.
• Automatic scaling and restarting of host nodes and containers.
• Zero downtime upgrades (rolling deploys).

Until recently the ecosystem around container orchestration has been fragmented, and the most popular alternatives have been:

• Kubernetes (Originaly from Google, now managed by CNCF, the Cloud Native Computing Foundation)
• Swarm (From the maker of Docker)
• Mesos (From Apache Software Foundation)
• Fleet (From CoreOS)

But the last year there has been a convergence towards Kubernetes as the preferred solution.

• 7 February
  • CoreOS announces that they are removing Fleet from Container Linux and recommends Kubernetes
• 27 July
  • Microsoft joins the CNCF
• 9 August
  • Amazon Web Services join the CNCF
• 29 August
  • VMware and Pivotal joins the CNCF
• 17 September
  • Oracle joins the CNCF
• 17 October
  • Docker announces native support for Kubernetes in addition to it’s own Swarm product
• 24 October
  • Microsoft Azure announces the managed Kubernetes service AKS
• 29 November
  • Amazon Web Services announces the managed Kubernetes service EKS

Especially the last two news items are important. Deploying and running your own Kubernetes-installation requires time and skills (Read how Stripe used 5 months to trust running Kubernetes in production, just for batch jobs.)

Until now the choice has been running your own Kubernetes cluster or using Google Container Engine which has been using Kubernetes since 2014. Many of us feel a certain discomfort by locking ourselves to one provider. But this is now changing when you can develop infrastructure on Kubernetes and choose between the 3 large cloud providers in addition to running your own cluster if wanted. *

* Kubernetes is a fast moving project, and features might be available on the different platforms on different timelines.

Getting started with Azure Kubernetes - AKS

Caveats

This guide is based on the documentation on Microsoft.com. Setting up a Azure Kubernetes cluster did not work in the beginning of December, but today, 23. December, it seems to work fairly well. But, upgrading the cluster from Kubernetes 1.7 to 1.8 for example does NOT work.

AKS is in Preview and Azure are working continuously to make AKS stable and to support as many Kubernetes-features as possible. Amazon Web Services has a similar closed invite-only Preview currently while working on stability and features.

Both Azure and AWS expresses expectations about their Kubernetes offerings will be ready for production in 2018.

Preparations

You need Azure-CLI (version 2.0.21 or newer) to execute the az commands:

• Download Azure-CLI here
• Information about Azure-CLI on MacOS and Linux here

All commands executed in Windows PowerShell.

Azure login

Log on to Azure:

1

az login

You will get a link to open in your browser together with an authentication code. Enter the code on the webpage and az login will save the login information so that you will not have to authenticate again on the same machine.

PS The login information gets saved in C:\Users\Username\.azure\. You have to make sure nobody can access these files. They will then have full access to your Azure account.

Activate ContainerService

Since AKS is in Preview/Beta, you explicitly have to activate it in your subscription to get access to the aks subcommands.

1
2

az provider register -n Microsoft.ContainerService
az provider show -n Microsoft.ContainerService

Create a resource group

Here we create a resource group named “my_aks_rg” in Azure region West Europe.

1

az group create --name my_aks_rg --location westeurope

Protip To see a list of all available Azure regions, use the command az account list-locations --output table. PS AKS might not be available in all regions yet!

Create Kubernetes cluster

1

az aks create --resource-group my_aks_rg --name my_cluster --node-count 3 --generate-ssh-keys --node-vm-size Standard_B2s --node-osdisk-size 128 --kubernetes-version 1.8.2

• --node-count
  • Number of agent(host) nodes available to run containers
• --generate-ssh-keys
  • Creates and prints a SSH key which can be used for SSHing directly to the agent nodes.
• --node-vm-size
  • Which size Azure VMs the agent nodes should be created as. To see available sizes use az vm list-sizes -l westeurope --output table and Microsofts webpages.
• --node-osdisk-size
  • Disk size of the agent nodes in GB. PS Containers can be stopped and moved to another host if Kubernetes finds it necessary or if a agent node disappears. All data saved locally in the container will be gone. If saving data permanently use Kubernetes PersistentVolumes and not the local agent node or container disks.
• --kubernetes-version
  • Which Kubernetes version to install. Azure does NOT necessarily install the last version by default, and currently upgrading with az aks upgrade does not work. Latest version available right now is 1.8.2. It’s recommended to use the latest available version since there is a lot of changes from version to version. The documentation is also much better for newer versions.

Save the output of the command in a file in a secure location. It contains keys that can be used to connect to the cluster with SSH. Even though that should not in theory be necessary.

Install kubectl

kubectl is the client which performs all operations against your Kubernetes cluster. Azure CLI can install kubectl for you:

1

az aks install-cli

After kubectl is installed we need to get login information so that kubectl can communicate with the Kubernetes cluster.

1

az aks get-credentials --resource-group my_aks_rg --name my_cluster

The login information is saved in C:\Users\Username\.kube\config. Keep these files secure as well.

Protip When you have several Kubernetes clusters you can change which one kubectl talks to with kubectl config get-contexts and kubectl config set-context my_cluster.

Inspect cluster

To check that the cluster and kubectl works we start with a couple of commands.

See all agent nodes and status:

1
2
3
4
5

> kubectl get nodes
NAME                       STATUS    AGE       VERSION
aks-nodepool1-16970026-0   Ready     15m       v1.8.2
aks-nodepool1-16970026-1   Ready     15m       v1.8.2
aks-nodepool1-16970026-2   Ready     15m       v1.8.2

See all services, pods and deployments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

> kubectl get all --all-namespaces

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   po/kubernetes-dashboard-6fc8cf9586-frpkn      1/1       Running   0          3d

NAMESPACE     NAME                          CLUSTER-IP     EXTERNAL-IP     PORT(S)           AGE
kube-system   svc/kubernetes-dashboard      10.0.161.132   <none>          80/TCP            3d

NAMESPACE     NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deploy/kubernetes-dashboard      1         1         1            1           3d

NAMESPACE     NAME                                    DESIRED   CURRENT   READY     AGE
kube-system   rs/kubernetes-dashboard-6fc8cf9586      1         1         1         3d

This is just some of the output from this command. You do not have to know what the resources in the kube-system namespace does. That is part of the intention when Microsoft is managing our cluster for us.

Namespaces In Kubernetes there is something called Namespaces. Resources in one namespace does not have automatic access to resources in another namespace. The services that runs Kubernetes itself use the namespace kube-system. The kubectl command by default only shows you resources in the default namespace, unless you specify --all-namespaces or --namespace=xx.

Start some nginx containers

An instance of a running container in Kubernetes is called a Pod.

nginx is a fast and flexible web server.

Now that the clsuter is up we can start rolling out services and deployments on it.

Lets start with creating a Deployment consiting of 3 containers all running the nginx:mainline-alpine image from Docker hub.

nginx-dep.yaml looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:mainline-alpine
        ports:
        - containerPort: 80

Load this into the cluster with kubectl create:

1

kubectl create -f https://raw.githubusercontent.com/StianOvrevage/gh-pages-blog/master/static/attachments/2017-12-23-managed-kubernetes-on-azure/nginx-dep.yaml

This command creates the resources described in the file. kubectl can read files either from your local disk or from a web URL.

After making changes to a resource definition (.yaml file), you can update the resources in the cluster with kubetl replace -f resource.yaml.

We can verify that the Deployment is ready:

1
2
3

> kubectl get deploy
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3         3         3            3           10m

We can also get the actual Pods that are running:

1
2
3
4
5

> kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-569477d6d8-dqwx5   1/1       Running   0          10m
nginx-deployment-569477d6d8-xwzpw   1/1       Running   0          10m
nginx-deployment-569477d6d8-z5tfk   1/1       Running   0          10m

Logger We can view logs from one pod with kubectl logs nginx-deployment-569477d6d8-xwzpw. But since we in this case don’t know which Pod ends up getting an incomming request we can view logs from all the Pods which have app=nginx label: kubectl logs -lapp=nginx. The use of app=nginx is our choice in nginx-dep.yaml when we configured spec.template.metadata.labels: app: nginx.

Making nginx available with a service

To send traffic to our new Pods we need to create a Service. A service consists of one or more Pods which are chosen based on different criteria, for example which labels they have and whether the Pods are Running and Ready.

Lets create a service which forwards traffic to all Pods with label app: nginx and are listening to port 80. In addition we make the service available via a LoadBalancer:

nginx-svc.yaml looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx

We tell Kubernetes to create our service with kubectl create as usual:

1

kubectl create -f https://raw.githubusercontent.com/StianOvrevage/gh-pages-blog/master/static/attachments/2017-12-23-managed-kubernetes-on-azure/nginx-svc.yaml

We can then wait and see which IP-address Azure assigns our service:

1
2
3

> kubectl get svc -w
NAME         CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE
nginx        10.0.24.11   13.95.173.255   80:31522/TCP   15m

PS It can take a few minutes for Azure to allocate and assign a Public IP for us. In the mean time <pending> will appear under EXTERNAL-IP.

A simple Welcome to nginx webpage should now be available on http://13.95.173.255 (remember to replace with your own External-IP).

We can also delete the service and deployment afterwards:

1
2

kubectl delete svc nginx
kubectl delete deploy nginx-deployment

Scaling the cluster

If we want to change the number of agent nodes running Pods we can do that via Azure-CLI:

1

az aks scale --name my_cluster --resource-group my_aks_rg --node-count 5

Currently all nodes will be created with the same size as when we created the cluster. AKS will probably get support for node-pools next year. That will allow for creating different groups of nodes with different size and operating systems, both Linux and Windows.

Delete cluster

You can delete the whole cluster like this:

1

az aks delete --name my_cluster --resource-group my_aks_rg --yes

Bonus material

Here is some bonus material if you want to go a bit further with Kubernetes.

Deploying services with Helm

Helm is a package manager and library of software that is ready to be deployed on a Kubernetes cluster.

Start by downloading the Helm-client. It will read login information etc. from the same location as kubectl automatically.

Install the Helm-server (Tiller) on the Kubernetes cluster and update the package library:

1
2

helm init
helm repo update

See available packages (Charts) with helm search.

Deploy MineCraft with Helm

Lets deploy a MineCraft server installation on our cluster, just because we can :-)

1

helm install --name stians --set minecraftServer.eula=true stable/minecraft

--set overrides one or more of the standard values configured in the package. The MineCraft package is made in a way where it does not start without accepting the user license agreement by setting the variable minecraftServer.eula. All the variables that can be set in the MineCraft package are documented here.

Then we wait for Azure to assign us a Public IP:

1
2

> kubectl get svc -w
stians-minecraft   10.0.237.0   13.95.172.192   25565:30356/TCP   3m

Now we can connect to our MineCraft server on 13.95.172.192:25565!

Kubernetes Dashboard

Kubernetes also has a graphic web user-interface which makes it a bit easier to see which resources are in the cluster, view logs and even open a remote shell inside a running Pod, among other things.

1
2

> kubectl proxy
Starting to serve on 127.0.0.1:8001

kubectl encrypts and tunnels the traffic to the Kubernetes API servers. The dashboard is available on http://127.0.0.1:8001/ui/.

Conclusion

I hope you enjoy Kubernetes as much as I have. The learning curve can be a bit steep in the beginning, but it does not take long before you are productive.

Look at the official guides on Kubernetes.io to learn more about defining different types of resources and services to run on Kubernetes. PS: There are big changes from version to version so make sure you use the documentation for the correct version!

Kubernetes also have a very active Slack-community on kubernetes.slack.com that is worthwhile to check out.