Stian Øvrevåge

Configuring Envoy as an edge proxy - through istio

Mon, 25 Nov 2024 00:00:00 GMT

How we implemented Envoy edge proxy best practices in istio-ingressgateway.

Introduction
Configuring overload manager and global connection limits using a custom Envoy bootstrap
Configuring buffer sizes and connection timeouts via EnvoyFilter
Appendices
Outro

This is a cross-post of a blog post also published on the Signicat Blog

We deployed this with Istio 1.23 and the Envoy edge proxy recommendations as of November 2024.

Introduction

If you are using istio as a Service Mesh for your Kubernetes clusters, chances are you are also using istio-ingressgateway to handle incoming traffic from the Internet.

Envoy however, which istio relies on, is not tuned for running at the edge by default:

Envoy is a production-ready edge proxy, however, the default settings are tailored for the service mesh use case, and some values need to be adjusted when using Envoy as an edge proxy.
— envoyproxy.io/docs

The Envoy edge proxy best practices document outlines specific recommended configuration parameters for running envoy (and thus istio-ingressgateway) on the edge.

It’s not immediately obvious how you would propagate these configurations through the regular istio installation and configuration procedures. There is an open feature request on GitHub asking for the ability to configure istio-ingressgateway according to best practices.

Here I’ll show you at least one way of getting these settings deployed.

We need two different approaches to achieve our goals. A custom bootstrap configuration and an EnvoyFilter.

Configuring overload manager and global connection limits using a custom Envoy bootstrap

To enable/configure Envoy overload manager and global connection limits we first create our config file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-envoy-custom-bootstrap-config
  namespace: istio-system
data:
  custom_bootstrap.yaml: |
    # Untrusted downstreams:
    overload_manager:
      refresh_interval: 0.25s
      resource_monitors:
      - name: "envoy.resource_monitors.fixed_heap"
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
          max_heap_size_bytes: 350000000 # 350000000=350MB
      - name: "envoy.resource_monitors.global_downstream_max_connections"
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
          max_active_downstream_connections: 25000
      actions:
      # Possible actions: https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/overload_manager/overload_manager#overload-actions
      - name: "envoy.overload_actions.shrink_heap"
        triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.9
      - name: "envoy.overload_actions.stop_accepting_requests"
        triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.95
      # Additional settings from https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/overload_manager/overload_manager
      - name: "envoy.overload_actions.disable_http_keepalive"
        triggers:
          - name: "envoy.resource_monitors.fixed_heap"
            threshold:
              value: 0.95
      # From https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/overload_manager/overload_manager#reducing-timeouts
      - name: "envoy.overload_actions.reduce_timeouts"
        triggers:
          - name: "envoy.resource_monitors.fixed_heap"
            scaled:
              scaling_threshold: 0.85
              saturation_threshold: 0.95
        typed_config:
          "@type": type.googleapis.com/envoy.config.overload.v3.ScaleTimersOverloadActionConfig
          timer_scale_factors:
            - timer: HTTP_DOWNSTREAM_CONNECTION_IDLE
              min_timeout: 2s
      # https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/overload_manager/overload_manager#load-shed-points
      loadshed_points:
        - name: "envoy.load_shed_points.tcp_listener_accept"
          triggers:
            - name: "envoy.resource_monitors.fixed_heap"
              threshold:
                value: 0.95
    # From https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge#best-practices-edge / https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/runtime#config-listeners-runtime
    # Also mentioned in https://istio.io/latest/news/security/istio-security-2020-007/#mitigation with much higher limits
    # Here we configure one limit for the "regular" public listener on port 8443 and a separate global limit that is higher to
    # avoid starving connections for admin and metrics and probes
    layered_runtime:
      layers:
      - name: static_layer_0
        static_layer:
          envoy:
            resource_limits:
              listener:
                0.0.0.0_8443:
                  connection_limit: 10000

Let’s save it to istio-envoy-custom-bootstrap-config.yaml.

There is a field here you MUST adjust to your environment. That is the max_heap_size_bytes which we set to about 90% of the configured K8s memory limit.

What this does is inform the overload manager of how much memory it has available, and is used for evaluating percentage of current usage compared to what it thinks it has available, that again triggers overload actions at certain thresholds.

You may also have to adjust the second to last line (0.0.0.0_8443) in case your public listener is named something else.

Now we install it in the cluster:

kubectl apply -n istio-system -f istio-envoy-custom-bootstrap-config.yaml

Then we can use an overlay to modify the Deployment that istioctl produces, before istioctl actually installs it in the cluster. This is the path and contents of what you need to add to your existing IstioOperator that you feed to istioctl:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:        
    ingressGateways:
      - name: istio-ingressgateway
        k8s:
          overlays:
            - kind: Deployment
              name: istio-ingressgateway
              patches:
                - path: spec.template.spec.containers.[name:istio-proxy].env[-1]
                  value:
                    name: ISTIO_BOOTSTRAP_OVERRIDE
                    value: /etc/istio/custom-bootstrap/custom_bootstrap.yaml
                - path: spec.template.spec.containers.[name:istio-proxy].volumeMounts[-1]
                  value:
                    mountPath: /etc/istio/custom-bootstrap
                    name: custom-bootstrap-volume
                    readOnly: true
                - path: spec.template.spec.volumes[-1]
                  value:
                    configMap:
                      name: istio-envoy-custom-bootstrap-config
                      defaultMode: 420
                      optional: false
                    name: custom-bootstrap-volume

Huge shoutout to our eminent Csaba Kárpáti which helped me out actually getting the overlay above work.

Configuring buffer sizes and connection timeouts via EnvoyFilter

We set the rest of the recommended configurations via an EnvoyFilter.

Create a file for it, named listener-filters-edge.yaml for example:

# Based on recommendations for edge deployments with untrusted downstreams:
# - https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge#best-practices-edge
# - https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/timeouts#faq-configuration-timeouts

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: listener-filters-edge
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: LISTENER
      match:
        context: GATEWAY
      patch:
        operation: MERGE
        value:
          per_connection_buffer_limit_bytes: 32768 # Doc examples 32 KiB # Default 1MB
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          name: "envoy.filters.network.http_connection_manager"
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
            # https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-field-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-request-headers-timeout
            request_headers_timeout: 10s                         # Default no timeout
            # https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-msg-config-core-v3-http1protocoloptions
            common_http_protocol_options:
              max_connection_duration: 60s                        # Default no timeout
              idle_timeout: 900s                                  # Default 1 hour. Doc example 900s
              headers_with_underscores_action: REJECT_REQUEST
            # https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#config-core-v3-http2protocoloptions
            http2_protocol_options:
              max_concurrent_streams: 100                         # Default 2147483647
              initial_stream_window_size: 65536                   # Doc examples 64 KiB - Default 268435456 (256 * 1024 * 1024)
              initial_connection_window_size: 1048576             # Doc examples 1 MiB - Same default as initial_stream_window_size
            # https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto.html#extensions-filters-network-http-connection-manager-v3-httpconnectionmanager
            stream_idle_timeout: 300s               # Default 5 mins. Must be disabled for long-lived and streaming requests
            request_timeout: 300s                   # Default no timeout. Must be disabled for long-lived and streaming requests
            use_remote_address: true
            normalize_path: true
            merge_slashes: true
            path_with_escaped_slashes_action: UNESCAPE_AND_REDIRECT

And install it like usual:

kubectl apply -n istio-system -f listener-filters-edge.yaml

Appendices

Appendix A - Displaying currently active Envoy edge configuration settings

Here is a handy script for printing the currently active settings to console. Useful for verifying the changes have actually made it all the way to Envoy.

CONFIG_FILE=igw_config.json

echo "Looking for pods labeled istio=ingressgateway"
ISTIO_INGRESSGATEWAY_POD=$(kubectl get pods -n istio-system -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}')

echo "Using $ISTIO_INGRESSGATEWAY_POD and dumping configuration to $CONFIG_FILE"

kubectl exec -n istio-system $ISTIO_INGRESSGATEWAY_POD curl http://localhost:15000/config_dump > $CONFIG_FILE

echo "Custom bootstrap configuration: "

printf "bootstrap.overload_manager.refresh_interval: "
cat $CONFIG_FILE | jq -r '.configs[0].bootstrap.overload_manager.refresh_interval'

printf "max_active_downstream_connections: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.resource_monitors[] | select(.name == "envoy.resource_monitors.global_downstream_max_connections") | .typed_config.max_active_downstream_connections'

printf "max_heap_size_bytes: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.resource_monitors[] | select(.name == "envoy.resource_monitors.fixed_heap") | .typed_config.max_heap_size_bytes'

printf "overload_actions.shrink_heap: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.shrink_heap") | .triggers[0].threshold.value'

printf "overload_actions.stop_accepting_requests: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.stop_accepting_requests") | .triggers[0].threshold.value'

printf "overload_actions.disable_http_keepalive: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.disable_http_keepalive") | .triggers[0].threshold.value'

printf "overload_actions.reduce_timeouts scaling_threshold: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.reduce_timeouts") | .triggers[0].scaled.scaling_threshold'

printf "overload_actions.reduce_timeouts saturation_threshold: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.reduce_timeouts") | .triggers[0].scaled.saturation_threshold'

printf "overload_actions.reduce_timeouts timer_scale_factors timer: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.reduce_timeouts") | .typed_config.timer_scale_factors[0].timer'

printf "overload_actions.reduce_timeouts timer_scale_factors min_timeout: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.actions[] | select(.name == "envoy.overload_actions.reduce_timeouts") | .typed_config.timer_scale_factors[0].min_timeout'

printf "load_shed_points.tcp_listener_accept: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.overload_manager.loadshed_points[] | select(.name == "envoy.load_shed_points.tcp_listener_accept") | .triggers[0].threshold.value'

printf "resource_limits.0.0.0.0_8443.connection_limit: "
cat $CONFIG_FILE | jq '.configs[0].bootstrap.layered_runtime.layers[] | select(.name == "static_layer_0") | .static_layer.envoy.resource_limits.listener."0.0.0.0_8443".connection_limit'

echo "EnvoyFilter configuration: "

printf "per_connection_buffer_limit_bytes: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.per_connection_buffer_limit_bytes'

printf "http2_protocol_options.max_concurrent_streams: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.http2_protocol_options.max_concurrent_streams'

printf "http2_protocol_options.initial_stream_window_size: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.http2_protocol_options.initial_stream_window_size'

printf "http2_protocol_options.initial_connection_window_size: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.http2_protocol_options.initial_connection_window_size'

printf "stream_idle_timeout: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.stream_idle_timeout'

printf "request_timeout: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.request_timeout'

printf "common_http_protocol_options.idle_timeout: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.common_http_protocol_options.idle_timeout'

printf "common_http_protocol_options.max_connection_duration: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.common_http_protocol_options.max_connection_duration'

printf "request_headers_timeout: "
cat $CONFIG_FILE | jq '.configs[2].dynamic_listeners[] | select(.name == "0.0.0.0_8443") | .active_state.listener.filter_chains[0].filters[0].typed_config.request_headers_timeout'

Save it to get-edge-config-values.sh and run it with for example bash get-edge-config-values.sh.

Appendix B - Overload manager metrics

Envoy can also export metrics related to overload manager, they can be enabled by adding overload to approximately this location in the IstioOperator spec:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        k8s:
          podAnnotations:
            proxy.istio.io/config: |-
              proxyStatsMatcher:
                inclusionPrefixes:
                - "overload"

Appendix C - istio installation and configuration at Signicat

At Signicat we use helm to template all resources going in to our istio installation. That includes resource types like:

IstioOperator
EnvoyFilters
ConfigMaps
Gateways
Sidecars and so on.

We don’t use helm to install anything. Only to generate a set of manifests that is then used as inputs to the appropriate tools, like feeding generated IstioOperator manifests to istioctl and EnvoyFilter manifests to kubectl.

This works fairly well and allows us to have the same set of base manifests with adjustable values and feature sets per environment and using standard helm that most platform engineers are already familiar with.

We also have a couple of other tricks to enable us to have zero-downtime blue-green upgrades to istio-ingressgateway that we may cover in a future post.

Outro

I hope this was helpful on your journey towards scale, resilience and reliability.

The next chapter in this saga would be once we complete extensive load testing with the new configurations compared to the defaults as well as trying to find optimal values. And associated Grafana dashboards are always nice!

Introduction

Sun, 24 Nov 2024 00:00:00 GMT

Home of technical deep dives, random thoughts and ideas of varying quality.

My name is Stian Øvrevåge and I’m a passionate technologist and wannabe adventurer.

In addition to posting on average one blog post a year I do other stuff related to aviation, photography and general nerding out.

The canonical overview of my online presence is located at https://stian.omg.lol/.

Summer of 2024

Sun, 21 Jul 2024 18:51:01 GMT

Well, as you can see, mountains and museums and stuff.

Snapshots of 2024

Wed, 29 May 2024 15:17:03 GMT

More random photos from 2024.

Kubernetes: Don't use NodeLocal DNSCache on GKE (without Cloud DNS)

Wed, 13 Dec 2023 00:00:00 GMT

A travel diary of how we diagnosed a latency spike problem on GKE and the surprising things we learned along the way.

Introduction
TL;DR:
Investigation
Appendices
Footnotes

This is a cross-post of a blog post also published on the Signicat Blog

Introduction

Last year I was freelancing as a consultant in Signicat and recently I returned, now as an actual employee!

The first week after returning, a tech lead for another team reaches out to me about a technical issue he’s been struggling with.

In this blog post I’ll try to guide you through the troubleshooting with actionable take-aways. It appears it’s going to be a long one with a lot of detours, so I’ve summarized our findings and recommendations here on the top starting right below this introduction. There are a few appendixes at the end that I’ve tried to make self-contained as to make them useful in other contexts and without necessarily having to follow the main story.

If you don’t want any spoilers but follow the bumpy journey from start to end, fill your coffee mug and skip ahead to Investigation.

TL;DR:

Due to an unfortunate combination of behaviour of CoreDNS (which NodeLocal DNSCache uses) and kube-dns (which is the default on GKE) I recommend NOT using them in combination.

Since GKE does not offer CoreDNS as a managed option for kube-dns (even though Kubernetes made CoreDNS the default in 1.13 in 2018) you are left with two options:

Not enabling NodeLocal DNSCache on GKE
Switching from kube-dns to Google Cloud DNS

What is this unfortunate combination you ask?

NodeLocal DNSCache / CoreDNS

NodeLocal DNSCache (in GKE at least) is configured with:

# kubectl get configmap --namespace kube-system node-local-dns -o yaml | yq .data.Corefile
cluster.local:53 {
  forward . __PILLAR__CLUSTER__DNS__ {
      force_tcp
      expire 1s

This means CoreDNS will upgrade any and all incoming DNS requests to TCP connections before connecting to dnsmasq (the first of two containers in a kube-dns Pod). CoreDNS reuses TCP connections if available. Unused connections in the connection pool should be cleaned up every 1 second (in theory). If no connections are available a new will be created, apparently with no upper bound.

This means new connections may be created en-masse when needed, but old connections can take a while (I’ve observed 5-10 seconds) before being cleaned up, and most importantly, closed.

kube-dns / dnsmasq

dnsmasq has a hardcoded maximum number of 20 workers. For us that means each kube-dns Pod is limited to 20 open connections.

kube-dns scaling is managed by a bespoke kube-dns autoscaler that by default in GKE is configured like:

# kubectl get configmap --namespace kube-system kube-dns-autoscaler -o yaml | yq .data.linear
{"coresPerReplica":256, "nodesPerReplica":16,"preventSinglePointFailure":true}

preventSinglePointFailure - Run at least two Pods.
nodesPerReplica - Run one kube-dns Pod for each 16 nodes.

This default setup will have two kube-dns Pods until your cluster grows beyond 32 nodes.

Two kube-dns Pods have a limit of 40 open TCP connections in total from the node-local-dns Pods running on each node. The node-local-dns Pods though are happy to try to open many more TCP connections.

GKE kube-dns docs mention “Performance limitations with kube-dns” as a known issue suggesting enabling NodeLocal DNSCache as a potential fix.

And GKE NodeLocal DNSCache docs mention “NodeLocal DNSCache timeout errors” as a known issue with the possible reasons being

An underlying network connectivity problem. (Spoiler: it probably isn’t)
Significantly increased DNS queries from the workload or due to node pool upscaling. (Spoiler: it probably isn’t)

With the fix being

“The workaround is to increase the number of kube-dns replicas by tuning the autoscaling parameters.”

Increasing the number of kube-dns Pods will reduce the frequency and impact but not eliminate it. Until we migrate to Cloud DNS we run kube-dns at a ratio of 1:1.5 of nodes by configuring autoscaling with "coresPerReplica":24 and using 16 core nodes. Resulting in 12 kube-dns Pods in a 17 node cluster.

By looking at the coredns_forward_conn_cache_misses_total metric I observe it at least increasing by more than 300 in a 15 second metric sampling window for one node-local-dns Pod. This means on average during those 15 seconds 20 new TCP connections were attempted since no existing connection could be re-used. (Footnote A)

This means even setting "nodesPerReplica":1 thus running one kube-dns Pod for each node may not be enough to guarantee not hitting the 20 process limit in dnsmasq occasionally.

I guess you could start lowering coresPerReplica to have more than one kube-dns Pod for each node, but now it’s getting silly.

To re-iterate; If you’re using NodeLocal DNSCache and kube-dns you should plan to migrate to Cloud DNS. You can alleviate the problem in the short term by scaling up kube-dns aggressively but it will not eliminate occasional latency spikes.

Am I affected?

How do I know if I’m affected by this?

You are using NodeLocal DNSCache (CoreDNS) and kube-dns (default on GKE)
You see log lines with i/o timeout in node-local-dns pods (kubectl logs -n kube-system node-local-dns-xxxxx)
You are hitting dnsmasq max procs. Create a graph container_processes{namespace="kube-system", container="dnsmasq"} or use my kube-dns Grafana dashboard. (Footnote B).

Investigation

The initial Problem

We have a microservice architecture and use Open Policy Agent (OPA) deployed as sidecars to process and validate request tokens.

However, some services were getting timeouts and he suspected it was an issue with istio, a service mesh that provides security and observability for traffic inside Kubernetes clusters.

The actual error message showing up in the logs was a generic context deadline exceeded (Client.Timeout exceeded while awaiting headers) which I recognize as a Golang error probably stemming from a http.Get() call or similar.

The first thing that comes to mind is that last year we made a change in our istio practices. Inside our clusters we call services in other namespaces (crossing team boundaries) using the same FQDN domain and path that our customers would use, not internal Kubernetes names. So internally we’re calling api.signicat.com/service and not service.some-team-namespace.

He had noticed that the problem seemed to have increased when doing the switch from internal DNS names to FQDN.

Istio / Envoy metrics

We started troubleshooting by enabling additional metrics in envoy to hopefully be able to confirm that the problem was visible as envoy timeouts or some other kind of error.

Envoy is the actual software that traffic goes through in an istio service mesh.

This turned out to be a dead end and we couldn’t find any signs of timeouts or errors in the envoy metrics.

Enabling these additional metrics was a bit of a chore and there were a few surprises. Have a look at the Enabling additional envoy metrics appendix for steps and caveats.

Reproducing and troubleshooting HTTP connection problems

At this point I’m thinking the problem may be something else. Failure to acquire a HTTP connection from the pool or a TCP socket maybe?

Using the httptrace golang library I created a small program that would continuously query both the internal some-service.some-namespace hostname and FQDN api.signicat.com/some-service, while logging and counting each phase of the HTTP request as well as it’s timings.

Here is a quick and dirty way of starting the program in an ephemeral Pod:

# Start a Pod named `conn-test` using the `golang:latest` image. Attach to it (`-i --tty`) and start `bash`. Delete it after disconnecting (`--rm`):
kubectl run conn-test --rm -i --tty --image=golang:latest -- bash
# Create and enter a directory for the program:
mkdir conn-test && cd conn-test
# Initialize a go environment for it:
go mod init conn-test
# Download the source:
wget https://raw.githubusercontent.com/signicat/blog-attachements/main/2023-gke-node-local-dns-cache/files/go-http-connection-test/main.go
# Download dependencies:
go mod tidy
# Configure it (refer to the source for more configuration options):
export URL_1=http://some-service.some-namespace.svc.cluster.local
# Start it:
go run main.go

If you need to run it for longer it probably makes sense to build it as a Docker image, upload it to an image registry and create a Deployment for it. It also exposes metrics in Prometheus format so you can also add scraping of the metrics either through annotations or a ServiceMonitor.

After a few hours the program crashed, and the last log message indicated that DNSStart was the last thing to happened. (The program crashed since I just log.Fataled if the request failed, which it did timing out. I since improved that, even though we already learned what we needed).

We did some manual DNS lookups using nslookup -debug api.signicat.com which (obviously, in retrospect) shows 6 failing DNS requests before finally getting it right:

api.signicat.com.$namespace.svc.cluster.local - NXDOMAIN
api.signicat.com.svc.cluster.local - NXDOMAIN
api.signicat.com.cluster.local - NXDOMAIN
api.signicat.com.$gcp-region.c.$gcp-project.internal - NXDOMAIN
api.signicat.com.c.$gcp-project.internal - NXDOMAIN
api.signicat.com.google.internal - NXDOMAIN
api.signicat.com - OK

See Overview of DNS on GKE for a lengthier explanation of why this happens.

The default timeout for HTTP requests in Golang is 5 seconds. DNS resolution should be fast enough and it shouldn’t be a problem to do 7 DNS lookups in that time. But if there is some sluggishness and variability in DNS resolution times the probability of having one slow lookup out of the 7 being slow increases the risk of causing a timeout. In addition, always doing 7 lookups puts additional strain on the DNS infrastructure potentially further exacerbating the probability of slow lookups.

From here on we have two courses of action. Figure out if we can reduce the number of DNS lookups as well as figure out if, and why, DNS resolution isn’t consistently fast.

See Reducing DNS lookups in Kubernetes and GKE for some thoughts on reducing these extraneous DNS lookups.

Troubleshooting potentially slow or broken DNS lookups

We use NodeLocal DNSCache which on GKE is enabled by clicking the right button.

NodeLocal DNSCache summary

It is deployed by a ReplicaSet causing one CoreDNS Pod (named node-local-dns-x in kube-system namespace) to be running on each node.

It adds a listener on the same IP as the kube-dns Service. Thereby intercepting traffic that would otherwise go to that IP.

The node-local-dns Pod caches both positive and negative results. Domains ending in .cluster.local are forwarded to kube-dns in the cluster (but through a new Service called kube-dns-upstream with a different IP). Requests for other domains are forwarded to 8.8.8.8 and 8.8.4.4.

NodeLocal DNSCache instances use a 2 second timeout when querying upstream DNS servers.

See Overview of GKE DNS with NodeLocal DNSCache for additional details.

We wanted to look at some metrics from the node-local-dns Pods but found that we didn’t have any! We fixed that but didn’t at the time learn anything new. See Enabling node-local-dns metrics for how to fix/enable node-local-dns metrics.

Stumbling over the logs of one of the node-local-dns-x Pods I notice:

[ERROR] plugin/errors: 2 archive.ubuntu.com.some-namespace.svc.cluster.local. A: dial tcp 172.18.165.130:53: i/o timeout

This tells us that at least lookups going to kube-dns in the cluster (172.18.165.130:53) are having problems.

So timeouts are happening on individual lookups, it’s not just the total duration of 7 lookups timing out. And lookups are not only happening inside the client application but in node-local-dns as well. We still don’t know if these lookups are lost or merely slow. But seen from the application anything longer than 2 seconds, that is timing out on node-local-dns, might as well be lost.

Just to be sure I checked that packets were not being dropped or lost on the nodes on both sides. And indeed the network seems fine. It’s been a long time since I’ve actually experienced problems due to packets being lost, but it’s always good to rule it out.

Next step is to capture the packets as they (hopefully) leave the node-local-dns Pod and (maybe) arriving at one of the kube-dns Pods.

See Network packet capture without dependencies for how to capture packets in Kubernetes.

The packets are indeed arriving at the dnsmasq container in the kube-dns Pods:

Query 0x31e8:

Leaves node-local-dns eth0:
10:53:11.134	10.0.0.107	172.20.13.4	DNS	0x31e8	Standard query 0x31e8 A archive.ubuntu.com.some-ns.svc.cluster.local

Arrives at dnsmasq eth0:
10:53:11.134	10.0.0.107	172.20.13.4	DNS	0x31e8	Standard query 0x31e8 A archive.ubuntu.com.some-ns.svc.cluster.local

Looking at DNS resolution time from the kube-dns container which dnsmasq forwards to shows that kube-dns is consistently answering queries extremely fast (not shown here).

But after some head scratching I look at the time between queries arriving at dnsmasq on eth0 before leaving again on lo for kube-dns and indeed there is a (relatively) long delay of 952ms between 10:53:11.134 and 10:53:12.086:

Leaves dnsmasq lo:
10:53:12.086	127.0.0.1	127.0.0.1	DNS	0x31e8	Standard query 0x31e8 A archive.ubuntu.com.some-ns.svc.cluster.local

In this case the query just sits around in dnsmasq for almost one second before being forwarded to kube-dns!

Why? As with most issues we start by checking if there are any issues with memory or CPU usage or CPU throttling (Footnote C).

Nope. CPU and memory usage is both low and stable but the 99 percentile DNS request duration is all over the place.

We also check and see that there is plenty of unused CPU available on the underlying nodes these pods are running on.

We also used dnsperf to benchmark and stress-test the various components and while fun, didn’t teach us anything new. See Using dnsperf to test DNS performance for more information on that particular side-quest.

Next up I want to increase the logging verbosity of dnsmasq to see if there are any clues to why it was (apparently) delaying processing DNS requests.

Have a look at Verbose logging on kube-dns for how to increase logging and Analyzing dnsmasq logs for how I analyzed the logs.

Analyzing the logs we learn that at least it appears that individual requests are fast and not clogging up the machinery.

In the meantime Google Cloud Support came back to us asking if we could run a command for i in $(seq 1 1800) ; do echo "$(date) Try: ${i} DnsmasqProcess: $(pidof dnsmasq | wc -w)"; sleep 1; done on the VM of the Pod. It also works to run this in a debug container attached to the dnsmasq container. The output looks like:

Wed Oct 11 14:50:57 UTC 2023 Try: 1 DnsmasqProcess: 9
Wed Oct 11 14:50:58 UTC 2023 Try: 2 DnsmasqProcess: 14
Wed Oct 11 14:50:59 UTC 2023 Try: 3 DnsmasqProcess: 15
Wed Oct 11 14:51:00 UTC 2023 Try: 4 DnsmasqProcess: 21
Wed Oct 11 14:51:01 UTC 2023 Try: 5 DnsmasqProcess: 21

It turns out that dnsmasq has a hard coded limit of 20 child processes. So every time we see 21 it means it will not spawn a new child to process incoming connections.

I plotted this on the same graph as the DNS request times observed from raw packet captures (See Analyzing DNS problems based on packet captures):

Graphing the number of dnsmasq processes and observed DNS request latency finally shows a stable correlation between our symptoms and a potential cause.

You can also get a crude estimation by graphing the container_processes{namespace="kube-system", container="dnsmasq"} metric, or use my kube-dns Grafana dashboard if you have cAdvisor/container metrics enabled:

Going back to our packet captures I see connections that are unused for many seconds before finally being closed by node-local-dns.

08:06:07.849	172.20.9.11	10.0.0.29	53 → 46123 [ACK] Seq=1 Ack=80 Win=43648 Len=0
08:06:09.849	10.0.0.29	172.20.9.11	46123 → 53 [FIN, ACK] Seq=80 Ack=1 Win=42624 Len=0
08:06:09.890	172.20.9.11	10.0.0.29	53 → 46123 [ACK] Seq=1 Ack=81 Win=43648 Len=0

However dnsmasq only acknowledges the request to close the connection, it does not actually close it yet. In order for the connection to be properly closed both sides have to FIN, ACK.

Many seconds later dnsmasq faithfully tries to return a response and finish the closing of the connection, but node-local-dns (CoreDNS) has promptly forgotten about the whole thing and replies with the TCP equivalent of a shrug (RST):

08:06:15.836	172.20.9.11	10.0.0.29	53	46123	Standard query response 0x7f12 No such name A storage.googleapis.com.some-ns.svc.cluster.local SOA ns.dns.cluster.local
08:06:15.836	172.20.9.11	10.0.0.29	53	46123	53 → 46123 [FIN, ACK] Seq=162 Ack=81 Win=43648 Len=0
08:06:15.836	10.0.0.29	172.20.9.11	46123	53	46123 → 53 [RST] Seq=81 Win=0 Len=0

Then I analyzed the whole packet capture to find the number of open connections at any time as well as the frequency and duration of these lingering TCP connections. See Analyzing concurrent TCP connections for details on how.

What we find is that the number of open connections at times surge way past 20 and that coincides with increased latency. In addition unused connections stay open for a problematic long time (5-10 seconds).

Since the closing of connections is initiated by CoreDNS. Is there any way we can make CoreDNS close connections faster or limit the number of connections it uses?

NodeLocal DNSCache (in GKE at least) is configured (kubectl get configmap --namespace kube-system node-local-dns -o yaml | yq .data.Corefile) with:

cluster.local:53 {
  forward . __PILLAR__CLUSTER__DNS__ {
      force_tcp
      expire 1s

So it is actually configured to “expire” connections after 1 second.

I’m not well versed in the CoreDNS codebase but it seems expiring connections is handled by a ticker. A ticker in go sends a signal every “tick” that can be used to trigger events for example. This means connections aren’t closed once they pass the 1 second mark, but the cleanup process runs every 1 second and then purges expired connections. So a connection can be idle for almost 2x the time (2 seconds in our case) before being cleaned up.

I still can’t explain why we see connections lingering on for 5-10 seconds though.

There are no configuration options indicating that it’s possible to limit the number of connections. Nor anything in the code that would suggest it’s a possibility. Adding it is probably not done in a day either as it would require making decisions on trade-offs about how to handle excess traffic volume. How much to queue, for how long. How to avoid the queue eating too much memory. What to do when the queue is full and so on and so on.

At this point I feel we have a pretty good grasp of how all of this conspires to cause the problems we observe. But unfortunately I can’t see any permanent solution that does not require modifying CoreDNS and ideally dnsmasq code itself. At the moment we don’t have the capacity to create and push through such changes upstream. If I could snap my fingers and magically get some new features they would be:

dnsmasq
- Make MAX_PROCS configurable.
CoreDNS
- Configurable max TCP connection pool size. Metrics on pool size and usage.
- Configurable queue size for requests waiting for a new connection from the pool. Metrics on queue capacity (max) and current size for tuning.

But until that becomes a reality I think the best option is to avoid using NodeLocal DNSCache in combination with kube-dns, and instead replace kube-dns with Cloud DNS.

Appendices

Appendix - Enabling additional envoy metrics

See istio.io for some high level info on proxy-level metrics and envoyproxy.io for complete list of available metrics.

Most likely we are interested in the upstream_cx_* and upstream_rq_* metrics.

By default metrics are only gathered and exposed for the xds-grpc cluster, and the full stats name looks like cluster.xds-grpc.upstream_cx_total. The xds-grpc cluster I assume is metrics for traffic between the istio-proxy containers in each Pod and the central istio services (istiod) used for configuration management.

A cluster in envoy is a grouping of backends and endpoints. Typically each Kubernetes Service will be it’s own cluster. There are also separate clusters named BlackHoleCluster, InboundPassthroughClusterIpv4 and PassthroughCluster.

When enabling metrics for other services (or clusters as they’re also called) they look like

cluster.outbound|80||my-service.my-namespace.svc.cluster.local.upstream_cx_total for outgoing traffic and
cluster.inbound|8080||.upstream_cx_total for incoming traffic.

Beware that traffic to for example api.signicat.com/some-service will be mapped to the internal Kubernetes Service DNS name in the metric, like some-service.some-team.svc.cluster.local.

Ports are also mapped. For outgoing traffic it will be the port in the Service. While for incoming traffic it will be the port the Pod is actually listening on, and not the one mapped in the Service.

Enabling additional envoy metrics

istio.io documents how to enable additional envoy metrics both globally in the mesh by configuring the IstioOperator object as well as on a per Deployment/Pod using the proxy.istio.io/config annotation.

WARNING: Be very careful when enabling additional metrics as they have a tendency to expose orders of magnitude more time-series than you might expect.

I apparently managed to get Prometheus OOMKilled even with my fairly limited testing on two deployments.

If you happen to have VictoriaMetrics in your monitoring stack you can monitor cardinality (the number of unique time-series, which is the thing that usually breaks a time-series database) in the VM UI:
kubectl -n metrics port-forward services/victoria-metrics-cluster-vmselect 8481:8481
And going to http://localhost:8481/select/0/vmui/?#/cardinality

Enable additional metrics on a Deployment or Pod

To enable additional metrics on a Deployment or Pod, add the following annotations:

metadata:
  annotations:
    proxy.istio.io/config: |-
      proxyStatsMatcher:
        inclusionSuffixes:
          - "upstream_rq_timeout"
          - "upstream_rq_retry"
          - "upstream_rq_retry_limit_exceeded"
          - "upstream_rq_retry_success"
          - "upstream_rq_retry_overflow"

Remember to add this to the Pod metadata (spec.template.metadata) if adding to a Deployment.

This results in a new time-series being exposed for each Service and Port (Cluster) that envoy has configured.

In our case we have ~500 services in our dev cluster and enabling these specific metrics adds ~3.800 new time-series for each Pod in the Deployment we added it to. The test Deployment I’m playing with has 6 Pods so ~23.000 new time-series from adding 5 additional metrics to 1 Deployment!

Another option is to use regex to enable additional metrics:

          inclusionRegexps:
          - ".*upstream_rq_.*"
          - ".*upstream_cx_.*"

But again, enabling these ~50 metrics on this specific Deployment will result in ~250.000 new time-series.

Be especially wary of metrics envoy exposes as histograms, such as upstream_cx_connect_ms and upstream_cx_length_ms as they result in many _bucket time-series. During my testing this resulted in 6-7 million new time-series in total.

It’s possible to specify only the connections we are interested in, which makes it manageable, cardinality wise. For example to only gather metrics from Services A through F in their corresponding namespaces:

    inclusionRegexps:
      - ".*(svc-a.ns-a.svc|svc-b.ns-b.svc|svc-c.ns-c.svc|svc-d.ns-d.svc|svc-e.ns-e.svc|svc-f.ns-f.svc).*.upstream_rq.*"
      - ".*(svc-a.ns-a.svc|svc-b.ns-b.svc|svc-c.ns-c.svc|svc-d.ns-d.svc|svc-e.ns-e.svc|svc-f.ns-f.svc).*.upstream_cx.*"

Enable additional metrics globally (please don’t!)

You can also enable additional metrics globally across the mesh. But that’s probably a very bad idea if you are running at any sort of scale. I estimated that enabling .*upstream_rq_.* and .*upstream_cx_.* in our dev cluster would result in 50M additional time-series at a minimum. Or 5-10x our current Prometheus usage.

If you are using the old Istio Operator way of installing and managing istio it should be enough to update the IstioOperator object in the cluster. If you are using istioctl (recommended) you must update the source IstioOperator manifest that is being fed to istioctl and run the appropriate istioctl commands again to update. Note that this also creates an IstioOperator object in the cluster with whatever config is used. But in this case it’s never used for anything other than reference. So updating the IstioOperator object in a cluster if managing istio with istioctl does nothing.

Viewing stats and metrics

Metrics should start becoming available in Prometheus with names like envoy_cluster_upstream_cx_total. Note that by default you’ll already see metrics from the xds-grpc cluster.

You can also get the stats directly from a sidecar. Either by querying the pilot-agent directly:

kubectl exec -n my-namespace my-pod -c istio-proxy -- pilot-agent request GET stats

Or querying the metrics endpoint exposed to Prometheus:

kubectl exec -n my-namespace my-pod -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus'

Note that these will not give you any additional metrics compared to those exposed to Prometheus. A metric that is not enabled will not show up, and all metrics that are enabled are also automatically exposed to Prometheus.

Notes on timeout and retry metrics

If istio is configured to be involved with timeouts and retries that is configured on the VirtualService level.

That means it will only take effect if using api.signicat.com/some-service (a route in a Istio VirtualService) and not some-service.some-namespace.svc.cluster.local (Kubernetes Service).

Istio will only count timeouts and retries for requests to a VirtualService route that has timeouts (and optionally retries) configured.

Additionally, the timeouts and retries seems to be enforced, and counted, in the Source istio-proxy (envoy), and not the Target.

Further work

It would be beneficial to be able to have envoy only show/export metrics that are non-zero. Since there is usually only a very small set of all possible Service-to-Service pairs that will actually regularly have traffic.

It’s possible to customize metrics using the Telemetry API (Customizing Istio Metrics with Telemetry API) but it seems limited to only working with metrics and their dimensions. Not the time-series values.

WASM plugins are probably not a good fit either and are experimental and causes a severe CPU and memory penalty.

Edit to add in 2024

We know today that the reason for so many time-series (reporting zero), as well as elevated memory usage in istio-proxy, is because we did not filter exposed Services between namespaces causing every istio-proxy to keep track of every pair of possible workloads in the whole cluster.

Appendix - Overview of DNS on GKE

kube-dns and CoreDNS confusion

The main difference between DNS on upstream Kubernetes and GKE is that Kubernetes switched to CoreDNS 5 years ago while GKE still uses the old kube-dns. It’s possible to add CoreDNS but it’s not possible to remove or disable kube-dns.

In upstream K8s, CoreDNS reached GA back in 2018 in K8s 1.11 and kube-dns was removed from kubeadm in K8s 1.21.

However for backwards compatibility CoreDNS still uses the name kube-dns, which makes things confusing for sure!

Part of the magic of Kubernetes is it’s DNS-based Service Discovery. Lets say we deploy an application that has a Service named helloworld to the namespace team-a. Other applications in the same namespace are able to connect to that service by calling for example http://helloworld. This makes it easy to build and configure a group of microservices that talk to each other without needing to know about namespaces or FQDNs. Applications in another namespace can also call that application using http://helloworld.team-a. (Footnote D).

This magic is achieved using the standard DNS search domain list feature. On Linux the /etc/resolv.conf file defines which DNS servers to use. It also includes a search option that works together with the ndots option. You can see this by executing cat /etc/resolv.conf in any Pod.

If a domain name contains fewer “dots” (“.”) than ndots is set to, the DNS resolver will first try the hostname with each of the appended domains in search. In Kubernetes (and GKE) ndots is by default set to 5.

In vanilla Kubernetes the search domains are <namespace>.svc.cluster.local svc.cluster.local cluster.local while on GKE it’s <namespace>.svc.cluster.local svc.cluster.local cluster.local <gcp-zone>.c.technology-dev-platform.internal c.<gcp-project>.internal google.internal.

This means if we call https://api.signicat.com (2 dots) it will first try to resolve api.signicat.com.some-namespace.svc.cluster.local and so on through the whole list, sequentally, before finally trying api.signicat.com and actually succeeding.

See Reducing DNS lookups in Kubernetes and GKE for thoughts on avoiding some of this.

Then the requests are sent to the dnsmasq container of a kube-dns Pod. dnsmasq is configured by a combination of command-line arguments defined in the kube-dns Deployment and data from the kube-dns ConfigMap. We have not changed the default kube-dns ConfigMap and this results in dnsmasq running with these arguments:

/usr/sbin/dnsmasq -k \
    --cache-size=1000 \
    --no-negcache \
    --dns-forward-max=1500 \
    --log-facility=- \
    --server=/cluster.local/127.0.0.1#10053 \
    --server=/in-addr.arpa/127.0.0.1#10053 \
    --server=/ip6.arpa/127.0.0.1#10053 \
    --max-ttl=30 \
    --max-cache-ttl=30 \
    --server /internal/169.254.169.254 \
    --server 8.8.8.8 \
    --server 8.8.4.4 \
    --no-resolv

This means all lookups for cluster.local, in-addr.arpa and ip6.arpa will be sent to 127.0.0.1:10053, which is the kube-dns container. Lookups for internal are sent to 169.254.169.254. All others are sent to 8.8.8.8 and 8.8.4.4.

In addition --no-negcache disables caching for lookups that were not found (NXDOMAIN). This is particularly interesting since that means when looking up for example api.signicat.com and the domains in the search list are first tried, they will all result in NXDOMAIN but dnsmasq will not cache those results but send them on to kube-dns. Every. Time. This may severely reduce the usefulness of the cache and create a constant volume of traffic to kube-dns itself.

It’s also worth noting that Google Public DNS servers have a default rate limit of 1500 QPS per IP.

Appendix - Reducing DNS lookups in Kubernetes and GKE

There are two reasons for the 7 DNS lookups instead of the ideal 1. First is the ndots which tells the DNS resolver in the container if a domain name has fewer dots than this, it will first try looking up the name by appending each entry in the search list sequentially. For Kubernetes ndots is set to 5 (why is explained by Tim Hockin here). So api.signicat.com only has 2 dots, and hence will first go through the list of search domains.

Secondly GKE adds 3 extra search domains in addition to the 3 standard ones in Kubernetes. Bringing the total to 6 before doing the “proper” DNS lookup.

The specific assumption we are deviating from leading to problems in our case is “We could mitigate some of the perf penalties by always trying names as upstream FQDNs first, but that means that all intra-cluster lookups get slower. Which do we expect more frequently? I’ll argue intra-cluster names, if only because the TTL is so low.”

An option that we’re currently exploring is using a trailing dot in the FQDN (api.signicat.com.) when calling other services. This explicitly tells DNS that this is a FQDN and should not search through the search domain lists for a hit first. This seems to work on some of our services but not all. Indicating that there isn’t any inherent problems doing this with regards to istio or other infrastructure. But there may be additional changes needed on some services to support this. I’m suspecting certain web application frameworks in certain languages not handling this well out of the box.

Appendix - Overview of GKE DNS with NodeLocal DNSCache

NodeLocal DNSCache is a feature of Kubernetes that primarily aims to improve performance, scale and latency by:

Potentially not traversing the network to another node
Skip iptables DNAT which sometimes caused problems
Upgrading connections from UDP to TCP which should reduce latency in case of dropped packets
Enabling negative caching

NodeLocal DNSCache is available as a GKE add-on.

NodeLocal DNSCache is deployed as a DaemonSet named node-local-dns in kube-system namespace. One node-local-dns-x Pod is created on each node in the cluster.

The node-local-dns-x Pods run CoreDNS and are configured by the node-local-dns ConfigMap (kubectl get cm node-local-dns -o yaml|yq .data.Corefile).

The configuration is similar to dnsmasq in that requests for cluster.local, in-addr.arpa and ip6.arpa are sent to kube-dns while the rest are sent to 8.8.8.8 and 8.8.4.4.

It binds to (re-uses) the same IP as the kube-dns Service. That way all requests from Pods on the node towards kube-dns will actually be handled by node-local-dns instead. And to actually communicate with kube-dns another Service named kube-dns-upstream is created that is a clone of the kube-dns Service but with a different IP.

Even though node-local-dns uses CoreDNS and kube-prometheus-stack has support for scraping CoreDNS it won’t necessarily work for node-local-dns. See the next appendix for how to scrape metrics from node-local-dns.

Appendix - Enabling node-local-dns metrics

We wanted to look at some metrics from node-local-dns, but found that we didn’t have any!

The node-local-dns pods do have annotations for scraping metrics:

metadata:
  annotations:
    prometheus.io/port: "9253"
    prometheus.io/scrape: "true"

But in our Prometheus we don’t use the kubernetes-pods scrape job config from the prometheus example. meaning that these targets will not be discovered or scraped by prometheus. (And we don’t want to add and allow for scraping this way).

The kube-prometheus-stack helm chart comes with both a Service and ServiceMonitor to scrape metrics from CoreDNS (which NodeLocal DNSCache uses).

However the Service and ServiceMonitor uses a hardcoded selector of k8s-app: kube-dns while node-local-dns Pods have k8s-app: node-local-dns.

We made some changes to these and deployed them to the cluster and now started getting metrics in the coredns_dns_* timeserieses.

You can add our updated Service and ServiceMonitor like this:

kubectl apply -f -n kube-system https://raw.githubusercontent.com/signicat/blog-attachements/main/2023-gke-node-local-dns-cache/files/node-local-dns-metrics-service.yaml
kubectl apply -f -n metrics https://raw.githubusercontent.com/signicat/blog-attachements/main/2023-gke-node-local-dns-cache/files/node-local-dns-servicemonitor.yaml

And metrics should start flowing within a couple of minutes.

In new versions of the CoreDNS dashboard that comes with kube-prometheus-stack you should be able to select the node-local-dns job.

I added the job template variable to the dashboard in this PR, which may not have made it into a new version of the kube-prometheus-stack chart yet. In the mean time you can use our updated CoreDNS dashboard which adds the necessary template variable as well as a couple of other improvements.

Appendix - Using dnsperf to test DNS performance

Trying to tease out more information on the problem we run some DNS load testing using dnsperf.

An alternative to dnsperf is resperf which is “a companion tool to dnsperf” designed for testing resolution performance of a caching DNS server. Whereas dnsperf is primarily meant to test authoritative DNS servers. Since resperf is more complicated to work with, and we want to test at a fairly low volume, we assume that dnsperf is good enough for now.

We spin up a new pod with Ubuntu to run dnsperf from

kubectl run dnsperf -n default --image=ubuntu:22.04 -i --tty --restart=Never

And install dnsperf

apt-get update && apt-get install -y dnsperf

We’ll be testing three different destinations.

kube-dns Service: First we use the IP of the kube-dns Service (172.18.0.10) which will be intercepted by the node-local-dns Pod on the same node. It will do it’s caching and the traffic is visible in our modified CoreDNS Grafana dashboard for that node.
kube-dns-upstream Service: Then we use the IP of the kube-dns-upstream Service (172.18.165.139) which is a copy of the kube-dns Service that node-local-dns uses when looking up .cluster.local domains. It has a different IP than kube-dns so that it won’t be intercepted by node-local-dns.
One kube-dns Pod: Then we use the IP of one specific kube-dns Pod (172.20.14.115) so that we can observe the behaviour of one instance without any load balancing sprinkling the traffic all over the place.

dnsperf takes a list of domains to look up in a file that looks like

archive.ubuntu.com A
test.salesforce.com A
storage.googleapis.com A

To create a file with the domains we actually look up I do a packet capture on one of the node-local-dns Pods for half an hour. Open it in Wireshark. Filter by dns.flags.response == 1. Add DNS -> Queries -> query -> Name as a Column. Export Packet Dissections as CSV. And use Excel and Notepad to get a file of 50.000 DNS names in that format. Upload that file to the newly created dnsperf Pod. Create a separate file with only cluster.local domains as well (cat domains-all.txt | grep "cluster.local" > domains-cluster-local.txt).

dnsperf does not expand/multiply domains by adding the domains from search-path in /etc/resolve.conf. The list we extracted from the packet capture already have these additional lookups though so it is representative nonetheless.

Run dnsperf:

dnsperf -d domains-all.txt -l 60 -Q 100 -S 5 -t 2 -s 172.18.0.10

Explanation of arguments:

-d domains-all.txt - Domain list file.
-l 60 - Length (duration) of test in seconds.
-Q 100 - Queries per second.
-S 5 - Print statistics every 5 seconds.
-t 2 - Timeout 2 seconds.
-s 172.18.0.18 - DNS server to query

Each test has different values for the arguments.

Results from running a series of tests ranging from 100 Queries per second (QPS) to 1500 QPS:

These results are from a second round of testing a couple of weeks after the first. On the first round I observed a lot (6%) timing out on just 200 QPS directly towards a kube-dns Pod. I’m not sure why the results are suddenly much better. Looking at the graphs from node-local-dns there is a significant improvement overall some days before the second round of testing. We have not done any changes that could explain the sudden improvement. I guess it’s just one of those things…

CoreDNS did a benchmark of kube-dns and CoreDNS and managed to get ~36.000 QPS on internal names and ~2.200 QPS on external names on kube-dns.

Appendix - Network packet capture without dependencies

I haven’t done packet capture in a Kubernetes cluster before. First I tried ksniff that this blog post describes. But no dice.

Another method that is much more reliable is running tcpdump in a debug container as mentioned here. He pipes tcpdump from a container directly to wireshark on his own machine. But since I’m using kubectl etc inside Ubuntu on WSL2 on Windows that is probably going to require much more setup.

Instead I’m opting for just saving packet captures in the container as files and copy them to a directory on my machine accessible by Windows.

# Spin up a new container using the ubuntu:22.04 image in the existing node-local-dns-7vqpp Pod. Try to attach to the process-namespace of the node-cache container if possible.
kubectl debug -c debug -n kube-system -it node-local-dns-7vqpp --image=ubuntu:22.04 --target=node-cache

As soon as the image is downloaded and container started your console should give you a new shell inside the container where we can start preparing:

# Update apt and install tshark. Enter "no" for "Should non-superusers be able to capture packets?"
apt-get update && apt-get install -y tshark

# Capture 100 packets from eth0 to verify that things are working
tshark -i eth0 -n -c 100

At first I captured 1 million packets without any filters. That resulted in a 6GB file which may be a bit on the large side when I’m just interested in some DNS lookups. So lets add a capture filter port 53 going forward.

Capturing DNS packets to and from a node-local-dns Pod:

# In the shell running in the debug container with tshark installed:
date; tshark -i eth0 -n -c 1000000 -f "port 53" -w node-local-dns-7vqpp-eth0.pcap

After either capturing 100.000 packets or I’m happy I stop the capture with Ctrl+C and I can download the pcap file in WSL:

# On your local machine:
kubectl cp -c debug kube-system/node-local-dns-7vqpp:node-local-dns-7vqpp-eth0.pcap /mnt/c/Data/node-local-dns-7vqpp-eth0.pcap

Then browse to C:\Data\ where you can open node-local-dns-7vqpp-eth0.pcap in Wireshark.

I print the current time on the Pod just before starting packet capture. Most of the time the correct UTC time will be recorded on the packets. But in case it isn’t and the time starts from 0, I can use that time to adjust the offset at least roughly. Making it easier to correlate packet captures from different Pods.

To capture on the receiving kube-dns Pod, both incoming DNS traffic to the dnsmasq container on port 53 on eth0, and between dnsmasq and kube-dns containers on lo port 10053:

kubectl debug -c debug -n kube-system -it kube-dns-d7bc86d4c-d2x8p --image=ubuntu:22.04 --target=dnsmasq

# Install tshark as shown above

# Start two packet captures running in the background:
date; tshark -i eth0 -n -c 1000000 -f "port 53" -w kube-dns-d7bc86d4c-d2x8p-eth0.pcap &
date; tshark -i lo -n -c 1000000 -f "port 10053" -w kube-dns-d7bc86d4c-d2x8p-lo.pcap &

# To stop these type `fg` in the shell to bring a background process to the foreground and stop it with `Ctrl+C`.
# Then `fg` and `Ctrl+C` again to stop the other.

Download the files to my local machine as before:

kubectl cp -c debug kube-system/kube-dns-d7bc86d4c-d2x8p:kube-dns-d7bc86d4c-d2x8p-eth0.pcap /mnt/c/Data/kube-dns-d7bc86d4c-d2x8p-eth0.pcap
kubectl cp -c debug kube-system/kube-dns-d7bc86d4c-d2x8p:kube-dns-d7bc86d4c-d2x8p-lo.pcap /mnt/c/Data/kube-dns-d7bc86d4c-d2x8p-lo.pcap

Check out Analyzing DNS problems based on packet captures for some tips and tricks on analyzing the packet captures.

Appendix - Verbose logging on `kube-dns`

The dnsmasq man page lists several interesting options such as --log-queries=extra and --log-debug!

But it’s not possible to make changes to the kube-dns Deployment since it’s managed by GKE. Any changes you make will be reverted immediately.

Instead we take the existing manifests for kube-dns, modify them and create a parallel deployment that we control:

kube-dns-deployment.yaml

Deployment named kube-dns-debug.
A couple of annotations commented out that would otherwise make the Deployment be instantly removed.
Keep existing k8s-app: kube-dns label on the Pods so they will receive traffic for the kube-dns-upstream Service.
An additional reason: debug label on the Pods so we can target them specifically.
Additional logging enabled with --log-queries=extra, --log-debug and --log-async=25. I enabled these one at a time but the log volume with everything enabled isn’t overwhelming.

kube-dns-debug-service.yaml

Service named kube-dns-debug with port udp+tcp/53 targeting only the Pods with the added reason: debug label. I used this service to run load tests only towards these Pods.

kube-dns-metrics-service.yaml

Service named kube-dns-debug-metrics with port tcp/10054 and tcp/10055 targeting the same Pods as above but for exposing metrics.

kube-dns-servicemonitor.yaml

ServiceMonitor named kube-dns-debug that targets the kube-dns-debug-metrics Service above.

Strictly speaking the kube-dns-debug Service, kube-dns-debug-metrics Service and kube-dns-debug ServiceMonitor isn’t necessary if using the k8s-app: kube-dns label on the new Pods. But it makes it possible to separate the two deployments completely by using another label on the Pods such as k8s-app: kube-dns-debug and thus avoid for example load tests affecting real cluster DNS traffic.

Now some DNS requests should arrive at the kube-dns-debug Pods with additional logging enabled.

It’s also possible to force all DNS traffic to this Pod by manually adding the reason: debug label as a selector on the kube-dns-upstream Service. It will stay that way for “a while” (hours, maybe days) before being reverted. Plenty of time to play around at least.

Appendix - Analyzing `dnsmasq` logs

Once we have increased the log verbosity of dnsmasq we can see if there’s anything to learn there.

First I used our updated CoreDNS dashboard to identify a time interval where we observed latency spikes. Then using loki, our centralized log storage, I download about 5000 lines of logs spanning about 50 seconds worth of logs. (Our loki is limited to loading 5000 lines, therefore it’s important to try to narrow down and find logs where we are actually experiencing issues).

2023-10-10T10:02:36+02:00 I1010 08:02:36.571446   1 nanny.go:146] dnsmasq[4811]: 315207 10.0.0.83/56382 forwarded metadata.google.internal.cluster.local to 127.0.0.1#10053
2023-10-10T10:02:36+02:00 I1010 08:02:36.571457   1 nanny.go:146] dnsmasq[4811]: 315207 10.0.0.83/56382 reply metadata.google.internal.cluster.local is NXDOMAIN
2023-10-10T10:02:36+02:00 I1010 08:02:36.571488   1 nanny.go:146] dnsmasq[4810]: 315107 10.0.0.83/24595 forwarded metadata.google.internal.cluster.local to 127.0.0.1#10053
2023-10-10T10:02:36+02:00 I1010 08:02:36.571495   1 nanny.go:146] dnsmasq[4810]: 315107 10.0.0.83/24595 reply metadata.google.internal.cluster.local is NXDOMAIN
2023-10-10T10:02:36+02:00 I1010 08:02:36.575810   1 nanny.go:146] dnsmasq[4810]: 315108 10.0.0.83/24595 query[A] metadata.google.internal.some-namespace.svc.cluster.local from 10.0.0.83
2023-10-10T10:02:36+02:00 I1010 08:02:36.576101   1 nanny.go:146] dnsmasq[4810]: 315108 10.0.0.83/24595 forwarded metadata.google.internal.some-namespace.svc.cluster.local to 127.0.0.1#10053
2023-10-10T10:02:36+02:00 I1010 08:02:36.576132   1 nanny.go:146] dnsmasq[4810]: 315108 10.0.0.83/24595 reply metadata.google.internal.some-namespace.svc.cluster.local is NXDOMAIN

Here we can see exact time (08:02:36.571446). Which dnsmasq process logged the line, identified by the process ID (dnsmasq[4811]). A number identifying an individual request (315207). And the client IP and port (10.0.0.83/56382). One process always maps to one TCP connection so PID and client IP and port will always match.

Just from this snippet we can see that multiple DNS requests are handled by each process and we have exact time for when individual requests were received from the client as well as exact time for replies.

I wrote a small (and ugly) program to analyze the logs. It gathers the duration of each request identified by the request number and each “session” identified by the process ID.

It filters out requests faster than 2 milliseconds and sessions shorter than 500 milliseconds. The output looks like:

Request: PID 5439 RequestNumber 356558 Domain . Duration 2.273ms
Request: PID 4914 RequestNumber 319022 Domain api.statuspage.io.some-namespace.svc.cluster.local Duration 2.619ms
Request: PID 4915 RequestNumber 319123 Domain api.statuspage.io.cluster.local Duration 3.088ms
Session: PID 4890 Duration 1.005229s
Session: PID 5022 Duration 852.861ms
Session: PID 5435 Duration 2.627174s

The vast majority of individual requests complete in microseconds, and none are slower than 10 milliseconds. This is an indication that the delays aren’t coming from the processing of individual requests.

Sessions however last much longer, regularly in the 1-3 second range. This isn’t necessarily a problem since it’s resource efficient to keep sessions longer and re-using them for many requests.

Appendix - Analyzing concurrent TCP connections

I want to see how many connections are open at any point (and not impacted by metric scraping intervals etc) as well as how long they tend to stay idle before being closed.

I made another small (and probably even uglier) program to analyze the packet captures (exported from Wireshark as CSV) from kube-dns and try to answer those questions.

While iterating through every packet it keeps a counter on how many distinct connections are observed as well as the time since the previous packet in the same connection was observed:

Time: 08:08:30.610010781 TimeShort: 08:08:30 Port: 61236 Action: open Result: none Flags: PSHACK IdleGroup: fast ConIdleTime: 564.26µs ActiveConnections: 11
Time: 08:08:30.610206151 TimeShort: 08:08:30 Port: 10806 Action: open Result: none Flags: ACK IdleGroup: fast ConIdleTime: 178.4µs ActiveConnections: 11
Time: 08:08:30.900267009 TimeShort: 08:08:30 Port: 62083 Action: open Result: none Flags: FINACK IdleGroup: slow ConIdleTime: 6.429595796s ActiveConnections: 11

Plotting ActiveConnections in the Excel graph we have from Analyzing DNS problems based on packet captures:

Up to the limit of 20 dnsmasq processes they follow the number of open TCP connections pretty closely. However for long periods of time there are way more open connections than dnsmasq is allowed to spawn new child processes to handle. This also overlaps with the sudden huge increases in latency. Another thing we can infer from this is that new TCP connections are successfully being opened from node-local-dns (CoreDNS) to dnsmasq, even though dnsmasq is unable to handle them yet. Probably the master dnsmasq process accepts the connections but blocking the request until there is room to spawn a new child process.

greping for “slow” we get all packets where the connection was idle for more than 1 second:

Time: 08:08:18.064874738 TimeShort: 08:08:18 Port: 19956 Action: open Result: none Flags: FINACK IdleGroup: slow ConIdleTime: 7.535928188s ActiveConnections: 9
Time: 08:08:18.064888758 TimeShort: 08:08:18 Port: 30168 Action: open Result: none Flags: FINACK IdleGroup: slow ConIdleTime: 7.577724318s ActiveConnections: 9
Time: 08:08:18.064909758 TimeShort: 08:08:18 Port: 41718 Action: open Result: none Flags: FINACK IdleGroup: slow ConIdleTime: 7.393646609s ActiveConnections: 9
Time: 08:08:18.064911088 TimeShort: 08:08:18 Port: 30386 Action: open Result: none Flags: FINACK IdleGroup: slow ConIdleTime: 7.535962768s ActiveConnections: 9
Time: 08:08:18.064920468 TimeShort: 08:08:18 Port: 21482 Action: open Result: none Flags: FINACK IdleGroup: slow ConIdleTime: 7.535946008s ActiveConnections: 9

This is particularly interesting since node-local-dns is configured to expire connections after 1 second. And in theory should be cleaned up after at most 2 seconds. I managed to keep myself from diving head first into that particular rabbit hole though.

The underlying dns Go library that CoreDNS uses also has a limit of 128 DNS queries for a single TCP connection before closing it.

Appendix - Analyzing DNS problems based on packet captures

Now that we have some packet captures we can start dissecting and analyzing and looking for the needle in the haystack.

I’ll be using Wireshark for this.

This youtube video shows a few neat tricks. Thanks!

If you are looking at the traffic between dnsmasq and kube-dns (lo network interface) go to Analyze -> Decode as and add a mapping from port 10053 to DNS to have Wireshark decode the packets correctly.
Start by applying the display filter dns.flags.response == 1 to only show DNS responses.
Find a DNS Response packet and find the [Time: n.nnn seconds] field, right click it and Apply as Column.
You can also add Transaction ID and the Name field (under Queries) as columns as well.

Then you can export as CSV for example in File -> Export Packet Dissections… -> As CSV.

Importing the data from the node-local-dns packet capture into Excel (yes, there’s no way escaping Excel!) and plotting the duration of each and every DNS lookup over a 20 minute period, colored by upstream server:

We clearly see huge spikes in latency. But it seems to only affect queries being forwarded to the two kube-dns Pods (172.20.1.38 & 172.20.7.57).

Another interesting finding is that it appears to happen at the exact same time on both kube-dns Pods. Weird. If we didn’t already know that (for at least some queries) the added duration happens inside the dnsmasq container, I would probably suspect a problem on the network.

Plotting the duration on requests arriving at dnsmasq on one of the kube-dns Pods shows the same pattern:

Another thing I noticed is that sometimes close to when request duration would spike, Wireshark warns about TCP Port numbers reused:

42878	19:09:03.987437048	127.0.0.1	127.0.0.1	TCP	78	44516	[TCP Port numbers reused] 44516 → 10053 [SYN] Seq=0 Win=43690 Len=0

However Wireshark doesn’t discriminate whether that was 20 minutes or 2 seconds ago. Only that it occurs in the same packet capture.

One hypothesis I had was that outgoing requests from dnsmasq to kube-dns would be stuck waiting for available TCP ports. I plotted the source port usage over time:

It did not strengthen my suspicion and some random checking shows that the TCP Port reuse is far enough spaced in time (minutes) to avoid problems. So for the time being I’m not digging further into this.

Footnotes

[A] This doesn’t strictly mean that 20 new TCP connections are required since many of them are probably retries.

[B] Note that even if the graph doesn’t hit 22 you may still be affected. The number of processes is counted only at the exact time the metrics are scraped, in our case 10 seconds. You can manually sample the process count by attaching a debug container to dnsmasq (kubectl debug -c debug -n kube-system -it kube-dns-6fb7c8866c-bxj7f --image=ubuntu:22.04 --target=dnsmasq) and running for i in $(seq 1 1800) ; do echo "$(date) Try: ${i} DnsmasqProcess: $(pidof dnsmasq | wc -w)"; sleep 1; done.

[C] CPU throttling would not be an issue in this case since kube-dns does not have CPU limits set.

[D] Although you should be cautious of calling services by name in other namespaces if they are owned by different teams. As that introduces coupling on what should be implementation details across team boundaries. In Signicat we always call the full FQDN and path if connecting to services owned by other teams.

Kubernetes Sidecar Config Drift

Tue, 03 May 2022 00:00:00 GMT

When using a Sidecar Injector (such as Istio), there is nothing that ensures that an update (potentially breaking) to a sidecar config template is applied/updated on Pods that have already been injected with a sidecar. This post describes the causes of this problem, as well as introducing a tool to mitigate it.

This is a cross-post of a blog post also published on the Signicat Blog

Huge thanks to one of my favorite clients, Signicat, and especially Jon, for allowing me to share some of the nitty gritty details of a challenge that I believe is probably quite widespread, yet under-appreciated, in modern Kubernetes cloud environments.

Last week, working on Signicat’s next generation cloud platform, I discovered that several individuals invented their own ways of mitigating what I now call Sidecar Configuration Drift.

To ease the pain I created k8s-sidecar-rollout to restart the required workloads and thereby updating their sidecar configurations.

This blog post is a bit of background information on what the causes of this problem is.

What is Sidecar Config Drift?

This means that after updating a sidecar config it may take a very long time until all Pods have the updated config. They may receive the updates at any undetermined time in the future. While the updates are pending things might not work as expected and things may not be compliant. When the update is finally applied to the Pod it may surface breaking changes.

I call this phenomena Sidecar Configuration Drift.

Background

Kubernetes - Declarative vs imperative

What makes Kubernetes so powerful is also what can make it hard and confusing to work with until your mindset has shifted.

That is the philosophy of being declarative instead of imperative like most of us are used to for the past 20 years.

Declarative means you tell Kubernetes HOW you want things to look. You DON’T tell Kubernetes WHAT to do.

For example you do not tell Kubernetes to scale your Deployment to 5 instances. You tell Kubernetes that your Deployment should have 5 instances. The difference is subtle but extremely important. In a highly dynamic cloud environment your instances might disappear or crash for a multitude of reasons. In this declarative mindset you should not care about that since Kubernetes is tasked with ensuring 5 instances and will (try to) provision new ones when it’s needed.

This is the apparent magic which makes Kubernetes.

This magic is technically solved with what we call Controllers. A controller is responsible for constantly comparing the Actual state and Desired state. To scale your Deployment to 5 instances you set replicas: 5 on the Deployment resource. If needed a controller will create a completely new ReplicaSet resource with 5 instances. Another controller will then create 5 Pods. And the scheduler will finally try to place and start those Pods on actual nodes. The ReplicaSet controller will scale down the old once the new has reached it’s desired state.

This constant process is called a reconciliation loop and it’s a critical feature.

Sidecars

Sidecar is a Kubernetes design pattern. A sidecar is simply a container that is living side-by-side with the main application container that can do tasks that are logically not part of the application. Examples of this can be log handling, monitoring agents, proxies. All containers in a Pod (including sidecars) share the same filesystem, kernel namespace, IP addresses etc.

More about sidecars: The Distributed System ToolKit: Patterns for Composite Containers

Sidecar injection

Sidecar injection is when a sidecar container is added to a Pod even if the sidecar isn’t defined in any of the higher level primitives, such as Deployment or StatefulSet.

When for example the ReplicaSet controller will Create a new Pod. If configured, the Kubernetes API will call one or more MutatingWebhooks. These webhooks can then change the Pod definition before they are saved (and picked up by the scheduler).

The Problem

The problem is when updating a sidecar injection template there is no system that runs a reconciliation loop.

The webhook just updates the Pod template. It does not keep track of which Pods have gotten which template or check if any template change would result in a different sidecar configuration.

The controllers ALSO does not continuously monitor if and how re-creating the same Pod (without sidecars) would result in a different Pod once the sidecars have been injected.

In effect sidecar injection does not follow the expected declarative pattern that the rest of Kubernetes does.

Consequences

If a platform team changes the istio sidecar template it will not actually take effect on a Pod until that Pod for some reason is re-created.

Let’s assume the istio-proxy sidecar template have been updated by the platform team. We roll it out and test it and it seems to work. But the change will break some applications running in the cluster.

That breakage will go un-noticed until:

The product team commits changes that triggers a re-deploy. The deployment will suddenly fail but it might not have anything to do with the actual changes the team did to the application. This is surely confusing!

The platform team for example upgrades a pool of worker nodes causing all `Pods` to be re-created on new nodes.

A Pod is re-created when a Kubernetes worker node crashes. In this scenario it appears the failure spawned into existence out of nowhere since neither the product team nor platform team actually “did” anything to trigger it.

Also worth noting is that any attempts at Rolling back a Deployment now containing failing Pods will not actually fix anything.

It’s the sidecar templates that needs to be rolled back and Pods probably need to be re-created again.

Mitigations

We can mitigate drift by:

Re-starting all Pods in the cluster whenever we update sidecar injection templates.

Sometimes we might forget to re-start so regularly re-start all Pods in the cluster anyway.

k8s-sidecar-rollout

To make these restarts easy and fast I’ve created https://github.com/StianOvrevage/k8s-sidecar-rollout .

It’s a tool that figures out (with your help) which workloads (Deployment, StatefulSet, DaemonSet) that needs to be rolled out again (re-started) and then rolls out for you. Head over to the GitHub repo for installation and complete usage instructions. Here is an example of how it can be used:

python3 sidecar-rollout.py \
    --sidecar-container-name=istio-proxy \
    --include-daemonset=true \
    --annotation-prefix=myCompany \
    --parallel-rollouts 10 \
    --only-started-before="2022-05-01 13:00" \
    --exclude-namespace=kube-system \
    --confirm=true

This will gather all Pods with a container named istio-sidecar belonging to a Deployment or DaemonSet that was started before 2022-05-01 13:00 (which may be when we updated the istio sidecar config template) excluding the kube-system namespace. It will patch the workloads with two annotations with myCompany prefix and run 10 rollouts in parallel.

The script that now re-starts Pods adds two annotations indicating that a restart to update sidecars has occurred as well as the time:

$ kubectl get pods -n product-team some-api-7cdc65482b-ged13 -o yaml | yq '.metadata.annotations'
    sidecarRollout.rollout.timestamp: 2022-05-03T18:05:31
    sidecarRollout.rollout.reason: Update sidecars istio-proxy

The idea is that if your Pods are suddenly failing, you can quickly check the annotations and see if it has anything to do with sidecar updates or not.

These annotations will of course disappear again when a Deployment is updated.

Yak shaving - Photo drips for my mom

Sun, 13 Feb 2022 00:00:00 GMT

TL;DR: I finally organized my photo archive and in an evening created a service to e-mail my mom a photo from the last 20 years every morning.

Yak shaving - Photo drips for my mom

Update: Check out https://github.com/StianOvrevage/photo-drips for ugly but working code.

Update May 2022: My mom told me this week that I must NEVER stop sending these daily photos <3

Background

TL;DR: I finally organized my photo archive and in an evening created a service to e-mail my mom a photo from the last 20 years every morning.

This started out a few weeks ago when I decided to reinstall Windows on my laptop.

The first hurdle was that my 2-3-4 different cloud storage subscriptions were all overdue for a clean-up. The one best suited had me throttled to 1Mbit/s since I was storing 20TB+ of data.

I’ve been taking a lot of photos and videos since I got my first digital camera 22 years ago. Even though I use Lightroom to keep some order there was some duplication and discontinuity. So (after upgrading my fibreoptic internet to 1Gbit, optimizing my home network and cleaning up space on my machine) I started to clean up and organize the various catalogues.

Looking at memories from 20 years ago made me realize I’m better at taking photos than “utilizing” them afterwards. What really is the point, then? I took a picture of one on the screen with my phone and sent on snapchat to my mom, not thinking much about it. But she was really thrilled, which made me really happy as well.

For my current client I’m nearing the end of my contract and for the last two months I’ve mainly been maintaining, documenting and handing over and I really miss building things and solving problems.

Recently a potential client asked about Python and AWS Lambda competence. It’s not something I work with daily and it’s not on my CV. But it made me think about all the various languages and tools I’ve used during the last decade.

My subconscious brain offers up an idea on how to a) build something b) brush up some Python and Lambda knowledge and c) brighten my moms day.

Concept

Every morning a photo from my archives is e-mailed to my mom, a photo drip.

There is also a gallery where she can look at the previous photo drips.

Process and goals

The primary objective was to have a working prototype as quickly as possible. So no premature optimization, refactoring or anything.

Photo selection and preparation

At around 3pm I started browsing photos from year 2000. It took me about 75 minutes to pick out about 300 photos from the first 20.000.

Tip: When browsing in Lightroom, press B to add to Quick Collection.

Export the pictures in the quick collection with a custom filename format like this 0003-2000-05-14.jpg.

This format ensures filenames are ordered from oldest to newest, and the date can later be extracted directly from the filename without doing any EXIF stuff.

Photos resized to 1500x1500. Never enlarge. Sharpen for screen.

Photo storage

It would have been quicker to set up a AWS EC2 virtual machine running all the components but that would be too easy.

The requirements for storage is: cheap, reliable, publicly available. So a standard AWS S3 bucket should do just fine.

Using the browser I upload all the photos in batch to a new bucket. In a sub-folder with a random name. That should provide an appropriate level of security and avoid strangers on the internet stumbling upon it. Since the bucket is publicly available.

I verify that I can load the pictures in my browser with the public S3 URL. It took a few attempts at getting the permissions right, and I suspect adding this policy was required even though everything in the settings was set to “Full public access”.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::memorydrops/*"
            ]
        }
    ]
}

Sending the e-mail

Sending e-mails directly with SMTP is really not an option anymore because of all the spam blocking systems. I know we need to use a third party service or something.

I first looked at AWS SES (simple-email-service). But nothing about it seemed simple. Then looked at a few of the stablished ones such as SendGrid and Mailgun. These tools have evolved a lot and now has a plethora of features aimed at marketing, transactional e-mails etc. Probably overkill and potentially time consuming while having no transferable knowledge or code when hitting a dead-end since they are all different APIs.

What about just using my own Gmail SMTP to ship?

The first tutorials about that required turning on “Less secure app access” for my account.

Not accepting that trade-off, I found https://levelup.gitconnected.com/an-alternative-way-to-send-emails-in-python-5630a7efbe84 where I learned that I can generate an “App password”, similar to an API Key for specific Google applications without disabling other security features. Jackpot!

The article also has working code that I shamelessly used as a starting point.

First I attached the photo of the day, but I really want it embedded. Even though I don’t really like linking images on a public URL (S3 bucket) because of potential browser and client issues that’s what I ended up with. The alternative of base64 encoding the data inline seemed like a chore. Besides I know mom uses Gmail in Chrome anyway so if it worked for me it would probably work for her.

I don’t want the system to be dependent on any state management, databases, etc.

Picking the right photo each day is simply select file N, where N is the days since the first deployment.

Ideally I would use AWS S3 API to list the contents of the bucket to get the available files. But to save some time the “photo index” is simply a text file listing the files:

ls > filelist.txt

I create a new Lambda function and paste the python script and filelist.txt directly in the Lambda code editor and deploy and test it. Working!

The Lambda function is the configured with a EventBridge trigger with the schedule cron(0 5 * * ? *) that should trigger the function at 0500 UTC every day.

Gallery

I also want to have a gallery where she can view the previous memory drops without having to shuffle through endless emails.

Started out by looking at VueJS for the frontend, which I have used before. But I have not used the new version yet and I suspected it might take a long time getting a project set up from scratch since unfortunately tutorials, tips, documentation in the frontend / JavaScript world has a tendency to be chronically outdated and unreliable.

Dropped that and opted for a very simple native JS photo gallery called zoomwall.js.

Only showing a selection of photos depending on the day however requires a bit more engineering. (Writing this I realize another acceptable approach would be to use native JS to manipulate the DOM directly.)

I implemented this by inlining the CSS and JS into one template HTML file that is rendered on-demand in a Python Lambda function. Then using AWS API Gateway to expose it as a normal webserver.

I wanted to use Jinja2 for templating instead of the built-in Python templating functions. Doing that causes some headache since the Lambda environment does not have Jinja2 installed.

Luckily creating a custom deployment package (.zip) including dependencies is rather trivial.

Conclusion

All in all this project was started with filtering photos at 3pm, having dinner from 5pm to 6pm and by 8.30pm everything was deployed. The following morning I had the memory drop in my inbox to great delight. An unexpected benefit of using my own Gmail for sending the e-mail is that mom can reply directly.

Expected costs

S3 offers 5GB of standard storage for free for 12 months. After that I expect the cost to be in the $0.x range per month.

API Gateway offers 1 Million API calls per month for free for 12 months. After that I expect the cost to be in the $0.x range per month.

Lambda offers 1 Million requests per month forever.

Performance has been sacrificed for the gallery to make it simple. Server-side rendering is not going to be as fast as client-side. Python is not the fastest alternative. Using a FaaS such as Lambda also introduces penalties and unknowns (cold starts, etc). Yet the gallery HTML loads in ~200ms and and seems instant.

Improvements

After deciding to share the code I spent an hour or two writing this document as well as some necessary clean-up and changes from the prototype.

The filelist.txt is not bundled with the code. It’s hosted in the S3 bucket along with the photos. That means I can update and add photos later without touching code.

That requires the requests package, so now both modules are packaged with dependencies and uploaded via AWS CLI instead of browser.

Some hard-coded URLs etc have been converted to environment variables.

Added the exif Python package to extract the original time and date of the photo to include in the e-mail.

Added an URL redirect from a prettier domain to the auto generated API Gateway hostname of the gallery. Did not bother with proper custom domain since that requires a lot of fiddling with SSL certificates.

Bugs and future improvements

The JS gallery seems slightly broken. Might replace with a better one.
The e-mail HTML is not pretty.
Make sender and recipient e-mails environment variables (but recipient is currently a Python list, so).
Make start date env var. Requires parsing date from user.
The usual: Check that required env vars are set on startup. Improve error handling and logging.

A side quest in API development, observability, Kubernetes and cloud with a hint of database

Sat, 06 Mar 2021 00:00:00 GMT

Quite often people ask me what I actually do. I have a hard time giving a short answer. Even to colleagues and friends in the industry. Here I will try to show and tell how I spent an evening digging around in a system I helped build for a client.

Quite often people ask me what I actually do. I have a hard time giving a short answer. Even to colleagues and friends in the industry.

Here I will try to show and tell how I spent an evening digging around in a system I helped build for a client.

Table of contents

Background
The (initial) problem
Side quest: Database optimizations
Determining the next bottleneck
Side quest: Cluster resources and burstable VMs
Conclusion

Background

I’m a consultant doing development, DevOps and cloud infrastructure.

For this specific client I mainly develop APIs using Golang to support new products and features as well as various exporting, importing and processing of data in the background.

I’m also the “ops” guy handling everything in AWS, setting up and maintaing databases, making sure the “DevOps” works and the frontend and analytics people can do their work with little friction. 99% of the time things work just fine. No data is lost. The systems very rarely have unforeseen downtime and the users can access the data they want with acceptable latency rarely exceeding 500ms.

A couple of times a year I assess the status of the architecture and set up new environments from scratch and update any documentation that has drifted. This is also a good time to do changes and add or remove constraints in anticipation of future business needs.

In short, the current tech stack that has evolved over a couple of years is:

Everything hosted on Amazon Web Services (AWS).
AWS managed Elastic Kubernetes Service (EKS) currently on K8s 1.18.
GitHub Actions for building Docker images for frontends, backends and other systems.
AWS Elastic Container Registry for storing Docker images.
Deployment of each system defined as a Helm chart alongside source code.
Actual environment configuration (Helm values) stored in repo along source code. Updated by GitHub Actions.
ArgoCD in cluster to manage status of all environments and deployments. Development environments usually automatically deployed on change. Push a button to deploy to Production.
Prometheus for storing metrics from the cluster and nodes itself as well as custom metrics for our own systems.
Loki for storing logs. Makes it easier to retrieve logs from past Pods and aggregate across multiple Pods.
Elastic APM server for tracing.
Pyroscope for live CPU profiling/tracing of Go applications.
Betteruptime.com for tracking uptime and hosting status pages.

I might write up a longer post about the details if anyone is interested.

The (initial) problem

A week ago I upgraded our API from version 1, that was deployed in January, to version 2 with new features and better architecture.

One of the endpoints of the API returns an analysis of an object we track. I have previously reduced the amount of database queries by 90% but it still requires about 50 database calls from three different databases. Getting and analyzing the data usually completes in about 3-400 milliseconds returning an 11.000 line JSON.

It’s also possible to just call /objects/analysis to get the analysis for all the 500 objects we are tracking. It takes 20 seconds but is meant for exports to other processes and not interactive use, so not a problem.

Since the product is under very active development the frontend guys just download the whole analysis for an object to show certain relevant information to users. It’s too early to decide on which information is needed more often and how to optimize for that. Not a problem.

So we need an overview of some fields from multiple objects in a dashboard / list. We can easily pull analysis from 20 objects without any noticable delay.

But what if we just want to show more, 50? 200? 500? The frontend already have the IDs for all the objects and fetches them from /objects/id/analysis. So they loop over the IDs and fire of requests simultaneously.

Analyzing the network waterfall in Chrome DevTools indicated that the requests now took 20-30 seconds to complete! But looking closer most of the time they were actually queued up in the browser. This is because Chrome only allows 6 concurrent TCP connection to the same origin when using HTTP1 (https://developers.google.com/web/tools/chrome-devtools/network/understanding-resource-timing).

Fixing the (initial) problem

HTTP2 should fix this problem easily. By default HTTP2 is disabled in nginx-ingress. I add a couple of lines enabling it and update the Helm deployment of the ingress controller.

Verifying the (initial) fix

Some common development tools doesn’t support HTTP2, such as Postman. So I found h2load which can both help me verify HTTP2 is working and I also get to measure the improvement, nice!

Note that I’m not using the analysis endpoint since I want to measure the change from HTTP1 to HTTP2 and it will become apparent later that there are other bottlenecks preventing us from a linear performance increase when just changing from HTTP1 to HTTP2.

Also note that this is somewhat naive since it requests the same URL over and over which can give false results due to any caching. But fortunately we don’t do any caching yet.

Baseline simple request - HTTP1 1 connections, 20000 requests

Using 1 concurrent streams, 1 client and HTTP1 I get an estimate of performance pre-http2:

h2load --h1 --requests=20000 --clients=1 --max-concurrent-streams=1 https://api.x.com/api/v1/objects/1

The results are as expected:

finished in 1138.99s, 17.56 req/s, 18.41KB/s
requests: 20000 total, 20000 started, 20000 done, 19995 succeeded, 5 failed, 0 errored, 0 timeout

For http2 we set max concurrent streams to the same as number of requests:

h2load --requests=200 --clients=1 --max-concurrent-streams=200 https://api.x.com/api/v1/objects/1

Which results in almost half the latency:

finished in 1.23s, 162.65 req/s, 158.06KB/s
requests: 200 total, 200 started, 200 done, 200 succeeded, 0 failed, 0 errored, 0 timeout

So HTTP2 is working and providing significant latency improvements. Success!

Baseline complex request - HTTP1 1 connections, 20000 requests

We start by establishing a baseline with 1 connection querying over and over.

h2load --h1 --requests=20000 --clients=1 --max-concurrent-streams=1

Verifying the fix for assumed workload

So we verified that HTTP2 gives us a performance boost. But what happens when we fire away 500 requests to the much heavier /analysis endpoint?

These graphs are not as pretty since the ones above. This is mainly due to the sampling interval of the metrics and that we need several datapoints to accurately determine the rate() of a counter.

Complex request - HTTP1 6 connections, 500 requests

finished in 32.25s, 14.88 req/s, 2.29MB/s
requests: 500 total, 500 started, 500 done, 500 succeeded, 0 failed, 0 errored, 0 timeout

In summary it so far seems to scale linearly with load. Most of the time is spent fetching data from the database. Still very predictable low latency on database queries and the resulting HTTP response.

Complex request - HTTP2 500 “connections”, 500 requests

So now we unleash the beast. Firing all 500 requests at the same time.

finished in 16.66s, 30.02 req/s, 3.55MB/s
requests: 500 total, 500 started, 500 done, 500 succeeded, 0 failed, 0 errored, 0 timeout

Important about Kubernetes and CPU limits
Even with CPU limits set to 1 (100% of one CPU), your container can still be throttled at much lower CPU usage. Check out this article for more information.

In an ideal world all 500 requests should start and complete in 2-300ms regardless. Since that is not happening it’s an indication that we are now hitting some other bottleneck.

Looking at the graphs it seems we are starting to saturate the database. The latency for every request is now largely dependent on the slowest of the 10-12 database queries it depends on. And as we are stressing the database the probability of slow queries increase. The latency for the whole process of fetching 500 requests are again largely dependent on the slowest requests.

So this optimization gives on average better performance, but more variability of the individual requests, when the system is under heavy load.

Side quest: Database optimizations

It seems we are saturating the database. Before throwing more money at the problem (by increasing database size) I like to know what the bottlenecks are. Looking at the traces from APM I see one query that is consistently taking 10x longer than the rest. I also confirm this in the AWS RDS Performance Insights that show the top SQL queries by load.

When designing the database schema I came up with the idea of having immutability for certain data types. So instead of overwriting row with ID 1, we add a row with ID 1 Revision 2. Now we have the history of who did what to the data and can easily track changes and roll back if needed. The most common use case is just fetching the last revision. So for simplicity I created a PostgreSQL view that only shows the last revision. That way clients don’t have to worry about the existense of revisions at all. That is now just an implementation detail.

When it comes to performance that turns out to be an important implementation detail. The view is using SELECT DISTINCT ON (id) ... ORDER BY id, revision DESC. However many of the queries to the view is ordering the returned data by time, and expect the data returned from database to already be ordered chronologically. Using EXPLAIN ANALYZE on the queries this always results in a full table scan instead of using indexes, and is what’s causing this specific query to be slow. Without going into details it seems there is no simple and efficient way of having a view with the last revision and query that for a subset of rows ordered again by time.

For the forseable future this does not actually impact real world usage. It’s only apparent under artificially large loads under the worst conditions. But now we know where we need to refactor things if performance actually becomes a problem.

Determining the next bottleneck

Whenever I fix one problem I like to know where, how and when the next problem or limit is likely to appear. When increasing the number of requests and streams I expected to see increasing latency. But instead I see errors appear like a cliff:

finished in 27.33s, 36.59 req/s, 5.64MB/s
requests: 5000 total, 1002 started, 1002 done, 998 succeeded, 4002 failed, 4000 errored, 0 timeout

Consulting the logs for both the nginx load balancer and the API there are no records of failing requests. Since nginx does not pass the HTTP2 connection directly to the API, but instead “unbundles” them into HTTP1 requests I suspect there might be issues with connection limits or even available ports from nginx to the API. But maybe it’s a configuration issue. By default nginx does not limit the number of connections to a backend (our API). . But, there is actually a default limit to the number of HTTP2 requests that can be served over a single connection - And it happens to be 1000.

I leave it at that. It’s very unlikely we’ll be hitting these limits any time soon.

Side quest: Cluster resources and burstable VMs

When load testing the first time around sometimes Grafana would also become unresponsive. That’s usually a bad sign. It might indicate that the underlying infrastructure is also reaching saturation. That is not good since it can impact what should be independent services.

Our Kubernetes cluster is composed of 2x t3a.medium on demand nodes and 2x t3a.medium spot nodes. These VM types are burstable. You can use 20% per vCPU sustained over time without problems. If you exceed those 20% you start consuming CPU credits faster than they are granted and once you run out of CPU credits processes will be forcibly throttled.

Of course Kubernetes does not know about this and expects 1 CPU to actually be 1 CPU. In addition Kubernetes will decide where to place workloads based on their stated resource requirements and limits, and not their actual resource usage.

When looking at the actual metrics two of our nodes are indeed out of CPU credits and being throttled. The sum of factors leading to this is:

We have not yet set resource requests and limits making it harder for Kubernetes to intelligently place workloads
Using burstable nodes having some additional constraints not visible to Kubernetes
Old deployments laying around consuming unnecessary resources
Adding costly features without assessing the overall impact

I have not touched on the last point yet. I started adding Pyroscope to our systems since I simply love monitoring All The Things. The documentation does not go into specifics but emphasizes that it’s “low overhead”. Remember that our budget for CPU usage is actually 40% per node, not 200%. The Pyroscope server itself consumes 10-15% CPU which seems fair. But investigating further the Pyroscope agent also consumes 5-6% CPU per instance. This graph shows the CPU usage of a single Pod before and after turning off Pyroscope profiling.

5-6% CPU overhead on a highly utilized service is probably worth it. But when the baseline CPU usage is 0% CPU and we have multiple services and deployments in different environments we are suddenly using 40-60% CPU on profiling and less than 1% on actual work!

The outcome of this is that we need to separate burstable and stable load deployments. Monitoring and supporting systems are usually more stable resource wise while the actual business systems much more variable, and suitable for burst nodes. In practice we add a node pool of non-burst VMs and use NodeAffinity to stick Prometheus, Pyroscope etc to those nodes. Another benefit of this is that the supporting systems needed to troubleshoot problems are now less likely to be impacted by the problem itself, making troubleshooting much easier.

Conclusion

This whole adventure only took a few hours but resulted in some specific and immediate performance gains. It also highlighted the weakest links in our application, database and infrastructure architecture.

End of 2020 rough database landscape

Fri, 27 Nov 2020 00:00:00 GMT

This post is an attempt to lay out the rough landscape of databases that you might encounter or consider as of late 2020.

There seems to exist a database for every niche, mood or emotion. And they seem to change just as fast.

How do you balance the urge for the new and shiny but without risking too much headache down the road?

This post is an attempt to lay out the rough landscape of databases that you might encounter or consider as of late 2020.

There will be broad generalizations for brevity.

The goal is not to be exhaustive or take all possible precautions. Consider it a starting point for further research and planning.

TLDR: Scroll to the diagrams or view the big picture.

Table of contents

Background
- Project phase overview
Planning
- Database categories
  - SQL
  - NoSQL
  - KeyValue
  - Timeseries
  - Graph
  - Other nice things
The Landscape
- SQL
- NoSQL
- KeyValue
- Timeseries
- Graph
Further reading
Conclusion

Background

I’m a consultant doing development, DevOps and cloud infrastructure. I also have the occasional side project trying out the Tech Flavor of the Month.

Project phase overview

The typical phases in projects I’m involved in follow no scientific or trademarked methodology, so YMMV:

Starting out

Get something working as fast as possible. Take all the shortcuts. Use some opinionated framework or platform.

Moving from development to production

People like it, people use it. Move the thing from a single “pet server” to a more robust cloud environment.

Scaling production

Bottlenecks and scaling problems start to emerge. Refactor or replace some pieces to remove the bottlenecks.

Challenges

Moving between these phases might be a major PITA if the wrong shortcuts were taken in the previous phases.

This of course applies to all technology choices and not just databases. But we have to start somewhere, right?

Planning

When starting out I try to envision all the phases of the project and which directions it may take in the future.

First I want the technology or software I choose to be instantly usable. A Docker image. Great. An apt-get install. Sweet. npm install. Sure, why not. Downloading a tarball. Installing some C dependencies. Setting some flags. Compiling. Symlinking and fixing permissions. Creating some configuration from scratch. Making my own systemd service definitions. Going back and doing every step again because it failed. Mkay, no thanks, I’m out.

At least for me it’s a plus if it’s easy to deploy on Kubernetes since I use it for everything already. I always have a cluster or three laying around so I can get a prototype or five up and running quickly before later spending money for cloud hosting.

Does the thing have momentum and a community? If it does it probably has high quality tooling either by the vendor or the open source community (preferably both). It probably also has lots of common questions answered on blogs and StackOverflow and Github issues.

So we managed to build something and the audience likes it.

How easy is it to move it from a production environment into something stable and low-maintenance? For databases that would typically involve using a managed service for hosting it. You do not want to be responsible for operating your own databases. Is it common enough that there are competitors in the marketplace offering it as a managed service? If there is only a single option expect prices to be very steep. Preferably also a managed service by one of the big known cloud platforms. They are usually cheaper. They are less likely to vanish. It might make integration with other systems easier later.

We hit some problems either because of raw scale or some type of usage we did not anticipate in the beginning.

Are there compatible implementations that might solve some common problems? Typically this is because an implementation has to make a decision about it’s trade-offs. For a database system this is usually around the CAP theorem. A database system (or anything that keeps state) can be:

Partition Tolerant - The system still works if a node or the network between nodes fail.
Available - All requests receive a response.
Consistent - The data we read is the current data and not an earlier state.

But, you can only have two at the same time. And distributed systems tends to need to be partition tolerant. So we are stuck between consistency and availability.

It might be a good to have an idea of the CAP tradeoffs an implementation has done, and whether there are compatible implementations with different tradeoffs that can be used if later we find out we need to tweak our trade-offs for speed and/or scale.

More information about CAP theorem here and here. Jepsen have also extensively tested many popular databases to see how they break and if they are true to their stated trade-offs.

Database categories

Databases can be roughly sorted into categories. I’ll keep it simple and use the everyday lingo and not go into details about semantics and definitions (forgive me).

https://www.prisma.io/dataguide/intro/comparing-database-types

SQL

The oldest category is the relational database, also known as SQL based on the typical interface used to access these databases.

In general these databases have tables with names, a set of pre-defined columns and an arbitrary number of rows. You should have an idea of the data types to be stored in each column (such as text or numbers).

The downside of this is that you have to start with a rough model of the data you want to store and work with. The benefit of this is that later you know something about the model of the data you are working with. Most of the time I’ll happily do this in the database rather than handle all the potential inconsistencies in all systems that use that database.

Main contenders: PostgreSQL. MySQL & MariaDB.

NoSQL

All the rage the last decade. You put data in you get data out. The data is structured but not necessarily predefined. Think JSON object with values, arrays and lists.

The benefit is productivity when developing. The drawback is that you may pay a price for those shortcuts later if you’re not careful.

Main contender: MongoDB.

KeyValue

Technically a sub-category of NoSQL, and should probably be called caches. But I feel it deserves it’s own category.

A hyper-fast hyper-simple type of database. It has two columns. A key (ID) and value. The value can be anything, a string, a number, an entire JSON object or a blob containing binary data.

These are typically used in combination with another type of database. Either by storing very commonly used data for even quicker access. Or for certain types of simple data that requires insane speed or throughput and you don’t want to overload the main database.

Main contender: Redis.

Timeseries

A lesser known type of database optimized for storing a time series. A time series is a specific data type where the index is typically the time of a measurement. And the measurement is a number.

A time series is almost never changed after the fact. So these databases can be optimized for writing huge amounts of new data and reading and calculating on existing data. At the cost of performance for deleting or updating old data which is sloooow. Since the values are always numbers that tend to change somewhat predictably compression and deduplication can save us massive amounts of storage.

Main contenders: Prometheus, InfluxDB, TimescaleDB (plugin for PostgreSQL).

Graph

Graph databases are cool. In a graph database the relationship between objects is a primary feature. Whereas in SQL you need to join an element from one table with another object in another table with some kind of common identifier.

For most simple use cases a regular SQL database will do fine. But when the number of objects stored (rows) and the number of intermediary tables (joins) become large it gets slow, or expensive, or both.

I don’t have much experience with graph databases but I suspect they are less suited to general tasks and should be reserved for solving specific problems.

Main contenders: Neo4j. Redis + RedisGraph.

PS: Graph databases and GraphQL are completely separate things.

Other nice things

When researching this post I’ve come across things that look promising but are hard to categorize or fall in their own very niche categories.

Dgraph - A GraphQL and backend in one.
PrestoDB - An SQL interface on top of whatever database or storage you want to connect.
RethinkDB - A NoSQL database focused on real-time streaming/updating clients.
FoundationDB - A transactional key-value store by Apple.
ClickHouse - An SQL database that stores data (on disk) in columns instead of rows. Makes for blazingly fast analytical and aggregation queries.
Amazon Quantum Ledger Database - A managed distributed ledger database (aka blockchain).
EDB Postgres Advanced Server - An Oracle compatible PostgreSQL variant.

The Landscape

How to use these maps:

Version compatibility are in parenthesis. I have not mapped every version and how much breaking they are compared to previous versions but included some notes where I know there might be issues.

API/Protocol/Interface - This is decided by the framework, tool or driver you want to use. Sometimes it might be easier to choose the framework first and then a fitting database protocol. Or you might be lucky to choose the database features you need first and then select frameworks, tools and drivers that support it.

I think interfaces are really important when creating and choosing technology. I had a presentation about it a while ago and I think it’s still relevant.

Engine - Database implementations that are independent but try to be compatible. If there are alternatives to the “original” implementation they might have done different tradeoffs with regards to the CAP theorem or solve other specific problems.

Big three managed - Available managed services by the big three clouds, Amazon (AWS), Google (GCP) or Microsoft (Azure). Having an option to host in the big three is most likely the cheapest method as well as having a variety of other managed services to build a complete system in a single cloud.

Vendor managed - If the database vendor or backing company offers an Official managed service. They are usually hosted on the big three. Potentially a large cost premium over the raw compute power.

Self-hosted - Implementations you can run on your own computer or server.

| Legend | |––––|—–| | | The checklist icon marks potential compatibility issues. For most use cases not a problem.
PS: The absence of this icon does not automatically mean compatibility. | | | I put the lightning icon on the self-hosted implementations that have what seems to be stable Kubernetes operators available. In short, a Kubernetes operator makes running a stateful system, such as a database, on Kubernetes much easier. It might allow for longer time before migrating to a managed service. |

SQL

Compatibility:

PostgreSQL - Yugabyte

PostgreSQL - CockroachDB

MySQL - MariaDB

Kubernetes Operators:

PostgreSQL (CrunchyData)

PostgreSQL (Zalando)

Yugabyte

CockroachDB

Percona PostgreSQL for MySQL & XtraDB

NoSQL

PS: There are some breaking changes from MongoDB 3.6 to 4 so make sure the tools you intend to use are compatible with the database version you intend on using.

Kubernetes Operators:

MongoDB

Percona Distribution for MongoDB

ScyllaDB

Elastic Stack

KeyValue

Kubernetes Operators:

Redis (Spotahome)

Timeseries

Kubernetes Operators:

Prometheus-Stack

VictoriaMetrics

Graph

Kubernetes Operators:

ArangoDB

Conclusion

Congratulations if you made it this far!

I did this research primarily to reduce my own analysis paralysis on various projects so I can get-back-to-building. If you learned something as well, great stuff!

And if you want my advice, just use PostgreSQL unless you really know about some special requirements that necessitates using something else :-)

Mini-post: Down-scaling Azure Kubernetes Service (AKS)

Tue, 04 Jun 2019 00:00:00 GMT

Looking into caveats of running Kubernetes on AKS with small node sizes and node counts.

We discovered today that some implicit assumptions we had about AKS at smaller scales were incorrect.

Suddenly new workloads and jobs in our Radix CI/CD could not start due to insufficient resources (CPU & memory).

Even though it only caused problems in development environments with smaller node sizes it still surprised some of our developers, since we expected the size of development clusters to have enough resources.

I thought it would be a good chance to go a bit deeper and verify some of our assumptions and also learn more about various components that usually “just works” and isn’t really given much thought. First I do a kubectl describe node <node> on 2-3 of the nodes to get an idea of how things are looking:

Resource                       Requests          Limits
--------                       --------          ------
cpu                            930m (98%)        5500m (585%)
memory                         1659939584 (89%)  4250M (228%)

So we are obviously hitting the roof when it comes to resources. But why?

Node overhead

We use Standard DS1 v2 instances as AKS nodes and they have 1 CPU core and 3.5 GiB memory.

The output of kubectl describe node also gives us info on the Capacity (total node size) and Allocatable (resources available to run Pods).

Capacity:
 cpu:                            1
 memory:                         3500452Ki
Allocatable:
 cpu:                            940m
 memory:                         1814948Ki

So we have lost 60 millicores / 6% of CPU and 1685MiB / 48% of memory. The next question is if this increases linearly with node size (the percentage of resources lost is the same regardless of node size) or is fixed (always reserves 60 millicores and 1685Mi of memory), or a combination.

I connect to another cluster that has double the node size (Standard DS2 v2) and compare:

Capacity:
 cpu:                            2
 memory:                         7113160Ki
Allocatable:
 cpu:                            1931m
 memory:                         4667848Ki

So for this the loss is 69 millicores / 3.5% of CPU and 2445MiB / 35% of memory.

So CPU reservations are close to fixed regardless of node size while memory reservations are influenced by node size but luckily not linearly.

What causes this “waste”? Reading up on kubernetes.io gives a few clues. Kubelet will reserve CPU and memory resources for itself and other Kubernetes processes. It will also reserve a portion of memory to act as a buffer whenever a Pod is going beyond it’s memory limits to avoid risking System OOM, potentially making the whole node unstable.

To figure out what these are configured to we log in to an actual AKS node’s console and run ps ax|grep kube and the output looks like this:

/usr/local/bin/kubelet --enable-server --node-labels=node-role.kubernetes.io/agent=,kubernetes.io/role=agent,agentpool=nodepool1,storageprofile=managed,storagetier=Premium_LRS,kubernetes.azure.com/cluster=MC_clusters_weekly-22_northeurope --v=2 --volume-plugin-dir=/etc/kubernetes/volumeplugins --address=0.0.0.0 --allow-privileged=true --anonymous-auth=false --authorization-mode=Webhook --azure-container-registry-config=/etc/kubernetes/azure.json --cgroups-per-qos=true --client-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/azure.json --cloud-provider=azure --cluster-dns=10.2.0.10 --cluster-domain=cluster.local --enforce-node-allocatable=pods --event-qps=0 --eviction-hard=memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5% --feature-gates=PodPriority=true,RotateKubeletServerCertificate=true --image-gc-high-threshold=85 --image-gc-low-threshold=80 --image-pull-progress-deadline=30m --keep-terminated-pod-volumes=false --kube-reserved=cpu=60m,memory=896Mi --kubeconfig=/var/lib/kubelet/kubeconfig --max-pods=110 --network-plugin=cni --node-status-update-frequency=10s --non-masquerade-cidr=0.0.0.0/0 --pod-infra-container-image=k8s.gcr.io/pause-amd64:3.1 --pod-manifest-path=/etc/kubernetes/manifests --pod-max-pids=-1 --rotate-certificates=false --streaming-connection-idle-timeout=5m

To log in to the console of a node, go to the MC_resourcegroup_clustername_region resource-group and select the VM. Then go to Boot diagnostics and enable it. Go to Reset password to create yourself a user and then Serial console to log in and execute commands.

We can see --kube-reserved=cpu=60m,memory=896Mi and --eviction-hard=memory.available<750Mi which adds up to 1646Mi which is pretty close to the 1685Mi that was the gap between Capacity and Allocatable.

We also do this on a Standard DS2 v2 node and get --kube-reserved=cpu=69m,memory=1638Mi and --eviction-hard=memory.available<750Mi.

So we can see that the memory of kube-reserved grows almost linearly and seems to always be about 20-25% while CPU reservations are almost the same. The memory eviction buffer is always fixed at 750Mi which would mean bigger resource waste as nodes decrease in size.

CPU

	Standard DS1 v2	Standard DS2 v2
VM capacity	1.000m	2.000m
kube-reserved	-60m	-69m
Allocatable	940m	1.931m
Allocatable %	94%	96.5%

Memory

	Standard DS1 v2	Standard DS2 v2
VM capacity	3.500Mi	7.113Mi
kube-reserved	-896Mi	-1.638Mi
Eviction buf	-750Mi	-750Mi
Allocatable	1.814Mi	4.667Mi
Allocatable %	52%	65%

Node pods (DaemonSets)

We have some Pods that run on every node, and they are installed by default by AKS. We get the resource limits of these by describing either the pods or the daemonsets.

CPU

	Standard DS1 v2	Standard DS2 v2
Allocatable	940m	1.931m
kube-system/calico-node	-250m	-250m
kube-system/kube-proxy	-100m	-100m
kube-system/kube-svc-redirect	-5m	-5m
Available	585m	1.576m
Available %	58%	81%

Memory

	Standard DS1 v2	Standard DS2 v2
Allocatable	1.814Mi	4.667Mi
kube-system/kube-svc-redirect	-32Mi	-32Mi
Available	1.782Mi	4.635Mi
Available %	50%	61%

So for Standard DS1 v2 nodes we have about 0.5 CPU and 1.7GiB memory per node for pods. And for Standard DS2 v2 nodes it’s about 1.5 CPU and 4.6GiB memory.

kube-system pods

Now lets add some standard Kubernetes pods we need to run. As far as I know these are pretty much fixed for a cluster and not related to node size or count.

Deployment	CPU	Memory
kube-system/kubernetes-dashboard	100m	50Mi
kube-system/tunnelfront	10m	64Mi
kube-system/coredns (x2)	200m	140Mi
kube-system/coredns-autoscaler	20m	10Mi
kube-system/heapster	130m	230Mi
Sum	460m	494Mi

Third party pods

Deployment	CPU	Memory
grafana	200m	500Mi
prometheus-operator	500m	1.000Mi
prometheus-alertmanager	100m	225Mi
flux	50m	64Mi
flux-helm-operator	50m	64Mi
Sum	900m	1.853Mi

Radix platform pods

Deployment	CPU	Memory
radix-api-prod/server (x2)	200m	400Mi
radix-api-qa/server (x2)	100m	200Mi
radix-canary-golang-dev/www	40m	500Mi
radix-canary-golang-prod/www	40m	500Mi
radix-platform-prod/public-site	5m	10Mi
radix-web-console-prod/web	10m	42Mi
radix-web-console-qa/web	5m	21Mi
radix-github-webhook-prod/webhook	10m	30Mi
radix-github-webhook-prod/webhook	5m	15Mi
Sum	415m	1.718Mi

If we add up the resource usage of these groups of workloads and see the total available resources on our 4 node Standard DS1 v2 clusters we are left with 0.56 CPU cores (14%) and 3GB of memory (22%):

Workload	CPU	Memory
kube-system	460m	494Mi
third-party	900m	1.853Mi
radix-platform	415m	1.718Mi
Sum	1.760m	4.020Mi
Available on 4x DS1	2.340m	7.128Mi
Available for workloads	565m	3.063Mi

Though surprising that we lost this much resources before being able to deploy our actual customer applications, it should still be a bit of headroom.

Going further I checked the resource requests on 8 customer pods deployed in 4 environments (namespaces). Even though none of them had a resource configuration in their radixconfig.yaml files they still had resource requests and limits. Not surprising since we use LimitRange to set default resource requests and limits. The surprise was that half of them had 50Mi of memory and the other half 500Mi, seemingly at random.

It turns out that we did an update to the LimitRange values a few days ago but that only applies to new Pods, so depending on if the Pods got re-created for any reason they may or may not have the old request of 500Mi, which in our case of small clusters will quickly drain the available resources.

Read more about LimitRange here: kubernetes.io , and here is the commit that eventually trickled down to reduce memory usage: github.com

Pod scheduling

Depending on the weight between CPU and memory requests and how often things get destroyed and re-created you may find yourself in a situation where you have enough resources in your cluster but new workloads are still Pending. This can happen when one resource type (e.g. CPU) is filled before another (e.g. memory), leading one or more resources to be stranded and unlikely to be utilized.

Imagine for example a cluster that is already utilized like this:

	CPU	Memory
node0	94%	86%
node1	80%	89%
node2	98%	60%

Scheduling a workload that requests 15% CPU and 20% memory cannot be scheduled since there are no nodes fulfilling both requirements. In theory there is probably a CPU intensive Pod on node2 that could be moved to node1 but Kubernetes does not do re-scheduling to optimize utilization. It can do re-scheduling based on Pod priority (medium.com) and there is an incubator project (akomljen.com) that can try to drain nodes with low utilization.

So for the foreseable future keeping in mind that resources can get stranded and that looking at the sum of cluster resources and sum of cluster resource demand might be misleading.

calico-node

The biggest source of waste on our small clusters is calico-node which is installed on every node and requests 25% of a CPU core while only using 2.5-3% CPU:

The request is originally set here github.com but I have not got into why that number was choosen. Next steps would be to do some benchmarking of calico-node to smoke out it’s performance characteristics to see if it would be safe to lower the resource requests, but that is out of scope for now.

Conclusion

By increasing node size from Standard DS1 v2 to Standard DS2 v2 we also increase the available CPU from 58% per node to 81% per node. Available memory increases from 50% to 61% per node.
With a total platform requirement of 3-4GB of memory and 4.6GB available on Standard DS2 v2 we might have more resources for actual workloads on a 1-node Standard DS2 v2 cluster than a 3-node Standard DS1 v2 cluster!
Beware of stranded resources limiting the utilization you can achieve across a cluster.

Disk performance on Azure Kubernetes Service (AKS) - Part 1: Benchmarking

Sat, 23 Feb 2019 00:00:00 GMT

In this first post on troubleshooting some disk performance issues on Azure Kubernetes Service (AKS) we will benchmark Azure Premium SSD to find how workloads affect performance and which metrics to monitor to know when troubleshooting potential disk issues.

Understanding the characteristics of disk performance of a platform might be more important than you think. If disk resources are not correctly matched to your workload, your performance will suffer and might lead you to incorrectly diagnose a problem as being related to CPU or memory.

The defaults might also not give you the performance you expect.

Disable Azure cache for workloads with high number of random writes
Use a P15 (256GB) or larger Premium SSD even though you might only need a fraction of it.

Table of contents

Background
- Metric Methodologies
- Storage Background
What to measure?
How to measure disk
How to measure disk on Azure Kubernetes Service
Test results
Conclusion

Microsoft Azure

If you don’t have a Azure subscription already you can try services for $200 for 30 days. The VM size Standard_B2s is Burstable, has 2vCPU, 4GB RAM, 8GB temp storage and costs roughly $38 / month. For $200 you can have a cluster of 3-4 B2s nodes plus traffic, loadbalancers and other additional costs.

See my blog post Managed Kubernetes on Microsoft Azure (English) for information on how to get up and running with Kubernetes on Azure.

I have no affiliation with Microsoft Azure except using them through work.

Corrections

February 2020: Some of my previous knowledge and assumptions were not correct when applied to a cloud + Docker environment, as explained by AKS PM Jesse Noller on GitHub.

One of the issues is that even accessing a “data disk” will incur IOPS on the OS disk, and throttling of the OS disk will also constraint IOPS on the data disks.

Background

I’m part of a team at Equinor building an internal PaaS based on Kubernetes running on AKS (Azure managed Kubernetes). We use Prometheus for monitoring each cluster as well as InfluxDB for collecting metrics from k6io which runs continous tests on our public endpoints.

A couple of weeks ago we discovered some potential problems with both Prometheus and InfluxDB with memory usage and restarts. High CPU usage of type iowait suggested that there might be some disk issues contributing to the problems.

iowait: “Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.” (hpe.com). You can see iowait on your Linux system by running top and looking at the wa percentage.

PS: You can have a disk IO bottleneck even with low iowait, and a high iowait does not always indicate a disk IO bottleneck (ibm.com).

First off we need to benchmark the underlying disk to get an understanding of it’s performance limits and characteristics. That is what we will cover in this post.

Metric Methodologies

There are two helpful methodologies when monitoring information systems. The first one is Utilization, Saturation and Errors (USE) from Brendan Gregg and the second one is Rate, Errors, Duration (RED) from Tom Wilkie. RED is best suited when observing workloads and transactions while USE is best suited for observing resources.

I’ll be using the USE method here. USE can be summarised as:

For every resource, check utilization, saturation, and errors.
- resource: all physical server functional components (CPUs, disks, busses, …)
- utilization: the average time that the resource was busy servicing work
- saturation: the degree to which the resource has extra work which it can’t service, often queued
- errors: the count of error events

Storage Background

Disk usage has two dimensions, throughput/bandwidth(BW) and operations per second (IOPS), and the underlying storage system will have upper limits of how much data it can receive (BW) and the number of operations it can perform per second (IOPS).

Background - harddrive types: harddrives come in two types, Solid State Disks (SSD) and spindle (HDD). A SSD disk is a microship capable of permanently storing data while a HDD uses spinning platters to store data. HDDs have a fixed rate of rotation (RPM), typically 5.400 and 7.200 RPM for lower cost drives for home use and higher cost 10.000 and 15.000 RPM drives for server use. Over the last 20 years of HDDs their storage density has increased, but the RPM has largely stayed the same. A disk with twice the density (500GB to 1TB for example) can read twice as much data on a single rotation and thus increase the bandwidth significantly. However, reading or writing a random block still requires waiting for the disk to spin enough to reach the relevant sector on the disk. So IOPS has not increased much for HDDs and is still a low 125-150 IOPS for a 10.000 RPM enterprise disk. A SSD does not have any moving parts so is able to reach MUCH higher IOPS. A low end Samsung 960 EVO with 500GB capacity costs $150 and can achieve a whopping 330.000 IOPS! (wikipedia.com)

Background - access patterns: The way a program uses storage also has a huge impact on the performance one can achieve. Sequential access is when we read or write a large file. When this happens the operating system and harddrive can optimize and “merge” operations so that we can read or write a much bigger chunk of data at a time. If we can read 1MB at a time 150 times per second we get 150MB/s of bandwidth. However, fully random access where the smallest chunk we read or write is a 4KB block the same 150 IOPS would only give a bandwidth of 0.6MB/s!

Background - cloud vs physical: Now we know what HDDs are limited to a low IOPS and low IOPS combined with a random access pattern gives us a low overall bandwidth. There is a huge gotcha here when it comes to cloud. On Azure when using Premium Managed SSD the IOPS you are given is a factor of the disk size you provision (microsoft.com). A 512GB disk is limited to 2.300 IOPS and 150MB/s. With 100% random access that only gives about 9MB/s of bandwidth!

Background - OS caching: To overcome some of the limitations of the underlying disk (mostly IOPS) there are potentially several layers of caching involved. Linux file systems can have writeback enabled which causes Linux to temporarily store data that is going to be written to disk in memory. This can give a big performance increase when there are sudden spikes of writes exceeding the performance of the underlying disk. It also increases the chance that operations can be merged where several write operations to areas of the disk that are nearby can be executed as one. This caching works best for sudden peaks and will not necessarily be enough if there is continous random writes to disk. This caching also means that even though an application thinks it has saved some data to disk it can be lost in the case of a power outage or other failure. Applications can also explicitly request direct access where every operation is persisted to disk before receiving a confirmation. This is a trade-off between performance and durability that needs to be decided based on the application itself and the environment.

Background - Azure caching: Azure also provides read and write cache for its disks which is enabled by default. As we will see soon for our use case it’s not a good idea to use.

What to measure?

These metrics are collected by the Prometheus node-exporter and follows it’s naming. I’ve also created a dashboard that is available on Grafana.com.

With the USE methodology as a guideline and the two separate but related “resources”, bandwidth and IOPS we can look for some useful metrics.

Utilization:

rate(node_disk_written_bytes_total) - Write bandwidth. The maximum is given by Azure and is 25MB/s for our disk size.
rate(node_disk_writes_completed_total) - Write operations. The maximum is given by Azure and is 120 IOPS for our disk size.
rate(node_disk_io_time_seconds_total) - Disk active time in percent. The time the disk was busy servicing requests. 100% means fully utilized.

Saturation:

rate(node_cpu_seconds_total{mode="iowait"} - CPU iowait. The percentage of time a CPU core is blocked from doing useful work because it’s waiting for an IO operation to complete (typically disk, but can also be network).

Useful calculated metrics:

rate(node_disk_write_time_seconds_total) / rate(node_disk_writes_completed_total) - Write latency. How long from a write is requested until it’s completed.
rate(node_disk_written_bytes_total) / rate(node_disk_writes_completed_total) - Write size. How big the average write operation is. 4KB is minimum and indicates 100% random access while 512KB is maximum and indicates sequential access.

How to measure disk

The best tool for measuring disk performance is fio, even though it might seem a bit intimidating at first due to it’s insane number of options.

Installing fio on Ubuntu:

apt-get install fio

fio executes jobs described in a file. Here is the top of our jobs file:

[global]
ioengine=libaio   # sync|libaio|mmap
group_reporting
thread
size=10g          # Size of test file
cpus_allowed=1    # Only use this CPU core
runtime=300s      # Run test for 5 minutes

[test1]
filename=/tmp/fio-test-file
direct=1          # If value is true, use non-buffered I/O. Non-buffered I/O usually means O_DIRECT
readwrite=write   # read|write|randread|randwrite|readwrite|randrw
iodepth=1         # How many operations to queue to the disk
blocksize=4k

The fields we will be changing for the various tests are direct, readwrite, iodepth and blocksize. Save the contents in a file named jobs.fio and we run a test with fio --sector test1 jobs.fio and wait until the test completes.

PS: To run these tests on higher performance hardware and better caching you might want to set runtime to 0 to have the test run continously and monitor the metrics until performance reaches a steady-state.

How to measure disk on Azure Kubernetes Service

For this testing we use a standard Prometheus installation collecting data from node-exporter and visualizing data in Grafana. The dashboard I created for the testing can be found here: https://grafana.com/dashboards/9852.

By default Kubernetes will schedule a Pod to any node that has enough memory and CPU for our workload. Since one of the tests we are going to run are on the OS disk we do not want the Pod to run on the same node as any other disk-intensive application, such as Prometheus.

Look at which Pods are running with kubectl get pods -o wide and look for a node that does not have any disk-intensive application.

Then we tag that node with kubectl label nodes aks-nodepool1-37707184-2 tag=disktest. This allows us later to specify that we want to run our testing Pod on that specific node.

A StorageClass in Kubernetes is a specification of a underlying disk that Pods can request usage of through volumeClaimTemplates. AKS comes with a default StorageClass managed-premium that has caching enabled. Most of these tests require the Azure cache disabled so create a new StorageClass managed-premium-retain-nocache:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: managed-premium-retain-nocache
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Retain
parameters:
  storageaccounttype: Premium_LRS
  kind: Managed
  cachingmode: None

You can add it to your cluster with:

kubectl apply -f /attachments/2019-02-23-disk-performance-on-aks-part-1/storageclass.yaml

Next we create a StatefulSet that uses a volumeClaimTemplate to request a 250GB Azure disk. This provisions a P15 Azure Premium SSD with 125MB/s bandwidth and 1100 IOPS:

kubectl apply -f /attachments/2019-02-23-disk-performance-on-aks-part-1/ubuntu-statefulset.yaml

Follow the progress of the Pod creation with kubectl get pods -w and wait until it is Running.

When the Pod is Running we can start a shell on it with kubectl exec -it disk-test-0 bash

Once inside bash on the Pod, we install fio:

apt-get update && apt-get install -y fio wget

And save the contents of in the Pod:

wget /attachments/2019-02-23-disk-performance-on-aks-part-1/jobs.fio

Now we can run the different test sections one by one. PS: If you don’t specify a section fio will run all the tests simultaneously, which is not what we want.

fio --section=test1 jobs.fio
fio --section=test2 jobs.fio
fio --section=test3 jobs.fio
fio --section=test4 jobs.fio
fio --section=test5 jobs.fio
fio --section=test6 jobs.fio
fio --section=test7 jobs.fio
fio --section=test8 jobs.fio
fio --section=test9 jobs.fio

Test results

Test 1 - Learning to dislike Azure Cache

Sequential write, 4K block size, Azure Cache enabled, OS cache disabled. See full fio test results.

I run the first tests on the OS disk of a Kubernetes node. The OS disks have Azure caching enabled.

The first 1-2 minutes of the test I get very good performance of 45MB/s and ~11.500 IOPS but that drops to 0 very quickly as the cache is full and busy writing things to the underlying disk. When that happens everything freezes and I cannot even execute shell commands. After stopping the test the system still hangs for a bit while the cache empties.

The maximum latency measured by fio was 108751k usec. Or about 108 seconds!

For the first try of these tests a 20-30 second period of very fast writes (250MB/s) caused a 7-8 minutes hang while the cache emptied. Trying again caused another pattern of lower peak performance with shorter hangs in between. Very unpredictable. I’m not sure what to make of this. It’s not acceptable that a Kubernetes node becomes unresponsive for many minutes following a short burst of writing. There are scattered recommendations online of disabling caching for write-heavy applications. Since I have not found any way to measure the Azure cache itself, the results are unpredictable and potentially very impactful as well as making it very hard to use the metrics we do have to evaluate application and storage behaviour I’ve concluded that it’s best to use data disks with caching disabled for our workloads (you cannot disable caching on an AKS node OS disk).

Test 2 - Disable Azure Cache - enable OS cache

Sequential write, 4K block size. Change: Azure cache disabled, OS caching enabled. See full fio test results.

If we swap the Azure cache for the Linux OS cache we see that iowait increases while the writing occurs. The application sees high write performance until the number of Dirty bytes reaches a threshold of about 3.7GB of memory. The performance of the underlying disk is 125MB/s and 250 IOPS. Here we are throttled by the 125MB/s limit of the Azure P15 Premium SSD.

Also notice that on sequential writes of 4K with OS caching the actual blocks written to disk is 512K which saves us a lot of IOPS. This will become important later.

Test 3 - Disable OS cache

Sequential write, 4K block size. Change: OS caching disabled. See full fio test results.

By disabling the OS cache (direct=1) the results are consistent and predictable. There is no iowait since the application does not have multiple writes pending at the same time. Because of the 2-3ms latency of the disks we are not able to get more than about 400 IOPS. This gives us a meager 1.5MB/s even though the disk is limited to 1100 IOPS and 125MB/s. To reach that we need multiple simultaneous writes or a bigger IO depth (queue). Disk active time is also 0% which indicates that the disk is not saturated.

Test 4 - Increase IO depth

Sequential write, 4K block size, OS caching disabled. Change: IO depth 16. See full fio test results.

For this test we only increase the IO depth from 1 to 16. IO depth is the number of write operations fio will execute simultaneously. Since we are using direct these will be queued by the OS for writing. We are now able to hit the performance limit of 1100 IOPS. Disk active time is now steady at 100% indicating that we have saturated the disk.

Test 5 - Larger block size, smaller IO depth

Sequential write, OS caching disabled. Change: 128K block size, IO depth 1. See full fio test results.

We increase the block size to 128KB and reduce the IO depth to 1 again. The write latency for larger blocks increase to ~5ms which gives us 200 IOPS and 28MB/s. The disk is not saturated.

Test 6 - Enable OS cache

Sequential write, 256K block size, IO depth 1. Change: OS caching enabled. See full fio test results.

We have now enabled the OS cache/buffer (direct=0). We can see that the writes hitting the disk are now merged to 512KB blocks. We are hitting the 125MB/s limit with about 250 IOPS. Enabling the cache also has other effects: CPU suddenly shows significant IO wait. The write latency shoots through the roof. Also note that the writing continued for 30-40 seconds after the test was done. This also means that the bandwidth and IOPS that fio sees and reports is higher than what is actually hitting the disk.

Test 7 - Random writes, small block size

IO depth 1, OS caching enabled. Change: Random write, 4K block size. See full fio test results.

Here we go from sequential writes to random writes. We are limited by IOPS. The average size of the blocks actually written to disks, and the IOPS required to hit the bandwidth limit is actually varying a bit throughout the test. The time taken to empty the cache is about as long as I ran the test (4-5 minutes).

Test 8 - Large block size

Random write, OS caching enabled. Change: 256K block size, IO depth 16. See full fio test results.

Increasing the block size to 256K makes us bandwidth limited to 125MB/s.

Conclusion

Access patterns and block sizes have a tremendous impact on the amount of data we are able to write to disk.

Managed Kubernetes on Microsoft Azure (English)

Fri, 29 Dec 2017 00:00:00 GMT

In this post we will set up a managed Kubernetes cluster from scratch using Azure CLI.

A few days ago I wrote a walkthrough of setting up Azure Container Service (AKS) in Norwegian. Someone asked me for an English version of that, and here it is.

Kubernetes(K8s) is becoming the de-facto standard for deploying container-based applications and workloads. Microsoft is currently in preview of their managed Kubernetes offering (Azure Kubernetes Service, AKS) which makes it easy to create a Kubernetes cluster and deploy workloads without the skill and time required to manage day-to-day operations of a Kubernetes-cluster, which today can be complex and time consuming.

In this post we will set up a Kubernetes cluster from scratch using Azure CLI. Table of contents

Background
- Docker containers
- Container orchestration
Getting started with Azure Kubernetes - AKS
Bonus material
- Deploying services with Helm
  - Deploy MineCraft with Helm
- Kubernetes Dashboard
Conclusion

Microsoft Azure

If you don’t have a Azure subscription already you can try services for $200 for 30 days. The VM size Standard_B2s is Burstable, has 2vCPU, 4GB RAM, 8GB temp storage and costs roughly $38 / month. For $200 you can have a cluster of 3-4 B2s nodes plus traffic, loadbalancers and other additional costs.

We have no affiliation with Microsoft Azure except their sponsorship of our startup DataDynamics with cloud services for 24 months in their BizSpark program.

Background

Docker containers

We will not do a deep dive on Docker containers in this post, but here is a summary for those who are not familiar with it.

Docker is a way to package software so that it can run on the most popular platforms without worrying about installation, dependencies and to a certain degree, configuration.

In addition, a Docker container uses the operating system of the host machine when it runs. Because of this it’s possible to run many more containers on the same host machine compared to running virtual machines.

Here is a incomplete and rough comparison between a Docker container and a virtual machine:

	Virtual machine	Docker container
Image size	from 200MB to many GB	from 10MB to 3-400MB
Startup time	60 seconds +	1-10 seconds
Memory usage	256MB-512MB-1GB +	2MB +
Security	Good isolation between VMs	Not as good isolation between containers
Building image	Minutes	Seconds

PS The numbers for virtual machines is taken from memory. I tried starting a MySQL virtual appliance on my laptop but VMware Player refuses to run because of Windows Hyper-V incompatibility. VMware Workstation refuses to run because of license issues and Oracle VirtualBox repeatedly gives me a nasty bluescreen. Hooray!

Protip The smallest and fastest Docker images are built on Alpine Linux. For the webserver Nginx the Alpine-based image is 15MB compared to 108MB for the normal Debian-based image. PostgreSQL:Alpine is 38MB compared to 287MB with “full” OS. Last version of MySQL is 343MB but will in version 8 support Alpine Linux as well.

To recap, some of the advantages of Docker containers are:

Compatibility across platforms, Linux, Windows, MacOS.
10-100x smaller size. Faster to download, build and upload.
Memory usage only for application and not base OS.
- Advantage when developing. Ability to run 10-20-30 containers on a development laptop.
- Advantage in production. Can reduce hardware/cloud costs considerably.
Near instant startup. Makes dynamic scaling of applications easier.

Download Docker for Windows here.

To start a MySQL database container from Windows CMD or Powershell:

docker run --name mysql -p 3306:3306 -e MYSQL_RANDOM_ROOT_PASSWORD=true mysql

Stop the container with:

docker kill mysql

You can search for already built Docker images on Docker Hub. It’s also possible to create private Docker repositories for your own software that you don’t want to be publicly available.

Container orchestration

Now that Docker container images has become the preferred way to package and distribute software on the Linux platform, there has emerged a need for systems to coordinate running and deploying these containers. Similar to the ecosystem of products VMware has built up around development and operation of virtual machines.

Container orchestration systems have the responsibility for:

Load balancing.
Service discovery.
Health checks.
Automatic scaling and restarting of host nodes and containers.
Zero downtime upgrades (rolling deploys).

Until recently the ecosystem around container orchestration has been fragmented, and the most popular alternatives have been:

Kubernetes (Originaly from Google, now managed by CNCF, the Cloud Native Computing Foundation)
Swarm (From the maker of Docker)
Mesos (From Apache Software Foundation)
Fleet (From CoreOS)

But the last year there has been a convergence towards Kubernetes as the preferred solution.

7 February
- CoreOS announces that they are removing Fleet from Container Linux and recommends Kubernetes
27 July
- Microsoft joins the CNCF
9 August
- Amazon Web Services join the CNCF
29 August
- VMware and Pivotal joins the CNCF
17 September
- Oracle joins the CNCF
17 October
- Docker announces native support for Kubernetes in addition to it’s own Swarm product
24 October
- Microsoft Azure announces the managed Kubernetes service AKS
29 November
- Amazon Web Services announces the managed Kubernetes service EKS

Especially the last two news items are important. Deploying and running your own Kubernetes-installation requires time and skills (Read how Stripe used 5 months to trust running Kubernetes in production, just for batch jobs.)

Until now the choice has been running your own Kubernetes cluster or using Google Container Engine which has been using Kubernetes since 2014. Many of us feel a certain discomfort by locking ourselves to one provider. But this is now changing when you can develop infrastructure on Kubernetes and choose between the 3 large cloud providers in addition to running your own cluster if wanted. *

* Kubernetes is a fast moving project, and features might be available on the different platforms on different timelines.

Getting started with Azure Kubernetes - AKS

Caveats

This guide is based on the documentation on Microsoft.com. Setting up a Azure Kubernetes cluster did not work in the beginning of December, but today, 23. December, it seems to work fairly well. But, upgrading the cluster from Kubernetes 1.7 to 1.8 for example does NOT work.

AKS is in Preview and Azure are working continuously to make AKS stable and to support as many Kubernetes-features as possible. Amazon Web Services has a similar closed invite-only Preview currently while working on stability and features.

Both Azure and AWS expresses expectations about their Kubernetes offerings will be ready for production in 2018.

Preparations

You need Azure-CLI (version 2.0.21 or newer) to execute the az commands:

All commands executed in Windows PowerShell.

Log on to Azure:

az login

You will get a link to open in your browser together with an authentication code. Enter the code on the webpage and az login will save the login information so that you will not have to authenticate again on the same machine.

PS The login information gets saved in C:\Users\Username\.azure\. You have to make sure nobody can access these files. They will then have full access to your Azure account.

Activate ContainerService

Since AKS is in Preview/Beta, you explicitly have to activate it in your subscription to get access to the aks subcommands.

az provider register -n Microsoft.ContainerService
az provider show -n Microsoft.ContainerService

Create a resource group

Here we create a resource group named “my_aks_rg” in Azure region West Europe.

az group create --name my_aks_rg --location westeurope

Protip To see a list of all available Azure regions, use the command az account list-locations --output table. PS AKS might not be available in all regions yet!

Create Kubernetes cluster

az aks create --resource-group my_aks_rg --name my_cluster --node-count 3 --generate-ssh-keys --node-vm-size Standard_B2s --node-osdisk-size 128 --kubernetes-version 1.8.2

--node-count
- Number of agent(host) nodes available to run containers
--generate-ssh-keys
- Creates and prints a SSH key which can be used for SSHing directly to the agent nodes.
--node-vm-size
- Which size Azure VMs the agent nodes should be created as. To see available sizes use az vm list-sizes -l westeurope --output table and Microsofts webpages.
--node-osdisk-size
- Disk size of the agent nodes in GB. PS Containers can be stopped and moved to another host if Kubernetes finds it necessary or if a agent node disappears. All data saved locally in the container will be gone. If saving data permanently use Kubernetes PersistentVolumes and not the local agent node or container disks.
--kubernetes-version
- Which Kubernetes version to install. Azure does NOT necessarily install the last version by default, and currently upgrading with az aks upgrade does not work. Latest version available right now is 1.8.2. It’s recommended to use the latest available version since there is a lot of changes from version to version. The documentation is also much better for newer versions.

Save the output of the command in a file in a secure location. It contains keys that can be used to connect to the cluster with SSH. Even though that should not in theory be necessary.

Install kubectl

kubectl is the client which performs all operations against your Kubernetes cluster. Azure CLI can install kubectl for you:

az aks install-cli

After kubectl is installed we need to get login information so that kubectl can communicate with the Kubernetes cluster.

az aks get-credentials --resource-group my_aks_rg --name my_cluster

The login information is saved in C:\Users\Username\.kube\config. Keep these files secure as well.

Protip When you have several Kubernetes clusters you can change which one kubectl talks to with kubectl config get-contexts and kubectl config set-context my_cluster.

Inspect cluster

To check that the cluster and kubectl works we start with a couple of commands.

See all agent nodes and status:

> kubectl get nodes
NAME                       STATUS    AGE       VERSION
aks-nodepool1-16970026-0   Ready     15m       v1.8.2
aks-nodepool1-16970026-1   Ready     15m       v1.8.2
aks-nodepool1-16970026-2   Ready     15m       v1.8.2

See all services, pods and deployments:

> kubectl get all --all-namespaces

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   po/kubernetes-dashboard-6fc8cf9586-frpkn      1/1       Running   0          3d

NAMESPACE     NAME                          CLUSTER-IP     EXTERNAL-IP     PORT(S)           AGE
kube-system   svc/kubernetes-dashboard      10.0.161.132   <none>          80/TCP            3d

NAMESPACE     NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deploy/kubernetes-dashboard      1         1         1            1           3d

NAMESPACE     NAME                                    DESIRED   CURRENT   READY     AGE
kube-system   rs/kubernetes-dashboard-6fc8cf9586      1         1         1         3d

This is just some of the output from this command. You do not have to know what the resources in the kube-system namespace does. That is part of the intention when Microsoft is managing our cluster for us.

Namespaces In Kubernetes there is something called Namespaces. Resources in one namespace does not have automatic access to resources in another namespace. The services that runs Kubernetes itself use the namespace kube-system. The kubectl command by default only shows you resources in the default namespace, unless you specify --all-namespaces or --namespace=xx.

Start some nginx containers

An instance of a running container in Kubernetes is called a Pod.

nginx is a fast and flexible web server.

Now that the clsuter is up we can start rolling out services and deployments on it.

Lets start with creating a Deployment consiting of 3 containers all running the nginx:mainline-alpine image from Docker hub.

nginx-dep.yaml looks like this:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:mainline-alpine
        ports:
        - containerPort: 80

Load this into the cluster with kubectl create:

kubectl create -f /attachments/2017-12-23-managed-kubernetes-on-azure/nginx-dep.yaml

This command creates the resources described in the file. kubectl can read files either from your local disk or from a web URL.

After making changes to a resource definition (.yaml file), you can update the resources in the cluster with kubetl replace -f resource.yaml.

We can verify that the Deployment is ready:

> kubectl get deploy
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3         3         3            3           10m

We can also get the actual Pods that are running:

> kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-569477d6d8-dqwx5   1/1       Running   0          10m
nginx-deployment-569477d6d8-xwzpw   1/1       Running   0          10m
nginx-deployment-569477d6d8-z5tfk   1/1       Running   0          10m

Logger We can view logs from one pod with kubectl logs nginx-deployment-569477d6d8-xwzpw. But since we in this case don’t know which Pod ends up getting an incomming request we can view logs from all the Pods which have app=nginx label: kubectl logs -lapp=nginx. The use of app=nginx is our choice in nginx-dep.yaml when we configured spec.template.metadata.labels: app: nginx.

Making nginx available with a service

To send traffic to our new Pods we need to create a Service. A service consists of one or more Pods which are chosen based on different criteria, for example which labels they have and whether the Pods are Running and Ready.

Lets create a service which forwards traffic to all Pods with label app: nginx and are listening to port 80. In addition we make the service available via a LoadBalancer:

nginx-svc.yaml looks like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx

We tell Kubernetes to create our service with kubectl create as usual:

kubectl create -f /attachments/2017-12-23-managed-kubernetes-on-azure/nginx-svc.yaml

We can then wait and see which IP-address Azure assigns our service:

> kubectl get svc -w
NAME         CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE
nginx        10.0.24.11   13.95.173.255   80:31522/TCP   15m

PS It can take a few minutes for Azure to allocate and assign a Public IP for us. In the mean time <pending> will appear under EXTERNAL-IP.

A simple Welcome to nginx webpage should now be available on http://13.95.173.255 (remember to replace with your own External-IP).

We can also delete the service and deployment afterwards:

kubectl delete svc nginx
kubectl delete deploy nginx-deployment

Scaling the cluster

If we want to change the number of agent nodes running Pods we can do that via Azure-CLI:

az aks scale --name my_cluster --resource-group my_aks_rg --node-count 5

Currently all nodes will be created with the same size as when we created the cluster. AKS will probably get support for node-pools next year. That will allow for creating different groups of nodes with different size and operating systems, both Linux and Windows.

Delete cluster

You can delete the whole cluster like this:

az aks delete --name my_cluster --resource-group my_aks_rg --yes

Bonus material

Here is some bonus material if you want to go a bit further with Kubernetes.

Deploying services with Helm

Helm is a package manager and library of software that is ready to be deployed on a Kubernetes cluster.

Start by downloading the Helm-client. It will read login information etc. from the same location as kubectl automatically.

Install the Helm-server (Tiller) on the Kubernetes cluster and update the package library:

helm init
helm repo update

See available packages (Charts) with helm search.

Deploy MineCraft with Helm

Lets deploy a MineCraft server installation on our cluster, just because we can :-)

helm install --name stians --set minecraftServer.eula=true stable/minecraft

--set overrides one or more of the standard values configured in the package. The MineCraft package is made in a way where it does not start without accepting the user license agreement by setting the variable minecraftServer.eula. All the variables that can be set in the MineCraft package are documented here.

Then we wait for Azure to assign us a Public IP:

> kubectl get svc -w
stians-minecraft   10.0.237.0   13.95.172.192   25565:30356/TCP   3m

Now we can connect to our MineCraft server on 13.95.172.192:25565!

Kubernetes Dashboard

Kubernetes also has a graphic web user-interface which makes it a bit easier to see which resources are in the cluster, view logs and even open a remote shell inside a running Pod, among other things.

> kubectl proxy
Starting to serve on 127.0.0.1:8001

kubectl encrypts and tunnels the traffic to the Kubernetes API servers. The dashboard is available on http://127.0.0.1:8001/ui/.

Conclusion

I hope you enjoy Kubernetes as much as I have. The learning curve can be a bit steep in the beginning, but it does not take long before you are productive.

Look at the official guides on Kubernetes.io to learn more about defining different types of resources and services to run on Kubernetes. PS: There are big changes from version to version so make sure you use the documentation for the correct version!

Kubernetes also have a very active Slack-community on kubernetes.slack.com that is worthwhile to check out.

Managed Kubernetes på Microsoft Azure (Norwegian)

Mon, 25 Dec 2017 00:00:00 GMT

I denne posten setter vi opp et managed Kubernetes cluster fra scratch ved bruk av Azure CLI.

Update 29. Dec: There is an English version of this post here.

Kubernetes (K8s) er i ferd med å bli de-facto standard for deployments av kontainer-baserte applikasjoner. Microsoft har nå preview av deres managed Kubernetes tjeneste (Azure Kubernetes Service, AKS) som gjør det enkelt å opprette et Kubernetes cluster og rulle ut tjenester uten å måtte ha kompetanse og tid til den daglige driften av selve Kubernetes-clusteret, som per i dag kan være relativt komplisert og tidkrevende.

I denne posten setter vi opp et Kubernetes cluster fra scratch ved bruk av Azure CLI. Table of contents

Bakgrunn
- Docker containers
- Container orchestration
Kom i gang med Azure Kubernetes - AKS
Bonusmateriale
- Rulle ut tjenester med Helm pakker
  - MineCraft server med Helm
- Kubernetes Dashboard
Konklusjon

Microsoft Azure

Hvis du ikke har Azure fra før kan du prøve tjenester for $200 i 30 dager. VM typen Standard_B2s er Burstable, har 2vCPU, 4GB RAM, 8GB temp storage og koster ~$38 / mnd. For $200 kan du ha et cluster på 3-4 B2s noder plus trafikkostnad, lastbalanserere og andre nødvendige tjenester.

Vi har ingen tilknytning til Microsoft bortsett fra at de sponser vår startup DataDynamics med cloud-tjenester i 24 mnd i deres BizSpark program.

Bakgrunn

Docker containers

Vi tar ikke for oss Docker containers i dybden i denne posten, men her er en kort oppsummering for de som ikke er kjent med teknologien.

Docker er en måte å pakketere programvare slik at det kan kjøres på samtlige populære platformer uten å måtte bruke mye tid på dependencies, oppsett og konfigurasjon.

I tillegg bruker en Docker container operativsystemet på vertsmaskinen når den kjører. Dette gjør at en kan kjøre mange flere containere på samme vertsmaskin sammenlignet med virtuelle maskiner.

Her er en ufullstendig og grov sammenligning mellom en Docker container og en virtuell maskin:

	Virtuel maskin	Docker container
Image størrelse	fra 200MB til mange GB	fra 10MB til 3-400MB
Oppstartstid	60 sekunder +	1-10 sekunder
Minnebruk	256MB-512MB-1GB +	2MB +
Sikkerhet	God isolasjon mellom VM	Dårligere isolasjon mellom containere
Bygge image	Minutter	Sekunder

PS Tallene for virtuelle maskiner er tatt fra hukommelsen. Jeg forsøkte å starte en MySQL virtuell appliance på min laptop men VMware Player nekter å kjøre pga inkompatibilitet med Windows Hyper-V. VMware Workstation nekter å kjøre pga utgått lisens og Oracle VirtualBox gir en nasty bluescreen gang på gang. Hooray!

Protip De minste og raskeste Docker imagene er bygget på Alpine Linux. For webserveren Nginx er det Alpine-baserte imaget 15MB mot det Debian-baserte imaget på 108MB. PostgreSQL:Alpine er 38MB mot 287MB. Siste versjon av MySQL er 343MB men vil i versjon 8 støtte Alpine Linux også.

Noen av fordelene med Docker containers er altså:

Kompatibilitet på tvers av platformer, Linux, Windows og MacOS.
10-100x mindre størrelse. Raskere å laste ned, raskere å bygge, raskere å laste opp.
Minnebruk kun for applikasjon og ikke eget OS.
- Fordel under utvikling, kan kjøre 10-20-30 Docker containere samtidig på en laptop.
- Fordel i produksjon, kan redusere hardware utgifter betraktelig.
Oppstart på få sekunder. Gjør dynamisk skalering av applikasjoner mye enklere.

Last ned Docker for Windows her.

Og start en MySQL database fra Windows CMD eller Powershell:

docker run --name mysql -p 3306:3306 -e MYSQL_RANDOM_ROOT_PASSWORD=true mysql

Stop containeren med:

docker kill mysql

En kan søke etter ferdige Docker images på Docker Hub. Det er også mulig å lage private Docker repositories for egen programvare som ikke skal være tilgjengelig for omverden.

Container orchestration

Etter som Docker containers har blitt den foretrukne måten å pakke og distribuere programvare på Linux platformen de siste par årene har det vokst frem et behov for systemer som kan samkjøre drift og utrulling av disse containerene. Ikke ulikt det økosystemet av produkter VMware har bygget opp rundt utvikling og drift av virtuelle maskiner.

Container orchestration systemene har som oppgave å sørge for:

Lastbalansering.
Service discovery.
Health checks.
Automatisk skalering og restarting av vertsmaskiner og containere.
Oppgraderinger uten nedetid (rolling deploy).

Frem til nylig har økosystemet rundt container orchestration vært fragmentert og de mest populære alternativene har vært:

Kubernetes (Opprinnelig fra Google, nå styrt av CNCF, Cloud Native Computing Foundation)
Swarm (Fra produsenten bak Docker)
Mesos (Fra Apache Software Foundation)
Fleet (Fra CoreOS)

Men det siste året har det vært en konvergens mot Kubernetes som foretrukket løsning.

7 februar
- CoreOS annonserer at de fjerner Fleet fra Container Linux og anbefaler Kubernetes
27 juli
- Microsoft slutter seg til CNCF
9 august
- Amazon Web Services slutter seg til CNCF
29 august
- VMware og Pivotal slutter seg til CNCF
17 september
- Oracle slutter seg til CNCF
17 oktober
- Docker annonserer native støtte for Kubernetes i tillegg til sitt eget Swarm produkt
24 oktober
- Microsoft Azure annonserer managed Kubernetes med tjenesten AKS
29 november
- Amazon Web Services annonserer managed Kubernetes med tjenesten EKS

De to siste nyhetene er spesielt viktige. Å drifte sin egen Kubernetes-installasjon krever tid og kompetanse. (Les hvordan Stripe brukte 5 måneder på å bli fortrolig med å drifte sitt eget Kubernetes cluster, bare for batch jobs.)

Frem til nå har valget vært mellom å drifte sitt eget Kubernetes cluster eller bruke Google Container Engine som har brukt Kubernetes siden 2014. Mange av oss føler et visst ubehag ved å låse oss til én tilbyder. Men dette er nå anderledes når en kan utvikle infrastruktur på Kubernetes, og velge tilnærmet fritt * mellom de 3 store cloud-tilbyderene i tillegg til å drifte selv om ønskelig.

* Kubernetes utvikles raskt, og funksjonalitet blir ofte ikke tilgjengelig på de ulike platformene samtidig.

Opprette Azure Kubernetes Cluster

Forbehold

Denne gjennomgangen tar utgangspunkt i dokumentasjonen på Microsoft.com. Å sette opp et Azure Kubernetes cluster fungerte ikke i starten av desember, men per dags dato, 23. desember, ser det ut til å fungere relativt bra. Men, oppgradering av cluster fra Kubernetes 1.7 til 1.8 fungerer for eksempel IKKE.

AKS er i Preview og Azure jobber kontinuerlig med å gjøre AKS stabilt og støtte så mange Kubernetes-funksjoner som mulig. Amazon Web Services har tilsvarende en lukket invite-only Preview per dags dato mens de også jobber med stabilitet og funksjonalitet.

Både Azure og AWS uttrykker forventning om at deres Kubernetes tjenester skal være klare for produksjonsmiljø ila 2018.

Forberedelser

Du behøver Azure-CLI (versjon 2.0.21 eller nyere) for å utføre kommandoene:

Alle kommandoer gjøres i Windows PowerShell.

Azure innlogging

Logg på Azure:

az login

Du får en link som du åpner i din browser samt en autentiseringskode. Skriv koden på nettsiden og az login lagrer påloggingsinformasjonen slik at du ikke behøver å autentisere igjen på samme maskin.

PS Pålogingsinformasjonen lagres i C:\Users\Brukernavn\.azure\. Du må selv passe på at ingen kopierer disse filene. Da får de full tilgang til din Azure konto.

Aktiver ContainerService

Siden AKS er i Preview/Beta må du eksplisitt aktivere det for å få tilgang til aks kommandoene.

az provider register -n Microsoft.ContainerService
az provider show -n Microsoft.ContainerService

Opprett en resource group

Her oppretter vi en resource group med navn “min_aks_rg” i Azure region West Europe.

az group create --name min_aks_rg --location westeurope

Protip For å se en liste over tilgjengelige Azure regioner, bruk kommandoen az account list-locations --output table. PS Det kan hende AKS ikke er tilgjengelig i alle regioner enda.

Opprette Kubernetes cluster

az aks create --resource-group min_aks_rg --name mitt_cluster --node-count 3 --generate-ssh-keys --node-vm-size Standard_B2s --node-osdisk-size 256 --kubernetes-version 1.8.2

--node-count
- Antall vertsmaskiner tilgjengelig for å kjøre containers
--generate-ssh-keys
- Oppretter og outputter en SSH key som kan brukes for å SSHe direkte til vertsmaskinene.
--node-vm-size
- Hvilken type Azure VM clusteret skal bestå av. For å se tilgjengelige størrelser bruk az vm list-sizes -l westeurope --output table og Microsofts nettsider.
--node-osdisk-size
- Disk størrelse på vertsmaskiner i GB. PS Conteinere kan bli stoppet og flyttet til en annen host ved behov eller hvis en vertsmaskin forsvinner. Alle data lagret lokalt i conteineren blir da borte. Hvis en skal lagre ting permanent må en bruke PersistentVolumes og ikke lokal disk på vertsmaskin.
--kubernetes-version
- Hvilken Kubernetes versjon som skal installeres. Azure installerer IKKE den siste versjonen som standard, og per dags dato fungerer ikke az aks upgrade tilstrekkelig. Siste tilgjengelige versjon per dags dato er 1.8.2. Det er en fordel å bruke siste versjon da det skjer store forbedringer i Kubernetes fra versjon til versjon. Dokumentasjon er også mye bedre for nyere versjoner.

Lagre teksten som kommandoen spytter ut i en fil på en trygg plass. Den inneholder nøkler som kan brukes for å kople til clusteret med SSH. Selv om det i teorien ikke skal være nødvendig.

Installer kubectl

kubectl er klienten som gjør alle operasjoner mot ditt Kubernetes cluster. Azure CLI kan installere kubectl for deg:

az aks install-cli

Etter kubectl er installert behøver vi å få påloggingsinformasjon slik at kubectl kan kommunisere med Kubernetes clusteret.

az aks get-credentials --resource-group min_aks_rg --name mitt_cluster

Påloggingsinformasjonen lagres i C:\Users\Brukernavn\.kube\config. Hold disse filene hemmelig også.

Protip Når en har flere ulike Kubernetes clusters kan en bytte hvilken kubectl skal snakke til med kubectl config get-contexts og kubectl config set-context mitt_cluster.

Inspiser cluster

For å se at clusteret og kubectl virker begynner vi med noen kommandoer.

Se alle vertsmaskiner og status:

> kubectl get nodes
NAME                       STATUS    AGE       VERSION
aks-nodepool1-16970026-0   Ready     15m       v1.8.2
aks-nodepool1-16970026-1   Ready     15m       v1.8.2
aks-nodepool1-16970026-2   Ready     15m       v1.8.2

Se alle tjenester, pods, deployments:

> kubectl get all --all-namespaces

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   po/kubernetes-dashboard-6fc8cf9586-frpkn      1/1       Running   0          3d

NAMESPACE     NAME                          CLUSTER-IP     EXTERNAL-IP     PORT(S)           AGE
kube-system   svc/kubernetes-dashboard      10.0.161.132   <none>          80/TCP            3d

NAMESPACE     NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deploy/kubernetes-dashboard      1         1         1            1           3d

NAMESPACE     NAME                                    DESIRED   CURRENT   READY     AGE
kube-system   rs/kubernetes-dashboard-6fc8cf9586      1         1         1         3d

Jeg har bare tatt et lite utdrag fra denne kommandoen. Du behøver ikke å forstå hva alle ressursene i kube-system namespacet gjør. Det er hensikten at du skal slippe det når Microsoft står for management av selve clusteret.

Namespaces I Kubernetes er det noe som heter Namespaces. Ressurser i ett namespace har ikke automatisk tilgang til ressurser i et annet namespace. Tjenestene som Kubernetes selv benytter installeres i namespacet kube-system. Kommandoen kubectl viser deg vanligvis bare ressurser i default namespace med mindre du spesifiserer --all-namespaces eller --namespace=xx.

Starte noen nginx containere

En instans av en kjørende container kalles i Kubernetes for en Pod.

nginx er en rask og fleksibel webserver.

Nå som clusteret er oppe å kjøre kan vi begynne å rulle ut tjenster og deployments på det.

Vi begynner med å lage en Deployment bestående av 3 containere som alle kjører nginx:mainline-alpine imaget fra Docker hub.

nginx-dep.yaml ser slik ut:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:mainline-alpine
        ports:
        - containerPort: 80

Last denne inn på clusteret med kubectl create:

kubectl create -f /attachments/2017-12-23-managed-kubernetes-on-azure/nginx-dep.yaml

Denne kommandoen oppretter ressursene beskrevet i filen. kubectl kan lese filer enten lokalt fra din maskin eller fra en URL.

Etter du har gjort endringer i en ressurs-definisjon (.yaml fil) kan du oppdatere ressursene i clusteret med kubectl replace -f ressurs.yaml

Vi kan verifisere at Deployment er klar:

> kubectl get deploy
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3         3         3            3           10m

Vi kan også hente de faktiske Pods som er startet:

> kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-569477d6d8-dqwx5   1/1       Running   0          10m
nginx-deployment-569477d6d8-xwzpw   1/1       Running   0          10m
nginx-deployment-569477d6d8-z5tfk   1/1       Running   0          10m

Logger Vi kan se logger fra én pod med kubectl logs nginx-deployment-569477d6d8-xwzpw. Men siden vi i dette tilfellet ikke vet hvilken Pod som ender opp med å få innkommende forespørsler kan vi se logger fra alle Pods som har app=nginx label: kubectl logs -lapp=nginx. At vi her bruker app=nginx har vi selv bestemt i nginx-dep.yaml når vi satt spec.template.metadata.labels: app: nginx.

Gjøre nginx tilgjengelig med en tjeneste

For å kommunisere med våre nye Pods behøver vi å opprette en tjeneste (Service). En tjeneste består av en eller flere Pods som velges basert på ulike kriterier, blant annet hvilke labels de har og om Podene det gjelder er Running og Ready.

Nå lager vi en tjeneste som ruter trafikk til alle Pods som har label app: nginx og som lytter på port 80. I tillegg gjør vi tjenesten tilgjengelig via en LoadBalancer:

nginx-svc.yaml ser slik ut:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx

Vi ber Kubernetes om å opprette tjeneten vår med kubectl create som vanlig:

kubectl create -f /attachments/2017-12-23-managed-kubernetes-on-azure/nginx-svc.yaml

Deretter kan vi se hvilken IP-adresse tjenesten vår har fått av Azure:

> kubectl get svc -w
NAME         CLUSTER-IP   EXTERNAL-IP     PORT(S)        AGE
nginx        10.0.24.11   13.95.173.255   80:31522/TCP   15m

PS Det kan ta et par minutter for Azure å tildele tjenesten vår en Public IP, i mellomtiden vil det stå <pending> under EXTERNAL-IP.

En enkel Welcome to nginx webside skal nå være tilgjengelig på http://13.95.173.255 (husk å bytt ut med din egen External-IP).

Vi har nå en lastbalansert nginx tjeneste med 3 servere klar til å ta imot trafikk.

For ordens skyld kan vi slette tjeneste og deployment etterpå:

kubectl delete svc nginx
kubectl delete deploy nginx-deployment

Skalere cluster

Hvis en ønsker å endre antall vertsmaskiner/noder som kjører Pods kan en gjøre det via Azure-CLI:

az aks scale --name mitt_cluster --resource-group min_aks_rg --node-count 5

For øyeblikket blir alle noder opprettet med samme størrelse som når clusteret ble opprettet. AKS vil antageligvis få støtte for node-pools i løpet av neste år. Da kan en opprette grupper av noder med forskjellig størrelse og operativsystem, både Linux og Windows.

Slette cluster

En kan slette hele clusteret slik:

az aks delete --name mitt_cluster --resource-group min_aks_rg --yes

Bonusmateriale

Her er litt bonusmateriale dersom du ønsker å gå enda litt videre med Kubernetes.

Rulle ut tjenester med Helm

Helm er en pakke-behandler og et bibliotek av programvare som er klart for å rulles ut i et Kubernetes-cluster.

Start med å laste ned Helm-klienten. Den henter påloggingsinformasjon osv fra samme sted som kubectl automatisk.

Installer Helm-serveren (Tiller) på Kubernetes clusteret og oppdater pakke-biblioteket:

helm init
helm repo update

Se tilgjengelige pakker (Charts) med: helm search.

Rulle ut MineCraft med Helm

La oss rulle ut en MineCraft installasjon på clusteret vårt, fordi vi kan :-)

helm install --name stian-sin --set minecraftServer.eula=true stable/minecraft

--set overstyrer en eller flere av standardverdiene som er satt i pakken. MineCraft pakken er laget slik at den ikke starter uten å ha sagt seg enig i brukervilkårene i variabelen minecraftServer.eula. Alle variablene som kan overstyres i MineCraft pakken er dokumentert her.

Så venter vi litt på at Azure skal tildele en Public IP:

> kubectl get svc -w
stian-sin-minecraft   10.0.237.0   13.95.172.192   25565:30356/TCP   3m

Og vipps kan vi kople til Minecraft på 13.95.172.192:25565.

Kubernetes Dashboard

Kubernetes har også et grafisk web-grensesnitt som gjør det litt lettere å se hvilke ressurser som er i clusteret, se logger og åpne remote-shell inne i en kjørende Pod, blant annet.

> kubectl proxy
Starting to serve on 127.0.0.1:8001

kubectl krypterer og tunnelerer trafikken inn til Kubernetes’ API servere. Dashboardet er tilgjengelig på http://127.0.0.1:8001/ui/.

Konklusjon

Jeg håper du har fått mersmak for Kubernetes. Lærekurven kan være litt bratt i begynnelsen men det tar ikke så veldig lang tid før du er produktiv.

Se på de offisielle guidene på Kubernetes.io for å lære mer om hvordan du definerer forskjellige typer ressurser og tjenester for å kjøre på Kubernetes. PS: Det gjøres store endringer fra versjon til versjon så sørg for å bruke dokumentasjonen for riktig versjon!

Kubernetes har også et veldig aktivt Slack-miljø på kubernetes.slack.com. Der er det også en kanal for norske Kubernetes brukere; #norw-users.

Next generation monitoring with OpenTSDB

Mon, 02 Jun 2014 19:56:40 GMT

Step by step guide on how to install a single-instance of OpenTSDB using the latest versions of Hadoop and HBase.

2021 Update: The specific tools discussed in this blog post should be considered obsolete by todays standards. You should investigate Prometheus, InfluxDB and TimescaleDB for your monitoring needs.

In this paper we will provide a step by step guide on how to install a single-instance of OpenTSDB using the latest versions of the underlying technology, Hadoop and HBase. We will also provide some background on the state of existing monitoring solutions.

Table of contents

Abstract
Background
The monitoring revolution
Setting up a single node OpenTSDB instance on Debian 7 Wheezy
Installing OpenTSDB
Feeding data into OpenTSDB
Performance comparison

Background

Since its inception in 1999 rrdtool (the underlying storage mechanism of once universal MRTG) has been the base of many popular monitoring solutions; Cacti, collectd, Ganglia, Munin, Observium, OpenNMS and Zenoss, to name a few.

There are a number of problems with the current approach and we will highlight some of these here.

Please note that this includes Graphite and its backend Whisper, which is based on the same basic design as rrdtool and has some of the same limitations.

Performance problems - Welcome to I/O-hell

When MRTG and rrdtool was created the preservation of disk space was more important than preservation of disk operations and the default collection interval was 5 minutes (which many are still using). The way rrdtool is designed it requires quite a few random reads and writes per datapoint. It also re-reads, computes the average, and writes old data again according to the RRA rules defined which causes additional I/O load. In 2014 memory is cheap, disk storage is cheap and CPU is fairly cheap. Disk I/O operations (IOPS) however are still very expensive in terms of hardware. The recent maturing of SSD provides extreme amounts of IOPS for a reasonable price, but the drive sizes are fractional. The result is that in order to scale IOPS-wise you currently need many low-space SSDs to get the required space, or many low-IOPS spindle drives to get the required IOPS:

Samsung EVO 840 1TB SSD - 98.000 IOPS - 470 USD

Seagate Barracuda 3TB - 240 IOPS - 110 USD

You would need $44.880 (408 drives) worth of spindle drives in order to match a single SSD drive in terms of I/O-performance. On the other hand a $2.000 array of spindle drives would get you a net ~54 TB of space. The cost of SSD to reach the same volume would be $25.380. Not to mention the cost of servers, power, provisioning, etc.

Note: This is the cheapest available bulk consumer drives and comparable OEM drives (SSD, spindle) for a HP server will be 6 to 30 times more expensive.

In rrdtool version 1.4, released in 2009, rrdcached was introduced as a caching daemon for buffering multiple data updates and reducing the number of random I/O operations by writing several related datapoints in sequence. It took a couple of years before this new feature was implemented in most of the common open source monitoring solutions.

For a good introduction into the internals of rrdtool/rrdcached updates and the problems with I/O scaling look at presentation by Sebastian Harl, How to Escape the I/O Hell

Scaling problems

Most of today’s monitoring systems do not easily scale-out. Scale-out, or scaling horizontally, is when you can add new nodes in response to increased load. Scaling up by replacing existing hardware with state-of-the-art hardware is both expensive and only buys you limited time before the next even more expensive necessary hardware upgrade. Many systems offer distributed polling but none offer the option of spreading out the disk load. For example; you can scale Zenozz for High Availability but not performance.

Loss of detail

Current RRD based systems will aggregate old data into averages in order to save storage space. Most technicians do not have the in depth knowledge in order to tune the rules for aggregation and will leave the default values as is. Using cacti as an example and looking at the cacti documentation we see that in a very short time, 2 months, data is averaged to a single data point PER DAY. For systems such as Internet backbones where traffic vary a lot from bottom (30% utilization for example) to peak (90% utilization for example) during a day only the average of 60% is shown in the graphs. This in turn makes troubleshooting by comparing old data difficult. It makes trending based on peaks/bottoms impossible and it may also lead to wrong or delayed strategic decisions on where to invest in added capacity.

Lack of flexibility

In order to collect, store and graph new kinds of metrics an operator would need a certain level of programming skills and experience with the internals of the monitoring system. Adding new metrics to the systems would range from hours to weeks depending on the skill and experience of the operator. Creating new graphs based on existing metrics is also very difficult on most systems. And not within reach for the average operator.

The monitoring revolution

We are currently at the beginning of a monitoring revolution. The advent of cloud computing and big data has created a need for measuring lots of metrics for thousands of machines at small intervals. This has sparked the creation of completely new monitoring components. One of the components where we now have improved alternatives is for efficient metric storage.

The first is OpenTSDB, a “Scalable, Distributed, Time Series Database” that begun development at StumbleUpon in 2011 and aimed at solving some of the problems with existing monitoring systems. OpenTSDB is built in top of Apache HBase which is a scalable and performant database that builds on top of Apache Hadoop. Hadoop is a series of tools for building large and scalable distributed systems. Back in 2010 Facebook already had 2000 machines in a Hadoop cluster with 21PB (that is 21.000.000 GB) of combined storage.

The second is an interesting newcommer, InfluxDB, that began development in 2013 and has the goal of offering scalability and performance without the requirements of HBase/Hadoop.

In addition to advances in performance these alternatives also decouple storage of metrics and display of graphs and abstract the interaction in simple and well-defined APIs. This makes it easy for developers to create improved frontends rapidly and this has already resulted in several very attractive open-source frontends such as Metrilyx (OpenTSDB), Grafana (InfluxDB, Graphite, soon OpenTSDB), StatusWolf (OpenTSDB), Influga (InfluxDB).

Setting up a single node OpenTSDB instance on Debian 7 Wheezy

In the rest of this paper we will set up a single node OpenTSDB instance. OpenTSDB builds on top of HBase and Hadoop and scales to very large setups easily. But it also delivers substantial performance on a single node which is deployed in less than an hour. There are plenty of guides on installing a Hadoop cluster but here we will focus on the natural first step of getting a single node running using recent releases of the relevant software:

OpenTSDB 2.0.0 - Released 2014-05-05
HBase 0.98.2 - Released 2014-05-01
Hadoop 2.4.0 - Released 2014-04-07

If you later require to deploy a larger cluster consider using a framework such as Cloudera CDH or Hortonworks HDP which are open-source platforms which package Apache Hadoop components and provides a fully tested environment and easy-to-use graphical frontends for configuration and management. It is recommended to have at least 5 machines in a HBase cluster supporting OpenTSDB.

This guide assumes you are somewhat familiar with using a Linux shell/command prompt.

Hardware requirements

CPU cores: Max (Limit to 50% of your available CPU resources)
RAM: Min 16 GB
Disk 1 - OS: 10 GB - Thin provisioned
Disk 2 - Data: 100 GB - Thin provisioned

Operating system requirements

This guide is based on a recently installed Debian 7 Wheezy 64bit installed without any extra packages. See the official documentation for more information.

All commands are entered as root user unless otherwise noted.

Pre-setup preparations

We start by installing a few tools that we will need later.

apt-get install wget make gcc g++ cmake maven

Create a new ext3 partition on the data disk /dev/sdb:

(echo "n"; echo "p"; echo ""; echo ""; echo ""; echo "t"; echo "83"; echo "w") | fdisk /dev/sdb

mkfs.ext3 /dev/sdb1

ext3 is the recommended filesystem for Hadoop.

Create a mountpoint /mnt/data1 and add it to the file system table and mount the disk:

mkdir /mnt/data1
echo "/dev/sdb1     /mnt/data1    ext3    auto,noexec,noatime,nodiratime   0   1" | tee -a /etc/fstab
mount /mnt/data1

Using noexec for the data partition will increase security as nothing on the data partition will be allowed to ever execute. Using noatime and nodiratime increases performance since the read access timestamps are not updated on every file access.

Installing java from packages

Installing java on Linux can be quite challenging due to licensing issues, but thanks to the guys over at Launchpad.net who are providing a repository with a custom java package this can now be done quite easy.

We start by adding the launchpad java repository to our /etc/apt/sources.list file:

echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list

Add the signing key and download information from the new repository:

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
apt-get update

Run the java installer:

apt-get install oracle-java7-installer

Follow the instructions on screen to complete the Java 7 installation.

Installing HBase

OpenTSDB has its own HBase installation tutorial here. It is very brief and does not use the latest versions or snappy compression.

Download and unpack HBase:

cd /opt
wget http://apache.vianett.no/hbase/hbase-0.98.2/hbase-0.98.2-hadoop2-bin.tar.gz
tar xvfz hbase-0.98.2-hadoop2-bin.tar.gz
export HBASEDIR=`pwd`/hbase-0.98.2-hadoop2/

Increase the system-wide limitations of open files and processes from the default of 1000 to 32000 by adding a few lines to /etc/security/limits.conf:

echo "root    -               nofile  32768" | tee -a /etc/security/limits.conf
echo "root    soft/hard       nproc   32000" | tee -a /etc/security/limits.conf
echo "*    -               nofile  32768" | tee -a /etc/security/limits.conf
echo "*    soft/hard       nproc   32000" | tee -a /etc/security/limits.conf

The settings above will only take effect if we also add a line to /etc/pam.d/common-session:

echo "session required  pam_limits.so" | tee -a /etc/pam.d/common-session

Install snappy

Snappy is a compression algorithm that values speed over compression ratio and this makes it a good choice for high throughput applications such as Hadoop/HBase. Due to licensing issues Snappy does not ship with HBase and need to be installed on top.

The installation process is a bit complicated and has caused headache for many people (me included). Here we will show a method of installing snappy and getting it to work with the latest version of HBase and Hadoop.

Compression algorithms in HBase Compression is the method of reducing the size of a file or text without losing any of the contents. There are many compression algorithms available and some focus on being able to create the smallest compressed file at the cost of time and CPU usage while other achieve reasonable compression ratio while being very fast. Out of the box HBase supports gz(gzip/zlib), snappy and lzo. Only gz is included due to licensing issues. Unfortunately gz is a slow and costly algorithm compared to snappy and lzo. In a test performed by Yahoo (see slides here, page 8) gz achieves 64% compression in 32 seconds. lzo 47% in 4.8 seconds and snappy 42% in 4.0 seconds. lz4 is another protocol considered for inclusion that is even faster (2.4 seconds) but requires much more memory. For more information look at the Apache HBase Handbook - Appendix C - Compression

Building native libhadoop and libsnappy

In order to use compression we need the common Hadoop library, libhadoop.so, and the snappy library, libsnappy.so. HBase ships without libhadoop.so and the libhadoop.so that ships in the Hadoop Package is only for 32 bit OS. So we need to compile these files ourself.

Start by downloading and installing ProtoBuf. Hadoop requres version 2.5+ which is not available as a Debian package unfortunately.

wget --no-check-certificate https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure; make; make install
export LD_LIBRARY_PATH=/usr/local/lib/

Download and compile Hadoop:

apt-get install zlib1g-dev
wget http://apache.uib.no/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz
tar zxvf hadoop-2.4.0-src.tar.gz
cd hadoop-2.4.0-src/hadoop-common-project/
mvn package -Pdist,native -Dskiptests -Dtar -Drequire.snappy -DskipTests

Copy the newly compiled native libhadoop library into /usr/local/lib, then create the folder in which HBase looks for it and create a shortcut from there to /usr/local/lib/libhadoop.so:

cp hadoop-common/target/native/target/usr/local/lib/libhadoop.* /usr/local/lib
mkdir -p $HBASEDIR/lib/native/Linux-amd64-64/
cd $HBASEDIR/lib/native/Linux-amd64-64/
ln -s /usr/local/lib/libhadoop.so* .

Install snappy from Debian packages:

apt-get install libsnappy-dev

Configuring HBase

Now we need to do some basic configuration before we can start HBase. The configuration files are in $HBASEDIR/conf/.

conf/hbase-env.sh

A shell script setting various environment variables related to how HBase and Java should behave. The file contains a lot of options and they are all documented by comments so feel free to look around in it.

Start by setting the JAVA_HOME, which points to where Java is installed:

export JAVA_HOME=/usr/lib/jvm/java-7-oracle/

Then increase the size of the Java Heap from the default of 1000 which is a bit low:

export HBASE_HEAPSIZE=8000

conf/hbase-site.xml

An XML file containing HBase specific configuration parameters.

<configuration>

   <property>
    <name>hbase.rootdir</name>
    <value>/mnt/data1/hbase</value>
  </property>
  
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/mnt/data1/zookeeper</value>
  </property>

</configuration>

Testing HBase and compression

Now that we have installed snappy and configured HBase we can verify that HBase is working and that the compression is loaded by doing:

$HBASEDIR/bin/hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy

This should output some lines with information and end with SUCCESS.

Starting HBase

HBase ships with scripts for starting and stopping it, namely start-hbase.sh and stop-hbase.sh. You start HBase with

$HBASEDIR/bin/start-hbase.sh

Then look at the log to ensure it has started without any serious errors:

tail -fn100 $HBASEDIR/bin/../logs/hbase-root-master-opentsdb.log

If you want HBase to start automatically on boot you can use a process management tool such as Monit or simply put it in /etc/rc.local:

/opt/hbase-0.98.2-hadoop2/bin/start-hbase.sh

Installing OpenTSDB

Start by installing gnuplot, which is used by the native webui to draw graphs:

apt-get install gnuplot

Then download and install OpenTSDB:

wget https://github.com/OpenTSDB/opentsdb/releases/download/v2.0.0/opentsdb-2.0.0_all.deb
dpkg -i opentsdb-2.0.0_all.deb

Configuring OpenTSDB

The configuration file is /etc/opentsdb/opentsdb.conf. It has some of the basic configuration parameters but not nearly all of them. Here is the official documentation with all configuration parameters.

The defaults are reasonable but we need to make a few tweaks, the first is to add this:

tsd.core.auto_create_metrics = true

This will make OpenTSDB accept previously unseen metrics and add them to the database. This is very useful in the beginning when feeding data into OpenTSDB. Without this you will have to use the command mkmetric for each metric you will store and get errors that might be hard to trace if the metric you create do not match what is actually sent.

Then we will add support for chunked requests via the HTTP API:

tsd.http.request.enable_chunked = true
tsd.http.request.max_chunk = 16000

Some tools and plugins (such as our own improved collectd to OpenTSDB plugin) send multiple data points in a single HTTP request for increased efficiency and requires this setting to be enabled.

Creating HBase tables

Before we start OpenTSDB we need to create the necessary tables in HBase:

env COMPRESSION=SNAPPY HBASE_HOME=$HBASEDIR /usr/share/opentsdb/tools/create_table.sh

Starting OpenTSDB

Since version 2.0.0 OpenTSDB ships as a Debian package and includes SysV init scripts. To start OpenTSDB as a daemon running in the background we run:

service opentsdb start

And then check the logs for any errors or other relevant information:

tail -f /var/log/opentsdb/opentsdb.log

If the server is started successfully the last line of the log should say:

13:42:30.900 INFO  [TSDMain.main] - Ready to serve on /0.0.0.0:4242

And you can now browse to your new OpenTSDB in a browser using http://hostname:4242 !

Feeding data into OpenTSDB

It is not within the scope of this paper to go into details about how to feed data into OpenTSDB but we will give a quick introduction here to get you started.

A note on metric naming in OpenTSDB Each datapoint has a metric name such as df.bytes.free and one or more tags such as host=server1 and mount=/mnt/data1. This is closer to the proposed Metrics 2.0 standard for naming metrics than the traditional naming of df.bytes.free.server1.mnt-data. This makes it possible to create aggregates across tags and combine data easily using the tags. OpenTSDB stores each datapoint with a given metric and tags in one HBase row per hour. But due to a HBase issue it still has to scan every row that matches the metric, ignoring the tags. Even though it will only return the data also matching the tags. This results in very much data being read and it will be very slow to read if there is a large number of data points for a given metric. The default for the collectd-opentsdb plugin is to use the read plugin name as metric, and other values as tags. In my case this results in 72.000.000 datapoints per hour for this metric. When generating a graph all of this data has to be read and evaluated before drawing a graph. 24 hours of data is over 1.7 billion datapoints for this single metric and results in a read performance of 5-15 minutes for a simple graph. A solution to this is to use shift-to-metric, as mentioned in the OpenTSDB user guide. Shift-to-metric is simply moving one or more data identifiers from tags to the metric in order to reduce the cardinality (number of values) for a metric, and hence the time required to read out the data we want. We have modified the collectd-opentsdb java plugin in order to shift the tags to metrics, and this increases read-performance by ~1000x down to 10-100ms. Read the section about collectd below for more information on our modified plugin.

tcollector

tcollector is the default agent for collecting and sending data from a Linux server to a OpenTSDB server. It is based on Python and plugins / addons can be written in any language. It ships with the most common plugins to collect information about disk usage and performance, cpu and memory statistics and also for some specific systems such as mysql, mongodb, riak, varnish, postgresql and others. tcollector is very lightweight and features advanced de-duplication in order to reduce unneeded network traffic.

The commands for installing dependencies and downloading tcollector are

aptitude install git python
cd /opt
git clone git://github.com/OpenTSDB/tcollector.git

Configuration is in the startup script tcollector/startstop, you will need to uncomment and set the value of TSD_HOST to point to your OpenTSDB server.

To start it run

/opt/tcollector/startstop start

This is also the command you want to add to /etc/rc.local in order to have the agent automatically start at boot. Logfiles are saved in /var/log/tcollector.log and they are rotated automatically.

peritus-tc-tools

We have developed a set of tcollector plugins for collecting statistics from

ISC DHCPd server, about number of DHCP events and DHCP pool sizes
OpenSIPS, total number of subscribers and registered user agents
Atmail, number of users, admins, sent and received emails, logins and errors

As well as a high performance replacement for smokeping called tc-ping.

These plugins are available for download from our GitHub page.

collectd-opentsdb

collectd is the system statistics collection daemon and is a widely used system for collecting metrics from various sources. There are several options for sending data from collectd to OpenTSDB but one way that works well is to use the collectd-opentsdb java write plugin.

Since collectd is a generic metric collection tool the original collectd-opentsdb plugin will use the plugin name (such as snmp) as the metric, and use tags such as host=servername, plugin_instance=ifHcInOctets and type_instance=FastEthernet0/1.

As mentioned in the note on metric naming in OpenTSDB this can be very inefficient when data needs to be read again resulting in read performance potentially thousands of times slower than optimal (<100ms). To alleviate this we have modified the original collectd-opentsdb plugin to store all metadata as part of the metric. This gives metric names such as ifHCInBroadcastPkts.sw01.GigabitEthernet0 and very good read performance.

The modified collectd-opentsdb plugin can be downloaded from our GitHub repository.

Monitoring OpenTSDB

To monitor OpenTSDB itself install tcollector as described above on the OpenTSDB server and set TSD_HOST to localhost in /opt/tcollector/startstop.

You can then go to http://opentsdb-server:4242/#start=1h-ago&end=1s-ago&m=sum:rate:tsd.rpc.received%7Btype=\*%7D&o=&yrange=%5B0:%5D&wxh=1200x600 to view a graph of amount of data received in the last hour.

Performance comparison

Lastly we include a little performance comparison between the latest version of OpenTSDB+HBase+Hadoop, a previous version of OpenTSDB+HBase+Hadoop that we have used for a while as well as rrdcached which ran in production for 4 years at a client.

The workload is gathering and storing metrics from 150 Cisco switches with 8200 ports/interfaces every 5 seconds. This equals about 15.000 points per second.

Collection

Even though it is not the primary focus, we include some data about collection performance for completeness. Collection is done using the latest version of collectd and the builtin SNMP plugin.

NB #1: There is a memory leak in the way collectd’s SNMP plugin uses the underlying libsnmp library and you might need to schedule a restart of the collectd service as a workaround for that if handling large workloads.

NB #2: Due to limitations in the libnetsnmp library you will run into problems if polling many (1000+) devices with a single collectd instance. A workaround is to run multiple collectd instances with fewer hosts. memory leak

Figure 2 shows that collection through SNMP polling consumes about 2200Mhz. We optimized some of the data types and definitions in collectd when moving to OpenTSDB and achieved a 20% performance increase in the polling as seen in Figure 3.

Writing to the native rrdcached write plugin consumes 1300Mhz while our modified collectd-opentsdb plugin consumes 1450Mhz. It is probably possible to create a much more efficient write plugin with more advanced knowledge of concurrency and using a lower level language such as C.

Storage

When considering storage performance we will look at CPU usage and disk IOPS since these are the primary drivers of cost in today’s datacenters.

collectd + rrdcached

CPU usage - 1300Mhz, see Figure 2 above.

OpenTSDB + Hbase 0.96 + Hadoop 1

OpenTSDB + HBase 0.98 + Hadoop 2

Conclusion

Even without tuning, a single instance OpenTSDB installation is able to handle significant amounts of data before running into IO problems. This comes at a cost of CPU, currently OpenTSDB will consume > 300% the amount of CPU cycles compared to rrdcached for storage. But this is offset by a 85-95% reduction in disk load. In absolute terms for our particular set up (one 2 year old HP DL360p Gen8 running VMware vSphere 5.5) CPU usage increased from 15% to 25% while reducing IOPS load from 70% to < 10%.

Fine tuning of parameters (such as Java GC) as well as detailed analysis of memory usage is outside the scope of this brief paper and detailed information may be found elsewhere (51,52,53) for those interested.

Stian Øvrevåge

Configuring Envoy as an edge proxy - through istio

Introduction

Configuring overload manager and global connection limits using a custom Envoy bootstrap

Configuring buffer sizes and connection timeouts via EnvoyFilter

Appendices

Appendix A - Displaying currently active Envoy edge configuration settings

Appendix B - Overload manager metrics

Appendix C - istio installation and configuration at Signicat

Outro

Introduction

My name is Stian Øvrevåge and I’m a passionate technologist and wannabe adventurer.

The canonical overview of my online presence is located at https://stian.omg.lol/.

Summer of 2024

Snapshots of 2024

Kubernetes: Don't use NodeLocal DNSCache on GKE (without Cloud DNS)

Introduction

TL;DR:

NodeLocal DNSCache / CoreDNS

kube-dns / dnsmasq

Am I affected?

Investigation

The initial Problem

Istio / Envoy metrics

Reproducing and troubleshooting HTTP connection problems

Troubleshooting potentially slow or broken DNS lookups

Appendices

Appendix - Enabling additional envoy metrics

Enabling additional envoy metrics

Enable additional metrics on a Deployment or Pod

Enable additional metrics globally (please don’t!)

Viewing stats and metrics

Notes on timeout and retry metrics

Further work

Appendix - Overview of DNS on GKE

Appendix - Reducing DNS lookups in Kubernetes and GKE

Appendix - Overview of GKE DNS with NodeLocal DNSCache

Appendix - Enabling node-local-dns metrics

Appendix - Using dnsperf to test DNS performance

Appendix - Network packet capture without dependencies

Appendix - Verbose logging on kube-dns

Appendix - Analyzing dnsmasq logs

Appendix - Analyzing concurrent TCP connections

Appendix - Analyzing DNS problems based on packet captures

Footnotes

Kubernetes Sidecar Config Drift

What is Sidecar Config Drift?

Background

Kubernetes - Declarative vs imperative

Sidecars

Sidecar injection

The Problem

Consequences

Mitigations

k8s-sidecar-rollout

Yak shaving - Photo drips for my mom

Yak shaving - Photo drips for my mom

Background

Concept

Process and goals

Photo selection and preparation

Photo storage

Sending the e-mail

Gallery

Conclusion

Improvements

Bugs and future improvements

A side quest in API development, observability, Kubernetes and cloud with a hint of database

Background

The (initial) problem

Fixing the (initial) problem

Verifying the (initial) fix

Baseline simple request - HTTP1 1 connections, 20000 requests

Baseline complex request - HTTP1 1 connections, 20000 requests

Verifying the fix for assumed workload

Complex request - HTTP1 6 connections, 500 requests

Complex request - HTTP2 500 “connections”, 500 requests

Side quest: Database optimizations

Determining the next bottleneck

Side quest: Cluster resources and burstable VMs

Appendix - Verbose logging on `kube-dns`

Appendix - Analyzing `dnsmasq` logs