Introduction

Leader election is a crucial pattern in distributed systems where multiple instances or nodes compete to perform certain tasks. In a Kubernetes cluster, leader election can be used to ensure that only one instance is responsible for executing leader-specific tasks at any given time. This blog post will explore how to implement a leader election mechanism in Kubernetes using lease locks.

Overview

The leader election mechanism implemented in Go code relies on Kubernetes coordination features, specifically Lease object in the coordination.k8s.io API Group. Lease locks provide a way to acquire a lease on a shared resource, which can be used to determine the leader among a group of nodes.

Repository

The example code, used for this blog is available on mjasion/golang-k8s-leader-example GitHub repository.

Code Walkthrough

The main function is the entry point of the program. It reads configuration values from environment variables and obtains the Kubernetes clientset by getting access to Kube-Api by ServiceAccount attached to Pod. The application is written to work in Kubernetes Pod, that’s why it is using rest.InClusterConfig() function.

The leader election configuration is set up using the LeaderElectionConfig struct from the Kubernetes client library. It specifies the lease lock, lease duration, renewal deadline, retry period, and callback functions for leader-specific tasks.

leaderElectionConfig := leaderelection.LeaderElectionConfig{
    Lock: &resourcelock.LeaseLock{
        LeaseMeta: metav1.ObjectMeta{
            Name:      lockName,
            Namespace: leaseNamespace,
        },
        Client: clientset.CoordinationV1(),
        LockConfig: resourcelock.ResourceLockConfig{
            Identity: os.Getenv("HOSTNAME"),
        },
    },
    LeaseDuration: time.Duration(leaseDuration) * time.Second,
    RenewDeadline: time.Duration(renewalDeadline) * time.Second,
    RetryPeriod:   time.Duration(retryPeriod) * time.Second,
    Callbacks: leaderelection.LeaderCallbacks{
        OnStartedLeading: onStartedLeading,
        OnStoppedLeading: onStoppedLeading,
    },
    ReleaseOnCancel: true,
}

The most important settings are the lease duration, renewal deadline, and retry period:

• The LeaseDuration specifies how long the lease is valid.
• The RenewDeadline specifies the amount of time that the current node has to renew the lease before it expires.
• The RetryPeriod specifies the amount of time that the current holder of a lease has last updated the lease.

The leader-specific tasks are performed in the onStartedLeading function, which is called when the current node becomes the leader. The updateServiceSelectorToCurrentPod function updates the service selector to include the current pod’s hostname.

func onStartedLeading(ctx context.Context) {
	log.Println("Became leader: ", os.Getenv("HOSTNAME"))
	clientset := getKubeClient()
	updateServiceSelectorToCurrentPod(clientset)
	go func() {
		for {
			select {
			case <-ctx.Done():
				log.Println("Stopped leader loop")
				return
			default:
				log.Println("Performing leader tasks...")
				time.Sleep(1 * time.Second)
			}
		}
	}()
}

The onStoppedLeading function is called when the current node stops being the leader. It can be used for cleanup tasks.

func onStoppedLeading() {
	log.Println("Stopped being leader")
}

A context and a wait group are created to manage goroutines. A goroutine is started to run the leader election using the leaderelection.RunOrDie function.

ctx, cancel := context.WithCancel(context.Background())
defer cancel()
wg := &sync.WaitGroup{}
wg.Add(1)

go func() {
    defer wg.Done()
	leaderelection.RunOrDie(ctx, leaderElectionConfig)
}()

cancel()
wg.Wait()

The program also sets up a Gin router and defines a root endpoint that returns the hostname of the current node, to easily check which Pod is being the leader.

Demo 1 - Deploying a single Pod

In this demo, we will deploy a single Pod to a Kubernetes cluster and observe how the leader election works.

As you can see here, the pod is elected as a leader and performs leader-specific tasks. The lease object contains the information about the current leader in the HOLDER column.

NAME                 HOLDER                               AGE
k8s-leader-example   k8s-leader-example-8dd646bb7-dsfmq   11s

Demo 2 - Deploying multiple Pods and killing the leader

In this demo, we will deploy multiple Pods to a Kubernetes cluster and observe how the leader election works. The settings used for this demo are as follows:

The leader election mechanism will attempt to renew the lease every 5 seconds. If the lease is not renewed within 5 seconds, the leader election mechanism will attempt to acquire the lease. If the lease is not acquired within 1 second, the leader election mechanism will retry to acquire the lease.

Running command kubectl get lease --watch allows to observe the leader election process. The lease object contains first the information about the previous leader, when the leader is killed, and then the information about the new leader.

Conclusion

Implementing leader election in Kubernetes using lease locks is an effective way to ensure that only one instance or node performs leader-specific tasks at a time. In this blog post, we explored the provided Go code that demonstrates how to implement leader election in a Kubernetes cluster.

By incorporating leader election into your distributed system, you can enhance its reliability and prevent conflicts that may arise from multiple instances attempting to execute the same tasks simultaneously.

Istio is a complex system. For the applications, the main component is the sidecar container Istio-Proxy, which proxies all traffic from all containers in Pod. And this can lead to some issues.

This post describes one of the most complicated problems I have encountered in my career.

The problem - Connection Reset 🐛

During Istio rollout on a huge system, with more than 40 different microservices, on a single endpoint, QA engineers found a bug. It was a POST endpoint, which was returning chunked data.

Istio was returning error 502, in logs an additional flag was visible: upstream_reset_before_response_started. The application logs confirmed that the result was correct.

In legacy Istio versions of the presented problem Istio were returning 503 error with UC flag.

Analyzing issue ⛏️

Let’s see the curl response and look at Istio-proxy logs:

kubectl exec -it curl-0 -- curl http://http-chunked:8080/wrong -v
< HTTP/1.1 502 Bad Gateway
< content-length: 87
< content-type: text/plain
< date: Sun, 24 Apr 2022 12:28:28 GMT
< server: istio-envoy
< x-envoy-decorator-operation: http-chunked.default.svc.cluster.local:8080/*
upstream connect error or disconnect/reset before headers. reset reason: protocol error

$ kubectl logs http-chunked-0 -c istio-proxy
[2022-04-24T12:23:37.047Z] "GET /wrong HTTP/1.1" 502 UPE upstream_reset_before_response_started{protocol_error} - "-" 0 87 1001 - "-" "curl/7.80.0" "3987a4cb-2e0e-4de6-af66-7e3447600c73" "http-chunked:8080" "10.244.0.17:8080" inbound|8080|| 127.0.0.6:39063 10.244.0.17:8080 10.244.0.14:35500 - default

Time for spying 🕵🏻‍♂️

To analyze the traffic we can use tcpdump and Wireshark. Istio-proxy runs as a sidecar, which routes whole incoming and outgoing traffic to pod through own proxy.

To sniff traffic there are 3 ways:

1. Running tcpdump in istio-proxy container,
2. Using kubectl plugin ksniff - a plugin to kubectl to dump packets from pod, github repo,
3. Adding additional container to pod, with root permission and tcpdump installed,

The first option will not work by default, because istio-proxy runs without root permission. The third is the backup if 1 and 2 would not work. Let’s try ksniff.

What is ksniff 🛠️

ksniff in three words is a plugin that:

• figures out what node is running pod with an app,
• deploys an own pod with an affinity to that node, bound to the host network,
• opens Wireshark on your laptop with a packet stream from the application.

Let’s execute it to sniff our application:

kubectl sniff http-chunked-0 -c istio-proxy -p -f '-i lo' -n default

Important parameters

• -p is a parameter to support sniffing even if the pod is non-privileged. See docs,
• -f '-i lo' passes filter to tcpdump, we want to sniff localhost interface inside the Pod.

If there is no issue, our system has Wireshark in PATH, ksniff should open a new window

Finding the root cause 🔎

Wireshark will continuously follow with new packet records. It makes it hard to figure out our particular call. We can use filters to help with searching. Knowing the request path, method, response code - we can use it to find our packet using filter:

http.request.uri == "/wrong"

It shows only a single packet, our request. Wireshark allows to show the whole TCP conversation:

• click right click on the packet,
• go to Conversation Filter,
• select TCP.

Wireshark will write a filter to show the whole communication between istio-proxy container and the application container!

Let’s see the above image. The first 3 records are the three-way handshake packets. Later is our GET request. The most interesting happens in the last two packets. Application container returns response HTTP 200 OK. istio-proxy then closes the connection with RST packet.

This is what we saw in the logs. The flag was upstream_reset_before_response_started{protocol_error}. But why? This still does not explain.

Swiss knife by Wireshark 🪛

It is hard to read the HTTP protocol from multiple packet bodies. But Wireshark also has a solution for that. We can see data from L7, the application one. In our case, it is the HTTP protocol.

Click with the right mouse on a single packet, go to the Follow tab, and select TCP Stream:

Now we can check what the request from istio-proxy looked like, and what was the response from the app. Do you have an idea from the above picture?

Look closer at the response, there is a double Transfer-Encoding header. One starts from uppercase, the second one does not.

Double transfer-encoding header - what does it mean❔

Searching over Istio issues I found this answer. The most important are the first 2 points:

1. two transfer-encoding: chunked is equivalent to transfer-encoding: chunked, chunked as per RFC,
2. transfer-encoding: chunked, chunked doesn’t have the same semantic as transfer-encoding: chunked

Why the response was taken as double-chunked? According to Transfer Codings in Section 4, transfer-coding names are case-insensitive.

Summary 📓

As you see, Istio stands as a guard 👮‍♂️ of the HTTP protocol. If the app is returning a double-chunked response, then Istio requires it, otherwise, it rejects processing the request. curl ignores this inconsistency.

This issue was one of the most difficult tasks, which I ever had :-)

Infrastructure to reproduce and example app 🏭

In Github repository I created example infrastructure to reproduce the problem.

Bootstrap of the infrastructure installs ArgoCD, Istio and the App. The sample app exposes two endpoints:

• /correct - endpoint, which creates a streamed response,
• /wrong - is doing same as above, but additionally it set value of the Transfer-Encoding header to Chunked(uppercase).

I would like to thank Przemysław for his help and for showing me how to use Wireshark efficiently during this issue.🤝🏻