99.co is Singapore’s fastest-growing real estate portal. We power our listings search feature with Elasticsearch (ES), a distributed search engine that can perform complicated search queries at a fast speed. Our backend is a microservices architecture running in Google Kubernetes Engine (GKE), which includes the search service.
Our search service was running on GKE, but our ES cluster was not. As seen in the diagram below, we used to run our ES cluster on Docker containers in GCP VM instances. This article will detail how we migrated our ES infrastructure to be fully hosted on a Kubernetes cluster.
While it is acceptable to run an ES cluster self-managed servers, problems might come when a maintenance is needed. Examples include: scaling up the machine’s resources (CPU/memory), performing software upgrades, or adding new ES nodes. We had to do rolling upgrade by spinning down each ES node and server, performing the maintenance, and starting up each machine and container one by one. While it is simple in theory, in practice, things could go south pretty quickly.
Early this July, we experienced a 6 hours outage on one of our three shards while upgrading our ES cluster. A network connection misconfiguration in istio-proxy stopped the rebalancing process and ultimately degraded our service performance below our 99.9% SLA. Although we managed to restore it before business hours, night outages are not something we ever want to go through again. Hence, our search for automation began!
Elastic Cloud on Kubernetes (ECK) is what we needed. Released to the general public on 16th January 2020, Elasticsearch is now fully compatible with Kubernetes. ECK utilizes a Kubernetes operator and Custom Resource Definition (CRD) to provide a high-level abstraction to deploy, package, and manage ES. ECK offers automated deployment to help reduce the risk of human error during the upgrade or maintenance cycle – which would have prevented our night outage.
ECK also offers many other advantages such as:
- Managing and monitoring multiple ES clusters
- Upgrading to versions with ease
- Scaling cluster capacity up and down
- Changing cluster configuration
- Dynamically scaling local storage (including Elastic Local Volume)
- Scheduling and performing backups
Our Elastic Cloud on Kubernetes setup
Below is a diagram of the Elastic Cloud on the Kubernetes cluster we built. Let’s go through each part individually.
Search Service Architecture with ECK
ECK’s Kubernetes CRD and operator have made the setup simple. We only need to add our configuration intoYAML files. Follow along for a step-by-step explanation.
- To start, we need to add ES’s CRD and operator to our Kubernetes cluster.
all-in-one.yamlfile is multiple YAML files bundled together. Kubernetes will create a namespace called
elastic-systemwhere the elastic-operator will be running along with other necessary objects.
- A critical part of the elastic-operator configuration is towards the end of the file, which uses StatefulSet. A pod called
elastic-operator-0will be running in the
elastic-systemnamespace. This operator is responsible for managing and monitoring the ES CRDs, such as scaling the cluster, changing cluster configuration.
- Next, we would see multiple CustomResourceDefinitions (CRDs), such as the Elasticsearch CRD below (not everything is pasted here). This CRD defines how to provide the specifications to the Elasticsearch application later on.
- Now, we should be able to see the
elastic-operator-0pod running in the
elastic-systemnamespace. We can tail the logs with the command below
- The next step is to deploy the Elasticsearch cluster itself!
- Before that, we need to create a separate NodePool in our Kubernetes cluster to host the ES nodes. This will help us isolate the ES nodes from our main NodePool, where most of our microservices are running. In case our microservices were running high on CPU/memory, the resources shared to ES might be consumed up and thus affecting the performance of the entire ES cluster. Another benefit is that we can fine-tune the NodePool resource accurately to the ES nodes, especially since they consume a lot of memory.
- To isolate the elastic-cloud NodePool, we use Kubernetes’ scheduling feature Taint with the NoSchedule effect. Taint must be added during NodePool creation. We also use the
tolerationsfields so that our Elasticsearch pods can tolerate this taint. The whole policy prevents other pods without tolerance to NoSchedule effect for
elastic-only=truefrom scheduling to this NodePool. As a result, only the ES pods can be scheduled in the elastic-cloud NodePool.
- Finally, let’s deploy the ES cluster with its nodes by creating a file called
- In Kubernetes nodeSets, we can specify sets of nodes that we want to deploy. For 99.co setup, we are using 1 nodeSets that have 3 nodes, running as the master node, data node, and ingest node. Internally, the ECK operator will create StatefulSets as each pod (ES Node) needs to be stateful with stable, persistent storage.
- We add initContainers to increase the kernel settings vm.max_map_count to 262144 with privileged mode. This increases the virtual address space that can be mapped to files. Without it, we may encounter out-of-memory exceptions. The documentation recommends it for production usage.
- nodeAffinity of requiredDuringSchedulingIgnoredDuringExecution requires the pods to be scheduled only in elastic-cloud NodePool. This, combined with the
tolerationsdiscussed earlier, will ensure the nodes run only in elastic-cloud NodePool instead of anywhere in the Kubernetes cluster.
- We define the CPU/memory requests and limits, just like how we define resources in Deployments. As for why we need to set the requests & limits, here is an excellent post on the topic.
- For storage, we ask the operator to request each pod a PersistentVolumeClaim of 80Gb with a specific Storage Class. This allows PersistentVolume to be dynamically provisioned without the need to create PV every time we need it. In this case, storageClass of
faster-southeast1-ais an SSD persistent disk type provided by GCE (Google Compute Engine). Another benefit of using Storage Class is that we can expand the volume automatically in case we are running out of space. In such a scenario, ECK will update the existing PVCs accordingly and recreate the StatefulSets automatically.
- We expose the Service as a LoadBalancer via HTTP. It would otherwise default to a ClusterIP service.
- HTTP over TLS is also enabled by default. Behind the scenes, the elastic-operator will create self-signed certificates. We disabled this as the GCP firewall is protecting our intra-services.
- Now, we run
kubectl apply -f elasticsearch.yaml, and voila!
- We can see the cluster object (note that
esis a custom resource definition coming from the
all-in-one.yamlthat we applied at the first step)
- We can also see the ES nodes running as pods
- We can see the StatefulSet and Service created as well. Since we use LoadBalancer as the service, we can call our pods by the external IP directly, in this case 10.148.0.117:9200. If we are only using ClusterIP, we can use
kubectl port-forward service/search-prod-es-http 9200and hit
- The elastic-operator also creates a bunch of Secrets for each cluster that we deploy.
- Next step is to deploy Kibana, which is simpler than the Elasticsearch itself. We create a
- Make sure that elasticsearchRef is filled with the ES cluster name from the
metadata.namefield in the
elasticsearch.yaml. Under the hood, Kibana will create a Deployment instead of StatefulSet, since Kibana doesn’t need any persistent storage.
- Deploy with
kubectl apply -f kibana.yaml, and we can see the kibana running.
Rolling Upgrade with ECK
At this point, our Elasticsearch cluster is already up and running. Let’s step back to addressing our initial problem, how does ECK manage to perform rolling upgrades to the nodes automatically?
The answer is using Update Strategy. Since we did not provide the
updateStrategy in our
elasticsearch.yaml, it will be defaulted to:
1, which means this ensures the cluster has no more than 1 unavailable Pod at any point of time.
-1, which is unbounded, all Pods can be recreated immediately.
With these settings, when changing the Elasticsearch YAML configuration, the Pod will automatically restart. This also ensures that we only have 1 Pod restarting at a time. Our cluster’s health will be marked as Yellow since we have 1 Pod/ES node down, but it is still enough to serve requests.
Here’s a deeper look into the elastic-operator log after we update the
The operator is disabling shards allocation and requesting for a synced flush, precisely what is recommended by the ES documentation when performing a manual rolling upgrade. The process is repeated for the rest of the pods. No more human actions are needed to perform a rolling upgrade 🎉!
Running Elasticsearch on Kubernetes allows developers/admins to utilize container orchestration by Kubernetes and apply best practices on managing Elasticsearch clusters by the Elastic Operator. While Kubernetes adds A level of complexity, it has the benefit of removing manual operations and offers peace of mind to the engineering team.