Deploying OpenSearch and Kafka (Kafdrop) on Kubernetes

Introduction

In the rapidly evolving landscape of cloud-native infrastructure and microservices architecture, deploying OpenSearch and Kafka in a Kubernetes environment presents both challenges and opportunities for modern DevOps teams. This comprehensive guide leverages cutting-edge cloud-native technologies to provide a step-by-step approach to deploying OpenSearch and Kafka (with Kafdrop for monitoring) on a Kubernetes cluster, addressing configuration complexities, security considerations, and network optimization strategies.

The rise of distributed systems, observability platforms, and event-driven architectures has made robust deployment techniques crucial for organizations embracing digital transformation and cloud-native development paradigms. </br>

Prerequisites:

Before diving into the deployment, ensure your infrastructure is prepared with the following cloud-native essentials:

• A running Kubernetes cluster (v1.30 or later) - optimized for scalability and resilience
• Helm installed (v3+) for streamlined package management
• kubectl configured for seamless cluster interactions
• Sufficient CPU and memory resources - critical for high-performance microservices
• Persistent storage provisioned to support stateful applications

Pro tip: Leverage infrastructure-as-code (IaC) and GitOps principles to manage your Kubernetes deployments efficiently, enabling repeatable and version-controlled infrastructure configurations.

Step 1: Deploying OpenSearch on Kubernetes

1.1 Install OpenSearch using Helm

Create a custom values file (custom-values-opensearch.yaml) with the necessary configurations:

yaml
clusterName: "opensearch-cluster"
nodeGroup: "master"

replicas: 1
persistence:
  enabled: false

extraEnvs:
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: "StrongPassword123"

opensearchSecurity:
  enabled: false

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "1"
    memory: "2Gi"

Now install OpenSearch:

sh
helm install my-opensearch opensearch/opensearch \
  --namespace opensearch --create-namespace \
  --values custom-values-opensearch.yaml

Check the status of OpenSearch pods:

sh
kubectl get pods -n opensearch

1.2 Common Issues & Fixes

Issue: OpenSearch Cluster Pending

If OpenSearch pods are stuck in Pending status, check node resources:

sh
kubectl describe node <node-name>

Ensure enough CPU and memory are available.

Issue: OpenSearch Fails to Start Due to Security Plugin

If logs show an error related to OPENSEARCH_INITIAL_ADMIN_PASSWORD:

• Define it in the values file under extraEnvs.
• Restart the deployment:

``sh kubectl rollout restart deployment my-opensearch -n opensearch ``

Issue: DNS Resolution for OpenSearch Fails

sh
nslookup opensearch-cluster-master.opensearch.svc.cluster.local

If this fails:

• Verify CoreDNS is running:

``sh kubectl get pods -n kube-system | grep coredns ``

• Add OpenSearch IP manually in /etc/hosts (temporary fix):

``sh echo "<opensearch-cluster-ip> opensearch-cluster-master.opensearch.svc.cluster.local" | sudo tee -a /etc/hosts ``

Step 2: Deploying OpenSearch Dashboards

2.1 Install OpenSearch Dashboards using Helm

Create a custom values file (custom-values-dashboard.yaml):

yaml
service:
  type: NodePort

opensearchHosts: "https://10.106.60.114:9200"

extraEnvs:
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: "StrongPassword123"

nodeSelector:
  kubernetes.io/hostname: "kladbservicesmaster1237"

Install OpenSearch Dashboards:

sh
helm install opensearch-dashboards opensearch/opensearch-dashboards \
  --namespace opensearch \
  --values custom-values-dashboard.yaml

Find the dashboard URL:

sh
export NODE_PORT=$(kubectl get --namespace opensearch -o jsonpath="{.spec.ports[0].nodePort}" services opensearch-dashboards)
export NODE_IP=$(kubectl get nodes --namespace opensearch -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT

2.2 Common Issues & Fixes

2.2.1 Issue: OpenSearch Dashboards Server Not Ready

Check logs:

sh
kubectl logs -n opensearch -l app.kubernetes.io/name=opensearch-dashboards

Ensure opensearchHosts is correctly set in custom-values-dashboard.yaml. </br> </br>

2.2.2 Issue: Connection Error in Logs

If logs contain ConnectionError: getaddrinfo EBUSY, try:

sh
kubectl exec -it -n opensearch $(kubectl get pod -n opensearch -l app.kubernetes.io/component=opensearch-cluster-master -o jsonpath="{.items[0].metadata.name}") -- curl -k -u admin:StrongPassword123 -X GET https://localhost:9200/_cluster/health?pretty

If Unauthorized, ensure OPENSEARCH_INITIAL_ADMIN_PASSWORD is set correctly.

</br>

Step 3: Deploying Kafka & Kafdrop

3.1 Install Kafka using Helm

sh
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-kafka bitnami/kafka --namespace kafka --create-namespace

3.2 Install Kafdrop for Monitoring

sh
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: kafdrop
  namespace: kafka
spec:
  selector:
    app: kafdrop
  ports:
    - protocol: TCP
      port: 9000
      targetPort: 9000
      nodePort: 32000
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafdrop
  namespace: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafdrop
  template:
    metadata:
      labels:
        app: kafdrop
    spec:
      containers:
      - name: kafdrop
        image: obsidiandynamics/kafdrop:latest
        ports:
        - containerPort: 9000
        env:
        - name: KAFKA_BROKERCONNECT
          value: "PLAINTEXT://my-kafka:9092"
EOF

Find Kafdrop URL:

sh
kubectl get svc -n kafka | grep kafdrop

Access Kafdrop:

sh
http://<NODE-IP>:32000

3.3 Common Issues & Fixes

3.3.1 Issue: Kafka Pods Stuck in Pending

Check node resource allocation:

sh
kubectl describe node <node-name>

Ensure there is enough CPU and memory. </br> </br>

3.3.2 Issue: Kafdrop UI Not Loading

Check logs:

sh
kubectl logs -n kafka -l app=kafdrop

Ensure KAFKA_BROKERCONNECT is set correctly.

</br>

Advanced Considerations for Enterprise Deployment

Observability and Monitoring

While our guide provides a foundational deployment strategy, enterprise-grade implementations should consider:

• Implementing comprehensive logging and tracing
• Integrating advanced monitoring solutions
• Ensuring robust security configurations
• Designing for high availability and disaster recovery

Performance Optimization

Modern distributed systems demand sophisticated scaling and performance strategies:

• Implement horizontal pod autoscaling
• Utilize cluster autoscaler for dynamic resource management
• Configure intelligent resource quotas
• Design for multi-region and multi-cloud deployments

Security Best Practices

In an era of increasing cybersecurity threats:

• Enable network policies
• Implement robust authentication mechanisms
• Regularly update and patch your infrastructure
• Use secrets management for sensitive configurations

Conclusion

This guide demonstrates the power of cloud-native technologies in simplifying complex distributed system deployments. By leveraging Kubernetes, OpenSearch, Kafka, and modern DevOps practices, organizations can build scalable, resilient, and efficient data infrastructure. The journey of cloud-native adoption is ongoing, requiring continuous learning, adaptation, and optimization. Stay curious, embrace emerging technologies, and continuously refine your infrastructure strategies.

Key Takeaways:

• Kubernetes provides a robust platform for complex deployments
• OpenSearch and Kafka are critical components of modern data architectures
• Automation and best practices are essential for successful implementations
• Continuous learning and adaptation are key to cloud-native success