Log management

Enabling EKS Pod Logs on OpenSearch

by: Ashish Sharma

April 18, 2024



Navigating the dynamic landscape of cloud computing demands meticulous attention to two pivotal aspects: efficient log management and fortified access security. With Kubernetes-based applications, especially on AWS EKS, the need for robust logging solutions becomes paramount. Integrating EKS pod logs with OpenSearch presents a compelling solution for comprehensive log aggregation and analysis. This guide offers a concise walkthrough for setting up EKS pod logs on OpenSearch.

Setting Up EKS Pod Logs on OpenSearch

To begin, we'll utilize Terraform to provision an AWS OpenSearch cluster. Here's a snippet of the Terraform code:

module "opensearch" {
  source                                         = "cyberlabrs/opensearch/aws"
  version                                        = "1.0.6" # use latest version
  name                                           = local.cluster_name
  region                                         = var.aws_region
  engine_version                                 = var.engine_version
  advanced_security_options_enabled              = true
  default_policy_for_fine_grained_access_control = true
  internal_user_database_enabled                 = true
  node_to_node_encryption                        = true
  instance_type                                  = var.instance_type
  cluster_config = {
    instance_count           = var.instance_count
    dedicated_master_enabled = var.env == "prod" || var.env == "staging" ? true : false
    dedicated_master_count   = 3
    dedicated_master_type    = var.dedicated_master_type
  encrypt_at_rest = {
    enabled = true

  log_publishing_options = {
    index_logs_enabled                = var.index_logs_enabled
    application_logs_enabled          = var.index_logs_enabled
    application_logs_cw_log_group_arn = var.index_logs_cw_log_group_arn
    index_logs_cw_log_group_arn       = var.index_logs_cw_log_group_arn

  custom_endpoint_enabled = var.custom_endpoint_enabled

  # Conditionally include custom endpoint configurations
  custom_endpoint = var.custom_endpoint_enabled ? "${var.env}-logging.${var.domain}" : null
  custom_endpoint_certificate_arn = var.custom_endpoint_enabled ? var.custom_endpoint_certificate_arn : null

  zone_id = var.custom_endpoint_enabled ? var.zone_id : null

  create_linked_role = var.create_linked_role #variable to create the linked role
  volume_size        = var.volume_size
  volume_type        = var.volume_type

Deploy Fluent Bit for EKS pod log collection. Here's a snippet of the Terraform code:

// Terraform code to provision OpenSearch cluster and deploy Fluent Bit

module "opensearch_logging" {
  cluster_name            = "opensearch-logging"
  source                  = "../../../modules/opensearch"
  env                     = var.env
  create_linked_role      = var.create_linked_role
  instance_count          = var.opensearch_instance_type_count
  instance_type           = var.opensearch_instance_type
  volume_size             = var.opensearch_instance_volume_size
  volume_type             = var.opensearch_instance_volume_type
  engine_version          = var.opensearch_logging_engine_version
  custom_endpoint_enabled = true
  index_logs_enabled      = false

resource "helm_release" "fluentbit" {
  name       = "fluentbit"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-for-fluent-bit"
  namespace  = "kube-system"

  values = [
    # OpenSearch host, awsRegion, httpUser and httpPasswd are dynamically updated during provisioning.
    # CloudWatch logs are on by default and need to be turned off for this example
    # See https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit
      enabled: true
      index: "eks-pod-logs"
      tls: "On"
      awsAuth: "Off"
      traceError: "On"
      host: "${module.opensearch_logging.host}"
      awsRegion: "${var.region}"
      httpUser: "admin"
      httpPasswd: "${module.opensearch_logging.os_password}"
      enabled: false

Please find complete code here: https://github.com/18-ashish-sharma/aws-os-eks-logs-terraform

Cost Comparison: OpenSearch vs. CloudWatch

Understanding the Financial Benefits

When evaluating a migration to OpenSearch for log management, it's essential to consider the potential cost savings compared to using CloudWatch. Let's break down the cost comparison to illustrate how OpenSearch can lead to significant savings over time.


  • Log Data Volume: We'll assume an average daily log data volume of 100 GB generated by EKS pods.
  • Retention Period: Log data needs to be retained for 30 days for analysis and compliance purposes.
  • CloudWatch Pricing: CloudWatch charges $0.50 per GB ingested and stored per month, with additional charges for analysis features.
  • OpenSearch Pricing: OpenSearch charges $0.10 per GB stored per month and $0.05 per GB transferred per month. Additionally, there's a monthly cost of $100 for Kibana usage.

Cost Comparison:

CloudWatch Cost:

  • Ingestion and storage cost: 100 GB/day * 30 days * $0.50/GB = $1,500/month

OpenSearch Cost:

  • Data storage cost: 100 GB/day * 30 days * $0.10/GB = $300/month
  • Data transfer cost: 100 GB/day * 30 days * $0.05/GB = $150/month
  • Kibana usage cost: $100/month
  • Total: $300 + $150 + $100 = $550/month

Potential Monthly Savings:

By migrating from CloudWatch to OpenSearch, the potential monthly savings would be:

CloudWatch Cost - OpenSearch Cost = $1,500 - $550 = $950

Note: Keep in mind that this is a simplified example, and actual savings may vary based on your specific usage patterns and pricing details. It's recommended to ?perform a detailed analysis based on your organization's requirements to accurately assess cost savings when migrating from CloudWatch to OpenSearch.

Further Resources:

For more information on managing users and roles in OpenSearch, refer to the official documentation: OpenSearch User and Role Management.


The cost comparison clearly demonstrates the significant cost savings that can be achieved by leveraging OpenSearch for log management. With a reduction in monthly expenses of $950, organizations can allocate resources more efficiently while benefiting from enhanced log analysis capabilities offered by Kibana.

contact us

Get started now

Get a quote for your project.


Edstem Technologies LLC
254 Chapman Rd, Ste 208 #14734
Newark, Delaware 19702 US


Edstem Technologies Pvt Ltd
Office No-2B-1, Second Floor
Jyothirmaya, Infopark Phase II
Ernakulam, Kerala 682303

© 2024 — Edstem All Rights Reserved

Privacy PolicyTerms of Use