Introduction
Navigating the dynamic landscape of cloud computing demands meticulous attention to two pivotal aspects: efficient log management and fortified access security. With Kubernetes-based applications, especially on AWS EKS, the need for robust logging solutions becomes paramount. Integrating EKS pod logs with OpenSearch presents a compelling solution for comprehensive log aggregation and analysis. This guide offers a concise walkthrough for setting up EKS pod logs on OpenSearch.
Setting Up EKS Pod Logs on OpenSearch
To begin, we'll utilize Terraform to provision an AWS OpenSearch cluster. Here's a snippet of the Terraform code:
hclmodule "opensearch" { source = "cyberlabrs/opensearch/aws" version = "1.0.6" # use latest version name = local.cluster_name region = var.aws_region engine_version = var.engine_version advanced_security_options_enabled = true default_policy_for_fine_grained_access_control = true internal_user_database_enabled = true node_to_node_encryption = true instance_type = var.instance_type cluster_config = { instance_count = var.instance_count dedicated_master_enabled = var.env == "prod" || var.env == "staging" ? true : false dedicated_master_count = 3 dedicated_master_type = var.dedicated_master_type } encrypt_at_rest = { enabled = true } log_publishing_options = { index_logs_enabled = var.index_logs_enabled application_logs_enabled = var.index_logs_enabled application_logs_cw_log_group_arn = var.index_logs_cw_log_group_arn index_logs_cw_log_group_arn = var.index_logs_cw_log_group_arn } custom_endpoint_enabled = var.custom_endpoint_enabled # Conditionally include custom endpoint configurations custom_endpoint = var.custom_endpoint_enabled ? "${var.env}-logging.${var.domain}" : null custom_endpoint_certificate_arn = var.custom_endpoint_enabled ? var.custom_endpoint_certificate_arn : null zone_id = var.custom_endpoint_enabled ? var.zone_id : null create_linked_role = var.create_linked_role #variable to create the linked role volume_size = var.volume_size volume_type = var.volume_type }
Deploy Fluent Bit for EKS pod log collection. Here's a snippet of the Terraform code:
hcl// Terraform code to provision OpenSearch cluster and deploy Fluent Bit module "opensearch_logging" { cluster_name = "opensearch-logging" source = "../../../modules/opensearch" env = var.env create_linked_role = var.create_linked_role instance_count = var.opensearch_instance_type_count instance_type = var.opensearch_instance_type volume_size = var.opensearch_instance_volume_size volume_type = var.opensearch_instance_volume_type engine_version = var.opensearch_logging_engine_version custom_endpoint_enabled = true index_logs_enabled = false } resource "helm_release" "fluentbit" { name = "fluentbit" repository = "https://aws.github.io/eks-charts" chart = "aws-for-fluent-bit" namespace = "kube-system" values = [ <<-EOT # OpenSearch host, awsRegion, httpUser and httpPasswd are dynamically updated during provisioning. # CloudWatch logs are on by default and need to be turned off for this example # See https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit --- opensearch: enabled: true index: "eks-pod-logs" tls: "On" awsAuth: "Off" traceError: "On" host: "${module.opensearch_logging.host}" awsRegion: "${var.region}" httpUser: "admin" httpPasswd: "${module.opensearch_logging.os_password}" cloudWatchLogs: enabled: false EOT ] }
Please find complete code here: https://github.com/18-ashish-sharma/aws-os-eks-logs-terraform
Cost Comparison: OpenSearch vs. CloudWatch
Understanding the Financial Benefits
When evaluating a migration to OpenSearch for log management, it's essential to consider the potential cost savings compared to using CloudWatch. Let's break down the cost comparison to illustrate how OpenSearch can lead to significant savings over time.
Assumptions:
- Log Data Volume: We'll assume an average daily log data volume of 100 GB generated by EKS pods.
- Retention Period: Log data needs to be retained for 30 days for analysis and compliance purposes.
- CloudWatch Pricing: CloudWatch charges $0.50 per GB ingested and stored per month, with additional charges for analysis features.
- OpenSearch Pricing: OpenSearch charges $0.10 per GB stored per month and $0.05 per GB transferred per month. Additionally, there's a monthly cost of $100 for Kibana usage.
Cost Comparison:
CloudWatch Cost:
- Ingestion and storage cost: 100 GB/day 30 days $0.50/GB = $1,500/month
OpenSearch Cost:
- Data storage cost: 100 GB/day 30 days $0.10/GB = $300/month
- Data transfer cost: 100 GB/day 30 days $0.05/GB = $150/month
- Kibana usage cost: $100/month
- Total: $300 + $150 + $100 = $550/month
Potential Monthly Savings:
By migrating from CloudWatch to OpenSearch, the potential monthly savings would be:
CloudWatch Cost - OpenSearch Cost = $1,500 - $550 = $950
Note: Keep in mind that this is a simplified example, and actual savings may vary based on your specific usage patterns and pricing details. It's recommended to ?perform a detailed analysis based on your organization's requirements to accurately assess cost savings when migrating from CloudWatch to OpenSearch.
Further Resources:
For more information on managing users and roles in OpenSearch, refer to the official documentation: OpenSearch User and Role Management.
Conclusion:
The cost comparison clearly demonstrates the significant cost savings that can be achieved by leveraging OpenSearch for log management. With a reduction in monthly expenses of $950, organizations can allocate resources more efficiently while benefiting from enhanced log analysis capabilities offered by Kibana.
