Introduction
Navigating the dynamic landscape of cloud computing demands meticulous attention to two pivotal aspects: efficient log management and fortified access security. With Kubernetes-based applications, especially on AWS EKS, the need for robust logging solutions becomes paramount. Integrating EKS pod logs with OpenSearch presents a compelling solution for comprehensive log aggregation and analysis. This guide offers a concise walkthrough for setting up EKS pod logs on OpenSearch.
Setting Up EKS Pod Logs on OpenSearch
To begin, we'll utilize Terraform to provision an AWS OpenSearch cluster. Here's a snippet of the Terraform code:
module "opensearch" {
source = "cyberlabrs/opensearch/aws"
version = "1.0.6" # use latest version
name = local.cluster_name
region = var.aws_region
engine_version = var.engine_version
advanced_security_options_enabled = true
default_policy_for_fine_grained_access_control = true
internal_user_database_enabled = true
node_to_node_encryption = true
instance_type = var.instance_type
cluster_config = {
instance_count = var.instance_count
dedicated_master_enabled = var.env == "prod" || var.env == "staging" ? true : false
dedicated_master_count = 3
dedicated_master_type = var.dedicated_master_type
}
encrypt_at_rest = {
enabled = true
}
log_publishing_options = {
index_logs_enabled = var.index_logs_enabled
application_logs_enabled = var.index_logs_enabled
application_logs_cw_log_group_arn = var.index_logs_cw_log_group_arn
index_logs_cw_log_group_arn = var.index_logs_cw_log_group_arn
}
custom_endpoint_enabled = var.custom_endpoint_enabled
# Conditionally include custom endpoint configurations
custom_endpoint = var.custom_endpoint_enabled ? "${var.env}-logging.${var.domain}" : null
custom_endpoint_certificate_arn = var.custom_endpoint_enabled ? var.custom_endpoint_certificate_arn : null
zone_id = var.custom_endpoint_enabled ? var.zone_id : null
create_linked_role = var.create_linked_role #variable to create the linked role
volume_size = var.volume_size
volume_type = var.volume_type
}Deploy Fluent Bit for EKS pod log collection. Here's a snippet of the Terraform code:
// Terraform code to provision OpenSearch cluster and deploy Fluent Bit
module "opensearch_logging" {
cluster_name = "opensearch-logging"
source = "../../../modules/opensearch"
env = var.env
create_linked_role = var.create_linked_role
instance_count = var.opensearch_instance_type_count
instance_type = var.opensearch_instance_type
volume_size = var.opensearch_instance_volume_size
volume_type = var.opensearch_instance_volume_type
engine_version = var.opensearch_logging_engine_version
custom_endpoint_enabled = true
index_logs_enabled = false
}
resource "helm_release" "fluentbit" {
name = "fluentbit"
repository = "https://aws.github.io/eks-charts"
chart = "aws-for-fluent-bit"
namespace = "kube-system"
values = [
<<-EOT
# OpenSearch host, awsRegion, httpUser and httpPasswd are dynamically updated during provisioning.
# CloudWatch logs are on by default and need to be turned off for this example
# See https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit
---
opensearch:
enabled: true
index: "eks-pod-logs"
tls: "On"
awsAuth: "Off"
traceError: "On"
host: "${module.opensearch_logging.host}"
awsRegion: "${var.region}"
httpUser: "admin"
httpPasswd: "${module.opensearch_logging.os_password}"
cloudWatchLogs:
enabled: false
EOT
]
}Please find complete code here: https://github.com/18-ashish-sharma/aws-os-eks-logs-terraform
Cost Comparison: OpenSearch vs. CloudWatch
Understanding the Financial Benefits
When evaluating a migration to OpenSearch for log management, it's essential to consider the potential cost savings compared to using CloudWatch. Let's break down the cost comparison to illustrate how OpenSearch can lead to significant savings over time.
Assumptions:
- Log Data Volume: We'll assume an average daily log data volume of 100 GB generated by EKS pods.
- Retention Period: Log data needs to be retained for 30 days for analysis and compliance purposes.
- CloudWatch Pricing: CloudWatch charges $0.50 per GB ingested and stored per month, with additional charges for analysis features.
- OpenSearch Pricing: OpenSearch charges $0.10 per GB stored per month and $0.05 per GB transferred per month. Additionally, there's a monthly cost of $100 for Kibana usage.
Cost Comparison:
CloudWatch Cost:
- Ingestion and storage cost: 100 GB/day * 30 days * $0.50/GB = $1,500/month
OpenSearch Cost:
- Data storage cost: 100 GB/day * 30 days * $0.10/GB = $300/month
- Data transfer cost: 100 GB/day * 30 days * $0.05/GB = $150/month
- Kibana usage cost: $100/month
- Total: $300 + $150 + $100 = $550/month
Potential Monthly Savings:
By migrating from CloudWatch to OpenSearch, the potential monthly savings would be:
CloudWatch Cost - OpenSearch Cost = $1,500 - $550 = $950
Note: Keep in mind that this is a simplified example, and actual savings may vary based on your specific usage patterns and pricing details. It's recommended to ?perform a detailed analysis based on your organization's requirements to accurately assess cost savings when migrating from CloudWatch to OpenSearch.
Further Resources:
For more information on managing users and roles in OpenSearch, refer to the official documentation: OpenSearch User and Role Management.
Conclusion:
The cost comparison clearly demonstrates the significant cost savings that can be achieved by leveraging OpenSearch for log management. With a reduction in monthly expenses of $950, organizations can allocate resources more efficiently while benefiting from enhanced log analysis capabilities offered by Kibana.
