Integrate Fluent Bit into CloudWatch for EKS

Introduction

Managing log data in a containerized environment like Amazon EKS can be a challenge. You need a solution that efficiently collects logs from your applications and data plane components, while offering centralized storage and analysis capabilities. This is where Integrate Fluent Bit into CloudWatch for EKS come in.

Prerequisites for Fluent Bit and CloudWatch Container Insights Integration

Before integrating FluentBit with CloudWatch Container Insights on your Amazon EKS cluster, you need to ensure several key components are in place. This ensures a smooth and successful configuration.

1. CloudWatch Agent with Container Insights (Enhanced Observability):

• This is the foundation for log collection and analysis within your EKS cluster. The CloudWatch agent with Container Insights enabled provides a mechanism to capture container logs and metrics.
• You can deploy the CloudWatch agent using the Amazon EKS add-on or by manually configuring it on your worker nodes.
• Make sure to activate the “enhanced observability” feature during setup. This unlocks additional capabilities for detailed log processing and analysis.
• Refer to the official AWS documentation for detailed instructions on setting up the CloudWatch agent with Container Insights (enhanced observability): https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html

2. IAM Permissions:

• Fluent Bit requires specific IAM permissions to interact with CloudWatch Logs and stream your container logs.
• You’ll need to create an IAM role for the service account used by Fluent Bit. This role should have policies attached that grant it access to create CloudWatch log groups and write logs to those groups.
• The recommended approach is to use the CloudWatchAgentServerPolicy IAM policy. Alternatively, you can create a custom policy with the necessary permissions.

3. Verify Existing Configuration (Optional):

• If you’re already using a log collection solution on your EKS cluster, it’s crucial to verify its configuration before deploying Fluent Bit.
• You might need to adjust existing configurations to avoid conflicts or ensure a smooth transition to Fluent Bit for log collection.

Installing the Amazon CloudWatch Observability EKS Add-on (Optional)

While configuring the CloudWatch agent with Container Insights (enhanced observability) manually provides granular control, the Amazon CloudWatch Observability EKS add-on offers a simpler and more automated approach.

Here’s a detailed breakdown of installing the CloudWatch Observability EKS add-on:

Benefits of Cloudwatch:

• Simplified Deployment: The add-on streamlines the process of deploying the CloudWatch agent with Container Insights. It handles configuration management and ensures consistent deployment across your EKS cluster.
• Reduced Complexity: You don’t need to manually configure individual worker nodes. The add-on takes care of provisioning and managing the CloudWatch agent automatically.
• Faster Time to Value: By automating deployment, the add-on allows you to start collecting logs and leveraging Container Insights features quicker.

Installation Steps:

1. Access the Amazon EKS Console: Navigate to the Amazon EKS service within the AWS Management Console.
2. Select your EKS Cluster: From the list of available clusters, choose the one where you want to enable Container Insights.
3. Navigate to Add-ons: Locate the “Add-ons” section within the cluster details.
4. Search for CloudWatch Observability: Use the search bar to find the “Amazon CloudWatch Observability” add-on.
5. Configure and Review: The add-on configuration options allow you to define specific settings, such as the desired log forwarding behavior and region for CloudWatch Logs storage. Carefully review the options before proceeding.
6. Deploy the Add-on: Once satisfied with the configuration, initiate the deployment process for the CloudWatch Observability add-on.
7. Verification: After successful deployment, verify that the CloudWatch agent with Container Insights (enhanced observability) is running on your cluster nodes. You can use the CloudWatch console or kubectl commands to confirm its presence.

Configuring Fluent Bit for CloudWatch Container Insights

Now that you have the prerequisites in place, let’s delve into configuring Fluent Bit for streaming logs to CloudWatch Logs with Container Insights. This process involves several steps:

1. IAM Role and Permissions

• Create an IAM Role: Begin by creating a new IAM role specifically for Fluent Bit. This role will grant it the necessary permissions to interact with CloudWatch Logs.
• Attach CloudWatchAgentServerPolicy: Attach the CloudWatchAgentServerPolicy IAM policy to the newly created role. This policy grants Fluent Bit the required permissions for creating CloudWatch log groups and writing logs to them.
• IRSA (Optional): If you’re using Instance Role Service Accounts (IRSA) for service accounts in your EKS cluster, you’ll need to attach the IAM role you created to the worker nodes. This ensures Fluent Bit running on the worker nodes can assume the role and access CloudWatch Logs.

Here are some helpful resources for managing IAM roles and policies:

• Creating IAM Roles: Click here https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html
• Attaching Policies to IAM Roles: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-and-attach-iam-policy.html
• Instance Role Service Accounts (IRSA): https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

2. Fluent Bit Configuration File

• Edit the Configuration File: Locate or create the Fluent Bit configuration file, typically named fluent-bit.conf. This file defines how Fluent Bit processes and forwards logs.
• Input Sources: Define input sources within the configuration file to specify where Fluent Bit should collect logs. Common examples include container logs (tail filter) and system logs (systemd filter).
  • You can configure specific paths or use filters to match container log files based on patterns.
• Output Plugin (CloudWatch): Configure the output plugin for CloudWatch Logs. The cloudwatch output plugin allows Fluent Bit to send collected logs to CloudWatch.
  • Specify the region where you want to store your logs in CloudWatch Logs.
  • Define the IAM role ARN (Amazon Resource Name) that Fluent Bit should assume for accessing CloudWatch Logs (the role you created in step 1).
  • Optionally, configure additional parameters like log group names, timestamps, and filtering rules within the output plugin configuration.

Here’s a sample Fluent Bit configuration snippet for reference (replace placeholders with your specific values):

[INPUT]

    Name tail

    Path /var/log/containers/*.log

[OUTPUT]

    Name cloudwatch

    Match *

    # Replace with your desired region

    Region us-east-1

    # Replace with the IAM role ARN you created

    Role_ARN arn:aws:iam::123456789012:role/fluent-bit-cloudwatch-role

    # Optional: Set a custom log group name prefix

    Log_Group_Prefix my-app-logs

[FILTER]

    Name parser

    Key_Name log

    Regex /(?<time>[^ ]*\s+[^- ]+ [^ ]+).* (?<message>.*)/

    Time_Key time

    Time_Format %Y-%m-%d %H:%M:%S

Important Note:

• Ensure the IAM role you created has the necessary permissions for the specific CloudWatch Logs actions you’re performing (e.g., creating log groups and writing logs).
• Refer to the official Fluent Bit documentation for detailed configuration options of input sources, output plugins, and filters: https://docs.fluentbit.io/

3. Deploy Fluent Bit as DaemonSet

• Package the Configuration: Once you’ve configured the Fluent Bit configuration file, package it along with the Fluent Bit binary for deployment.
• Deploy as DaemonSet: Deploy Fluent Bit as a DaemonSet on all worker nodes within your EKS cluster. This ensures a running instance of Fluent Bit on each node, collecting and forwarding logs to CloudWatch Logs.
• Deployment Tools: You can leverage deployment tools like kubectl to create and manage the Fluent Bit DaemonSet within your EKS cluster.

Here are some resources for deploying DaemonSets in Kubernetes:

• Creating DaemonSets: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
• Using kubectl with DaemonSets: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

By following these steps, you’ll have successfully configured Fluent Bit to collect container logs and stream them to CloudWatch Logs through Container Insights. This enables centralized log management, improved observability, and deeper insights into the health and performance of your EKS applications.

Conclusion

This guide has equipped you with the knowledge to integrate Fluent Bit with CloudWatch for EKS Container Insights for your Amazon EKS cluster. By following the steps outlined above, you’ve gained the ability to:

• Efficiently collect container logs and application data.
• Stream logs directly to CloudWatch Logs for centralized storage and analysis.
• Leverage Container Insights for enhanced observability and deeper insights into your EKS cluster health and performance.

This powerful combination empowers you to gain better control over your log management strategy within your EKS environment. With Fluent Bit’s lightweight approach and CloudWatch Container Insights’ robust features, you can ensure a streamlined and informative logging experience for your containerized applications.

FAQs

What is CloudWatch Container Insights?

CloudWatch Container Insights is a feature provided by Amazon Web Services (AWS) for monitoring and analyzing containerized applications running on Amazon Elastic Kubernetes Service (EKS). It enables users to collect, aggregate, and analyze logs and performance metrics from containers.

Why Integrate Fluent Bit into CloudWatch for EKS?

Integrating Fluent Bit with CloudWatch Container Insights allows for efficient and streamlined log collection and monitoring within Amazon EKS clusters. It enables real-time insights into the performance and health of containerized applications, enhancing observability and troubleshooting capabilities.

How do I integrate Fluent Bit with CloudWatch Container Insights?

Integration involves deploying Fluent Bit as a daemonset in your Amazon EKS cluster and configuring it to collect logs and metrics from containerized applications. Fluent Bit then forwards this data to CloudWatch Container Insights, where it’s processed and made available for analysis.

What are the benefits of using Fluent Bit with CloudWatch Container Insights?

By leveraging Fluent Bit for log collection and forwarding, you can benefit from its lightweight footprint and efficient resource utilization. This integration simplifies the setup for log monitoring in Amazon EKS and ensures seamless integration with CloudWatch Container Insights for comprehensive visibility into containerized environments.

Does integrating Fluent Bit with CloudWatch Container Insights incur additional costs?

While Fluent Bit itself is open-source and free to use, CloudWatch Container Insights may incur additional costs based on the volume of logs and metrics ingested and stored. It’s essential to review AWS pricing details to understand any potential cost implications of using this integration.

Are there any prerequisites for integrating Fluent Bit with CloudWatch Container Insights?

Yes, before integrating, ensure you have an Amazon EKS cluster set up and running with the necessary IAM permissions to interact with CloudWatch services. Additionally, you need to have basic familiarity with Kubernetes manifests for deploying Fluent Bit as a daemonset.

Is there any official documentation or guides available for this integration?

Yes, AWS provides detailed documentation and guides for integrating Fluent Bit with CloudWatch Container Insights on Amazon EKS. These resources include step-by-step instructions, configuration examples, and best practices to ensure a smooth integration process.

Latest Post:

• Fluent Bit Operations and Best Practices
• Integrate Fluent Bit into CloudWatch for EKS
• Fluent Bit and OpenSearch
• Demystifying Log Aggregation with Loki
• Fluent Bit for Kubernetes Logging