Fluent Bit and OpenSearch

Arzhost lifetime hosting with ultra-fast NVMe SSD storage and LiteSpeed servers
Fluent Bit and OpenSearch guide

Introduction

Fluent Bit and OpenSearch are powerful open-source tools that can be combined to create a robust log analytics solution. Fluent Bit is a lightweight log processor and forwarder, while OpenSearch is a search and analytics engine built from the foundation of Elasticsearch. Together, they provide a scalable and efficient way to collect, process, and analyze log data from various sources.

Installing Fluent Bit with Package Managers

A. Prerequisites

  • Linux system: Fluent Bit is primarily designed for Linux environments.
  • Root or sudo privileges: You’ll need administrative access to install packages.

B. Choosing a Package Manager

The specific installation steps will depend on your Linux distribution’s package manager. Here are common scenarios:

  • Debian/Ubuntu: Use apt-get or apt.
  • Red Hat/CentOS/Fedora: Use yum or dnf.
  • Arch Linux: Use pacman.

C. Installation Steps

  1. Update package lists:
sudo apt-get update  # Debian/Ubuntu

sudo yum update      # Red Hat/CentOS/Fedora

sudo pacman -Syu     # Arch Linux

sudo apt-get install fluent-bit  # Debian/Ubuntu

sudo yum install fluent-bit       # Red Hat/CentOS/Fedora

sudo pacman -S fluent-bit         # Arch Linux

II. Deploying Fluent Bit on Kubernetes

A. Prerequisites

  • Running Kubernetes cluster: Ensure you have a Kubernetes cluster up and running.
  • Helm (package manager for Kubernetes): Helm is highly recommended for managing Fluent Bit and OpenSearch on Kubernetes. If you don’t have it, follow the instructions from the Helm project to install it.

B. Adding the Fluent Helm Chart Repository

  1. Add the Fluent Helm Chart repository to Helm

helm repo add fluent https://fluent.github.io/helm-charts

helm repo update

C. Deploying Fluent Bit with Helm

  1. Create a values file (optional):
    A values file allows you to customize Fluent Bit’s configuration for your deployment. You can find an example values file in the Fluent Helm Chart repository. You can create a custom values file or copy the example and modify it as needed.
  2. Deploy Fluent Bit:
  3. Bash

helm install fluent-bit fluent/fluent-bit [–values values.yaml]

  • Replace [values.yaml] with the path to your custom values file (if used).

Explanation:

  • helm install: This command initiates the installation process.
  • fluent-bit: This is the name you’re assigning to your FluentBit deployment.
  • fluent/fluent-bit: This specifies the Helm chart from the fluent repository to be used for the installation.
  • –values values.yaml (optional): This flag is used if you have a custom values file for configuration.
Installing Fluentbit

III. Verification

A. Using a Package Manager

  1. Check the service status:
sudo systemctl status fluent-bit  # Systemd-based systems

B. Using Kubernetes

kubectl get pods -n <fluent-bit-namespace>  # Replace with your namespace

kubectl logs fluent-bit-<pod-name> -n <fluent-bit-namespace>  # Replace with pod name

By following these steps, you should have successfully installed and deployed Fluent Bit and OpenSearch on your system, either using a package manager or through a Kubernetes deployment with Helm. Remember to consult the official Fluent Bit documentation for more advanced configuration options and troubleshooting.

Understanding the Fluent Bit Helm Chart Options

If you’re deploying Fluent Bit in Kubernetes, using the Helm chart allows for easier configuration, upgrades, and rollbacks. But knowing what each value does is crucial for success.

Key Helm Chart Values

  • backend.type: Choose opensearch, elasticsearch, or others
  • input: Configure tail, systemd, or http
  • filters: Include Kubernetes metadata or custom parsing
  • resources: Set memory and CPU limits to prevent pod eviction

You can customize these through a values.yaml file or directly in the Helm install command.

Helm Chart Best Practices

Always pin your Helm chart version. New releases may introduce breaking changes. Use helm template to preview the generated Kubernetes manifests before applying them.

Configuring the OpenSearch Output Plugin in Fluent Bit

Fluent Bit’s and OpenSearch output plugin allows you to efficiently send collected logs and data to your OpenSearch instance. Here’s a detailed guide on configuring it:

I. Configuration File

Fluent Bit uses a configuration file (fluent-bit.conf) to define its behavior. This file typically resides in /etc/fluent-bit/ or /fluent-bit/.

II. Adding the OpenSearch Output Section

Within the fluent-bit.conf file, define an OUTPUT section to specify the OpenSearch connection details and data format. Here’s an example configuration:

[OUTPUT]

    Name opensearch  # Name for this output plugin instance

    Match *          # Matches all incoming records (adjust as needed)

    Host "localhost"  # Replace with your OpenSearch host address

    Port 9200         # Replace with your OpenSearch port (default 9200)

    Index my_index   # Index name in OpenSearch to store data

    # Optional: Configure authentication if required

    # http_User "username"

    # http_Passwd "password"

    # Optional: Additional configuration options

    # Buffer_Limit 512  # Adjust buffer size for efficiency (default 8192)

    # Flush_Interval 5   # Flush data to OpenSearch every 5 seconds (default unset)

Explanation:

  • Name: A unique name for this output plugin instance (e.g., opensearch).
  • Match: This pattern determines which records are sent to this output plugin. * matches all records. You can use more specific patterns (e.g., kube.* for Kubernetes logs) for filtering.
  • Host: The hostname or IP address of your OpenSearch server.
  • Port: The port number where OpenSearch is listening (default 9200).
  • Index: The name of the index in OpenSearch where you want to store the data.

III. Optional Configuration

  • http_User and http_Passwd: If your OpenSearch instance requires authentication, provide your username and password here.
  • Buffer_Limit: You can adjust the buffer size used to hold data before sending it to OpenSearch. A larger buffer can improve efficiency but may introduce latency.
  • Flush_Interval: Specify the interval (in seconds) at which Fluent Bit should flush data to OpenSearch. Use this to control the trade-off between latency and batching.

IV. Restarting Fluent Bit

After making changes to the configuration file, restart Fluent Bit to apply the new settings:

sudo systemctl restart fluent-bit  # Systemd-based systems

V. Data Format

By default, Fluent Bit sends data to OpenSearch in JSON format. You can customize the format using Fluent Bit’s record processing filters. Refer to the Fluent Bit documentation for more details on data formatting.

VI. Testing

Once configured, send some test logs to Fluent Bit to verify data is being sent to OpenSearch. You can use a tool like echo to generate test data:

echo “This is a test log” | fluent-bit -i stdin

Then, check your OpenSearch instance under the specified index (my_index in this example) to confirm the test log is present.

Matching Log Streams with output match in Fluent Bit

Fluent Bit allows fine-grained control over where different types of logs go—thanks to its match directive in the output section.

How Match Works

Each output block in Fluent Bit can define a match parameter that specifies which tag(s) the output applies to. Tags are set in the input or filter sections and control the routing of logs.

For example:
[OUTPUT]
Name opensearch
Match kube.*

This ensures only logs with a tag beginning with kube. are forwarded to OpenSearch. You can use wildcards for flexible matching.

Use Cases for Multiple Outputs

You can send app logs to OpenSearch, system logs to Elasticsearch, and error logs to an S3 bucket—all from one Fluent Bit instance. Just define separate [OUTPUT] blocks with different Match rules.

Sending Logs from Fluent Bit to Elasticsearch as an Alternative Output

While OpenSearch is growing in popularity, many organizations still use Elasticsearch. Fluent Bit fully supports output to both systems using slightly different configuration blocks.

Basic Configuration for Elasticsearch Output

The syntax is very similar:
[OUTPUT]
Name es
Match *
Host your.elasticsearch.server
Port 9200
Index logs
Type _doc

This can coexist with an OpenSearch output block if your routing is properly set via Match.

Fluent Bit vs Fluent Bit Elasticsearch Output Syntax

Though similar, OpenSearch sometimes requires additional parameters like AWS credentials or signed headers when used with Amazon OpenSearch. Elasticsearch on-prem is usually simpler to configure

Common Issues When Using Fluent Bit with OpenSearch

Even with proper setup, it’s common to hit a few snags when connecting Fluent Bit to OpenSearch. Let’s look at some of the most frequently encountered issues—and how to fix them quickly.

Fluent Bit Is Not Sending Logs to OpenSearch

This is often due to a misconfigured output section or missing credentials. Ensure your configuration includes correct Host, Port, and Index values in the [OUTPUT] section. If you’re using TLS, verify that tls and tls.verify settings are correctly applied.

Failed to Connect to OpenSearch

Double-check the OpenSearch URL and port. By default, OpenSearch uses port 9200. If Fluent Bit is running in Kubernetes, ensure network policies or service mesh configurations aren’t blocking access.

Logs Not Appearing in OpenSearch Dashboard

This may relate to mismatched logstash_format, incorrect time-based indexing, or unsupported field mappings. Try enabling verbose logging in Fluent Bit to view the JSON payloads being sent.

Fluent Bit vs Fluentd: Which Is Better for OpenSearch?

Many teams compare Fluent Bit and Fluentd when deciding how to forward logs to OpenSearch. While they serve similar purposes, their performance and usage models differ significantly.

Lightweight vs Flexible

Fluent Bit is designed for lightweight edge log shipping—ideal for container environments or embedded systems. Fluentd, on the other hand, is a more full-featured tool written in Ruby with hundreds of plugins, including fluent-plugin-opensearch.

Performance and Resource Usage

Fluent Bit consumes less memory and CPU, making it better suited for high-volume, resource-constrained environments. Fluentd might be overkill for small or simple deployments, but it shines in complex, plugin-heavy setups.

Conclusion

In conclusion, you’ve successfully configured Fluent Bit to send data to your OpenSearch instance. This allows you to leverage OpenSearch’s powerful search and analysis capabilities for your logs and collected data. Remember to test your configuration and fine-tune settings like buffer size and flush interval to optimize performance based on your data volume and needs.

For further customization, explore Fluent Bit’s record processing filters to tailor the data format sent to OpenSearch. Refer to the official Fluent Bit documentation for in-depth information on advanced configuration options and troubleshooting.

FAQs

What is OpenSearch?

OpenSearch is a distributed search and analytics engine based on Apache Lucene. It is a fork of Elasticsearch and provides capabilities for indexing, searching, and analyzing large volumes of data in real-time.

Why integrate Fluent Bit and OpenSearch?

Integrating Fluent Bit and OpenSearch allows for seamless collection and analysis of logs and metrics generated by applications, containers, and infrastructure components. This integration enables centralized logging and monitoring solutions in complex environments.

How do I install Fluent Bit?

Fluent Bit can be installed on various platforms including Linux, Windows, and macOS. Installation methods may vary depending on the platform. Typically, you can install it using package managers like apt, yum, or brew, or manually by downloading and extracting the binaries.

How do I install OpenSearch?

OpenSearch can be deployed either as a standalone instance or as a part of a cluster. It’s typically deployed on Linux-based systems. You can download the OpenSearch distribution from the official website and follow the installation instructions provided in the documentation.

How do I configure Fluent Bit to send data to OpenSearch?

Configuration of Fluent Bit and OpenSearch involves specifying input sources, such as log files or Docker container logs, defining filters for data processing, and configuring output plugins to send the processed data to OpenSearch. Specific configuration details may vary based on your use case and environment.

Can Fluent Bit handle high volumes of data?

Yes, Fluent Bit is designed to efficiently handle high volumes of data, making it suitable for environments with large-scale log and metric collection requirements. It achieves high performance by utilizing a lightweight architecture and optimized data processing pipelines.

Latest Post:

Alex

Alex

Alex Davies is a data enthusiast who loves simplifying complex topics. He believes clear logs are essential for smooth operations. Follow his blog for approachable guides and tips on using Fluent Bit to gain valuable insights from your data.

Share:

More Posts