Tailpipe: An Open Source SIEM for Instant Log Insights with DuckDB

Introduction

Tailpipe is an open source Security Information and Event Management (SIEM) tool that provides instant log insights. Powered by DuckDB, it enables users to query and analyze millions of events in seconds, directly from their terminal. Tailpipe is designed to be lightweight and developer-friendly, offering SQL-based analysis for logs collected from cloud, container, and application sources. It leverages DuckDB's in-memory analytics and Parquet's optimized storage for fast, local, and efficient operations.

Installation

Getting started with Tailpipe is straightforward. You can install it using Homebrew on macOS or a shell script for Linux and Windows (WSL).

# MacOS
brew install turbot/tap/tailpipe

# Linux or Windows (WSL)
sudo /bin/sh -c "$(curl -fsSL https://tailpipe.io/install/tailpipe.sh)"

After installation, you can install plugins for your favorite services, such as AWS, Azure, or GCP, from the Tailpipe Hub. For example, to install the AWS plugin:

tailpipe plugin install aws

Examples

Tailpipe allows you to define connections and partitions to collect logs from various sources. Here is an example configuration for AWS CloudTrail logs:

connection "aws" "logging_account" {
  profile = "my-logging-account"
}

partition "aws_cloudtrail_log" "my_logs" {
  source "aws_s3_bucket" {
    connection = connection.aws.logging_account
    bucket     = "aws-cloudtrail-logs-bucket"
  }
}

After configuring, you can collect logs and then enter interactive query mode to run SQL queries:

tailpipe collect aws_cloudtrail_log
tailpipe query

A sample SQL query to analyze CloudTrail logs:

select
  event_source,
  event_name,
  count(*) as event_count
from
  aws_cloudtrail_log
where
  not read_only
group by
  event_source,
  event_name
order by
  event_count desc;

Why Use Tailpipe?

Tailpipe offers several compelling advantages for log analysis and security:

• SQL-Powered Insights: Query and analyze logs instantly using familiar SQL syntax.
• Fast and Local: Leverages DuckDB and Parquet for high-performance, in-memory analytics directly on your machine.
• Extensible Ecosystem: Benefit from an ecosystem of prebuilt intelligence, including MITRE ATT&CK-aligned queries, detections, and dashboards.
• Detections as Code: Define security detections as code, integrate with Powerpipe for advanced monitoring, and extend functionality with plugins.
• Multi-Cloud Support: Collects logs from AWS, Azure, GCP, and other sources.

Links

• Official Website
• GitHub Repository
• Documentation
• Demo Video
• Tailpipe Hub (Plugins & Queries)
• Join the Community on Slack