logmerger: A TUI Utility for Merging and Viewing Multiple Log Files

Introduction

logmerger is a powerful TUI (Text User Interface) utility written in Python, designed to simplify the complex task of analyzing multiple log files simultaneously. It intelligently merges log entries from various sources into a single, chronologically ordered timeline, providing a unified view for easier debugging and system monitoring. This tool is particularly useful for developers and system administrators who need to correlate events across different log streams.

Installation

To get started with logmerger, you can easily install it using pip:

pip install logmerger

This command installs logmerger as a shell/console command, allowing you to run it directly. For additional support, such as merging pcap files, install with the [pcap] extra:

pip install logmerger[pcap]

Examples

logmerger offers both an interactive TUI and a command-line output option, catering to different analysis needs.

Interactive TUI

To view logs interactively, simply run logmerger with your log files:

logmerger log1.txt log2.txt

This command opens a browsable merged display, powered by the textual Python library, allowing you to navigate through your logs with ease.

(Note: Press 'h' within the TUI for help on all key commands.)

Output to Stdout

You can also direct the merged logs to standard output or a file using the --output option. For example, to send to stdout:

logmerger --output - log1.txt log2.txt

This will produce a clean, merged text output like this:

  Timestamp                 Files/Log1.Txt                        Files/Log2.Txt
 ????????????????????????????????????????????????????????????????????????????????????????????????????
  2023-07-14 08:00:01.000   WARN   Connection lost due to         INFO   Request processed
                            timeout                               successfully
  2023-07-14 08:00:03.000                                         INFO   User authentication
                                                                  succeeded
  2023-07-14 08:00:04.000   ERROR  Request processed
                            unsuccessfully
                             Something went wrong
                             Traceback (last line is latest):
                                 sample.py: line 32
                                     divide(100, 0)
                                 sample.py: line 8
                                     return a / b
                             ZeroDivisionError: division by zero                           
  2023-07-14 08:00:06.000   INFO   User authentication            DEBUG  Starting data
                            failed                                synchronization
  2023-07-14 08:00:08.000   DEBUG  Starting data                  INFO   Processing incoming request
                            synchronization
  2023-07-14 08:00:11.000   INFO   Processing incoming request    DEBUG  Performing database backup
                            INFO   Processing incoming request
                            (a little more...)
  2023-07-14 08:00:14.000   DEBUG  Performing database backup     WARN   Invalid input received:
                                                                  missing required field

Why Use logmerger?

logmerger stands out as an essential tool for anyone dealing with distributed systems or complex applications. Its ability to merge logs chronologically across different files eliminates the manual effort of cross-referencing timestamps, significantly speeding up debugging and incident response. The interactive TUI provides a user-friendly interface for exploration, while its support for various log formats, including compressed files, CSV, JSONL, and specialized types like pcap, makes it highly versatile. Whether you need a quick overview or detailed forensic analysis, logmerger offers a robust solution for efficient log management.

Links

• GitHub Repository: ptmcg/logmerger
• PyPI Project Page: logmerger