Log Collector

🛠️ Python tool for Junos log analysis: extracts .tgz bundles and filters by timestamp/keyword, generating a single .txt file as output. It speeds up log analysis and groups logs accordingly.

💡 Who needs a log collector anyway?

Network troubleshooting often requires analyzing Junos .tgz bundles that contain dozens of compressed log files. Manually extracting these and running grep or searching through text editors is time-consuming and prone to human error.

I built logc to solve three specific problems:

• Efficiency: Automates the extraction and recursive searching of multiple logs in seconds.
• Precision: Uses specific timestamp and/or keyword logic to narrow down logs to the exact window of a network event, reducing "noise".
• Portability: Designed with zero external dependencies so it can be used immediately on any device with Python installed.

It works as follows: it checks for a .tgz file, inspects all log files therein, and extracts all logs into a single file based on timestamp and or keyword. The output is a structured file with the name of each log file at the beginning of each section and the relevant logs underneath the headers. As an aside note, this is my first project in Python. Building it was fun, and I'm here for any fixes that may be necessary.

📦 Installing, Updating, and/or Uninstalling logc

The recommended way to install logc is using pipx. This ensures the tool works on Windows, Mac, and Linux by automatically managing your environment and system PATH.

1. Set up pipx (first time only - just for Windows)

If you don't have pipx installed, run:

python -m pip install --user pipx
python -m pipx ensurepath

Note: Restart your terminal after running ensurepath.

2. Install the tool:

To install logc, run the following command:

pipx install git+https://github.com/rust-it-cr/log-collector.git

Note: This installation method requires Git to be installed on your local machine. However, you do not need a GitHub account to download the tool.

3. Update the tool:

To update the tool, simply run the following command:

pipx upgrade logc

If already up to date, your terminal will display the following message:

"logc is already at latest version x.y.z (location: <location on your PC>)"

4. Uninstall the tool:

If for some reason you don't want to use this tool any longer, uninstalling it is as simple as running this command:

pipx uninstall logc

🧩 Dependencies

This tool is built entirely using the Python Standard Library.

• No third-party packages are required.
• Easy to deploy in environments where Junos-generated .tgz log files must be analyzed.

🛠 Usage

Once installed, use the logc command in your terminal.

Help options:

View all available filters and options:

logc -h

🛠 Examples

1. Searching for specific errors

If you need to find every instance of a BGP flap across on a file (or several thereof) in the bundle:

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -f "messages" "bgp_logs" -k "BGP_IO_ERROR"

2. Searching for logs within a specific time

If you need to find all the logs from a specifit timestamp or time range across different files (or just one):

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -f "chassisd" -t "Oct  6 to Oct  8"

3. Combining filters:

You can also filter by both keywords and timestamps if that's what you need:

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -f "default-log-messages" -t "2025-01-01T00" -k "crash" and "version" and "upgrade" 

4. Case-insensitive searching:

If needed, you can perform a case-insensitive search if you don't remember if the keyword is lower- or upper-case, of a combination thereof:

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -f "kmd-logs" -t "Jan 1  12" -k "vpn" or "ipsec" or "ike" -i

5. Wildcard matching:

If needed, you can use a wildcard to gather the files that match your expression:

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -w "chassisd" -k "fpc" and "pic"

In this case, -w "chassisd" matches all files starting with that pattern, e.g., chassisd, chassisd.0.gz, chassisd.1.gz, etc.

You can also do a match on several expressions to parse many different files:

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -w "chassisd" "messages" "jsrpd" -t "Jan 16 06"

This will match on all variety of files starting with "chassisd", "messages", or "jsrpd".

There's an special use for the -w parameter:

logc -s "/home/user_name/Downloads/logs.tgz" -d "/home/user_name/Downloads/output.txt" -w all -k "vpn" or "bgp" or "l2vpn" or "chassisd" -i

The all parameter matches on all parsable files within the greater .tgz file for a really general serach.

🧪 Testing & Error Handling

This project uses pytest and the standard src layout. To run tests, you must install the project in editable mode so the test suite can locate the package logic.

1. Install the package and dependencies:

pip install pytest
pip install -e .

2. Run the tests:

Always run the tests from the project root directory (where the pyproject.toml file is located). This ensures the logc_tool package is correctly discovered.

For Windows users:

python -m pytest

For MacOS/Linux users:

pytest

[IMPORTANT] Do not run the tests from inside the tests/ folder. Running from the root directory allows pytest to properly map the src/ layout and find all test files automatically.

3. Unknown error handling:

Also, this tool has a way of handling unknown errors gracefully. If that happens, you will see the following output and a file in your "Desktop" folder (which then you can send me for debugging purposes):

logc -s "C:\Users\user_name\Downloads\corrupted-logs.tgz" -d "C:\Users\user_name\Downloads\no-file.txt" -f "messages" -k "ge-0/0/0"

'An error has occurred!'
'Error: <a technical description of the error>'
'Check the logc_error.log file for technical details and check the official guide (logc -h) for a guide on how to use this program.'

📜 License

This project is licensed under the GNU Lesser General Public License v3.0 or later.

• See the COPYING file for the full GPLv3 text.

• See the COPYING.LESSER file for the LGPLv3 additional permissions.