Welcome gitxray 1.0.16 - Exciting new HTML report and features.

Author: llavarello

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## Release v1.0.16 (October 18th, 2024)
+* Added a brand new HTML output format/report by default, making results a lot easier to navigate! Custom search bar instead of relying on DataTables which can be super slow for large HTML files. We're now also groupping results by Category across all contributors and highlighting results which contain a WARNING keyword.
+* Added certain association results to Contributor results, not all to prevent extra noise.
+* Added the ability to specify a directory for output instead of a file, gitxray creating the filename for you.
+* Removed the concept of 'Verbose' results, merging them with the non-verbose categories.
+* Removed the need for repositories and organizations to start with https://github.com (Thanks to @mattaereal for pointing that out!)
+
 ## Release v1.0.15 (September 20th, 2024)
 
 * Added searching for similar repository names in GitHub, Warning if another repository with the same name and better reputation is found.

README.md

@@ -1,45 +1,52 @@
 # Welcome to Gitxray 
 Gitxray (short for Git X-Ray) is a multifaceted security tool designed for use on GitHub repositories. It can serve many purposes, including OSINT and Forensics. `gitxray` leverages public GitHub REST APIs to gather information that would otherwise be very time-consuming to obtain manually. Additionally, it seeks out information in unconventional places.
 
-The Octocat getting X-Rayed  | [![Build Workflows](https://github.com/kulkansecurity/gitxray/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/kulkansecurity/gitxray) [![Latest Version in PIP](https://img.shields.io/pypi/v/gitxray.svg)](https://pypi.org/project/gitxray) [![Python Versions](https://img.shields.io/pypi/pyversions/gitxray.svg)](https://pypi.org/project/gitxray) [![License](https://img.shields.io/pypi/l/gitxray.svg)](https://github.com/kulkansecurity/gitxray/blob/main/LICENSE)
---- | ---
-![Gitxray Logo](https://kulkansecurity.github.io/gitxray/images/logo_gitxray.png "Gitxray Logo") | ![Gitxray Console](https://kulkansecurity.github.io/gitxray/images/console_gitxray.png "Gitxray Console")
+[![Build Workflows](https://github.com/kulkansecurity/gitxray/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/kulkansecurity/gitxray) [![Latest Version in PIP](https://img.shields.io/pypi/v/gitxray.svg)](https://pypi.org/project/gitxray) [![Python Versions](https://img.shields.io/pypi/pyversions/gitxray.svg)](https://pypi.org/project/gitxray) [![License](https://img.shields.io/pypi/l/gitxray.svg)](https://github.com/kulkansecurity/gitxray/blob/main/LICENSE) 
+--- 
+![Gitxray HTML Report](https://kulkansecurity.github.io/gitxray/images/html_report_gitxray.png "Gitxray HTML Report")
 <div style="clear: both;"></div>
 
 # Use cases
 Gitxray can be used to, for example:
+
 - Find sensitive information in contributor profiles disclosed by accident within, for example, Armored PGP Keys, or Key Names.
 
-` gitxray -r https://github.com/some-org/some-repository -v -f user_input`
 - Identify threat actors in a Repository. You may spot co-owned or shared accounts, as well as inspect public events to spot fake Stargazers.
 
-` gitxray -r https://github.com/some-org/some-repository -v -f keys,association,starred`
 - Identify fake or infected Repositories. It can detect tampered commit dates as well as, for example, Release assets updated post-release.
 
-` gitxray -r https://github.com/some-org/some-repository -v -f warning`
 - Forensics use-cases, such as filtering results by date in order to check what else happened on the day of an incident.
 
-` gitxray -r https://github.com/some-org/some-repository -v -f 2024-09-01`
-- And a lot more! Run a full X-Ray in Verbose mode to collect a ton of data.
+- And a lot more! Run a full X-Ray in to collect a ton of data.
+
+` gitxray -r https://github.com/some-org/some-repository`
+
+- If you rather use text output, you may want to filter output with filters:
+
+` gitxray -r https://github.com/some-org/some-repository -f user_input -outformat text`
 
-` gitxray -r https://github.com/some-org/some-repository -v`
+` gitxray -r https://github.com/some-org/some-repository -f keys,association,starred -outformat text`
+
+` gitxray -r https://github.com/some-org/some-repository -f warning -outformat text`
+
+` gitxray -r https://github.com/some-org/some-repository -f 2024-09-01 -outformat text`
 
 Please refer to the Documentation for additional use-cases and introductory information.
 
 # Documentation
 - [https://kulkansecurity.github.io/gitxray/](https://kulkansecurity.github.io/gitxray/)
 - [https://www.gitxray.com/](https://www.gitxray.com/)
 
-# Installing and running Gitxray
+# Installing, Updating, and running Gitxray
 
 gitxray was written with no use of external package dependencies other than the `requests` library.
 
 ## PyPI (PIP) Way
 
-`gitxray` is on PyPI and can be installed with:
+`gitxray` is on PyPI and can be installed and updated with:
 
 ```bash
-pip install gitxray
+pip install gitxray --upgrade
 ```
 
 Once installed, simply run gitxray from your command line by typing:
@@ -49,9 +56,12 @@ gitxray -h
 
 ## Run your first full X-Ray
 ```bash
-gitxray -o https://github.com/kulkansecurity -v
+gitxray -o https://github.com/kulkansecurity
 ```
 
+![Gitxray Console](https://kulkansecurity.github.io/gitxray/images/console_gitxray.png "Gitxray Console") 
+<div style="clear: both;"></div>
+
 ## Installing from source
 
 You may also run `gitxray` directly by cloning or downloading its GitHub repository and running.
@@ -68,11 +78,11 @@ python3 -m gitxray.gitxray
 
 One of the following must be specified:
 
-* `-r, --repository [URL]` - Specify a single repository URL to check. The URL must begin with `https://`. **Example**: `--repository https://github.com/example/repo`
+* `-r, --repository [URL]` - Specify a single repository to check. The URL may optionally begin with `https://github.com/`. **Example**: `--repository https://github.com/example/repo`
 
 * `-rf, --repositories-file [FILEPATH]` - Provide a file path containing a list of repositories, each on a new line. The file must exist. **Example**: `--repositories-file ./list_of_repos.txt`
 
-* `-o, --organization [URL]` - Specify an organization URL to check all repositories under that organization. The URL must begin with `https://`. **Example**: `--organization https://github.com/exampleOrg`
+* `-o, --organization [URL]` - Specify an organization to check all repositories under that organization. The URL may optionally begin with `https://github.com/`. **Example**: `--organization https://github.com/exampleOrg`
 
 ### Optional Arguments
 
@@ -84,13 +94,17 @@ You'll find these optional but very handy in common gitxray usage.
 
 - `-f, --filters [KEYWORDS]` - Comma-separated keywords to filter the results by, such as 'user_input', 'association', or 'mac'. **Example**: `--filters user_input,association,mac`
 
-#### Verbose and Debug
-- `-v, --verbose` - Enable verbose output which, for example, provides a detailed list of public events instead of a summary. **Example**: `--verbose`
+#### Output and Formats
+
+- `-out, --outfile [FILEPATH]` - Specify the file path for the output log. Cannot be a directory. **Example**: `--outfile ./output.log`
+
+- `-outformat, --output-format [FORMAT]` - Set the format for the log file. Supported formats are `html`, `text` and `json`. Default is `html`. **Example**: `--output-format json`
+
+#### Debug
 
 - `--debug` - Enable Debug mode for a detailed and extensive output. **Example**: `--debug`
 
-#### Output and Formats
+# Terms of Use
 
-- `-out, --outfile [FILEPATH]` - Specify the file path for the output log. Cannot be a directory. **Example**: `--outfile ./output.log`
+The user is solely responsible for ensuring that this tool is used in compliance with applicable laws and regulations, including obtaining proper authorization for repository scanning and the distribution of any results generated. Unauthorized use or sharing of results may violate local, national, or international laws.
 
-- `-outformat, --output-format [FORMAT]` - Set the format for the log file. Supported formats are `text` and `json`. Default is `text`. **Example**: `--output-format json`

docs/features.md

@@ -2,17 +2,21 @@
 
 Because of the amount of data it analyzes, `gitxray` can be a bit overwhelming at first. Let's look at a few examples of potential awesome findings which can better explain why you're here and why `gitxray` is awesome ♥. 
 
+## A user-friendly HTML report 📊
+
+`gitxray` now offers a default output format of `html`, creating a [Bootstrap](https://www.getbootstrap.com)-backed HTML report which offers easy navigation through Repository, Contributor and non-Contributor results.<div style="clear: both;"></div> ![Gitxray HTML Report](images/html_report_gitxray.png "HTML Report Gitxray")<div style="clear: both;"></div>
+
 ## Unintended disclosures in Contributor profiles &#128064;
 
 `gitxray` reports under a `user_input` category any user-supplied data that repository Contributors may have exposed via their GitHub accounts inadevertently. This is normally the case of PGP and SSH key name fields, which unfortunately are used by Users to record hostnames, computer models, password locations (e.g. in 1Password), or even the _password itself_ to a given key (which we all know might be the same password used elsewhere). To make things more interesting, `gitxray` also identifies any "excess" data found before, or after, PGP Armored keys published in a User's GitHub account. Wondering what that data normally is? Erroneous copy/pastes from the command line while exporting in ASCII/Armored format their keys. And what might that contain? Most of the times, a shell prompt revealing a local username, a hostname and a directory path. May I remind you all of this data is Public-facing.
 
 You may focus specifically on these types of findings by filtering results with:
 ```py
-gitxray -o https://github.com/SampleOrg -v -f user_input
+gitxray -o https://github.com/SampleOrg -f user_input
 ```
-or, for a specific repository (remember, _Verbose is always optional_): 
+or, for a specific repository:
 ``` py
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -f user_input
+gitxray -r https://github.com/SampleOrg/SampleRepo -f user_input
 ```
 
 ## Spotting shared, co-owned or fake Contributors &#128123;
@@ -24,11 +28,11 @@ Open source projects are under attack, with malicious actors hiding in plain sig
 You can focus specifically on association findings by filtering for `association` with:
 
 ``` 
-gitxray -o https://github.com/SampleOrg -v -f user_input
+gitxray -o https://github.com/SampleOrg -f association 
 ```
-or targetting a specific Repository with (_Verbose is always optional_):
+or targetting a specific Repository with:
 ```
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -f user_input
+gitxray -r https://github.com/SampleOrg/SampleRepo -f association
 ```
 
 ### Important 
@@ -41,15 +45,17 @@ Associations MUST NOT be directly and blindly used to report fake or shadow acco
 
 ## Forensics: What happened on the day of an incident? &#128269;
 
-Because `gitxray` collects data from multiple sources of activity including Commits, Comments, Workflow Runs, Issues, Deployments and more; and because Verbose mode in `gitxray` shows activity in a standarized YYYY-MM-DD format, it becomes possible to use Filtering in order to place focus on specific activity happening at a specific point in time.
+Because `gitxray` collects data from multiple sources of activity including Commits, Comments, Workflow Runs, Issues, Deployments and more; and because `gitxray` shows activity in a standarized YYYY-MM-DD format, it becomes possible to use Filtering in order to place focus on specific activity happening at a specific point in time.
 
 For example, by running `gitxray` with the following arguments, only results from that specific date are returned. You may place focus on a day, or even a month:
 
 ``` 
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -f 2024-08
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -f 2024-09-01
+gitxray -r https://github.com/SampleOrg/SampleRepo -f 2024-08 -outformat text
+gitxray -r https://github.com/SampleOrg/SampleRepo -f 2024-09-01 -outformat text
 ```
 
+An outformat of type `text` can help in this specific use-case more than the defaul `html` report.
+
 ## Analyzing Commit Hours to Identify Anomalies &#128347;
 
 `gitxray` provides a summary of contributor commit hours, allowing you to profile contributor activity and detect potential anomalies. This feature helps you understand typical patterns and flag unusual behavior for further investigation.
@@ -62,10 +68,10 @@ gitxray -r https://github.com/SampleOrg/SampleRepo -v -f 2024-09-01
 
 * Reclaimed Usernames: Trusted contributors might have had their accounts deleted and then re-registered by malicious actors. GitHub allows a username to be re-released after 90 days, making this a possible attack vector. Learn more about GitHub’s account deletion policy [here](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-personal-account/deleting-your-personal-account#deleting-your-personal-account).
 
-Although we always recommend running a full unfiltered verbose X-Ray, it is possible to focus on unreliable historic activity by filtering for Warning keywords:
+It is possible to focus on unreliable historic activity by filtering for Warning keywords:
 
 ``` 
-gitxray -o https://github.com/SampleOrg -v -f warning
+gitxray -o https://github.com/SampleOrg -f warning
 ```
 
 ## X-Raying GitHub Workflows &#9881; 
@@ -93,17 +99,15 @@ Another `gitxray` feature is the ability to list a TOP 3 of GitHub accounts that
 These findings, if any exist, are reported under a `contributors` category along with additional information related to other Repository Contributors. You can focus specifically on findings from the contributors category by filtering for `contributors` with:
 
 ``` 
-gitxray -o https://github.com/SampleOrg -v -f contributors 
+gitxray -o https://github.com/SampleOrg -f contributors 
 ```
-or targetting a specific Repository with (_Verbose is always optional_):
+or targetting a specific Repository with:
 ``` bash
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -f contributors
+gitxray -r https://github.com/SampleOrg/SampleRepo -f contributors
 ```
 ## Fake Starring, Private repos gone Public and more &#128584; 
 
-GitHub shares publicly [up to 90 days of past Events](https://docs.github.com/en/rest/activity/events?apiVersion=2022-11-28) for any User account, which include actions such as Repository creation, Watching, Committing, Pull Requesting, and more. `gitxray` summarizes these events for you and prints them out under a `90d_events` category in the results included for each Contributor, summarized in order to reduce the amount of data listed by default. 
-
-The summary however can be expanded into a full list of Events by merely turning on _Verbose mode_ (the -v flag). Using _Verbose mode_ is convenient in order to get details on **WHAT** was actioned upon.
+GitHub shares publicly [up to 90 days of past Events](https://docs.github.com/en/rest/activity/events?apiVersion=2022-11-28) for any User account, which include actions such as Repository creation, Watching, Committing, Pull Requesting, and more. `gitxray` includes these events under a `90d_events` category in the results included for each Contributor.
 
 For example, Events you may come across that would be interesting include:
 
@@ -118,9 +122,9 @@ To find Contributors who recently switched from Private to Public a repository o
 gitxray -o https://github.com/SampleOrg -f starred,private
 ```
 
-And you could then enable _Verbose_ (or before, you decide) and target a specific Repository Contributor to get more information:
+And you could then target a specific Repository Contributor to get more information:
 ``` 
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -c some_user
+gitxray -r https://github.com/SampleOrg/SampleRepo -c some_user
 ```
 ## Lots of e-mail addresses &#128231; and Profiling data &#128100;
 
@@ -134,13 +138,13 @@ Additionally, Personal Information (e.g. social networks) voluntarily made Publi
 
 Finally, the `profiling` category tends to display information related to the account itself (e.g. creation date, last updated, and more.)
 
-You may focus specifically on `emails`, `personal`, and `profiling` fields with (Verbose is optional):
+You may focus specifically on `emails`, `personal`, and `profiling` fields with:
 ```py
-gitxray -o https://github.com/SampleOrg -v -f emails,personal,profiling
+gitxray -o https://github.com/SampleOrg -f emails,personal,profiling
 ```
 or, for a specific repository, with: 
 ``` py
-gitxray -r https://github.com/SampleOrg/SampleRepo -v -f emails,personal,profiling
+gitxray -r https://github.com/SampleOrg/SampleRepo -f emails,personal,profiling
 ```
 
 ## Looking out for malicious Releases and Assets &#128065; 

docs/index.md

@@ -3,8 +3,7 @@ Gitxray (short for Git X-Ray) is a multifaceted security tool designed for use o
 
 The Octocat getting X-Rayed  | [![Build Workflows](https://github.com/kulkansecurity/gitxray/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/kulkansecurity/gitxray) [![Latest Version in PIP](https://img.shields.io/pypi/v/gitxray.svg)](https://pypi.org/project/gitxray) [![Python Versions](https://img.shields.io/pypi/pyversions/gitxray.svg)](https://pypi.org/project/gitxray) [![License](https://img.shields.io/pypi/l/gitxray.svg)](https://github.com/kulkansecurity/gitxray/blob/main/LICENSE)
 --- | ---
-![Gitxray Logo](https://kulkansecurity.github.io/gitxray/images/logo_gitxray.png "Gitxray Logo") | ![Gitxray Console](https://kulkansecurity.github.io/gitxray/images/console_gitxray.png "Gitxray Console")
-<div style="clear: both;"></div>
+![Gitxray Logo](https://kulkansecurity.github.io/gitxray/images/logo_gitxray.png "Gitxray Logo") | ![Gitxray Console](https://kulkansecurity.github.io/gitxray/images/console_gitxray.png "Gitxray Console")<br/> ![Gitxray HTML Report](images/html_report_gitxray.png "HTML Report Gitxray")<div style="clear: both;"></div>
 
 # What is it for?
 * Identifying threat actors in a Repository. [You may spot co-owned or shared accounts](/features/#spotting-shared-co-owned-or-fake-contributors), as well as inspect public events to [spot fake Stargazers](/features/#fake-stars-private-repos-gone-public-and-more).