Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/gitlab #13

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Feat/gitlab #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4cebac9
Use circleci conditiona logic
mrzarquon Sep 22, 2021
72b5a46
rename project and run poetry update
mrzarquon Sep 24, 2021
beb82d7
initial commit
mrzarquon Feb 17, 2022
154c1f0
initial commit
mrzarquon Feb 23, 2022
40f8788
Updated repo_diff to create csv with repository difference.
Robthreefold Mar 11, 2022
7c996f9
repair broken github integration
mrzarquon Mar 11, 2022
ee3b938
fixup snyk test
mrzarquon Mar 11, 2022
e5efa28
Merge branch 'main' into feat/gitlab
mrzarquon Mar 11, 2022
fba41c5
cleaned up docs
mrzarquon Mar 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .circleci/continue_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,11 @@ workflows:
jobs:
- snyk-poetry-test
dockerfile-tests:
when: << pipeline.parameters.run-docker-test >>
when:
condition:
or:
- << pipeline.parameters.run-docker-test >>
- equal: [main, << pipeline.git.branch >>]
jobs:
- snyk-docker-test

Expand All @@ -35,20 +39,21 @@ jobs:
- checkout
- python/install-packages:
pkg-manager: poetry
args: --no-dev
- snyk/scan:
fail-on-issues: true
monitor-on-build: false
severity-threshold: critical
token-variable: SNYK_TOKEN
additional-arguments: --policy-path=.snyk
additional-arguments: --policy-path=.snyk --file=poetry.lock

- when:
condition:
equal: [main, << pipeline.git.branch >>]
steps:
- run:
name: Snyk Monitor
command: snyk monitor --policy-path=.snyk

command: snyk monitor --policy-path=.snyk --file=poetry.lock


snyk-docker-test:
Expand Down
7 changes: 6 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,9 @@
output/*
*.DS_Store*
example_secrets.sh
.vscode
.vscode
.dccache
bin
lib
__pycache__
__*
188 changes: 105 additions & 83 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,127 +4,148 @@

Helping answer which repositories aren't monitored by Snyk?

This works by retrieving a list of all projects in a given Snyk Group (so all projects in all orgs belonging to the same Snyk Group) and associating them with a list of repositories found in a given GitHub Organization. If one wants to check against multiple GitHub Organizations, currently run this script multiple times, providing a different GitHub Org each time.
This works by retrieving a list of all projects in a given Snyk Group (so all projects in all orgs belonging to the same Snyk Group) and associating them with a list of repositories found in a given GitHub Organization (see below section on GitLab support). If one wants to check against multiple GitHub Organizations, currently run this script multiple times, providing a different GitHub Org each time.

Once the list is generating, the script can output a CSV or JSON file of the repositories that do not appear to have any projects in Snyk. There is an optional --with-projects flag that output all repositories and the count (in CSV) of projects or a link to every project (in JSON).

The data also includes the last update of the repository itself. Many organizations have old or stale repositories, so Snyk not having any projects for a repository last updated in 2016 might not be as important as one updated yesterday.

## Requirements
In order to run this script one needs a python environment with the Snyk, Github, and Typer libraries. Use the provided Dockerfile to build and run a container with this script setup in it. Refer to the [Docker](#user-content-running-with-docker) section for more information.

To run this script one needs a python environment with the Snyk, Github, and Typer libraries. Use the provided Dockerfile to build and run a container with this script setup in it. Refer to the [Docker](#user-content-running-with-docker) section for more information.

### Snyk Requirements

- Snyk Access Token: Either generate an service account token or [retrieve your own](https://docs.snyk.io/snyk-api-info/authentication-for-api)
- Snyk Group ID: Available under the Settings view of your Group page at [app.snyk.io](https://app.snyk.io/)

### GitHub Requirements

- GitHub Token: A token with access to list / view all repositories inside the organization you are checking. See GitHub's documentation on generating a [personal access token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token).
- Name of the GitHub organization you are checking for repositories

### GitLab Support

Support for GitLab is identical to GitHub, except use the GitLab personal access token instead of a GitHub one. The output omits the fork status of a repo because that isn't as determine from GitLab.

### Environment Variables

While you can pass the GitHub and Snyk tokens as commandline arguments to the script, it is best to use them as environment variables so they aren't stored in your workstation's command history.
While you can pass the GitHub and Snyk tokens as commandline arguments to the script, it' i's best to use them as environment variables so they aren't stored in your workstation's command history.

Use [example_secrets.sh](example_secrets.sh) if you need a simple way to copy/paste your tokens into a file and then load them into your environment for use.
Use [example_secrets.sh](example_secrets.sh) for quick way to copy/paste your tokens into a file and then load them into your environment for use.

Refer to --help for more environment variables:

```shell
> repo_diff.py --help
Usage: repo_diff.py [OPTIONS] COMMAND [ARGS]...

Options:
--snyk-token TEXT Snyk Token with access to the Group whose projects you
to want to audit against a Github Organization [env
var: SNYK_TOKEN; required]
--snyk-group TEXT Only projects associated with the Snyk Group are
checked for [env var: SNYK_GROUP; required]
--github-token TEXT GitHub Token with read access to the GitHub Org you
wish to audit [env var: GITHUB_TOKEN; required]
--github-org TEXT Name of the GitHub Org whose repositories you want to
check [env var: GITHUB_ORG; required]
--with-projects Include repositories that have projects
--out-file FILENAME Full path to where the output should be saved [env
var: REPO_DIFF_OUTPUT; required]
--format [csv|json] [default: csv]
--origin TEXT Specify which Snyk integrations to check for projects
[default: github, github-enterprise]
--help Show this message and exit.
--snyk-token TEXT Snyk Token with access to the Group whose
projects you to want to audit against a
Github Organization [env var: SNYK_TOKEN;
required]
--snyk-group TEXT Only projects associated with the Snyk Group
are checked for [env var: SNYK_GROUP;
required]
--scm-token TEXT Access Token to the SCM platform you wish to
audit (GitHub and GitLab currently
supported) [env var: SCM_TOKEN; required]
--scm-org TEXT Name of the GitHub Org or GitLab Group whose
repos you want to check against [env var:
SCM_ORG; required]
--scm-url TEXT Provide url as https://gitlab.example.com
etc [env var: SCM_URL]
--with-projects Include repositories that have projects
--out-file FILENAME Full path to where the output should be
saved [env var: REPO_DIFF_OUTPUT; required]
--format [csv|json] [default: FileFormat.csv]
--origin [github|github-enterprise|gitlab]
[default: Origin.github]
--help Show this message and exit.
```

## Running with Docker

1) Build the container with docker, a command like this should suffice:<p>
`docker build --pull --no-cache --force-rm -f Dockerfile -t repo_diff .`
```shell
❯ docker build --pull --no-cache --force-rm -f Dockerfile -t repo_diff .
[+] Building 24.0s (17/17) FINISHED
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 37B
=> [internal] load .dockerignore
=> => transferring context: 2B
=> [internal] load metadata for docker.io/library/python:3.9-slim
=> CACHED [requirements 1/6] FROM docker.io/library/python:3.9-slim@sha256:cd1045dbabff11dab74379e25f7974aa76
=> [internal] load build context
=> => transferring context: 100B
=> CACHED [runtime 2/7] WORKDIR /app
=> [requirements 2/6] RUN python -m pip install -U pip poetry
=> [runtime 3/7] COPY *.py .
=> [requirements 3/6] WORKDIR /src
=> [requirements 4/6] COPY pyproject.toml pyproject.toml
=> [requirements 5/6] COPY poetry.lock poetry.lock
=> [requirements 6/6] RUN poetry export -f requirements.txt --without-hashes -o /src/requirements.txt
=> [runtime 4/7] COPY --from=requirements /src/requirements.txt .
=> [runtime 5/7] RUN python -m pip install -U pip
=> [runtime 6/7] RUN pip install -r requirements.txt
=> [runtime 7/7] RUN mkdir /output
=> exporting to image
=> => exporting layers
=> => writing image sha256:09687a554bb04397641490538185324be3eab462e31c2e80a59312a0b143a483
=> => naming to docker.io/library/repo_diff
```

2) Run the container with the `docker run` command, ensuring to:
- Pass a local volume (`-v "${PWD}/output":/app/output`) for the csv output to be saved to
- Pass the SNYK_TOKEN and GITHUB_TOKEN environment variables (`-e SNYK_TOKEN -e GITHUB_TOKEN`)
- Delete the container image after use (`--rm`)
- Specify the `--snyk-group`, `--github-org`, and `--out-file` options for the script
```shell
❯ docker run --rm -v "${PWD}/output":/app/output -e SNYK_TOKEN -e GITHUB_TOKEN -it repo_diff \
--out-file output/repos-with-no-projects.csv \
--snyk-group 36863d40-ba29-491f-af63-7a1a7d79e411 \
--github-org snyk-playground

Searching Snyk for Projects from snyk-playground repositories
Searching 7 Snyk Org(s) with ('github', 'github-enterprise') Projects [####################################] 100%
Checking for Projects from the 27 repos in snyk-playground [####################################] 100%
Formatting and writing results to /output/output.csv
```

3) An example output.csv should look something like this:
```
Repository Name,Last Updated,Is Fork,Snyk Project Count
snyk-playground/snyk-container-scan-docker,2018-12-06 22:59:40,False,0
snyk-playground/config-repo,2021-08-06 09:51:10,False,0
snyk-playground/webhook-test-repository,2021-06-30 13:12:17,False,0
snyk-playground/pygithub-import-parser,2021-08-25 12:27:04,False,0
snyk-playground/docs,2021-07-24 04:55:03,False,0
snyk-playground/demo_snyk-transitive-ignore,2021-08-08 04:29:13,False,0
```

4) After loading the file into excel or similar it renders like this:
![Numbers.app render of CSV](https://github.com/snyk-tech-services/snyk-repo-diff/blob/main/img/table.png?raw=true)
1. Build the container with docker, a command like this should suffice:<p>
`docker build --pull --no-cache --force-rm -f Dockerfile -t repo_diff .`

Example:

````shell
❯ docker build --pull --no-cache --force-rm -f Dockerfile -t repo_diff .
[+] Building 24.0s (17/17) FINISHED
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 37B
=> [internal] load .dockerignore
=> => transferring context: 2B
=> [internal] load metadata for docker.io/library/python:3.9-slim
=> CACHED [requirements 1/6] FROM docker.io/library/python:3.9-slim@sha256:cd1045dbabff11dab74379e25f7974aa76
=> [internal] load build context
=> => transferring context: 100B
=> CACHED [runtime 2/7] WORKDIR /app
=> [requirements 2/6] RUN python -m pip install -U pip poetry
=> [runtime 3/7] COPY *.py .
=> [requirements 3/6] WORKDIR /src
=> [requirements 4/6] COPY pyproject.toml pyproject.toml
=> [requirements 5/6] COPY poetry.lock poetry.lock
=> [requirements 6/6] RUN poetry export -f requirements.txt --without-hashes -o /src/requirements.txt
=> [runtime 4/7] COPY --from=requirements /src/requirements.txt .
=> [runtime 5/7] RUN python -m pip install -U pip
=> [runtime 6/7] RUN pip install -r requirements.txt
=> [runtime 7/7] RUN mkdir /output
=> exporting to image
=> => exporting layers
=> => writing image sha256:09687a554bb04397641490538185324be3eab462e31c2e80a59312a0b143a483
=> => naming to docker.io/library/repo_diff ```

````

2. Run the container with the `docker run` command, ensuring to:

- Pass a local volume (`-v "${PWD}/output":/app/output`) for the csv output to be saved to
- Pass the SNYK_TOKEN and SCM_TOKEN environment variables (`-e SNYK_TOKEN -e SCM_TOKEN`)
- Delete the container image after use (`--rm`)
- Specify the `--snyk-group`, `--scm-org`, and `--out-file` options for the script

```shell
❯ docker run --rm -v "${PWD}/output":/app/output -e SNYK_TOKEN -e SCM_TOKEN -it repo_diff \
--out-file output/repos-with-no-projects.csv \
--snyk-group 36863d40-ba29-491f-af63-7a1a7d79e411 \
--scm-org snyk-playground

Searching Snyk for Projects from snyk-playground repositories
Searching 7 Snyk Org(s) with ('github', 'github-enterprise') Projects [####################################] 100%
Checking for Projects from the 27 repos in snyk-playground [####################################] 100%
Formatting and writing results to /output/output.csv
```

3. An example output.csv should look something like this:

```
Repository Name,Last Updated,Is Fork,Snyk Project Count
snyk-playground/snyk-container-scan-docker,2018-12-06 22:59:40,False,0
snyk-playground/config-repo,2021-08-06 09:51:10,False,0
snyk-playground/webhook-test-repository,2021-06-30 13:12:17,False,0
snyk-playground/pygithub-import-parser,2021-08-25 12:27:04,False,0
snyk-playground/docs,2021-07-24 04:55:03,False,0
snyk-playground/demo_snyk-transitive-ignore,2021-08-08 04:29:13,False,0
```

4. After loading the file into excel or similar it renders like this:
![Numbers.app render of CSV](https://github.com/snyk-tech-services/snyk-repo-diff/blob/main/img/table.png?raw=true)

## Advanced usage

repo_diff.py can take multiple origins (`--origin github --origin github-enterprise` is default), since Snyk can also monitor projects in the the CLI (common for gradle and poetry projects), to check that a repo is being monitored *somewhere* in your Snyk group, you can add the cli origin. This is not guaranteed to find all possible projects (the attribute, remoteRepoUrl, can be set manually by the user), but will in the situations that a attribute hasn't been overriden.
repo*diff.py can take multiple origins (`--origin github --origin github-enterprise` is default), since Snyk can also monitor projects in the the CLI (common for gradle and poetry projects), to check that a repo is being monitored \_somewhere* in your Snyk group, you can add the cli origin. This is not guaranteed to find all possible projects (the attribute, remoteRepoUrl, can be set manually by the user), but will in the situations that a attribute hasn't been overriden.

```shell
❯ python repo_diff.py --out-file output/test.json --format json --with-projects \
--origin cli \
--origin github \
--origin github-enterprise
Searching Snyk for Projects from snyk-playground repositories
Searching 7 Snyk Org(s) with ('cli', 'github', 'github-enterprise') Projects [####################################] 100%
Searching 7 Snyk Org(s) with ('cli', 'github', 'github-enterprise') Projects [####################################] 100%
Checking for Projects from the 27 repos in snyk-playground [####################################] 100%
Formatting and writing results to output/test.json
```
Expand All @@ -134,4 +155,5 @@ Formatting and writing results to output/test.json
This script and its contents are provided as a best effort example of how to use Snyk and Github's python sdk's to generate data from both services APIs.

## License
[License: Apache License, Version 2.0](LICENSE)

[License: Apache License, Version 2.0](LICENSE)
6 changes: 3 additions & 3 deletions example_secrets.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
# add snyk token here
export SNYK_TOKEN="BD832F91-A742-49E9-BC1E-411E0C8743EA"

# add github token here
export GITHUB_TOKEN="4BB6849A-9D18-4F38-B769-0E2490FA89CA"
# add github or gitlab token here
export SCM_TOKEN="4BB6849A-9D18-4F38-B769-0E2490FA89CA"

# save this file
# and load it with the command: source example_secrets.sh
# now check that these variables are set running the commands:
# echo $SNYK_TOKEN
# echo $GITHUB_TOKEN
# echo $SCM_TOKEN
42 changes: 42 additions & 0 deletions foobar.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
Repository Name,Last Updated,Snyk Project Count
snyk-playground/codefresh-pipeline-snyk-app-docker-scan,2018-10-24 16:51:26,0
snyk-playground/config-repo,2022-01-25 23:17:15,0
snyk-playground/snyk-container-scan-docker,2018-12-06 22:59:40,0
snyk-playground/snyk-sync,2022-02-17 12:26:36,0
snyk-playground/snyk-audit-to-csv,2021-10-19 14:53:52,0
snyk-playground/snyk-in-dockerfile-examples,2021-12-22 22:47:54,0
snyk-playground/test_repo,2022-02-11 03:09:17,0
snyk-playground/sync-config,2022-02-07 16:24:29,0
snyk-playground/sync-aws-deploy,2022-02-07 16:23:39,0
snyk-playground/org-project-import,2022-01-17 05:44:15,0
snyk-playground/mqttclientdashboard-akamai-fastpurge-msod,2020-09-04 23:38:24,0
snyk-playground/snyk-python-base,2021-12-10 15:37:39,0
snyk-playground/bazel-examples,2021-11-28 07:46:23,0
snyk-playground/license-validator,2019-01-17 13:04:35,0
snyk-playground/custom-snyk-action,2021-12-02 15:36:09,0
snyk-playground/org-import-bad,2022-02-14 14:22:40,0
snyk-playground/sampleRequirements3InProject,2022-02-11 03:08:20,0
snyk-playground/nightmare,2021-07-06 11:35:50,0
snyk-playground/repo-with-jira-config,2022-01-17 05:44:22,0
snyk-playground/webhook-test-repository,2021-06-30 13:12:17,0
snyk-playground/canvas-lms,2021-05-30 22:18:05,0
snyk-playground/npm-check-updates,2018-02-14 04:24:11,0
snyk-playground/goof,2022-01-17 01:51:34,0
snyk-playground/example-bazel-monorepo,2021-11-05 01:51:04,0
snyk-playground/monorepo-simple,2022-01-07 20:23:18,0
snyk-playground/multi-project-code,2021-11-17 10:39:59,0
snyk-playground/medium-sev-project,2020-07-18 21:16:27,0
snyk-playground/org-import-instance,2021-11-24 12:45:23,0
snyk-playground/org-import-branch-override,2022-01-17 05:33:11,0
snyk-playground/Dapper,2018-03-10 12:52:24,0
snyk-playground/docs,2021-07-24 04:55:03,0
snyk-playground/pygithub-import-parser,2021-08-25 12:27:04,0
snyk-playground/monorepo-kotlin,2022-01-26 05:48:29,0
snyk-playground/java-goof,2021-10-26 00:15:43,0
snyk-playground/training-app,2018-08-03 17:37:31,0
snyk-playground/testrepo,2018-02-16 20:06:17,0
snyk-playground/typescript,2022-01-18 19:04:35,0
snyk-playground/test-gradle,2019-05-15 19:56:43,0
snyk-playground/docker-goof,2021-08-25 13:37:41,0
snyk-playground/demo_snyk-transitive-ignore,2021-08-08 04:29:13,0
snyk-playground/cs-tools,2022-02-28 12:56:29,0
Loading