Skip to content

Adds model verification testing #467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Adds model verification testing #467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
15828fa
adds verify_models folder for integration testing
PulkitMishra Aug 21, 2021
fa1ae99
adds script for creating virtual environment and a pyre connection
PulkitMishra Aug 21, 2021
2bfe666
adds license
PulkitMishra Aug 22, 2021
807f1ef
adds library dependencies for all defined models
PulkitMishra Aug 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions tools/pysa_integration_tests/model_verify.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Copyright (c) Facebook, Inc. and its affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

import subprocess
import venv
import os
import shutil
import tempfile
from pathlib import Path
from ...api.connection import PyreConnection
from ...api.query import get_invalid_taint_models

cur_dir = Path(__file__).parent
envpath = cur_dir / ".env"
requirements_path = cur_dir / "verify_models" / "requirements.txt"
activate = envpath / "bin" / "activate"
command = f"pip install -r {requirements_path}"

if os.path.exists(envpath):
shutil.rmtree(envpath)
os.mkdir(envpath)
venv.create(envpath, with_pip=True)


subprocess.check_call(["bash", "-c", f"source {activate}; {command}"])

pyre_connection = PyreConnection(cur_dir / "verify_models")

temp_file = tempfile.TemporaryFile(prefix=".pyre_configuration", suffix=".local", dir=cur_dir)
temp_file.write(b'{"source_directories": ["verify_models"]}')
Copy link
Contributor

@abishekvashok abishekvashok Aug 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put in taint models path as well pointing to the repositories stubs? This helps us make the action verify newer models as well.


print(get_invalid_taint_models(pyre_connection))
Copy link
Contributor

@abishekvashok abishekvashok Aug 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What output do you get? Can we have a GitHub action that runs this script? We could mimic the pysa action and run this script instead of running the deliberately vulnerable flask app's script.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes having a github action is the next step @abishekvashok but first need to find a way to make pyre_configuration.local play well with pyre_configuration at the root folder

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If doesn't play well, feel free to delete the .pyre_configuration file at the root of the dir and create a file at the your folder. For an action it should get you by :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's possible and is my last resort. Some clever use of tempfile seems cleaner though

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PulkitMishra To avoid the issue with the .pyre_configuration located in the root folder the trick is to create a temporary directory in the global temporary folder (with something like tempfile.gettempdir() ) and create a folder there in which we can store the .pyre_configuration file. We can then pass to the PyreConnection the path of this folder so it will not conflict with the global .pyre_configuration.
We can create a function which does this in https://github.com/facebook/pyre-check/blob/main/api/connection.py and invoke it passing tempfile.gettempdir() as root directory and the path to the virtualenvironment we created as a source dir. Example function:

def create_temporary_pyre_connection(
    root_directory: pathlib.Path
    targets: Optional[List[str]] = None,
    source_directories: Optional[List[str]] = None,
) -> PyreConnection:
  # We can compute the sha1 as folder name to avoid creating a new directory every time and use the same if the source 
  directories and targets do not change
  root_directory.mkdir(exist_ok=True)
  pyre_configuration_directory = (
          root_directory
          / hashlib.sha1(
              ",".join((targets or []) + (source_directories or [])).encode()
          ).hexdigest()
      ) 
    pyre_configuration_directory.mkdir(exist_ok=True)
    with open(
        str(pyre_configuration_directory / ".pyre_configuration.local"), "w"
    ) as configuration:
        configuration_contents: Dict[str, Any] = {}
        if targets:
            configuration_contents["targets"] = targets
        if source_directories:
            configuration_contents["source_directories"] = source_directories
        json.dump(configuration_contents, configuration, indent=2)
    return PyreConnection(pyre_configuration_directory)

shutil.rmtree(envpath)
8 changes: 8 additions & 0 deletions tools/pysa_integration_tests/verify_models/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Overview

This folder helps model_verify.py in model validation.

# Contents

1. requirements.txt : File that contain the library dependencies needed for the models defined.
2. main.py : File with an empty main function to help run pyre on the folder.
7 changes: 7 additions & 0 deletions tools/pysa_integration_tests/verify_models/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (c) Facebook, Inc. and its affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

def main():
pass
44 changes: 44 additions & 0 deletions tools/pysa_integration_tests/verify_models/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
Django
mysql-connector-python
lxml
aiohttp
boto3
s3fs
python-amazon-simple-product-api
dill
falcon
Pillow
Markdown
aiofiles
fabric
fabric2
furl
mysqlclient
pandas
psycopg2
PyMySQL
pymssql
asyncssh
jsonpickle
pexpect
paramiko
Mako
Chameleon
Genshi
trender
Cheetah3
airspeed
chevron
SQLAlchemy
tornado
Jinja2
MarkupSafe
WTForms
Flask-WTF
djangorestframework
Flask
Werkzeug
PyYAML
requests
pycurl
pyre-check