-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add E2E Test Suite #43
Conversation
@robgpita are we adding the whole collection FIM library to the reop or the bear minimum test cases that we need? |
@ar-siddiqui yes, I added the whole collection for simplifying the regression testing (comparing against data that has already been generated without removing specific reaches and libraries). We can pare this down, but more specific instructions must be included as to which files need to be deleted. Do you have any ideas on how to keep the testdata lightweight, but test all of the flows2fim functionality? Also, the reference flows files comprise the majority of the data footprint (240ish MB) and can be simplified to a subset of collections we intend to use for regression testing. Those collections have yet to be identified, so I did not delete unnecessary rows in the reference .csv flows files. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great. This will be of great help in development.
Please address few minor comments.
This PR will close #33.
Reference data used for regression testing has been added. This adds 245M to the repo, which will slow down the
git pull
command. The collection (mip_17110008) used for regression testing was the smallest collection with all of the appropriate files and libraries. Unfortunately, the rating curves table in theripple.gpkg
is empty, so tests for theflows2fim validate
command will have to occur with either a new set of reference data, ( instructions have been included), or as part of the next milestone (#35), when the ability to access S3 resources is incorporated as part of the test suite.The bulk of this test suite is embodied within a bash script. This script has 3 options for arguments : controls, fim, & validate which will test their respective
flows2fim
utility. If no argument is provided, allflows2fim
utilities will be tested.The github actions workflow file has also been updated to run
test_suite_linux.sh
for eachflows2fim
utility, using the test matrix shown below forflows2fim
, as well as the test suite's functionality itself.Contents of test_matrix.md:
Test Matrix
-scsv
-scvs
,-o
-db
only-scsv
empty-f
empty-f
columns swapped-f
no values-rel false
-c
-l
-o
-c
file