Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add E2E Test Suite #43

Merged
merged 28 commits into from
Feb 11, 2025
Merged

Add E2E Test Suite #43

merged 28 commits into from
Feb 11, 2025

Conversation

robgpita
Copy link
Collaborator

@robgpita robgpita commented Feb 6, 2025

This PR will close #33.

Reference data used for regression testing has been added. This adds 245M to the repo, which will slow down the git pull command. The collection (mip_17110008) used for regression testing was the smallest collection with all of the appropriate files and libraries. Unfortunately, the rating curves table in the ripple.gpkg is empty, so tests for the flows2fim validate command will have to occur with either a new set of reference data, ( instructions have been included), or as part of the next milestone (#35), when the ability to access S3 resources is incorporated as part of the test suite.

The bulk of this test suite is embodied within a bash script. This script has 3 options for arguments : controls, fim, & validate which will test their respective flows2fim utility. If no argument is provided, all flows2fim utilities will be tested.

The github actions workflow file has also been updated to run test_suite_linux.sh for each flows2fim utility, using the test matrix shown below for flows2fim, as well as the test suite's functionality itself.

Contents of test_matrix.md:

Test Matrix

generate outputs regression tests Assert errors
controls
Generate 2,5,10,25,50,100 yr controls.csv files
missing -scsv
missing -scvs, -o
-db only
-scsv empty
-f empty
-f columns swapped
-f no values
fim
Generate 2,5,10,25,50,100 yr fim.tif files
Generate fim files in different output formats
-rel false
missing -c
missing -l
missing -o
empty -c file
validate

@robgpita robgpita requested a review from ar-siddiqui February 6, 2025 17:48
@ar-siddiqui
Copy link
Owner

@robgpita are we adding the whole collection FIM library to the reop or the bear minimum test cases that we need?
I don't think we should add 254MB of data to the repo.

@robgpita
Copy link
Collaborator Author

robgpita commented Feb 10, 2025

@ar-siddiqui yes, I added the whole collection for simplifying the regression testing (comparing against data that has already been generated without removing specific reaches and libraries). We can pare this down, but more specific instructions must be included as to which files need to be deleted. Do you have any ideas on how to keep the testdata lightweight, but test all of the flows2fim functionality?

Also, the reference flows files comprise the majority of the data footprint (240ish MB) and can be simplified to a subset of collections we intend to use for regression testing. Those collections have yet to be identified, so I did not delete unnecessary rows in the reference .csv flows files.

Copy link
Owner

@ar-siddiqui ar-siddiqui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. This will be of great help in development.

Please address few minor comments.

INSTALL.md Outdated Show resolved Hide resolved
testdata/reference_data/db/ripple.gpkg-shm Outdated Show resolved Hide resolved
scripts/test_suite_linux.sh Outdated Show resolved Hide resolved
@ar-siddiqui ar-siddiqui merged commit da6b7c2 into ar-siddiqui:main Feb 11, 2025
1 of 2 checks passed
@robgpita robgpita deleted the e2e-test branch February 11, 2025 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a Test Cases Matrix
2 participants