Skip to content

Developer Guide

Chaunte W. Lacewell edited this page Jan 5, 2024 · 4 revisions

Contributions are more than welcome, and greatly appreciated.

We expect any contributions to be done through Pull-Requests. We also appreciated comments and issues through the Issues tracker.

General Code Conventions

  • No trailing whitespace; no trailing empty lines
  • No TABs
  • Indent set at four spaces

C++ Specific

  • “private:” and “public:” tags aligned with “class” keyword
  • Class name start with a capital letter
  • Filename all lowercase and using dash as word separator
  • Header file use “#pragma once”
  • Private members start with an underscore (“_”) prefix
  • Keep your lines to about 80 characters or less in length
  • No copy-and-paste from an open source projects. Clean code is an important requirement. If inserting code taken from other source, specify licenses together with the pull request, and we will evaluate if there is not conflicts.
  • When there is something not covered in these instructions, we follow this guide

Workflow Conventions

We follow this workflow model.

  • Develop/debug on a branch, started at "develop" (e.g., git checkout develop; git checkout –b ).

  • Prefer smaller, independent commits (make them ideally suited for “cherry-picking”)

    • First line of commit message descriptive of the commit; end with a period (‘.’)
    • One empty line, if more detailed explanations are needed
    • More detailed explanations in as many additional lines as needed
    • Use present tense and active voice (“Fix bug #8848” and not “Bug #8848 fixed” or “Fixed bug #8848”)
    • Keep your lines in the commit message to about 80 characters or less in length
  • When ready, create a pull request against the "develop" branch.

Tests

After you are ready for commit, run the tests from the tests folder. Make sure all the tests passes, if not, please fix the code, or the failing tests, or contact the person who wrote the test for help. Make sure that when running the tests, nothing else is printed on the screen other than test information.

Local Storage Tests

There are scripts to run the C++ and Python tests for local storage. Run sh run_tests.sh from the tests directory for the C++ tests. You should see on screen is something like:

Running C++ tests...
[==========] Running 214 tests from 18 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AutoReplicate
[ RUN      ] AutoReplicate.default_replicate
[       OK ] AutoReplicate.default_replicate (1414 ms)
[----------] 1 test from AutoReplicate (1414 ms total)

[----------] 1 test from ExampleHandler
[ RUN      ] ExampleHandler.simplePing
[       OK ] ExampleHandler.simplePing (86 ms)
[----------] 1 test from ExampleHandler (86 ms total)

[----------] 2 tests from AddImage
[ RUN      ] AddImage.simpleAdd
[       OK ] AddImage.simpleAdd (419 ms)
[ RUN      ] AddImage.simpleAddx10
[       OK ] AddImage.simpleAddx10 (570 ms)
[----------] 2 tests from AddImage (990 ms total)

[----------] 1 test from UpdateEntity
[ RUN      ] UpdateEntity.simpleAddUpdate
[       OK ] UpdateEntity.simpleAddUpdate (448 ms)
[----------] 1 test from UpdateEntity (448 ms total)

...

[----------] 10 tests from CLIENT_CPP_CSV
[ RUN      ] CLIENT_CPP_CSV.parse_csv_entity
[       OK ] CLIENT_CPP_CSV.parse_csv_entity (240 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_connection
[       OK ] CLIENT_CPP_CSV.parse_csv_connection (12 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_images
[       OK ] CLIENT_CPP_CSV.parse_csv_images (1662 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_descriptor_set
[       OK ] CLIENT_CPP_CSV.parse_csv_descriptor_set (27967 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_descriptor
[       OK ] CLIENT_CPP_CSV.parse_csv_descriptor (10 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_bb
[       OK ] CLIENT_CPP_CSV.parse_csv_bb (81 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_video
[       OK ] CLIENT_CPP_CSV.parse_csv_video (1495 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_invalid_entity
[       OK ] CLIENT_CPP_CSV.parse_csv_invalid_entity (1 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_invalid_image
Error: Failed to open file.
[       OK ] CLIENT_CPP_CSV.parse_csv_invalid_image (3 ms)
[ RUN      ] CLIENT_CPP_CSV.parse_csv_invalid_video
Failed to open file: ../tests/test_videos/Megamind_invalid.avi
[       OK ] CLIENT_CPP_CSV.parse_csv_invalid_video (2 ms)
[----------] 10 tests from CLIENT_CPP_CSV (31476 ms total)

[----------] 1 test from CLIENT_CPP_Video
[ RUN      ] CLIENT_CPP_Video.add_single_video
[       OK ] CLIENT_CPP_Video.add_single_video (1061 ms)
[----------] 1 test from CLIENT_CPP_Video (1061 ms total)

[----------] 3 tests from BLOB
[ RUN      ] BLOB.add_Blob
[       OK ] BLOB.add_Blob (40 ms)
[ RUN      ] BLOB.update_Blob
[       OK ] BLOB.update_Blob (8 ms)
[ RUN      ] BLOB.find_Blob
[       OK ] BLOB.find_Blob (3 ms)
[----------] 3 tests from BLOB (52 ms total)

[----------] Global test environment tear-down
[==========] 214 tests from 18 test suites ran. (896859 ms total)
[  PASSED  ] 214 tests.

Run sh run_python_tests.sh from the tests/python directory for the Python tests.

You should see on screen is something like:

Running Python tests...
test_addBoundingBox (TestBoundingBox.TestBoundingBox) ... ok
test_addBoundingBoxWithImage (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBox (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxBlob (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxBlobComplex (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxByCoordinates (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxCoordinates (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxesInImage (TestBoundingBox.TestBoundingBox) ... ok
test_updateBoundingBox (TestBoundingBox.TestBoundingBox) ... ok
test_updateBoundingBoxCoords (TestBoundingBox.TestBoundingBox) ... ok
test_FindEntity_link_constraints_float (TestConnections.TestConnections) ... ok
test_FindEntity_link_constraints_string (TestConnections.TestConnections) ... ok
test_addDescriptorsx1000 (TestDescriptors.TestDescriptors) ... ok
test_addSet (TestDescriptors.TestDescriptors) ... ok
test_addSetAndDescriptors (TestDescriptors.TestDescriptors) ... ok
test_addSetAndDescriptorsDimMismatch (TestDescriptors.TestDescriptors) ... ok
test_classifyDescriptor (TestDescriptors.TestDescriptors) ... ok
test_addDescriptorsx1000FaissIVFFlat (TestEngineDescriptors.TestDescriptors) ... ok
test_addDescriptorsx1000TileDBDense (TestEngineDescriptors.TestDescriptors) ... ok
test_addDescriptorsx1000TileDBSparse (TestEngineDescriptors.TestDescriptors) ... ok
test_addDifferentSets (TestEngineDescriptors.TestDescriptors) ... ok
test_FindWithSortBlock (TestEntities.TestEntities) ... ok
test_FindWithSortKey (TestEntities.TestEntities) ... ok
test_addEntityWithLink (TestEntities.TestEntities) ... ok
test_addFindEntity (TestEntities.TestEntities) ... ok
test_addfindEntityWrongConstraints (TestEntities.TestEntities) ... ok
test_runMultipleAdds (TestEntities.TestEntities) ... ok
test_addEntityWithBlob (TestEntitiesBlobs.TestEntitiesBlob) ... ok
test_addEntityWithBlobAndFind (TestEntitiesBlobs.TestEntitiesBlob) ... ok
test_addEntityWithBlobNoBlob (TestEntitiesBlobs.TestEntitiesBlob) ... ok
test_findDescByBlob (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobAndConstraints (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobNoLabels (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobNoResults (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobUnusedRef (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobWithLink (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConst_blobTrue (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConst_get_id (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConst_multiple_blobTrue (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConstraints (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescUnusedRef (TestFindDescriptors.TestFindDescriptors) ... ok
test_addImage (TestImages.TestImages) ... ok
test_addImageWithLink (TestImages.TestImages) ... ok
test_findEntityImage (TestImages.TestImages) ... ok
test_findImage (TestImages.TestImages) ... ok
test_findImageNoBlob (TestImages.TestImages) ... ok
test_findImageRefNoBlobNoPropsResults (TestImages.TestImages) ... ok
test_findImageResults (TestImages.TestImages) ... ok
test_findImage_multiple_results (TestImages.TestImages) ... ok
test_updateImage (TestImages.TestImages) ... ok
test_concurrent (TestRetail.TestEntities) ... skipped 'Skipping class until fixed'
test_create_connection (TestTestCommand.TestTestCommand) ... ok
test_disconnect (TestTestCommand.TestTestCommand) ... ok
test_vdms_existing_connection (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_get_last_response (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_get_last_response_str (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_non_existing_connection (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_non_json_query (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_print_last_response (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_query_disconnected (TestVDMSClient.TestVDMSClient) ... ok
test_FindFramesByFrames (TestVideos.TestVideos) ... ok
test_FindFramesByInterval (TestVideos.TestVideos) ... ok
test_FindFramesInvalidParameters (TestVideos.TestVideos) ... ok
test_FindFramesMissingParameters (TestVideos.TestVideos) ... ok
test_addVideo (TestVideos.TestVideos) ... ok
test_addVideoFromLocalFile_file_not_found (TestVideos.TestVideos) ... ok
test_addVideoFromLocalFile_invalid_command (TestVideos.TestVideos) ... ok
test_addVideoFromLocalFile_success (TestVideos.TestVideos) ... skipped 'Skipping class until fixed'
test_addVideoWithLink (TestVideos.TestVideos) ... ok
test_extractKeyFrames (TestVideos.TestVideos) ... ok
test_findVid_multiple_results (TestVideos.TestVideos) ... ok
test_findVideo (TestVideos.TestVideos) ... ok
test_findVideoNoBlob (TestVideos.TestVideos) ... ok
test_findVideoResults (TestVideos.TestVideos) ... ok
test_updateVideo (TestVideos.TestVideos) ... ok

----------------------------------------------------------------------
Ran 75 tests in 58.802s

S3 Storage Tests

There are also scripts to run the C++ and Python tests for S3 storage. For testing the S3 storage, we use MinIO and MinIO Client. Prior to running this test, when using MinIO, it is necessary to install the MinIO components. Assuming the codebase is located in /vdms, run the following:

curl -L -o /vdms/minio https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x /vdms/minio
mkdir -p /vdms/minio_files/minio-bucket


# Install the MinIO Client mc command line tool used by scripts for creating buckets
curl -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x /usr/local/bin/mc

To properly run the testing scripts, we require the username for your S3/MinIO storage. It is okay to use the default username (AWS_ACCESS_KEY_ID) and password (AWS_SECRET_ACCESS_KEY) for MinIO but we need these values for the test. Run sh run_aws_tests.sh -u ${AWS_ACCESS_KEY_ID} -p ${AWS_SECRET_ACCESS_KEY} from the tests directory for the C++ tests.

You should see on screen is something like:

Running C++ tests...
Note: Google Test filter = RemoteConnectionTest.*
[==========] Running 19 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 19 tests from RemoteConnectionTest
[ RUN      ] RemoteConnectionTest.RemoteWriteFilename
Added object 'test_images/large1.jpg' to bucket: minio-bucket
[       OK ] RemoteConnectionTest.RemoteWriteFilename (6263 ms)
[ RUN      ] RemoteConnectionTest.RemoteReadWriteBuffer
Successfully retrieved 'test_images/large1.jpg' from 'minio-bucket'.
Added object 'test_images/large1.jpg' to bucket: minio-bucket
[       OK ] RemoteConnectionTest.RemoteReadWriteBuffer (6141 ms)
[ RUN      ] RemoteConnectionTest.RemoteListRetrieveFile
Successfully retrieved 'test_images/large1.jpg' from 'minio-bucket'.
[       OK ] RemoteConnectionTest.RemoteListRetrieveFile (6061 ms)
[ RUN      ] RemoteConnectionTest.RemoteWriteVideoFilename
Added object 'test_videos/Megamind.avi' to bucket: minio-bucket
[       OK ] RemoteConnectionTest.RemoteWriteVideoFilename (6172 ms)
[ RUN      ] RemoteConnectionTest.RemoteReadVideoFilename
Successfully retrieved 'test_videos/Megamind.avi' from 'minio-bucket'.
[       OK ] RemoteConnectionTest.RemoteReadVideoFilename (6071 ms)
[ RUN      ] RemoteConnectionTest.ImageRemoteWritePNG
Added object 'pngs/test_image.png' to bucket: minio-bucket
[       OK ] RemoteConnectionTest.ImageRemoteWritePNG (6228 ms)
[ RUN      ] RemoteConnectionTest.ImageRemoteReadPNG
Successfully retrieved 'pngs/test_image.png' from 'minio-bucket'.
[       OK ] RemoteConnectionTest.ImageRemoteReadPNG (6213 ms)
[ RUN      ] RemoteConnectionTest.ImageRemoteRemovePNG
[       OK ] RemoteConnectionTest.ImageRemoteRemovePNG (6050 ms)
[ RUN      ] RemoteConnectionTest.ImageRemoteWriteJPG
Added object 'jpgs/large1.jpg' to bucket: minio-bucket
[       OK ] RemoteConnectionTest.ImageRemoteWriteJPG (6124 ms)
[ RUN      ] RemoteConnectionTest.ImageRemoteReadJPG
Successfully retrieved 'jpgs/large1.jpg' from 'minio-bucket'.
[       OK ] RemoteConnectionTest.ImageRemoteReadJPG (6186 ms)
[ RUN      ] RemoteConnectionTest.ImageRemoteRemoveJPG
[       OK ] RemoteConnectionTest.ImageRemoteRemoveJPG (6052 ms)
[ RUN      ] RemoteConnectionTest.TDBImageWriteS3
[       OK ] RemoteConnectionTest.TDBImageWriteS3 (6219 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedWriteFilename
WRITE: The RemoteConnection has not been started
[       OK ] RemoteConnectionTest.RemoteDisconnectedWriteFilename (6056 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedReadBuffer
READ: The RemoteConnection has not been started
Added object 'test_images/large1.jpg' to bucket: minio-bucket
[       OK ] RemoteConnectionTest.RemoteDisconnectedReadBuffer (6064 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedWriteBuffer
WRITE: The RemoteConnection has not been started
Successfully retrieved 'test_images/large1.jpg' from 'minio-bucket'.
[       OK ] RemoteConnectionTest.RemoteDisconnectedWriteBuffer (6059 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedListFiles
WRITE: The RemoteConnection has not been started
[       OK ] RemoteConnectionTest.RemoteDisconnectedListFiles (6055 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedRetrieveFile
WRITE: The RemoteConnection has not been started
[       OK ] RemoteConnectionTest.RemoteDisconnectedRetrieveFile (6058 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedWriteVideoFilename
WRITE: The RemoteConnection has not been started
[       OK ] RemoteConnectionTest.RemoteDisconnectedWriteVideoFilename (6057 ms)
[ RUN      ] RemoteConnectionTest.RemoteDisconnectedReadVideoFilename
READ_Video: The RemoteConnection has not been started
[       OK ] RemoteConnectionTest.RemoteDisconnectedReadVideoFilename (6054 ms)
[----------] 19 tests from RemoteConnectionTest (116191 ms total)

[----------] Global test environment tear-down
[==========] 19 tests from 1 test suite ran. (116192 ms total)
[  PASSED  ] 19 tests.

Run sh run_python_aws_tests.sh -u ${AWS_ACCESS_KEY_ID} -p ${AWS_SECRET_ACCESS_KEY} from the tests/python directory for the Python tests.

You should see on screen is something like:

Running Python AWS S3 tests...
test_addBoundingBox (TestBoundingBox.TestBoundingBox) ... ok
test_addBoundingBoxWithImage (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBox (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxBlob (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxBlobComplex (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxByCoordinates (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxCoordinates (TestBoundingBox.TestBoundingBox) ... ok
test_findBoundingBoxesInImage (TestBoundingBox.TestBoundingBox) ... ok
test_updateBoundingBox (TestBoundingBox.TestBoundingBox) ... ok
test_updateBoundingBoxCoords (TestBoundingBox.TestBoundingBox) ... ok
test_FindEntity_link_constraints_float (TestConnections.TestConnections) ... ok
test_FindEntity_link_constraints_string (TestConnections.TestConnections) ... ok
test_addDescriptorsx1000 (TestDescriptors.TestDescriptors) ... ok
test_addSet (TestDescriptors.TestDescriptors) ... ok
test_addSetAndDescriptors (TestDescriptors.TestDescriptors) ... ok
test_addSetAndDescriptorsDimMismatch (TestDescriptors.TestDescriptors) ... ok
test_classifyDescriptor (TestDescriptors.TestDescriptors) ... ok
test_addDescriptorsx1000FaissIVFFlat (TestEngineDescriptors.TestDescriptors) ... ok
test_addDescriptorsx1000TileDBDense (TestEngineDescriptors.TestDescriptors) ... ok
test_addDescriptorsx1000TileDBSparse (TestEngineDescriptors.TestDescriptors) ... ok
test_addDifferentSets (TestEngineDescriptors.TestDescriptors) ... ok
test_FindWithSortBlock (TestEntities.TestEntities) ... ok
test_FindWithSortKey (TestEntities.TestEntities) ... ok
test_addEntityWithLink (TestEntities.TestEntities) ... ok
test_addFindEntity (TestEntities.TestEntities) ... ok
test_addfindEntityWrongConstraints (TestEntities.TestEntities) ... ok
test_runMultipleAdds (TestEntities.TestEntities) ... ok
test_addEntityWithBlob (TestEntitiesBlobs.TestEntitiesBlob) ... ok
test_addEntityWithBlobAndFind (TestEntitiesBlobs.TestEntitiesBlob) ... ok
test_addEntityWithBlobNoBlob (TestEntitiesBlobs.TestEntitiesBlob) ... ok
test_findDescByBlob (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobAndConstraints (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobNoLabels (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobNoResults (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobUnusedRef (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByBlobWithLink (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConst_blobTrue (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConst_get_id (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConst_multiple_blobTrue (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescByConstraints (TestFindDescriptors.TestFindDescriptors) ... ok
test_findDescUnusedRef (TestFindDescriptors.TestFindDescriptors) ... ok
test_addImage (TestImages.TestImages) ... ok
test_addImageWithLink (TestImages.TestImages) ... ok
test_findEntityImage (TestImages.TestImages) ... ok
test_findImage (TestImages.TestImages) ... ok
test_findImageNoBlob (TestImages.TestImages) ... ok
test_findImageRefNoBlobNoPropsResults (TestImages.TestImages) ... ok
test_findImageResults (TestImages.TestImages) ... ok
test_findImage_multiple_results (TestImages.TestImages) ... ok
test_updateImage (TestImages.TestImages) ... ok
test_concurrent (TestRetail.TestEntities) ... skipped 'Skipping class until fixed'
test_create_connection (TestTestCommand.TestTestCommand) ... ok
test_disconnect (TestTestCommand.TestTestCommand) ... ok
test_vdms_existing_connection (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_get_last_response (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_get_last_response_str (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_non_existing_connection (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_non_json_query (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_print_last_response (TestVDMSClient.TestVDMSClient) ... ok
test_vdms_query_disconnected (TestVDMSClient.TestVDMSClient) ... ok
test_FindFramesByFrames (TestVideos.TestVideos) ... ok
test_FindFramesByInterval (TestVideos.TestVideos) ... ok
test_FindFramesInvalidParameters (TestVideos.TestVideos) ... ok
test_FindFramesMissingParameters (TestVideos.TestVideos) ... ok
test_addVideo (TestVideos.TestVideos) ... ok
test_addVideoFromLocalFile_file_not_found (TestVideos.TestVideos) ... ok
test_addVideoFromLocalFile_invalid_command (TestVideos.TestVideos) ... ok
test_addVideoFromLocalFile_success (TestVideos.TestVideos) ... skipped 'Skipping class until fixed'
test_addVideoWithLink (TestVideos.TestVideos) ... ok
test_extractKeyFrames (TestVideos.TestVideos) ... ok
test_findVid_multiple_results (TestVideos.TestVideos) ... ok
test_findVideo (TestVideos.TestVideos) ... ok
test_findVideoNoBlob (TestVideos.TestVideos) ... ok
test_findVideoResults (TestVideos.TestVideos) ... ok
test_updateVideo (TestVideos.TestVideos) ... ok

----------------------------------------------------------------------
Ran 75 tests in 320.375s
Clone this wiki locally