Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5 Data reader enhancements #2328

Draft
wants to merge 3 commits into
base: develop
Choose a base branch
from
Draft

HDF5 Data reader enhancements #2328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
90cb6e7
Added a check in the HDF5 data reader to check that the metadata for
bvanessen Sep 26, 2023
de52603
Work in progress on making a more robust HDF5 data reader that can ha…
bvanessen Feb 27, 2024
0c1aa70
Added missing file.
bvanessen Feb 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -608,7 +608,7 @@ if (LBANN_WITH_HWLOC)
endif (LBANN_WITH_HWLOC)

if (LBANN_WITH_CONDUIT)
find_package(Conduit 0.7.0 CONFIG REQUIRED)
find_package(Conduit 0.8.9 CONFIG REQUIRED)
message(STATUS "Found CONDUIT: ${Conduit_DIR}")

if (NOT TARGET conduit::conduit OR NOT TARGET conduit::conduit_mpi)
Expand Down
2 changes: 1 addition & 1 deletion include/lbann/data_ingestion/data_store_conduit.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -541,7 +541,7 @@ class data_store_conduit
// methods follow
//=========================================================================

void start_exchange_data_by_sample(size_t current_pos, size_t mb_size);
void start_exchange_data_by_sample(uint64_t current_pos, uint64_t mb_size);
void finish_exchange_data_by_sample();

void setup_data_store_buffers();
Expand Down
5 changes: 5 additions & 0 deletions include/lbann/data_ingestion/readers/data_reader_HDF5.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ class DataReaderHDF5WhiteboxTester;
#define HDF5_METADATA_KEY_TRANSPOSE "transpose"
#define HDF5_METADATA_KEY_COERCE "coerce"
#define HDF5_METADATA_KEY_PACK "pack"
#define HDF5_METADATA_KEY_PARALLEL_IO "parallel_io"
#define HDF5_METADATA_KEY_PARALLEL_SPLIT "parallel_split"
/** Valid string values for a metadata file */
#define HDF5_METADATA_VALUE_COERCE_FLOAT "float"
#define HDF5_METADATA_VALUE_COERCE_DOUBLE "double"
Expand All @@ -55,6 +57,7 @@ class DataReaderHDF5WhiteboxTester;
#define HDF5_METADATA_VALUE_LAYOUT_HWC "hwc"
#define HDF5_METADATA_VALUE_LAYOUT_CDHW "cdhw"
#define HDF5_METADATA_VALUE_LAYOUT_DHWC "dhwc"
#define HDF5_METADATA_VALUE_TRUE "true"

namespace lbann {

Expand All @@ -78,6 +81,8 @@ static std::set<std::string> const hdf5_metadata_valid_keys = {
HDF5_METADATA_KEY_TRANSPOSE,
HDF5_METADATA_KEY_COERCE,
HDF5_METADATA_KEY_PACK,
HDF5_METADATA_KEY_PARALLEL_IO,
HDF5_METADATA_KEY_PARALLEL_SPLIT
};

/**
Expand Down
1 change: 1 addition & 0 deletions include/lbann/utils/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ set_full_path(THIS_DIR_HEADERS
cloneable.hpp
commify.hpp
compiler_control.hpp
conduit_extensions.hpp
dataset.hpp
describable.hpp
description.hpp
Expand Down
Loading