+ + +
+
+ +
+
+ +
+ +
+ + +
+ +
+ + +
+
+ + + + + +
+ +
+

Dataset#

+
+

Overview#

+

SED comes with the ability to download and extract any URL-based datasets. By default, users can use the datasets:

+
    +

  • WSe2

    • +

  • TaS2

    • +

  • Gd_W110

    • +

  • W110

    • +
+

It is easy to extend this list using a JSON file.

+
+
+
+

Getting Datasets#

+
+

Importing Required Modules#

+
import os
+from sed.dataset import dataset
+
+
+
+
+

get() Method#

+

The get method requires only the dataset name, but an alternative root_dir can be provided.

+

Try interrupting the download process and restarting it to see that it resumes from where it stopped.

+
dataset.get("WSe2", remove_zip=False)
+
+
+

Example Output:

+
Using default data path for "WSe2": "<user_path>/datasets/WSe2"
+
+3%|▎         | 152M/5.73G [00:02<01:24, 71.3MB/s]
+
+Using default data path for "WSe2": "<user_path>/datasets/WSe2"
+
+100%|██████████| 5.73G/5.73G [01:09<00:00, 54.3MB/s]
+
+Download complete.
+
+
+

By default, not providing remove_zip=False will delete the zip file after extraction:

+
dataset.get("WSe2")
+
+
+

Setting use_existing=False allows downloading the data to a new location instead of using existing data.

+
dataset.get("WSe2", root_dir="new_datasets", use_existing=False)
+
+
+

Example Output:

+
Using specified data path for "WSe2": "<user_path>/new_datasets/datasets/WSe2"
+Created new directory at <user_path>/new_datasets/datasets/WSe2
+
+
+

Interrupting extraction here behaves similarly, resuming from where it stopped.

+

If the extracted files are deleted, rerunning this command below will re-extract from the zip file:

+
dataset.get("WSe2", remove_zip=False)
+
+
+

Example Output:

+
Using default data path for "WSe2": "<user_path>/datasets/WSe2"
+WSe2 data is already fully downloaded.
+
+5.73GB [00:00, 12.6MB/s]
+
+Download complete.
+Extracting WSe2 data...
+
+100%|██████████| 113/113 [02:41<00:00, 1.43s/file]
+
+WSe2 data extracted successfully.
+
+
+

and this does not delete the zip file.

+
+
+
+
+

remove() Method#

+

The remove method allows removing some or all instances of existing data.

+

Remove only one instance:

+
dataset.remove("WSe2", instance=dataset.existing_data_paths[0])
+
+
+

Example Output:

+
Removed <user_path>/datasets/WSe2
+
+
+

Remove all instances:

+
dataset.remove("WSe2")
+
+
+

Example Output:

+
WSe2 data is not present.
+
+
+
+
+
+

Useful Attributes#

+
+

Available Datasets#

+
dataset.available
+
+
+

Example Output:

+
['WSe2', 'TaS2', 'Gd_W110', 'W110']
+
+
+
+
+

Data Directory#

+
dataset.dir
+
+
+

Example Output:

+
'<user_path>/datasets/WSe2'
+
+
+
+
+

Subdirectories#

+
dataset.subdirs
+
+
+

Example Output:

+
['<user_path>/datasets/WSe2/Scan049_1',
+ '<user_path>/datasets/WSe2/energycal_2019_01_08']
+
+
+
+
+

Existing Data Paths#

+
dataset.existing_data_paths
+
+
+

Example Output:

+
['<user_path>/new_dataset/datasets/WSe2',
+ '<user_path>/datasets/WSe2']
+
+
+
+
+
+
+

Example: Adding Custom Datasets#

+
+

DatasetsManager#

+

Allows adding or removing datasets in a JSON file at different levels (module, user, folder). It also checks all levels to list available datasets.

+
import os
+from sed.dataset import DatasetsManager
+
+
+
+

Adding a New Dataset#

+

This example adds a dataset to both the folder and user levels. Setting rearrange_files=True moves all files from subfolders into the main dataset directory.

+
example_dset_name = "Example"
+example_dset_info = {
+    "url": "https://example-dataset.com/download",  # Not a real path
+    "subdirs": ["Example_subdir"],
+    "rearrange_files": True
+}
+
+DatasetsManager.add(data_name=example_dset_name, info=example_dset_info, levels=["folder", "user"])
+
+
+

Example Output:

+
Added Example dataset to folder datasets.json
+Added Example dataset to user datasets.json
+
+
+

Verify that datasets.json is created:

+
assert os.path.exists("./datasets.json")
+dataset.available
+
+
+

Example Output:

+
['Example', 'WSe2', 'TaS2', 'Gd_W110']
+
+
+
+
+

Removing a Dataset#

+

Remove the Example dataset from the user JSON file:

+
DatasetsManager.remove(data_name=example_dset_name, levels=["user"])
+
+
+

Example Output:

+
Removed Example dataset from user datasets.json
+
+
+

Adding an already existing dataset will result in an error:

+
DatasetsManager.add(data_name=example_dset_name, info=example_dset_info, levels=["folder"])
+
+
+

Example Output:

+
ValueError: Dataset Example already exists in folder datasets.json.
+
+
+
+
+

Downloading the Example Dataset#

+
dataset.get("Example")
+
+
+

Example Output:

+
Using default data path for "Example": "<user_path>/datasets/Example"
+Created new directory at <user_path>/datasets/Example
+Download complete.
+Extracting Example data...
+
+100%|██████████| 4/4 [00:00<00:00, 28.10file/s]
+
+Example data extracted successfully.
+
+
+
+
+

Download to Another Location#

+
dataset.get("Example", root_dir="new_datasets", use_existing=False)
+
+
+

Example Output:

+
Using specified data path for "Example": "<user_path>/new_datasets/datasets/Example"
+Created new directory at <user_path>/new_datasets/datasets/Example
+
+
+
+
+

Removing an Instance#

+
print(dataset.existing_data_paths)
+path_to_remove = dataset.existing_data_paths[0]
+dataset.remove(data_name="Example", instance=path_to_remove)
+
+
+

Example Output:

+
Removed <user_path>/new_datasets/datasets/Example
+
+
+

Verify that the path was removed:

+
assert not os.path.exists(path_to_remove)
+
+
+
print(dataset.existing_data_paths)
+
+
+

Example Output:

+
['<user_path>/datasets/Example']
+
+
+
+
+
+
+
+

Default datasets.json#

+
{
+    "WSe2": {
+        "url": "https://zenodo.org/record/6369728/files/WSe2.zip",
+        "subdirs": [
+            "Scan049_1",
+            "energycal_2019_01_08"
+        ]
+    },
+    "Gd_W110": {
+        "url": "https://zenodo.org/records/10658470/files/single_event_data.zip",
+        "subdirs": [
+            "analysis_data",
+            "calibration_data"
+        ],
+        "rearrange_files": true
+    },
+    "W110": {
+        "url": "https://zenodo.org/records/12609441/files/single_event_data.zip",
+        "subdirs": [
+            "analysis_data",
+            "calibration_data"
+        ],
+        "rearrange_files": true
+    },
+    "TaS2": {
+        "url": "https://zenodo.org/records/10160182/files/TaS2.zip",
+        "subdirs": [
+            "Scan0121_1",
+            "energycal_2020_07_20"
+        ]
+    },
+    "Au_Mica": {
+        "url": "https://zenodo.org/records/13952965/files/Au_Mica_SXP.zip"
+    },
+    "Test": {
+        "url": "http://test.com/files/file.zip",
+        "subdirs": [
+            "subdir"
+        ],
+        "rearrange_files": true
+    }
+}
+
+
+
+
+ + +
+ + + + + + + +
+ + + + + + + +
+
+ +
+ +