LSUN download Data #2748
LSUN download Data #2748
Conversation
| @@ -82,6 +82,7 @@ def __init__( | |||
| # for each class, create an LSUNClassDataset | |||
| self.dbs = [] | |||
| for c in self.classes: | |||
| self._download(c) | |||
There was a problem hiding this comment.
The download should be optional. Other datasets have a download=True flag in the constructor that indicates whether the download should happen or not.
There was a problem hiding this comment.
Okay I will make the changes.
Thanks for your support and guidance.
| if not(os.path.isfile(os.path.join(self.root, file_name))): | ||
| download_url(url, self.root, file_name) | ||
| print("Extracting File") | ||
| extract_archive(file_path, self.root, True) | ||
| print("Done!!!") |
There was a problem hiding this comment.
You can make this simpler by using
vision/torchvision/datasets/utils.py
Lines 242 to 249 in 588f7ae
| @@ -93,6 +94,16 @@ def __init__( | |||
| self.indices.append(count) | |||
|
|
|||
| self.length = count | |||
|
|
|||
| def _download(self, c): | |||
| url = 'http://dl.yf.io/lsun/scenes/{}'.format(c + '_lmdb.zip') | |||
There was a problem hiding this comment.
Before the download you could check if the lmdb file is present and if its MD5 checks out with
vision/torchvision/datasets/utils.py
Lines 38 to 43 in 588f7ae
If that is the case you don't need to download anything.
There was a problem hiding this comment.
@pmeier Do I have to find md5 hash, if yes how? otherwise I can just directly check if the file exists or not
There was a problem hiding this comment.
Do I have to find md5 hash [...] otherwise I can just directly check if the file exists or not
Although not terribly common, download errors happen. Without a checksum, you have no idea if the file maybe is corrupt. Thus it we prefer to have a MD5 checksum for most files. In your case that would mean for the .zip and .lmdb files of every class.
if yes how?
You can download and extract all archives with the download logic you already have. Afterwards you can use a variety of tools. Depending on your setup you might have md5sum already installed. If you don't want additional software, you can use torchvision.datasets.utils.calculate_md5
vision/torchvision/datasets/utils.py
Line 26 in de90862
Codecov Report
@@ Coverage Diff @@
## master #2748 +/- ##
==========================================
+ Coverage 73.05% 73.12% +0.07%
==========================================
Files 96 96
Lines 8298 8317 +19
Branches 1291 1293 +2
==========================================
+ Hits 6062 6082 +20
Misses 1838 1838
+ Partials 398 397 -1
Continue to review full report at Codecov.
|
|
Closing this since it is stale and the prototype version of LSUN in #5390 includes downloads. |
Added the function to download LSUN data from the site, extract it and then delete zip file