[FEATURE REQUEST] Partial loading in storage2cache()?

[jmmelko]

Hi,

HAPI is great, but not for offline usage.

Because, using it offline you would want to work with a exhaustive .par file, but only request lines within a certain wavenumber range.

The problem is that, as far as I understand, the storage2cache() method loads the entire .par file.

(btw, the pos=None parameter is not even used in this method, so you could at least add

if pos:
    for i in range(0, min([pos, line_count])):
        InfileData.readline())

)

A lot of time would be saved by loading only lines within a certain wavenumber range (except maybe when the seeked range is at the end of the .par file).

I am not sure it is possible with the current code as it is, because lines are all read as strings, and then converted to numbers.

For this reason, I continue to use our former matlab code which is faster because it has this functionality, which is sad because I like Python better.

Am I the only one to needthis functionality?

tx

[erwanp]

I think HAPI2 will include HDF5 support ; it would become very easy to only select specific waveranges.

If it can help : to improve performances in RADIS, we benchmarked a few HDF5 librairies to handle large line databases; and I'd definitly suggest to give a try to Vaex which proved extremelly efficient in our case (2nd link below) :

• Test_database_loading.ipynb
• Test_vaex.ipynb