Integrating ASDF file support

[embray]

Greetings folks,

The Advanced Scientific Data Format (ASDF) is being developed at the Space Telescope Science Institute. It is used in the JWST calibration pipeline and is the native format for the upcoming Roman Space Telescope mission.

We are working to provide support for commonly used data analysis software and SExtractor++ is one of them, as there is high demand for this.

We've already developed a fork of SE++ that integrates ASDF support using the newly developed libasdf C library (still in alpha, but with an initial release coming soon). The fork enhances SE++ by adding improved generalization for reading image data from different file formats, and integrates well into measurement and detection images (this improved image file framework could be used for other file formats as well, I believe).

It also integrates provisional WCS support for ASDF files. This is a bit tricky because ASDF being a rather general file format does not have any one standard (like FITS) for WCS. Currently it just supports the particular case of a FITS-compatible WCS embedded in an ASDF file, though I have ideas for where to go next with that tbd later.

We hope this contribution will allow ASDF to be supported in SExtractor++. Our aim is to provide continued support for ASDF as a format natively supported in SExtractor++. Please let me know if your community is amenable to such a contribution, and I can provide a pull request or series of PRs.

[mkuemmel]

Hi @embray,

thanks for opening this discussion and the work on SE++. We discussed this last week, and we are happy for this effort which opens SE++ to Roman data. We checked into ASDF, and while this is certainly promising things seem to be much in flux right now:

• there is only one python reference implementation of the format;
• the WCS formatting is not yet solved.
  You just made a pull request, so the code is already working? If ASDF is supported by SE++, would you continue the support?
  We were wondering whether it would be better to create a specific branch for this project.

[embray]

A specific branch might be good to start with, I'm happy to help work on this within whatever development / release process works for your project. The main goal is to get it eventually distributed in such a way that it's easy enough for users to access the functionality.

The C implementation exists and is what is used here. Most of the API, especially the parts used here, should remain fairly stable but you suggested it is a bit in flux and the code may need continued tweaking as the libasdf API settles. STScI has resources dedicated to providing support for this.

As for WCS this is the tricky part and maybe it helps to provide more background:

• Many ASDF files are using something called "GWCS" which was developed originally mainly for JWST data products, which required a more complex WCS representation than was possible easily with FITS WCS

• GWCS only has an implementation in Python (which both reads serialized GWCS transforms and executes them--in ASDF the GWCS is serialized in the YAML metadata

• There have been lots of discussions about C or C++ implementations of GWCS (or wrapping an existing library in a compatible manner), but not settled solution

• At the end of the day it's still the same basic idea, especially for images: pixel coordinates in -> celestial coordinates out, or vice-versa. But GWCS allows some arbitrarily complex in-between steps.

• GWCS also has a format for embedding a FITS-compatible WCS, and this can be dumped into WCSLIB pretty easily, which I've done and tested.

• For now, the solution is to only support the FITS-compatible WCS format if it exists. I also new configuration options for locating a WCS within an ASDF file

In the specific case of Roman Space Telescope there are "Level-2" data products that have the full GWCS, and "Level-3" products that have been fully processed and embed an approximated FITS-compatible WCS. So for now this would only work with Level-3 products.

Longer term the idea would be to make the WCS interface less FITS-specific (while of course maintaining the WCSLIB-based implementation) and have the ability to drop in other WCS implementation(s). One path I've considered is making use of SE++'s already built-in Python interpreter to use the gwcs library, but that's also a bit of a hassle as it would also require pulling in Astropy. I see something like that as a longer-term goal though.