Returned wcs header has 100s of header cards added {astrometry.net}

[Mucephie]

I'm finding that returned solved WCS headers have hundreds of header keys added. Here is an example of the cards numbers:

No.    Name      Ver    Type               Cards   Dimensions   Format
  0  PRIMARY       1     PrimaryHDU     718   (3072, 2047)   int16 (rescales to uint16)   
  1  MASK             1     ImageHDU            8   (3072, 2047)   uint8   
  2  UNCERT        1     ImageHDU            9   (3072, 2047)   float64   

An example of one of the added keys:

COMMENT   /data/INDEXES/index-5200-LITE/index-5206-46.fits                      
COMMENT Index(353):                                                             
COMMENT   /data/INDEXES/index-5200-LITE/index-5206-47.fits                      
COMMENT Field name: job.axy

It seems that this behaviour comes from this line in the monitor_submission function of the astrometryNet class

wcs = fits.Header.fromstring(wcs_response.text)

Its possible that the response text has both, the useful wcs header information, and some unneeded solve log information that then gets parsed into the header.

[bsipocz]

cc @mwcraig

[mwcraig]

Thanks for the ping @bsipocz -- I'll take a look at this tomorrow morning (Friday) along with the open PR for this module.

[mwcraig]

@Mucephie -- you are correct that astrometry.net returns a bunch of log information.

It isn't clear to me that the default behavior should be to remove them, though -- I've personally never used the log information but was reluctant to delete the cards in case the log was useful to someone.

Would it be helpful to have a way to disable adding those?

[Mucephie]

@mwcraig It would be very helpful to disable them, or at-least filter out the hundreds of "comment index" entries, and "comment index.fits" entries.

A sort of compromise would be to add the astrometry.Net job ID to the fits header so the log could be accessed as needed from the web, or a possible future astroquery/astrometryNet function that takes a job ID and returns the log info if anyone needs to access this information in a pipeline.

An example of the URL to grab a full job log is
https://nova.astrometry.net/joblog/8494841
Where 8494841 is the job ID number and the output to the browser is a .txt file.

[keflavich]

@Mucephie Are you looking for just the WCS solution without any additional header arguments? You can get that with, e.g.;

from astropy.wcs import WCS
WCS(header).to_header()

which will give a header with only WCS-relevant keywords.

My take is that astroquery should return all information the online service provides. If it isn't straightforward to remove some of that information, we can provide convenience tools, but we should still make sure users have access to anything the service provides, and I prefer that we provide too much information by default rather than too little.

If the problem is that these comment cards are significantly increasing the size of the data being transferred over the internet, though, I could see a case for leaving them out.

[Mucephie]

@keflavich This is a potential solution, but I'd much prefer this be a flag in the initial solving call, and have the possibility of adding the job ID to the header by default. And there is some information I'd like to retain that I don't think WCS(header).to_header() would retain, like pixel scale from astrometry. That method would also remove the original images header info.
edit:: Correction, I just realised that the original header information is not returned at all like it is using the web portal. I might need to find away to merge two headers.

Internet throughput could be an issue for others who have real-time pipelines that are doing very fast solves during observations, but I don't think the amount of text in the log will be the bottleneck in that pipeline.

[dstndstn]

You could always tell it the scale of your image, and then it would use fewer index files and produce fewer header cards ;)

[dstndstn]

PS, I was surprised to see that indeed it does not have the original header cards merged in.

[dstndstn]

PPS, you can get pixel scale trivially by sqrt(det(CD)) treating CD[12]_[12] as a matrix

[dstndstn]

PPPS I filed a ticket to add this merged header as an available output. I am, however, extremely slow at implementing stuff.
dstndstn/astrometry.net#303