SE++ v0.21 assoc mode with no detection image

[mShuntov]

Hi guys,

Great to see the new update and the possibility of bypassing the detection image with the assoc mode!

However, I struggle to make it work, and the syntax of how to do it is not very clear to me.

I am interested in running SE++ to fit models on a list of measurement images in different bands. I construct an assoc file that contains ra, dec, group_id, flux, a_image, theta_image, axratio that are measured from an initial SE classic run and are to be used as initial values in the model fit. The sources are grouped manually -- sources that should be fit together are assigned same group_id.

Can you please clarify to me what is the syntax to run SE++ in assoc mode only and bypassing the detection image?

I have tried to simply not pass a detection image as an argument in the command line like:

sourcextractor++ --python-config-file $python_config \
         --python-arg $TILE 'sersic_full_assoc' \
         --assoc-catalog=$assoc_catname \
         --assoc-columns=1,2 \
         --assoc-filter=matched \
         --assoc-radius=20 \
         --assoc-coord-type=WORLD \
         --assoc-copy=1,2,3,4,5,6,7,8,9,10,11,12,13,14 \
         --assoc-group-id=4 \
         --assoc-source-sizes=11 \
         --log-file=$output_catname"_assoc.log" \
	  --output-catalog-filename $output_catname".fits" \
	  --output-properties=$properties \
	  --magnitude-zero-point ${mag_zeropoint} \
         --model-fitting-iterations 500 \
	 --sampling-scale-factor 1 \
         --sampling-max-fit-size 5024 \
         --tile-size=131072 \
         --tile-memory-limit=131072 \
         --output-flush-size=100 \
         --check-image-model-fitting=${checkimgname}"_model.fits" \
         --check-image-residual=${checkimgname}"_resid.fits" \
	  --thread-count 48 \
	   --log-level DEBUG \

But I get the error

Multithreading DEBUG : Starting output thread
/var/spool/torque/mom_priv/jobs/298979-6.cmaster.SC: line 115: 16731 Segmentation fault      (core dumped)

This happens even if I disable the multithreading (--thread-count=0). However, If I provide a detection image, everything works fine.

In case it matters, my model is

    x = FreeParameter(lambda o: o.assoc_value_0) 
    y = FreeParameter(lambda o: o.assoc_value_1) 
    
    rad = FreeParameter(lambda o: o.assoc_value_10, Range(lambda v, o: (.001*v, 2.0*v), RangeType.EXPONENTIAL))
    sersic = FreeParameter( 2.0, Range((0.3, 5.0), RangeType.LINEAR))
    ratio = FreeParameter(lambda o: o.assoc_value_13, Range( (0.08, 1.0), RangeType.LINEAR) )
    angle = FreeParameter( lambda o: o.assoc_value_12)

    i = 0
    flux = {}
    mag = {}

    for band,group in mesgroup: 

        flux[i] = FreeParameter(lambda o: o.assoc_value_9)
        mag[i] = DependentParameter(lambda f, zp=mag_zeropoint[band]: -2.5 * np.log10(f) + zp, flux[i] )
        
        # the centroid is fixed for all bands
        add_model(group, 
                  SersicModel(x, y, flux[i], rad, ratio, angle, sersic ) )

        add_output_column(MagColNames_list[i], mag[i])
        add_output_column(FluxColNames_list[i], flux[i])
        i+=1
        
    ra, dec, wc_rad, wc_angle, wc_ratio = get_world_parameters(x, y, rad, angle, ratio)

Could you please provide/describe a simple example of how to run the assoc mode without a detection image?

Cheers,
Marko.

[mkuemmel]

Hi @mShuntov, a bold, and no easy move, especially since the documentation is sub-optimal (meaning non-existent). Also I have always problems setting a run up.

Two possible problems:

• column indexing ion the command line is 1-indexed, but in the python config file it is 0-indexed. But I think you took care of that;
• you need to set --segmentation-algorithm=ASSOC and --grouping-algorithm=ASSOC

The second might be the problem. If you continue to have problems I could provide a test data set (originally from @marcschefer ) and you could come from that side.

Hope this helps.

[mkuemmel]

...on a second look, please ensure that FlexibleModelFitting is in your output properties.
Properties that refer to a detection image (such as PixelCentroid or ShapeParameters) are just neglected, but there could be problems as well.

[mShuntov]

Thanks so much @mkuemmel ! Setting --segmentation-algorithm=ASSOC and --grouping-algorithm=ASSOC solves the problem and now it runs! However, there is a different problem now that makes things crash and I tried everything (including removing all but FlexibleModelFitting properties) but it still persists with the following error:

Progress  INFO : Segmentation: 0
Progress  INFO : Detected: 1
Progress  INFO : Deblended: 0
Progress  INFO : Measured: 0
Progress  INFO : Elapsed: 00:04:18
Multithreading DEBUG : Starting output thread
SourceWithOnDemandProperties DEBUG : No TaskFactory for property: SourceXtractor::DetectionFrame [ 0 ] 
SourceXtractor ERROR : Failed to process the assoc catalog!
Property SourceXtractor::DetectionFrame [ 0 ]  not found!
Multithreading DEBUG : Stopping output thread
Multithreading DEBUG : All worker threads done!
ElementsProgram DEBUG : # Exit Code: 1

the assoc file is a simple .txt file. It can read it, because if there was a problem with the format it would raise a different error (e.g. FATAL : Can't either open or read assoc catalog). Here are the first 4 lines of the assoc file for example:

149.97308062327846	2.460798324420142	6	17065	2511.86767578125	2511.86767578125	1729.3203125	970.9883422851562	2252.554931640625	1995.3197021484375	33.54143524169922	8.886938095092773	-67.1298599243164	0.2649540305137634
149.90855484382504	2.483329285465301	6	17147	612.8126220703125	612.8126220703125	1786.3358154296875	2053.62158203125	258.48394775390625	1064.813232421875	18.700122833251953	11.409201622009277	-20.684810638427734	0.6101137399673462
149.9333569659592	2.477113305472032	6	17294	2898.418212890625	2898.418212890625	6094.47607421875	3523.963134765625	1565.0478515625	3396.06494140625	30.590499877929688	23.856542587280273	-87.5073471069336	0.7798677086830139

columns correspond to
ra dec tile group_id flux_115w flux_150w flux_277w flux_444w flux_f814w flux_mean a_image b_image theta axratio

I tried snooping in the source code for this error but didn't find the reason..

So, yes it might be helpful if you can share a working example and I try to figure out what's going on.

Thanks!

[mkuemmel]

Hi @mShuntov, please try two things:

• add --partition-multithreshold=false to your command. If this works then it is a bug and we open an issue;
• if you still have problems then use the demo here:
  https://cloud.physik.lmu.de/index.php/s/XYTs7AX7TdFKsF6 to have kind of a fresh start; there is one command that works (--> run.sh);

Hope this helps.

[mShuntov]

Hi @mkuemmel, the first point (adding --partition-multithreshold=false) solves everything, and now it runs! Thanks so much!

[mkuemmel]

Thanks to you that we found out the problem! I opened #576 and mark my last contribution as answer.

[mShuntov]

Hi guys, I've been running and figuring out the new ASSOC mode, but some things still elude me. I notice persistently that the model and residual images around bright and big objects all have box shapes, like in the figure below. I am pushing the --sampling-max-fit-size 16384 and the prior on the size is very loose (max value 10 arcsec).

How is the box size of the model fit defined? I've found somewhere this definition "fit is done inside a disk whose size is defined by the isophote, dilated by the FWHM of the PSF, and a 20% margin". How does this work in the ASSOC mode when there is no detection image to define the isophote? Is it maybe defined on the measurement images? Or is it taken from the sizes in the input assoc file? Maybe --assoc-source-sizes and --assoc-default-pixel-size control this?

Additionally, what should the units of the size be in the input assoc file? Because if they are in pixels measured on a high-resolution image (0.03 as/pix) and then passed on to models on lower-res (0.1 as/pix) images, there is usually no information to do the transform. Also, what if each measurement image in the group has a different pixel scale?

Cheers,
Marko.

[mShuntov]

Hi @mkuemmel, I forgot to come back to you on that, but yes indeed this was helpful and I sorted my problem. This adds another degree of freedom where one can control the size of the fitting area by providing --assoc-source-sizes . In my case I decided to add another column in the assoc file that sets this size. Now I don't see these boxy model/residuals and fits look much better.

I found out that in some cases, very large fitting areas cause fmf_stop_reason 0, so one needs to be careful about --assoc-source-sizes`.

I am now trying to sort out a different issue, which is when in some bands there are areas with no coverage (in my case flux=0) the model returns the initial value with non-vanishing errors.

Thanks for the help!

[mShuntov]

To add on to this issue that I mentioned: I was previously flagging these sources if aper or auto flux = 0, which leads me to the question:

Is there really no way of doing aperture and auto photometry in the ASSOC mode without a detection image?

[mkuemmel]

@marcschefer is in vacation and can answer the questions for aper/auto photometry.

For aperture photmetry it might actually work, have you tried?

Auto photometry is more complicated since it requires Kron parameters that are determined on the detection image.

[mkuemmel]

@mShuntov note that we will test aperture photometry in no-detection mode in #581

[mkuemmel]

Look at this: #586
@mShuntov feel free to test!

[mShuntov]

Hi guys, I am coming back to this discussion with another question because I am trying to understand the details of this assoc mode feature.

In general, everything it works really really well for me and I am very happy with the results. But, running on 30+ bands I need to look into the nitty-gritty details and try to speed things up wherever possible.

It seems that the fitting area (managed by --assoc-source-sizes) is very important for the speed. Since I have a list of images that range from 0.03 "/pix to 0.15 "/pix, I wonder how --assoc-source-sizes that is defined in pixels of the first image in the list (i guess) that for me has 0.03"/pix is translated to the other images that have 0.15 "/pix. Because a fitting area of 200 pixels on 0.03 "/pix image is appropriate but is way too big for 0.15 "/pix image.

I checked the output fmf_fitting_areas (taking the array that corresponds to the high-res and low-res images) and it seems that it's the same fitting areas in both bands. If this is the case, then I guess things can be optimized by scaling with the image pixel scale.

Is this something hard coded or can it be controlled via the configuration?

Cheers!

[mkuemmel]

The fitting size should scale with the image scale. From you plot that does not seem to be the case. Maybe @marcschefer can have a look at this? Could you provide some example images?

[marcschefer]

The sizes are defined as pixels of the reference image (which is either the detection image if there is one or the first measurement image). They are scaled to the other images.

[mkuemmel]

I can confirm what @marcschefer described:
The size of the fitting box is adjusted when dealing with images of a different pixel scale.
It's not clear why this does not show in your diagram.

Note that there is another mechanism to influence the fitting area in the no-detection case. When dealing with rotated images, the fitting area is made to enclose the fitting area in the first image projected to the measurement area. The rotation part of the projection enlarges the fitting area in the measurement image.

This can explain the points in the upper left of your diagram.

But still the different scale should show up.

[mShuntov]

Thanks @mkuemmel and @marcschefer for the explaination!
Yes, it's curious why this doesn't show up in my plot, but that is less important now for me now. I think also found some benefits of keeping the fitting area large, so I will stick with what I was doing before.

[mkuemmel]

Here the situation in my example:

The second image with index 1 has a scale 3 times larger than the first one. There is a constant panning, hence the offset from the blue line.