True divide runtime warning in analysis\filterbarcodes

[aschrader5]

I have been getting a true divide runtime warning during the filterbarcodes task. Would you know what is causing this? Any help would be appreciated.
-Alex
Error:
Job 9: Running AdaptiveFilterBarcodes 0
C:\Users\alexws2\Anaconda3\envs\merlin_env\lib\site-packages\matplotlib\axes_axes.py:8192: RuntimeWarning: invalid value encountered in true_divide
vals = 0.5 * width * vals / vals.max()
C:\Users\alexws2\Anaconda3\envs\merlin_env\lib\site-packages\pandas\core\series.py:856: RuntimeWarning: divide by zero encountered in log10
result = getattr(ufunc, method)(*inputs, **kwargs)
MERlin - the MERFISH decoding pipeline
Running AdaptiveFilterBarcodes
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:122: RuntimeWarning: divide by zero encountered in true_divide
blankFraction = blankHistogram / totalHistogram
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:122: RuntimeWarning: invalid value encountered in true_divide
blankFraction = blankHistogram / totalHistogram
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:130: RuntimeWarning: overflow encountered in true_divide
blankBarcodeCount + codingBarcodeCount)
C:\Users\alexws2\Anaconda3\envs\merlin_env\lib\site-packages\scipy\optimize\zeros.py:341: RuntimeWarning: Tolerance of 0.09999999999999998 reached.
warnings.warn(msg, RuntimeWarning)
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:122: RuntimeWarning: invalid value encountered in true_divide
blankFraction = blankHistogram / totalHistogram
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:130: RuntimeWarning: overflow encountered in true_divide
blankBarcodeCount + codingBarcodeCount)

[emanuega]

This error arises because it's unable to find a threshold that achieves the desired misidentification rate. This could be because there are few blanks or few decoded barcodes so that although it could find thresholds that achieve a greater than the desired misidentification rate and less than the desired identification, there is no threshold where the misidentification rate is within the specified tolerance. In the v0.1.2 branch we exposed the tolerance parameter for the GenerateAdaptiveFilter task. The default is 0.001.

[aschrader5]

Than you for your quick response. It seems that setting the tolerance to 0.1 has removed the error in the optimize method. However I am now getting a true divide error during the decode task. The full error plot is shown below:
Job 14: Running Decode 0
MERlin - the MERFISH decoding pipeline
MERlin - the MERFISH decoding pipeline
Running Decode
Running GenerateAdaptiveThreshold
c:\users\alexws2\merlin\merlin\data\dataorganization.py:137: FutureWarning: item has been deprecated and will be removed in a future version
return self.data[self.data['readoutName'] == bitName].index.item()
c:\users\alexws2\merlin\merlin\util\registration.py:114: RuntimeWarning: invalid value encountered in true_divide
m = np.divide(-(fdv + fdu), (fdu - fdv))
c:\users\alexws2\merlin\merlin\util\registration.py:117: RuntimeWarning: invalid value encountered in true_divide
unsmoothm = np.divide(dIdv + dIdu, dIdu - dIdv)
c:\users\alexws2\merlin\merlin\util\registration.py:114: RuntimeWarning: divide by zero encountered in true_divide
m = np.divide(-(fdv + fdu), (fdu - fdv))
[Wed Oct 30 22:27:52 2019]
Finished job 14.
510 of 518 steps (98%) done
C:\Users\alexws2\Anaconda3\envs\merlin_env\lib\site-packages\pandas\core\series.py:856: RuntimeWarning: divide by zero encountered in log10
result = getattr(ufunc, method)(*inputs, **kwargs)
C:\Users\alexws2\Anaconda3\envs\merlin_env\lib\site-packages\matplotlib\axes_axes.py:8192: RuntimeWarning: invalid value encountered in true_divide
vals = 0.5 * width * vals / vals.max()

[emanuega]

This is a warning coming from the Decode task, but the cases when this warning arises are handled in lines 115-127 of registration.py, so there is nothing you have to do.

[aschrader5]

Ok, just wanted to make sure those errors were normal.

However, I am still seeing errors in the partitionbarcodes task, even when I do not see the optimization error. Do you know what would cause this? and how would I go about fixing it?
Thank you,
Alex
For example:
Job 16: Running PartitionBarcodes 0

c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:124: RuntimeWarning: divide by zero encountered in true_divide
blankFraction = blankHistogram / totalHistogram
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:124: RuntimeWarning: invalid value encountered in true_divide
blankFraction = blankHistogram / totalHistogram
c:\users\alexws2\merlin\merlin\analysis\filterbarcodes.py:132: RuntimeWarning: overflow encountered in true_divide
blankBarcodeCount + codingBarcodeCount)

[emanuega]

That is a warning that arises because there are bins in the histogram that contain no barcodes resulting in division be 0. The error doesn't alarm me, but I expect it to come from the AdaptiveFilterBarcodes task instead of PartitionBarcodes. It is possible your PlotPerformance task is running concurrently to your PartitionBarcodes task and is actually the source of the warning.

Of note, the WatershedSegment task in merlin is currently most optimized for brain samples and I am not sure how well the segmentation will perform if you are applying it to other data.

[aschrader5]

Interesting, I am using a slightly edited version of the merlin_parameters file you had posted in the merlin example folder. I will try it again tomorrow with a fresh version of the parameters file.

Could this error also be caused by data with low SNR values? During my experiment I noticed that the SNR of my images was decreasing overtime.

I am not really paying much attention to the watershed method, since I do not have a good PolyT map of the mRNA in my system.

Thank you again,
Alex

[emanuega]

Typically we spots visible over the background through the experiment I find that I can get something to decode. How far is your decoding going? With the warnings you've shown, I still expect it to make it all the way through.

How many fields of view are you decoding? I am not sure how well the scale factor and chromatic aberration optimization would work on just a few fields of view, which could contribute to poor decoding.

[aschrader5]

I am only decoding a single FOV (for simplicity), I have a total of 9 fovs that I can analyze if necessary.

If I limit my analysis to bar codes with more then 2 pixels, I still get the tolerance error during the adaptivefilterbarcode task. This is why I am a bit worried about my data being to dim to identify spots.

What does your scale factors look like? This is what mine looks like:

This is also what my intensity distribution looks like:

Finally, this is what my abundance plots look like:

Thank you again for your help,
Alex

[emanuega]

The intensity vs area plot doesn't look promising. If this is a measurement with ~16-20 bits then I would expect to see a better defined higher intensity population. I'd suggest looking at your aligned fiducial images to make sure the alignment is working well and I'd also suggest trying all 10 fovs.

[aschrader5]

We are using the same probes that were used in the 16 bit MERFISH experiments in U2OS cells.

Also, in order to align the fiducial bead images I had to increase the highpass_sigma from 3 to 8. But the images do look like they were aligned correctly when using a highpass sigma of 8.

Would it be possible for you to provide a single FOV of example data? I want to compare your SNR to mine and see what a proper output of your processing software looks like.

Thank you again,
Alex

[aschrader5]

Quick question, would having small regions that are at 65k intensity in one of the images cause the image analysis issues I have been having?

[emanuega]

Having small regions where the camera was saturated should not cause significant problems. For a 16 bit measurement in U2OS cells I am surprised there are SNR concerns, especially if the sample is cleared. I've attached an image (as a png file) from a single FOV for a single imaging round for a 16 bit U2OS measurement for comparison. I don't believe that decoding a single FOV will provide proper output from the processing software since I expect the scale factor and chromatic aberration correction perform better with more than 1 field of view. I'd again suggest trying to analyze all of your fovs.

[aschrader5]

Good evening,
I have ran all 9 FOVs I have and I am still seeing barcodes that are randomly assigned outside of the cells. I have been running 10 optimization steps as well for my 9 FOVs, is that to many optimization steps?
Thank you for your help,
Alex
Below is one of the later bits I have imaged:

[emanuega]

I agree that your image looks like it has low SNR and it is unclear that the spots we do see in your image are true spots rather than stray probes. I'd recommend optimizing your experimental workflow until you get an image that looks more like the one I posted.

Using more optimization steps that is needed won't negatively impact the decoding quality. The optimized parameters will converge and then stay constant. I almost always use 10.

[aschrader5]

Ok, I will look into that.
Quick question, during your experiments do you see the first round of imaging being much brighter then the following rounds?
Thank you,
Alex

[aschrader5]

Good evening,

Would it be possible for you to send me the original Tif files from multiple different rounds of your experiment? Like bits 1, 8, 16 for the same FOV. We want to compare your images to ours to evaluate how much brighter your spots are compared to your background and how that compares to our images. Do you find that you need to increase the exposure time as you image the later bits?
It would be easiest to send the images to me via email at alexws2 @ illinois.edu.

Thank you again for your help,
Alex