Unable to identify image files matching regular expression

[Sergio Sarnataro]

Hello,

I plan to use MERlin for some experiment that we will perform in the lab in the very next future, so I was trying to use it on some published data in order to familiarize with the tool.

The image I wanted to use as a "toy" data is "aligned_images0.tif" from here:
https://download.brainimagelibrary.org/cf/1c/cf1c1a431ef8d021

The name I gave to this file in my local folder is "aligned_images_0.tif" and it is in the folder: /DATA_HOME/experiment1

When I run this command:
merlin -a test_decode_and_segment.json -m microscope.json -o dataorganization.csv -c codebook.fasta -n 1 experiment1

I get the following error:
MERlin - the MERFISH decoding pipeline
Traceback (most recent call last):
File "/home/ssarnataro/MERlin/merlin/data/dataorganization.py", line 272, in _map_image_files
self.fileMap = self._dataSet.load_dataframe_from_csv('filemap')
File "/home/ssarnataro/MERlin/merlin/core/dataset.py", line 361, in load_dataframe_from_csv
with open(savePath, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ssarnataro/Work/data/MERFISH_Atlas_Zhang_et_al/MERlin/ANALYSIS_HOME//experiment1/filemap.csv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ssarnataro/miniconda3/envs/merlin_env/bin/merlin", line 33, in
sys.exit(load_entry_point('merlin', 'console_scripts', 'merlin')())
File "/home/ssarnataro/MERlin/merlin/merlin.py", line 111, in merlin
analysisHome=_clean_string_arg(args.analysis_home)
File "/home/ssarnataro/MERlin/merlin/core/dataset.py", line 1008, in init
self, dataOrganizationName)
File "/home/ssarnataro/MERlin/merlin/data/dataorganization.py", line 71, in init
self._map_image_files()
File "/home/ssarnataro/MERlin/merlin/data/dataorganization.py", line 309, in map_image_files
currentType))
merlin.core.dataset.DataFormatException: Unable to identify image files matching regular expression (?P[\w|-]+)(?P[0-9]+).tif for image type aligned_images_0.

It is like my regular expression is not correct.

Note: I do not have any fiducial image, the images are already aligned.

I attach the dataorganization.csv file that I created for that.
dataorganization.zip

Do you have any idea on what is wrong?

Thank you very much

[Hazen Babcock]

I think you need to change a few things in your dataorganization file:

• aligned_images_0 -> aligned_images, the "0" is the fov so if you don't want to include it in imageType.
• the imagingRound column should just be 0 as you combined all the imaging rounds into a single file.
• fiducialImageType should also be aligned_images, not aligned_images_0.

[Sergio Sarnataro]

Thanks for your feedback.

Unfortunately, it still does not work

[Hazen Babcock]

Maybe remove the underscore? "alignedImages" instead of "aligned_images", the extra underscore might be confusing the regex?

[Sergio Sarnataro]

The fact is that, if I compare this regular expression: (?P[\w|-]+)_(?P[0-9]+)
with this name: aligned_images_0
I get full compatibility (see image attached):

So, what I expect, independently of the goodness of my data, files, table and so on, the tool should find it.

Am I wrong?

[Hazen Babcock]

Well, IDK.

You can add some print statements to the MERlin code (merlin/data/dataorganization.py) to try and figure out what the issue is.

                matchingFiles = False
                for currentFile in fileNames:
                    matchedName = matchRE.match(os.path.split(currentFile)[-1])
                    if matchedName is not None:
                        transformedName = matchedName.groupdict()
                        if transformedName['imageType'] == currentType:
                            if 'imagingRound' not in transformedName:
                                transformedName['imagingRound'] = -1
                            transformedName['imagePath'] = currentFile
                            matchingFiles = True
                            fileData.append(transformedName)

For example:
print(matchedName, currentFile, currentType)

My guess is that it is failing because currentType is "aligned_images_0" and transformedName['imageType'] is "aligned_images".