Potential TensorFlow issue
andreped opened this issue · comments
The initial issue was observed for the GSI-RADS sofware which Raidionics is derived from (GSI-RADS, Issue #2).
A user tried to install the latest binary release (v1.2.0) and run diagnosis on a local MRI.
She got the error:
Failed to load the native TensorFlow runtime
In addition, for the Raidionics software, the following ImportError was observed:
ImportError: No module named '_pywrap_tensorflow_internal'
This is likely due to a hidden import issue, which can be solved by adding a hook, but I see that others have tried something similar with no success (for TensorFlow).
Potential fix in recent commit 0773a3b.
See Actions for corresponding binary release(s) stored as Artifact (NOTE: Need to be logged in to both see and download the Artifact "Package", which contains the installers for all three operating systems).
Good afternoon. I'm a new user of both software and yes I got this error message: Failed to load the native TensorFlow runtime. I'm using a windows 10 operative system on a normal HP laptop.
@Ricatu12 I remember you contacted me on e-mail.
Hopefully, you will be able to test the software on a different computer, such that you are at least able to run your experiments.
Until then, I will be trying to fix this bug relevant for the machine you tested. However, it is challenging as I am unable to reproduce this bug. Will let you know if I have a new version for you to test.
Found the problem!
Your CPU (Celeron(R) N4020 CPU @ 1.10GHz) does not support AVX. If you go here and scroll down, you will see this:
This issue was talked about in this thread:
pyinstaller/pyinstaller#3754 (comment)
Solution: What we need to do is to use a TensorFlow version that is built without AVX support. Not sure how that will affect runtime and whatnot, but I could test this.
I will make an attempt and let you know when a new nightly release is ready for testing.
The reason why you got this issue and why most don't is because most modern CPUs come with AVX support. However, some older CPUs as well as low-end CPUs don't support AVX.
Only way to get AVX support is to use a machine with a CPU that has AVX support.
However, I have attempted to fix this in Raidionics, by swapping the prebuilt TensorFlow wheel, which is built with AVX support by default, with a different wheel built with SSE (which more CPUs should support - at least your's).
I have pushed a potential fix for this, quite the quick fix, just for you to test: ab6aa6d
You can download the new nightly release from here (click on "Package"):
https://github.com/dbouget/Raidionics/actions/runs/2265342438
Please, let me know how it goes. If you get the same issue, I think I know what to do.
Then the fix I did worked.
What you are seeing here is not really an error, but a warning. Should still run.
I assume the program just crashes, without any prompt?
When you run the software, you should be given some logging info. Stating where it is in the process. Can you tell me where it crashed?
Alternatively we could do a zoom meeting where you show me your screen, to debug this further.
OK, but then the software works as it should. Great!
The *.nii.gz files are NIFTI files (segmentation volumes) that can be opened in 3D Slicer or similar. Are you familiar with 3D Slicer? There should also be a report generated that is stored on disk.
When you run diagnosis, it creates a timestamped directory and saved the result in where you chose the output directory to be, for instance in my case the results are stored in E:\Downloads\05052022_131023 given that I set the output directory to E:\Downloads.
In this directory, when I run diagnosis (tested it for a single patient), it looks like this:
But you did not get a report.txt file? Try running diagnosis in single-mode and report what you get.
I think I know the issue, and it is not easy to solve on our end.
Essentially, Raidionics likely depends on some libraries that are built with AVX support. I assumed it was only TensorFlow, but that was rather naive. Nothing else is returned because the algorithm fails at some place. The simplest way to debug this is to run Raidionics from the terminal. That way you will get prompted what went wrong.
In the future we will add a more suitable logging solution which we will prompt what went wrong. Right now I believe nothing i prompted.
But yeah, seems like we are unable to support CPUs that don't have AVX support at the moment.
I would suggest you to run your experiments on a different machine. Remember to use the latest release then (and not the nightly). Release is available from here:
https://github.com/dbouget/Raidionics/releases
Closing for now.
@Ricatu12 were you successful in using Raidionics and doing your study on a separate computer?
Hi! Unfortunately it's not working on all the laptops I have. I put in stand by Il mar 11 ott 2022, 09:39 André Pedersen @.***> ha scritto:
…
@Ricatu12 Oh, that's sad to hear. Is it only segmentation you need? You don't need the reporting? Because we might have a version which can perform segmentation of M1 Macs very soon. At least it works on our initial trials.
No I need the segmentations for my study.
@Ricatu12 OK, great! Because we already have that working on our end - also on M1 chip. The TensorFlow issue was solved by using a different inference engine which is compatible on M1 chips.
We just need to build and test that the software works on M1, which we cannot do until we have a machine with M1 chip. Might get one tomorrow.
If that works, I will let you know, and I will give you the link to download a test version for you to try. Then hopefully it all works out :]
@Ricatu12 We just bought and setup a Mac Mini with M1 chip. Hence, we should be able to build and test if Raidionics builds and runs well on M1 tomorrow. If all goes well, we will have a version for you to test! Fingers crossed :]
Hi @Ricatu12,
We have a version for you to try (can be downloaded here). On our M1 machine we are now able to perform segmentation.
I would recommend you to uninstall the previous version, and manually delete the .raidionics folder located inside your home path (something like /Users/<user_account>/.raidionics)
Haven't had the time to make any additional tests, so some GUI parts might be clunky, and the reporting does not work at all.
The fastest way to see if it works is to do the following when starting Raidionics:
- Click File > Download test data, and select where to save the test data.
- Click Single patient
- Click Import patient (in the bottom-left corner), then Other data type (*.nii), then select the two files inside ApprovedExample/Raw (located where you chose to save the test data during the first step).
- For the Case27-T1 MRI Series (in the right-hand panel), select T1-CE as Sequence type, then save the patient by clicking the floppy disk icon (left-hand side panel, to the right side of the automatically generated patient name).
- Click the Actions tab in the right-hand panel, then click Preoperative segmentation and choose the tumor type.
- When the process has finished, click the
Datatab (right-hand side panel), select the T1 MRI, and then click on Annotations, where you should see two elements (one brain segmentation and one tumor segmentation).
Let me know if it works (or not), or if you need any extra help.
@Ricatu12 Just updating you to let you know that we have made a pre-release now, which should work on your macbook with M1 chip:
https://github.com/dbouget/Raidionics/releases/tag/v1.2.0-alpha
See dbouget's reply for how to get started using it. Then let us know how it goes :]