Scaling generated 3D-Structures
TheRealMephisto opened this issue · comments
Hi @LukasMosser,
I was wondering if it is possible to scale the size of the generated 3D-Structures.
The answer should clearly be "Yes!", but I'm not sure how to do it.
Can you tell me which parameters passed to the generator.py are relevant for scaling?
Thanks in advance!
Hi!
Use the --imsize flag followed by an integer. 1 will give you an image the size of your training images. Larger integers will give you larger outputs. This integer corresponds to the spatial scale of the latent vector.
Hope this answers your question!
Hi Lukas,
thanks for your nice project. I tried your code with my own 3D porous material. I was able to run through the simulation. I found that reasonable microstrucuture can be generated by setting --imsize flag with value of 1. However, when i tried for a larger number, the microstructure became larger but appeared to be strange to me. It seems patterned microstructure (repeated in x,y and z directions). Did i set parameters wrong? Please have a look at the attached images for imsize number 1, 4 and 9.
Regards,
Yan
.
Hi @zilinyan,
I think you are doing the right thing. During training imsize should be set to 1.
I think there may be one of two things happening here:
- Your network has not converged yet, how are you evaluating this at the moment?
- Your training images are not large enough to capture structural correlations of the microstructure. I have seen similar behavior early on when my training image was too small (32^3 compared to 64^3 used in the final version.
Does this help? I personally would also not use the classical formulation of Goodfellow to try this.
There are numerous better algorithms now such as Spectracl Normalisation GAN that have much more stable learning algorithms.
Let me know if this helps
Hi @LukasMosser,
thank you for your advice!
However, there seems to be a limitation, since i get a "CUDA error: out of memory" for --imsize parameters bigger than 21.
Sadly, the generated images are much smaller than the original training file.
Is there an easy way to (re)program the network in order to successively generate microstructures?
Greetings!
Yes, there is a memory limitation.
There's a few ways to get around it possibly.
A) I would try checkpointing in pytorch to see if you can unload parts of the network and intermediate results
B) more ram, generate via cpu or on a 24 gb gpu
C) try out single and half precision conversions of the network. That should reduce the memory consumption. But might also degrade results.
Lukas
Hi @LukasMosser ,
Thanks for your advice. I enlarged my training images size from 64^3 to 128^3 and extended the training epochs (20000). For the moment, I judged the convergence by checking the loss_D. I simply compared my predicted microstructure with the original one by eyes. I know that a rigorous method is detailed in your paper, but I am not at that step yet.
My concerns are:
-
My Loss_D and Loss_G seem to fluctuate very much (many spikes), even I averaged the Loss_D and G values over all the batches (not sure I understood your code right). In your paper, you have relatively smoother curves. Did you read results directly from training_curves.cvs?
-
The accuracies of the Generators seem to be related to the loss_G curves as shown in the attached figures. I have to check each generator to find a satisfying one. My experience is that the Generators that lead to an abrupt increase of loss_G should not be considered. As you state in your paper, the loss_G does not reflect the quality of the predicted microstructures. How did you decide an optimal Generator?
-
In my case, my training images are binary volumes. During the training, the predicted images are gray images (0-2000 iterations). After that, they become binary images-like? Is this normal? What is more, after 2000 iterations, the small images (imsize=1, fake_TI_.) are similar to original ones but the larger ones (imsize=7, fake_samples_) are strange. Do you verify your model based on the fake_sample_.hdf5 data or the fake_TI_.hdf5?
Greetings
PS: the small ones are slices from the fake_TI_* images, the bigger ones are from fake_samples_*
Hi @zilinyan,
I think you are doing the right thing. During training imsize should be set to 1.
I think there may be one of two things happening here:
- Your network has not converged yet, how are you evaluating this at the moment?
- Your training images are not large enough to capture structural correlations of the microstructure. I have seen similar behavior early on when my training image was too small (32^3 compared to 64^3 used in the final version.
Does this help? I personally would also not use the classical formulation of Goodfellow to try this.
There are numerous better algorithms now such as Spectracl Normalisation GAN that have much more stable learning algorithms.
Let me know if this helps
Hi @zilinyan
Thank you for sharing your findings. This very much reflects my own experience of training the networks.
Extremely erratic behavior in the loss and the need to check pretty much each iteration for convergence.
The way I searched for suitable generator iterations is the following:
- Store Checkpoints every 10 epochs (if you have enough memory you can store each iteration)
- For each checkpoint, I sampled ~100 images, segmented them using Otsu and computed 2-point correlation functions. At the origin, this includes a check on the porosity of the generated samples.
- If the two-point correlation function was satisfactory I moved on to flow properties. (which are much more time consuming and only worth trying for the very best generator checkpoints)
All of this can definitely be automated and I would recommend doing so.
You are correct in your observation, the resulting images are not perfectly binary, this is due to the continuous nature of the network output. I used Otsu thresholding to stay consistent for each iteration, but you could probably also simply threshold at 0.5 as the histogram of each generated sample should have most of the weight close to zero and close to one.
To clarify based on your Loss vs Image Quality plot -> You are now generating images that do resemble your inputs but for very high generator iterations your generator collapses and the output doesn't make sense anymore? It seems the output at least visually after 500 and 1000 iterations seems reasonable?
I personally would verify my images based on the properties I am interested in e.g. porosity, 2-point correlation, morphological properties and then stop training, there's no point in hoping for the network to stabilize, hence also my suggestion to go for a more stable training method such as WGAN-GP, WGAN-LP, or others.
Hi @LukasMosser,
Thanks a lot for your expertise. Your comments are very helpful. Yes, it seems my generator collapses sometimes, I will have a look at the WGAN.
Cheers