run demo_llff.sh got stuck.
YiChenCityU opened this issue · comments
`Configuring optimizers
| Name | Type | Params
0 | loss | MSELoss | 0
1 | render_fn | RenderLightfield | 14.9 M
2 | reg_tensorf | TensoRF | 0
14.9 M Trainable params
0 Non-trainable params
14.9 M Total params
59.567 Total estimated model params size (MB)
Restored all states from the checkpoint file at /mnt/externalDisk/NeRF/hyperreel/checkpoints/llff_trex/last.ckpt
Validating
Epoch 4: 0%| | 0/107 [00:00<?, ?it/s]/home/eason/anaconda3/envs/nerf/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py:129: UserWarning: You're resuming from a checkpoint that ended before the epoch ended. This can cause unreliable results if further training is done. Consider using an end-of-epoch checkpoint or enabling fault-tolerant training: https://pytorch-lightning.readthedocs.io/en/stable/advanced/fault_tolerant_training.html
rank_zero_warn(
Epoch 5: 93%|████████████████Loading image 4██████████████████████████████████████████████████████████████████████████████████ | 100/107 [00:00<00:00, 336.43it/s]
Loading image 5 [00:00, ?it/s]
Loading image 0
Loading image 2
Loading image 1
Loading image 3
Loading image 6
Validation DataLoader 0: 0%| | 0/7 [00:00<?, ?it/s]
`
Hi, When I trained the llff dataset, and run demo_llff.sh. The program got stuck like this? Do you know the reason?
Thanks very much.
Let me look into this on my end...
In the meantime, can I ask: are you running the demo script locally, or remotely (over ssh)? The script should launch a GUI, which you'll only be able to see if you run the script locally from your workstation.
This is the result I got after training the model for 20 epochs, and running the demo_llff.sh script
IMG_0919.MOV
Please let me know if you're still having difficulty running the demo, and I'd be happy to try to assist.
Hi, Breuckelen. I found the reason why the GUI cannot start. I have to recreate the conda env from beginning. Thanks very much.
What do you mean by recreating the conda environment from scratch?