Giters

MaybeShewill-CV / attentive-gan-derainnet

Unofficial tensorflow implemention of "Attentive Generative Adversarial Network for Raindrop Removal from A Single Image (CVPR 2018) " model https://maybeshewill-cv.github.io/attentive-gan-derainnet/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

got error when run train_model.py

jifenghu opened this issue · comments

commented

Hi,
First, I want to say thank you for share this model, and here is the issue I met.
my device is an i7-7700HQ CPU and a 1050ti GPU, and I use the data sample you prepared to run the train_model .

And the error is like:

(tensorflow) D:\attentive-gan-derainnet-master>python tools/train_model.py --dataset_dir D:\attentive-gan-derainnet-master\data_provider
D:\software\Anaconda\envs\tensorflow\lib\site-packages\h5py_init_.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
VGG16 Network init complete
2019-03-18 14:34:37.549633: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
I0318 14:34:38.240563 309452 train_model.py:255] Global configuration is as follows:
I0318 14:34:38.402141 309452 train_model.py:256] {'TEST': {'IMG_HEIGHT': 240, 'GPU_MEMORY_FRACTION': 0.8, 'IMG_WIDTH': 360, 'BATCH_SIZE': 1, 'TF_ALLOW_GROWTH': False}, 'TRAIN': {'CROP_IMG_HEIGHT': 240, 'GPU_MEMORY_FRACTION': 0.95, 'IMG_WIDTH': 376, 'IMG_HEIGHT': 256, 'LEARNING_RATE': 0.0002, 'GPU_NUM': 1, 'BATCH_SIZE': 1, 'CPU_MULTI_PROCESS_NUMS': 6, 'CROP_IMG_WIDTH': 360, 'TF_ALLOW_GROWTH': True, 'EPOCHS': 100010}}
I0318 14:34:40.289161 309452 train_model.py:264] Training from scratch
Traceback (most recent call last):
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call
return fn(*args)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun
run_metadata)
**tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[Node: val_IteratorGetNext = IteratorGetNextoutput_shapes=[[1,240,360,3], [1,240,360,3], [1,240,360,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:**

Traceback (most recent call last):
File "tools/train_model.py", line 341, in
train_model(args.dataset_dir, weights_path=args.weights_path)
File "tools/train_model.py", line 294, in train_model
train_psnr, train_summary_op, val_summary_op]
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 877, in run
run_metadata_ptr)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run
run_metadata)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[Node: val_IteratorGetNext = IteratorGetNextoutput_shapes=[[1,240,360,3], [1,240,360,3], [1,240,360,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'val_IteratorGetNext', defined at:
File "tools/train_model.py", line 341, in
train_model(args.dataset_dir, weights_path=args.weights_path)
File "tools/train_model.py", line 117, in train_model
val_input_tensor, val_label_tensor, val_mask_tensor = val_dataset.inputs(CFG.TRAIN.BATCH_SIZE, 1)
File "D:\attentive-gan-derainnet-master\data_provider\data_feed_pipline.py", line 296, in inputs
return iterator.get_next(name='{:s}_IteratorGetNext'.format(self._dataset_flags))
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 410, in get_next
name=name)), self._output_types,
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2107, in iterator_get_next
output_shapes=output_shapes, name=name)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op
op_def=op_def)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): End of sequence
[[Node: val_IteratorGetNext = IteratorGetNextoutput_shapes=[[1,240,360,3], [1,240,360,3], [1,240,360,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

which I don't know how to figure it out, please help if you see that~ thank you very much~

commented

@jifenghu Did you use the GPU or the CPU to train the nwtwork?:)

commented

@MaybeShewill-CV i think i used gpu to train

commented

@jifenghu Devide information shows you use CPU:)

commented

@MaybeShewill-CV oh，I see but how can I run it on GPU? I used the training code you provided, should i check my tensorflow version or add sth else?

commented

@jifenghu You may google or stack overflow to solver it:)

commented

@MaybeShewill-CV Sorry to bother you again but here is another error like
Traceback (most recent call last):
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[{{node val_IteratorGetNext}}]]

and for some reson I can't google or get on stack overflow

commented

@jifenghu How do you start the training process and what command do you use? Have you already generated the tfrecords?

commented

@jifenghu I have tested it just now noting works unusual. Please read README carefully to see if there are someting you've missed:)
Screenshot from 2019-03-18 22-07-09

commented

@MaybeShewill-CV first I run the commend like this :
python data_provider/data_feed_pipline.py --dataset_dir D:\attentive-gan-derainnet-master\data_provider --tfrecords_dir D:\attentive-gan-derainnet-master\data_provider

and the result is:
python data_provider/data_feed_pipline.py --dataset_dir D:\attentive-gan-derainnet-master\data_provider --tfrecords_dir D:\attentive-gan-derainnet-master\data_provider
I0319 08:01:25.944896 293632 data_feed_pipline.py:110] Generating training example tfrecords
I0319 08:01:25.949981 293632 data_feed_pipline.py:122] Generate training example tfrecords complete
I0319 08:01:25.949981 293632 data_feed_pipline.py:125] Generating validation example tfrecords
I0319 08:01:25.950978 293632 data_feed_pipline.py:137] Generate validation example tfrecords complete
I0319 08:01:25.951421 293632 data_feed_pipline.py:140] Generating testing example tfrecords
I0319 08:01:25.951421 293632 data_feed_pipline.py:152] Generate testing example tfrecords complete

and the run python tools/train_model.py --dataset_dir D:\attentive-gan-derainnet-master\data_provider

and the process is correct until :
I0319 07:57:19.940103 123684 train_model.py:264] Training from scratch
Traceback (most recent call last):
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\software\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[{{node train_IteratorGetNext}} = IteratorGetNextoutput_shapes=[[1,240,360,3], [1,240,360,3], [1,240,360,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[{{node train_IteratorGetNext/_34}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_132_train_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

also I have reinstalled my tensorflow-gpu and import well

commented

image
Hello, I have encountered the same problem as jifenghu; when I made the tfrecords file, I only used the three images in the trianing_data_example folder. Is the reason for the error? If jifenghu solves this problem, can you please tell me the solution?