No history neither models or generated images saved, we need to save model, history and generated images
Jimut123 opened this issue · comments
(aniket_mask_rcnn3) tamal@tamalmj:/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3$ python3 mask_rcnn3.py
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2021-03-17 19:37:27.586105: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-17 19:37:27.809481: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699850000 Hz
2021-03-17 19:37:27.811202: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eddfc39d70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:37:27.811284: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-03-17 19:37:27.902438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-17 19:37:28.314156: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.314918: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eddfc24fb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:37:28.314951: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2021-03-17 19:37:28.325146: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.326027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:37:28.336059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:37:28.381215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:37:28.437417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:37:28.455969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:37:28.491466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:37:28.547836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:37:28.669696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:37:28.670119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.671616: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.672821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:37:28.689438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:37:28.698546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:37:28.698633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:37:28.698669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:37:28.706308: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.708503: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:28.710162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7128 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2139: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2239: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:553: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py:202: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:600: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:203: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-03-17 19:37:34.833779: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.834105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:37:34.834176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:37:34.834194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:37:34.834209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:37:34.834224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:37:34.834240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:37:34.834255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:37:34.834270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:37:34.834329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.834629: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.834895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:37:34.834928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:37:34.834934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:37:34.834939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:37:34.835014: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.835326: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:37:34.835599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7128 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:207: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:216: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:223: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
./dataset/basophil/train/images
./dataset/eosinophil/train/images
./dataset/erythroblast/train/images
./dataset/ig/train/images
./dataset/lymphocyte/train/images
./dataset/monocyte/train/images
./dataset/neutrophil/train/images
./dataset/platelet/train/images
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
Train network heads
Starting at epoch 0. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1937/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1122: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1125: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
Epoch 1/10
2021-03-17 19:38:14.326052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:38:15.369728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:38:20.659674: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2021-03-17 19:38:24.072361: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.072436: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.139314: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.141612: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.548642: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.549016: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.947337: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:24.947511: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:25.020786: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:38:25.020836: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
1/50 [..............................] - ETA: 25:34 - loss: 17.9140 - rpn_class_loss: 1.3314 - rpn_bbox_loss: 3.6698 - mrcnn_class_loss: 12.9128 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+0 2/50 [>.............................] - ETA: 12:41 - loss: 16.5364 - rpn_class_loss: 1.2819 - rpn_bbox_loss: 3.6159 - mrcnn_class_loss: 11.6386 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+0 3/50 [>.............................] - ETA: 8:24 - loss: 14.5689 - rpn_class_loss: 1.2034 - rpn_bbox_loss: 3.5276 - mrcnn_class_loss: 9.8379 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+00 50/50 [==============================] - 57s 1s/step - loss: 3.6597 - rpn_class_loss: 0.1321 - rpn_bbox_loss: 1.2943 - mrcnn_class_loss: 0.8699 - mrcnn_bbox_loss: 0.7085 - mrcnn_mask_loss: 0.6548 - val_loss: 4.8457 - val_rpn_class_loss: 0.0292 - val_rpn_bbox_loss: 3.1592 - val_mrcnn_class_loss: 0.1857 - val_mrcnn_bbox_loss: 0.7262 - val_mrcnn_mask_loss: 0.7454
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1265: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.
Epoch 2/10
50/50 [==============================] - 24s 482ms/step - loss: 2.1932 - rpn_class_loss: 0.0168 - rpn_bbox_loss: 0.5727 - mrcnn_class_loss: 0.2502 - mrcnn_bbox_loss: 0.6622 - mrcnn_mask_loss: 0.6913 - val_loss: 3.7600 - val_rpn_class_loss: 0.0411 - val_rpn_bbox_loss: 1.7578 - val_mrcnn_class_loss: 0.2487 - val_mrcnn_bbox_loss: 1.0156 - val_mrcnn_mask_loss: 0.6968
Epoch 3/10
50/50 [==============================] - 24s 484ms/step - loss: 2.1187 - rpn_class_loss: 0.0280 - rpn_bbox_loss: 0.7429 - mrcnn_class_loss: 0.1732 - mrcnn_bbox_loss: 0.4814 - mrcnn_mask_loss: 0.6932 - val_loss: 3.7112 - val_rpn_class_loss: 0.0285 - val_rpn_bbox_loss: 1.8525 - val_mrcnn_class_loss: 0.4423 - val_mrcnn_bbox_loss: 0.7077 - val_mrcnn_mask_loss: 0.6802
Epoch 4/10
50/50 [==============================] - 24s 481ms/step - loss: 2.0721 - rpn_class_loss: 0.0153 - rpn_bbox_loss: 0.5532 - mrcnn_class_loss: 0.2496 - mrcnn_bbox_loss: 0.6140 - mrcnn_mask_loss: 0.6401 - val_loss: 2.6142 - val_rpn_class_loss: 0.0177 - val_rpn_bbox_loss: 0.7381 - val_mrcnn_class_loss: 0.1102 - val_mrcnn_bbox_loss: 1.0741 - val_mrcnn_mask_loss: 0.6741
Epoch 5/10
50/50 [==============================] - 25s 491ms/step - loss: 1.7990 - rpn_class_loss: 0.0114 - rpn_bbox_loss: 0.2935 - mrcnn_class_loss: 0.3418 - mrcnn_bbox_loss: 0.4821 - mrcnn_mask_loss: 0.6702 - val_loss: 2.0388 - val_rpn_class_loss: 0.0165 - val_rpn_bbox_loss: 0.6126 - val_mrcnn_class_loss: 0.2847 - val_mrcnn_bbox_loss: 0.4592 - val_mrcnn_mask_loss: 0.6658
Epoch 6/10
50/50 [==============================] - 25s 490ms/step - loss: 1.7345 - rpn_class_loss: 0.0104 - rpn_bbox_loss: 0.3701 - mrcnn_class_loss: 0.2780 - mrcnn_bbox_loss: 0.4213 - mrcnn_mask_loss: 0.6547 - val_loss: 3.3186 - val_rpn_class_loss: 0.0350 - val_rpn_bbox_loss: 1.3366 - val_mrcnn_class_loss: 0.6055 - val_mrcnn_bbox_loss: 0.6839 - val_mrcnn_mask_loss: 0.6576
Epoch 7/10
50/50 [==============================] - 24s 483ms/step - loss: 1.6058 - rpn_class_loss: 0.0093 - rpn_bbox_loss: 0.3452 - mrcnn_class_loss: 0.2652 - mrcnn_bbox_loss: 0.3348 - mrcnn_mask_loss: 0.6513 - val_loss: 2.0079 - val_rpn_class_loss: 0.0183 - val_rpn_bbox_loss: 0.7288 - val_mrcnn_class_loss: 0.2194 - val_mrcnn_bbox_loss: 0.4197 - val_mrcnn_mask_loss: 0.6217
Epoch 8/10
50/50 [==============================] - 24s 481ms/step - loss: 1.6592 - rpn_class_loss: 0.0183 - rpn_bbox_loss: 0.4256 - mrcnn_class_loss: 0.2830 - mrcnn_bbox_loss: 0.2853 - mrcnn_mask_loss: 0.6471 - val_loss: 2.1766 - val_rpn_class_loss: 0.0128 - val_rpn_bbox_loss: 0.7118 - val_mrcnn_class_loss: 0.1009 - val_mrcnn_bbox_loss: 0.6933 - val_mrcnn_mask_loss: 0.6577
Epoch 9/10
50/50 [==============================] - 24s 483ms/step - loss: 1.5084 - rpn_class_loss: 0.0051 - rpn_bbox_loss: 0.3078 - mrcnn_class_loss: 0.2647 - mrcnn_bbox_loss: 0.3115 - mrcnn_mask_loss: 0.6194 - val_loss: 2.2111 - val_rpn_class_loss: 0.0076 - val_rpn_bbox_loss: 0.6412 - val_mrcnn_class_loss: 0.4763 - val_mrcnn_bbox_loss: 0.5034 - val_mrcnn_mask_loss: 0.5826
Epoch 10/10
50/50 [==============================] - 24s 483ms/step - loss: 1.6972 - rpn_class_loss: 0.0210 - rpn_bbox_loss: 0.4783 - mrcnn_class_loss: 0.2437 - mrcnn_bbox_loss: 0.3190 - mrcnn_mask_loss: 0.6352 - val_loss: 1.5141 - val_rpn_class_loss: 0.0080 - val_rpn_bbox_loss: 0.4483 - val_mrcnn_class_loss: 0.1314 - val_mrcnn_bbox_loss: 0.3691 - val_mrcnn_mask_loss: 0.5572
Traceback (most recent call last):
File "mask_rcnn3.py", line 282, in <module>
train(model, "./dataset")
File "mask_rcnn3.py", line 257, in train
hist_df_nh = pd.DataFrame(history_network_heads.history)
AttributeError: 'NoneType' object has no attribute 'history'
(aniket_mask_rcnn3) tamal@tamalmj:/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3$ python3 mask_rcnn3.py
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2021-03-17 19:49:43.913311: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-17 19:49:43.941271: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699850000 Hz
2021-03-17 19:49:43.941646: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5651829d9fc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:49:43.941669: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-03-17 19:49:43.973444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-17 19:49:44.133701: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.134086: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5651805c5c00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-17 19:49:44.134103: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2021-03-17 19:49:44.134286: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.134588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:49:44.168549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:49:44.197237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:49:44.240665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:49:44.259555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:49:44.278373: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:49:44.289792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:49:44.316497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:49:44.316911: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.319354: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.321147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:49:44.321412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:49:44.323283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:49:44.323344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:49:44.323373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:49:44.323747: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.325237: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:44.326523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7138 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2139: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2239: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:553: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py:202: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:600: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:203: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-03-17 19:49:50.674465: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.674789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:02:00.0
2021-03-17 19:49:50.674864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-03-17 19:49:50.674884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:49:50.674902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-03-17 19:49:50.674921: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-03-17 19:49:50.674939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-03-17 19:49:50.674957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-03-17 19:49:50.674974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:49:50.675034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.675333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.675598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-03-17 19:49:50.675631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-17 19:49:50.675637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-03-17 19:49:50.675642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2021-03-17 19:49:50.675706: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.676012: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-17 19:49:50.676299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7138 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:02:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:207: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:216: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:223: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
./dataset/basophil/train/images
./dataset/eosinophil/train/images
./dataset/erythroblast/train/images
./dataset/ig/train/images
./dataset/lymphocyte/train/images
./dataset/monocyte/train/images
./dataset/neutrophil/train/images
./dataset/platelet/train/images
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
Train network heads
Starting at epoch 0. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1949/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1122: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1125: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
Epoch 1/4
2021-03-17 19:50:18.633214: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-03-17 19:50:18.764498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-17 19:50:19.431184: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2021-03-17 19:50:22.532819: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.532862: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.533247: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.533287: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.608080: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.608192: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.04GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.612358: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:22.612779: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:23.048745: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:23.048780: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.41GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-03-17 19:50:23.400585: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 993.83M (1042110720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-03-17 19:50:23.401123: I tensorflow/stream_executor/cuda/cuda_driver.cc:831] failed to allocate 894.45M (937899776 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
50/50 [==============================] - 50s 997ms/step - loss: 2.8902 - rpn_class_loss: 0.3050 - rpn_bbox_loss: 0.9136 - mrcnn_class_loss: 0.3932 - mrcnn_bbox_loss: 0.7135 - mrcnn_mask_loss: 0.5649 - val_loss: 4.3391 - val_rpn_class_loss: 0.0424 - val_rpn_bbox_loss: 1.3799 - val_mrcnn_class_loss: 0.6488 - val_mrcnn_bbox_loss: 1.5979 - val_mrcnn_mask_loss: 0.6701
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/callbacks.py:1265: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.
Epoch 2/4
50/50 [==============================] - 24s 476ms/step - loss: 2.5034 - rpn_class_loss: 0.0294 - rpn_bbox_loss: 0.7938 - mrcnn_class_loss: 0.3023 - mrcnn_bbox_loss: 0.6994 - mrcnn_mask_loss: 0.6785 - val_loss: 3.2076 - val_rpn_class_loss: 0.0212 - val_rpn_bbox_loss: 1.0725 - val_mrcnn_class_loss: 0.3652 - val_mrcnn_bbox_loss: 1.0609 - val_mrcnn_mask_loss: 0.6879
Epoch 3/4
50/50 [==============================] - 24s 478ms/step - loss: 2.0304 - rpn_class_loss: 0.0228 - rpn_bbox_loss: 0.4881 - mrcnn_class_loss: 0.2915 - mrcnn_bbox_loss: 0.5484 - mrcnn_mask_loss: 0.6796 - val_loss: 4.0582 - val_rpn_class_loss: 0.0321 - val_rpn_bbox_loss: 2.6589 - val_mrcnn_class_loss: 0.1285 - val_mrcnn_bbox_loss: 0.5701 - val_mrcnn_mask_loss: 0.6687
Epoch 4/4
50/50 [==============================] - 24s 479ms/step - loss: 2.1744 - rpn_class_loss: 0.0287 - rpn_bbox_loss: 0.7892 - mrcnn_class_loss: 0.2251 - mrcnn_bbox_loss: 0.4589 - mrcnn_mask_loss: 0.6725 - val_loss: 2.6046 - val_rpn_class_loss: 0.0262 - val_rpn_bbox_loss: 1.4513 - val_mrcnn_class_loss: 0.0517 - val_mrcnn_bbox_loss: 0.4163 - val_mrcnn_mask_loss: 0.6590
Train all layers
Starting at epoch 4. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1949/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
conv1 (Conv2D)
bn_conv1 (BatchNorm)
res2a_branch2a (Conv2D)
bn2a_branch2a (BatchNorm)
res2a_branch2b (Conv2D)
bn2a_branch2b (BatchNorm)
res2a_branch2c (Conv2D)
res2a_branch1 (Conv2D)
bn2a_branch2c (BatchNorm)
bn2a_branch1 (BatchNorm)
res2b_branch2a (Conv2D)
bn2b_branch2a (BatchNorm)
res2b_branch2b (Conv2D)
bn2b_branch2b (BatchNorm)
res2b_branch2c (Conv2D)
bn2b_branch2c (BatchNorm)
res2c_branch2a (Conv2D)
bn2c_branch2a (BatchNorm)
res2c_branch2b (Conv2D)
bn2c_branch2b (BatchNorm)
res2c_branch2c (Conv2D)
bn2c_branch2c (BatchNorm)
res3a_branch2a (Conv2D)
bn3a_branch2a (BatchNorm)
res3a_branch2b (Conv2D)
bn3a_branch2b (BatchNorm)
res3a_branch2c (Conv2D)
res3a_branch1 (Conv2D)
bn3a_branch2c (BatchNorm)
bn3a_branch1 (BatchNorm)
res3b_branch2a (Conv2D)
bn3b_branch2a (BatchNorm)
res3b_branch2b (Conv2D)
bn3b_branch2b (BatchNorm)
res3b_branch2c (Conv2D)
bn3b_branch2c (BatchNorm)
res3c_branch2a (Conv2D)
bn3c_branch2a (BatchNorm)
res3c_branch2b (Conv2D)
bn3c_branch2b (BatchNorm)
res3c_branch2c (Conv2D)
bn3c_branch2c (BatchNorm)
res3d_branch2a (Conv2D)
bn3d_branch2a (BatchNorm)
res3d_branch2b (Conv2D)
bn3d_branch2b (BatchNorm)
res3d_branch2c (Conv2D)
bn3d_branch2c (BatchNorm)
res4a_branch2a (Conv2D)
bn4a_branch2a (BatchNorm)
res4a_branch2b (Conv2D)
bn4a_branch2b (BatchNorm)
res4a_branch2c (Conv2D)
res4a_branch1 (Conv2D)
bn4a_branch2c (BatchNorm)
bn4a_branch1 (BatchNorm)
res4b_branch2a (Conv2D)
bn4b_branch2a (BatchNorm)
res4b_branch2b (Conv2D)
bn4b_branch2b (BatchNorm)
res4b_branch2c (Conv2D)
bn4b_branch2c (BatchNorm)
res4c_branch2a (Conv2D)
bn4c_branch2a (BatchNorm)
res4c_branch2b (Conv2D)
bn4c_branch2b (BatchNorm)
res4c_branch2c (Conv2D)
bn4c_branch2c (BatchNorm)
res4d_branch2a (Conv2D)
bn4d_branch2a (BatchNorm)
res4d_branch2b (Conv2D)
bn4d_branch2b (BatchNorm)
res4d_branch2c (Conv2D)
bn4d_branch2c (BatchNorm)
res4e_branch2a (Conv2D)
bn4e_branch2a (BatchNorm)
res4e_branch2b (Conv2D)
bn4e_branch2b (BatchNorm)
res4e_branch2c (Conv2D)
bn4e_branch2c (BatchNorm)
res4f_branch2a (Conv2D)
bn4f_branch2a (BatchNorm)
res4f_branch2b (Conv2D)
bn4f_branch2b (BatchNorm)
res4f_branch2c (Conv2D)
bn4f_branch2c (BatchNorm)
res4g_branch2a (Conv2D)
bn4g_branch2a (BatchNorm)
res4g_branch2b (Conv2D)
bn4g_branch2b (BatchNorm)
res4g_branch2c (Conv2D)
bn4g_branch2c (BatchNorm)
res4h_branch2a (Conv2D)
bn4h_branch2a (BatchNorm)
res4h_branch2b (Conv2D)
bn4h_branch2b (BatchNorm)
res4h_branch2c (Conv2D)
bn4h_branch2c (BatchNorm)
res4i_branch2a (Conv2D)
bn4i_branch2a (BatchNorm)
res4i_branch2b (Conv2D)
bn4i_branch2b (BatchNorm)
res4i_branch2c (Conv2D)
bn4i_branch2c (BatchNorm)
res4j_branch2a (Conv2D)
bn4j_branch2a (BatchNorm)
res4j_branch2b (Conv2D)
bn4j_branch2b (BatchNorm)
res4j_branch2c (Conv2D)
bn4j_branch2c (BatchNorm)
res4k_branch2a (Conv2D)
bn4k_branch2a (BatchNorm)
res4k_branch2b (Conv2D)
bn4k_branch2b (BatchNorm)
res4k_branch2c (Conv2D)
bn4k_branch2c (BatchNorm)
res4l_branch2a (Conv2D)
bn4l_branch2a (BatchNorm)
res4l_branch2b (Conv2D)
bn4l_branch2b (BatchNorm)
res4l_branch2c (Conv2D)
bn4l_branch2c (BatchNorm)
res4m_branch2a (Conv2D)
bn4m_branch2a (BatchNorm)
res4m_branch2b (Conv2D)
bn4m_branch2b (BatchNorm)
res4m_branch2c (Conv2D)
bn4m_branch2c (BatchNorm)
res4n_branch2a (Conv2D)
bn4n_branch2a (BatchNorm)
res4n_branch2b (Conv2D)
bn4n_branch2b (BatchNorm)
res4n_branch2c (Conv2D)
bn4n_branch2c (BatchNorm)
res4o_branch2a (Conv2D)
bn4o_branch2a (BatchNorm)
res4o_branch2b (Conv2D)
bn4o_branch2b (BatchNorm)
res4o_branch2c (Conv2D)
bn4o_branch2c (BatchNorm)
res4p_branch2a (Conv2D)
bn4p_branch2a (BatchNorm)
res4p_branch2b (Conv2D)
bn4p_branch2b (BatchNorm)
res4p_branch2c (Conv2D)
bn4p_branch2c (BatchNorm)
res4q_branch2a (Conv2D)
bn4q_branch2a (BatchNorm)
res4q_branch2b (Conv2D)
bn4q_branch2b (BatchNorm)
res4q_branch2c (Conv2D)
bn4q_branch2c (BatchNorm)
res4r_branch2a (Conv2D)
bn4r_branch2a (BatchNorm)
res4r_branch2b (Conv2D)
bn4r_branch2b (BatchNorm)
res4r_branch2c (Conv2D)
bn4r_branch2c (BatchNorm)
res4s_branch2a (Conv2D)
bn4s_branch2a (BatchNorm)
res4s_branch2b (Conv2D)
bn4s_branch2b (BatchNorm)
res4s_branch2c (Conv2D)
bn4s_branch2c (BatchNorm)
res4t_branch2a (Conv2D)
bn4t_branch2a (BatchNorm)
res4t_branch2b (Conv2D)
bn4t_branch2b (BatchNorm)
res4t_branch2c (Conv2D)
bn4t_branch2c (BatchNorm)
res4u_branch2a (Conv2D)
bn4u_branch2a (BatchNorm)
res4u_branch2b (Conv2D)
bn4u_branch2b (BatchNorm)
res4u_branch2c (Conv2D)
bn4u_branch2c (BatchNorm)
res4v_branch2a (Conv2D)
bn4v_branch2a (BatchNorm)
res4v_branch2b (Conv2D)
bn4v_branch2b (BatchNorm)
res4v_branch2c (Conv2D)
bn4v_branch2c (BatchNorm)
res4w_branch2a (Conv2D)
bn4w_branch2a (BatchNorm)
res4w_branch2b (Conv2D)
bn4w_branch2b (BatchNorm)
res4w_branch2c (Conv2D)
bn4w_branch2c (BatchNorm)
res5a_branch2a (Conv2D)
bn5a_branch2a (BatchNorm)
res5a_branch2b (Conv2D)
bn5a_branch2b (BatchNorm)
res5a_branch2c (Conv2D)
res5a_branch1 (Conv2D)
bn5a_branch2c (BatchNorm)
bn5a_branch1 (BatchNorm)
res5b_branch2a (Conv2D)
bn5b_branch2a (BatchNorm)
res5b_branch2b (Conv2D)
bn5b_branch2b (BatchNorm)
res5b_branch2c (Conv2D)
bn5b_branch2c (BatchNorm)
res5c_branch2a (Conv2D)
bn5c_branch2a (BatchNorm)
res5c_branch2b (Conv2D)
bn5c_branch2b (BatchNorm)
res5c_branch2c (Conv2D)
bn5c_branch2c (BatchNorm)
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/framework/indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/training_generator.py:49: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the `keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:758: The name tf.sets.set_intersection is deprecated. Please use tf.sets.intersection instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:760: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:772: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Re-starting from epoch 4
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
original_image shape: (1024, 1024, 3) min: 0.00000 max: 255.00000 uint8
image_meta shape: (21,) min: 1.00000 max: 1024.00000 float64
gt_class_id shape: (1,) min: 1.00000 max: 1.00000 int32
gt_bbox shape: (1, 4) min: 357.00000 max: 673.00000 int32
gt_mask shape: (1024, 1024, 1) min: 0.00000 max: 1.00000 bool
Processing 1 images
image shape: (1024, 1024, 3) min: 0.00000 max: 255.00000 uint8
molded_images shape: (1, 1024, 1024, 3) min: -123.70000 max: 137.20000 float64
image_metas shape: (1, 21) min: 0.00000 max: 1024.00000 int64
anchors shape: (1, 261888, 4) min: -0.35390 max: 1.29134 float32
True: BA
Prediction
['EO'] ===> 0.9152153
['EO'] ===> 0.6645939
['EO'] ===> 0.64412767
['EO'] ===> 0.63139707
['EO'] ===> 0.57328814
Aniket you need to work in this tomorrow, hope the issue gets resolved by then, uploaded the code...
Check Mask_RCNN/logs directory
Just some models.h5
No history. Also need to save some detected images.
Sent from Yahoo Mail on Android
On Wed, 17 Mar 2021 at 8:05 pm, Aniket ***@***.***> wrote:
Check Mask_RCNN/logs directory
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Check this to save images matterport/Mask_RCNN#134
What is inside the logs directory? Give a list
Room e chole esechi... Kalke dekhish...
Amake slides banate hbe porsur jonne.
Sent from Yahoo Mail on Android
On Wed, 17 Mar 2021 at 8:14 pm, Aniket ***@***.***> wrote:
What is inside the logs directory? Give a list
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Accha