MaskRCNN3
Jimut123 opened this issue · comments
2021-03-17 17:56:30.644710: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/tamal/.mujoco/mujoco200/bin:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/home/tamal/.mujoco/mujoco200/bin
2021-03-17 17:56:30.644736: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using TensorFlow backend.
Traceback (most recent call last):
File "mask_rcnn3.py", line 267, in <module>
model_dir=MODEL_DIR)
File "/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/mrcnn/model.py", line 1837, in __init__
self.keras_model = self.build(mode=mode, config=config)
File "/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/mrcnn/model.py", line 1856, in build
shape=[None, None, config.IMAGE_SHAPE[2]], name="input_image")
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/input_layer.py", line 178, in Input
input_tensor=tensor)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/engine/input_layer.py", line 87, in __init__
name=self.name)
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 541, in placeholder
x = tf.placeholder(dtype, shape=shape, name=name)
AttributeError: module 'tensorflow' has no attribute 'placeholder'
Try Tensorlfow 1.4.0
ERROR: Could not find a version that satisfies the requirement tensorflow==1.4.0 (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 1.15.0rc0, 1.15.0rc1, 1.15.0rc2, 1.15.0rc3, 1.15.0, 1.15.2, 1.15.3, 1.15.4, 1.15.5, 2.0.0a0, 2.0.0b0, 2.0.0b1, 2.0.0rc0, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.1.0rc0, 2.1.0rc1, 2.1.0rc2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0rc0, 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.2.2, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.3.2, 2.4.0rc0, 2.4.0rc1, 2.4.0rc2, 2.4.0rc3, 2.4.0rc4, 2.4.0, 2.4.1)
ERROR: No matching distribution found for tensorflow==1.4.0
ARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow/python/compat/compat.py:175: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Using TensorFlow backend.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2021-03-17 18:19:00.090136: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-17 18:19:00.117074: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699850000 Hz
2021-03-17 18:19:00.117502: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5572f17545d0 executing computations on platform Host. Devices:
2021-03-17 18:19:00.117526: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
/content/dataset/basophil/train/images
Traceback (most recent call last):
File "mask_rcnn3.py", line 273, in
train(model, "/content/dataset")
File "mask_rcnn3.py", line 218, in train
dataset_train.load_blood(dataset_dir, "train")
File "mask_rcnn3.py", line 171, in load_blood
image_ids = list(map(lambda x: x.split('.')[0], next(os.walk(image_dir))[2]))
StopIteration
Starting at epoch 0. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1820/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
Traceback (most recent call last):
File "mask_rcnn3.py", line 273, in
train(model, "./dataset")
File "mask_rcnn3.py", line 246, in train
layers='heads')
File "/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/mrcnn/model.py", line 2354, in train
self.compile(learning_rate, self.config.LEARNING_MOMENTUM)
File "/home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/mrcnn/model.py", line 2199, in compile
self.keras_model.add_metric(loss, name)
AttributeError: 'Model' object has no attribute 'add_metric'
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Using TensorFlow backend.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2139: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:2239: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:553: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/utils.py:202: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py:600: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:203: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-03-17 18:26:22.000920: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-17 18:26:22.021257: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3699850000 Hz
2021-03-17 18:26:22.021639: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5651c7f85910 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-17 18:26:22.021686: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:207: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:216: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:223: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
2021-03-17 18:26:23.662876: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 51380224 exceeds 10% of system memory.
./dataset/basophil/train/images
./dataset/eosinophil/train/images
./dataset/erythroblast/train/images
./dataset/ig/train/images
./dataset/lymphocyte/train/images
./dataset/monocyte/train/images
./dataset/neutrophil/train/images
./dataset/platelet/train/images
./dataset/basophil/val/images
./dataset/eosinophil/val/images
./dataset/erythroblast/val/images
./dataset/ig/val/images
./dataset/lymphocyte/val/images
./dataset/monocyte/val/images
./dataset/neutrophil/val/images
./dataset/platelet/val/images
Train network heads
Starting at epoch 0. LR=0.001
Checkpoint Path: /home/tamal/Research/Jimut/bloseg/Aniket_MASK_RCNN/MaskRCNN_3/Mask_RCNN/logs/blood20210317T1826/mask_rcnn_blood_{epoch:04d}.h5
Selecting layers to train
fpn_c5p5 (Conv2D)
fpn_c4p4 (Conv2D)
fpn_c3p3 (Conv2D)
fpn_c2p2 (Conv2D)
fpn_p5 (Conv2D)
fpn_p2 (Conv2D)
fpn_p3 (Conv2D)
fpn_p4 (Conv2D)
In model: rpn_model
rpn_conv_shared (Conv2D)
rpn_class_raw (Conv2D)
rpn_bbox_pred (Conv2D)
mrcnn_mask_conv1 (TimeDistributed)
mrcnn_mask_bn1 (TimeDistributed)
mrcnn_mask_conv2 (TimeDistributed)
mrcnn_mask_bn2 (TimeDistributed)
mrcnn_class_conv1 (TimeDistributed)
mrcnn_class_bn1 (TimeDistributed)
mrcnn_mask_conv3 (TimeDistributed)
mrcnn_mask_bn3 (TimeDistributed)
mrcnn_class_conv2 (TimeDistributed)
mrcnn_class_bn2 (TimeDistributed)
mrcnn_mask_conv4 (TimeDistributed)
mrcnn_mask_bn4 (TimeDistributed)
mrcnn_bbox_fc (TimeDistributed)
mrcnn_mask_deconv (TimeDistributed)
mrcnn_class_logits (TimeDistributed)
mrcnn_mask (TimeDistributed)
WARNING:tensorflow:From /home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
Traceback (most recent call last):
File "mask_rcnn3.py", line 273, in <module>
train(model, "./dataset")
File "mask_rcnn3.py", line 246, in train
layers='heads')
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py", line 2354, in train
File "/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/mask_rcnn-2.1-py3.7.egg/mrcnn/model.py", line 2199, in compile
AttributeError: 'Model' object has no attribute 'add_metric'
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
/home/tamal/anaconda3/envs/aniket_mask_rcnn3/lib/python3.7/site-packages/skimage/transform/_warps.py:830: FutureWarning: Input image dtype is bool. Interpolation is not defined with bool data type. Please set order to 0 or explicitely cast input image to another data type. Starting from version 0.19 a ValueError will be raised instead of this warning.
order = _validate_interpolation_order(image.dtype, order)
pip install -U scikit-image==0.16.2
Lol managed to run but will take 20 mins per epochs with the current RAM. Looks CUDA isn't installed