ERROR: Variable conv_1/kernel already exists, disallowed - When using with FLASK and GUNICORN
emilwidlund opened this issue · comments
This model works perfectly fine with Flask and its internal development server. But as soon as I use it with a production server like Gunicorn, it seems to throw a Tensorflow-error complaining about the use of a certain variable.
Here's the full stack. Thanks.
10:01:46 PM web.1 | 2018-07-06 22:01:46.254103: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
10:01:51 PM web.1 | [2018-07-06 22:01:51,235] ERROR in app: Exception on / [POST]
10:01:51 PM web.1 | Traceback (most recent call last):
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
10:01:51 PM web.1 | response = self.full_dispatch_request()
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
10:01:51 PM web.1 | rv = self.handle_user_exception(e)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
10:01:51 PM web.1 | reraise(exc_type, exc_value, tb)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
10:01:51 PM web.1 | raise value
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
10:01:51 PM web.1 | rv = self.dispatch_request()
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
10:01:51 PM web.1 | return self.view_functions[rule.endpoint](**req.view_args)
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/main.py", line 27, in process_image
10:01:51 PM web.1 | prediction = classify(temp.getvalue(), 'open_nsfw/data/open_nsfw-weights.npy')
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/open_nsfw/classify_nsfw.py", line 41, in classify
10:01:51 PM web.1 | model.build(weights_path=model_weights, input_type=input_type)
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/open_nsfw/model.py", line 50, in build
10:01:51 PM web.1 | kernel_size=7, stride=2, padding='valid')
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/open_nsfw/model.py", line 158, in __conv2d
10:01:51 PM web.1 | self.__get_weights(name, "biases"), dtype=tf.float32))
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/layers/convolutional.py", line 621, in conv2d
10:01:51 PM web.1 | return layer.apply(inputs)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 828, in apply
10:01:51 PM web.1 | return self.__call__(inputs, *args, **kwargs)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 699, in __call__
10:01:51 PM web.1 | self.build(input_shapes)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/layers/convolutional.py", line 144, in build
10:01:51 PM web.1 | dtype=self.dtype)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 546, in add_variable
10:01:51 PM web.1 | partitioner=partitioner)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/training/checkpointable.py", line 436, in _add_variable_with_custom_getter
10:01:51 PM web.1 | **kwargs_for_getter)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1317, in get_variable
10:01:51 PM web.1 | constraint=constraint)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1079, in get_variable
10:01:51 PM web.1 | constraint=constraint)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 425, in get_variable
10:01:51 PM web.1 | constraint=constraint)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 394, in _true_getter
10:01:51 PM web.1 | use_resource=use_resource, constraint=constraint)
10:01:51 PM web.1 | File "/Users/emilwidlund/.local/share/virtualenvs/glotter-picasso-aMAwiIpj/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 733, in _get_single_variable
10:01:51 PM web.1 | name, "".join(traceback.format_list(tb))))
10:01:51 PM web.1 | ValueError: Variable conv_1/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/open_nsfw/model.py", line 158, in __conv2d
10:01:51 PM web.1 | self.__get_weights(name, "biases"), dtype=tf.float32))
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/open_nsfw/model.py", line 50, in build
10:01:51 PM web.1 | kernel_size=7, stride=2, padding='valid')
10:01:51 PM web.1 | File "/Users/emilwidlund/Documents/glotter/glotter-picasso/open_nsfw/classify_nsfw.py", line 41, in classify
10:01:51 PM web.1 | model.build(weights_path=model_weights, input_type=input_type)
Looks like the server is trying to load the model a second time within the same tensorflow session.
Shouldn't be too hard to fix. Is it possible for you to share your code with me?
Thanks for your reply!
I'm basically trying to build a simple REST-api that takes an image and run the prediction against it and then return the sfw/nsfw scores.
Here is the simple code so far:
from flask import Flask, request, jsonify
from PIL import Image
from open_nsfw.classify_nsfw import classify
import io
app = Flask(__name__)
@app.route('/', methods=['POST'])
def process_image():
if request.method == 'POST':
# Declare the form-image as variable 'file'
file = request.files['image']
# Open the image with the PIL-module
image = Image.open(file)
# Create an empty bytesIO which will hold the image in memory
temp = io.BytesIO()
# Save the image to the empty bytesIO-object
# This way, we are able to save the image in memory with a reduced quality
image.save(temp, format='JPEG', quality=40)
prediction = classify(temp.getvalue(), 'open_nsfw/data/open_nsfw-weights.npy')
return jsonify(
sfw=float(prediction[0]),
nsfw=float(prediction[1])
)
This also meant that I slightly modified your code to take image data instead of an image path reference. This issue is not related to my modification though.
The error you've posted above means that model.build is getting called twice within the same tensorflow session, like:
model = OpenNsfwModel()
with tf.Session() as sess:
...
model.build() # fine
...
model.build() # error
....I guess the interesting part of your code is in the classify method. Maybe the session is not getting closed or reused when using gunicorn?
You could also try to create a new graph for each session:
with tf.Session(graph=tf.Graph()) as sess: instead of with tf.Session() as sess:
Thanks a bunch! The latter solution worked fine by creating a new graph each session. Thanks!