Error after change lattice size
LumingSun opened this issue · comments
Hi, I'm using CannedRegressor with CalibratedLatticeConfig, everything is fine when the lattice_size in FeatureConfig is set to 2, but after I change lattice_size to 4, I got the following error
`Traceback (most recent call last):
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1812, in _create_c_op
c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid value in tensor used for shape: -2147483648
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data1/luming.slm/cardinality/lattice_regression/lattice_regression.py", line 457, in
estimator.train(input_fn=train_input_fn)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1175, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1204, in _train_model_default
self.config)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1163, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/estimators.py", line 729, in calibrated_lattice_model_fn
dtype=dtype)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/estimators.py", line 652, in _calibrated_lattice_model_fn
model = premade.CalibratedLattice(model_config=model_config, dtype=dtype)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/premade.py", line 235, in init
dtype=dtype)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/premade_lib.py", line 579, in build_lattice_layer
lattice_input)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 757, in call
self._maybe_build(inputs)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 2098, in _maybe_build
self.build(input_shapes)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_layer.py", line 399, in build
dtype=self.dtype)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 448, in add_weight
caching_device=caching_device)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 750, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 145, in make_variable
shape=variable_shape if variable_shape else None)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 260, in call
return cls._variable_v1_call(*args, **kwargs)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 221, in _variable_v1_call
shape=shape)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 199, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2597, in default_variable_creator
shape=shape)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 264, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1518, in init
distribute_strategy=distribute_strategy)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1651, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_layer.py", line 711, in call
dtype=dtype)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_lib.py", line 492, in linear_initializer
weights = batch_outer_operation(one_d_weights, operation=tf.add)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_lib.py", line 333, in batch_outer_operation
result = tf.reshape(result, shape=new_shape)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 195, in reshape
result = gen_array_ops.reshape(tensor, shape, name)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8234, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3485, in _create_op_internal
op_def=op_def)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1975, in init
control_input_ops, op_def)
File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1815, in _create_c_op
raise ValueError(str(e))
ValueError: Invalid value in tensor used for shape: -2147483648`
It's really strange. And I got the same error message when I set the lattice size as 2, but add several more common features (that have no monotonic constraints). Any idea about this? Thanks.
The error message is somewhat strange and hard to debug without more context. Can you please share a minimal code snippet that would reproduce this error? Say a Colab notebook. Which version of TF and the TFL library are you using?
ValueError: Invalid value in tensor used for shape: -2147483648
One clue is that -2147483648 is the minimum value of int32.
Thanks for replying. I tried and found that it might be caused by large number of features and lattice size. I wanted to solve a task with 20 features, and It's ok when there are 20 input features with lattice size 2 (2^20), but fail with lattice size 3 (3^20 3486784401). And it also fail with 14 feature lattice size 5, 5^14 (6103515625) but OK with 4^14 (268435456) and 8^10 (1073741824).