TensorFlow 2.0 [Condition x == y did not hold element-wise:]

Question

I am training a chess program using TensorFlow 2 and Keras. Previously, I had this working if I loaded the data in the same script as the training of the model, but as the dataset became larger, it was much faster to pickle the data and then reload it. When I did that, I now get this set of errors even after trimming my model down to 1 layer to simplify the problem.

Can anyone shed light on the meaning of the 'Condition x == y did not hold element wise' error message?

Here is the code:

import tensorflow as tf
import numpy as np
import pandas as pd
import chess
from util import *
from sklearn.model_selection import train_test_split
import pickle
import time

bb = chess.Board()

print("Reading examplesMatrix...")
examplesMatrix = pickle.load(open('examplesMatrix.sav','rb'))
examplesMatrix = examplesMatrix.T

print("Reading Y_vec...")
Y_vec = pickle.load(open('Yvector.sav','rb'))

print(examplesMatrix.shape)
print(Y_vec.shape)

X_train, X_test, y_train, y_test = train_test_split(examplesMatrix, Y_vec, test_size=0.2)

start_time = time.time()

model = tf.keras.models.Sequential([
    #tf.keras.layers.Conv1D(kernel_size=8, filters=100),
    tf.keras.layers.Dense(activation='relu', units=600)
    #tf.keras.layers.Dense(units=400, activation='relu'),
    #tf.keras.layers.Dense(units=200, activation='relu'),
    #tf.keras.layers.Dense(units=100, activation='relu'),
    #tf.keras.layers.Dense(3, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(X_train, y_train) #, epochs=5)
model.evaluate(X_test, y_test)

print("Execution time: %s seconds ---" % (time.time() - start_time))
fname = "/home/jwales/eclipse-workspace/djwdata/science/chessmaster/keras_full_29OctC1.sav"
print("Saving as: "+fname)
model.save(fname)

And here is the output: (Note that my boards are 9x8 so I have an extra 8 elements for engineered features like total board score, move number, and whose move it is)

Reading examplesMatrix...
Reading Y_vec...
(1212827, 8, 9)
(1212827, 1)
2019-10-29 10:47:17.469971: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-29 10:47:17.491088: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2808000000 Hz
2019-10-29 10:47:17.491373: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4b4e870 executing computations on platform Host. Devices:
2019-10-29 10:47:17.491409: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
Train on 970261 samples
2019-10-29 10:47:18.220470: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: assertion failed: [] [Condition x == y did not hold element-wise:] [x (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/Shape_1:0) = ] [32 1] [y (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/strided_slice:0) = ] [32 8]
     [[{{node loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/assert_equal/Assert/Assert}}]]
    32/970261 [..............................] - ETA: 3:45:10Traceback (most recent call last):
  File "kt_trainer.py", line 68, in <module>
    model.fit(X_train, y_train) #, epochs=5)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 520, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 2, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError:  assertion failed: [] [Condition x == y did not hold element-wise:] [x (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/Shape_1:0) = ] [32 1] [y (loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/strided_slice:0) = ] [32 8]
     [[node loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/assert_equal/Assert/Assert (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_distributed_function_716]

Function call stack:
distributed_function

Answer 1

The Dense layer expects the flattened data.

Try:

   tf.keras.layers.Flatten() 

before calling the dense layer.