How can I use categorical one-hot labels for training with Keras?

Question

I have inputs that look like this:

[
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
...]

of shape (1, num_samples, num_features), and labels that look like this:

[
[0, 1]
[1, 0]
[1, 0]
...]

of shape (1, num_samples, 2).

However, when I try to run the following Keras code, I get this error: ValueError: Error when checking model target: expected dense_1 to have 2 dimensions, but got array with shape (1, 8038, 2). From what I've read, this appears to stem from the fact that my labels are 2D, and not simply integers. Is this correct, and if so, how can I use one-hot labels with Keras?

Here's the code:

num_features = 463
trX = np.random(8038, num_features)
trY = # one-hot array of shape (8038, 2) as described above

def keras_builder():  #generator to build the inputs
    while(1):
        x = np.reshape(trX, (1,) + np.shape(trX))
        y = np.reshape(trY, (1,) + np.shape(trY))
        print(np.shape(x)) # (1, 8038, 463)
        print(np.shape(y)) # (1, 8038, 2)
        yield x, y

model = Sequential()
model.add(LSTM(100, input_dim = num_features))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit_generator(keras_builder(), samples_per_epoch = 1, nb_epoch=3, verbose = 2, nb_worker = 1)

Which promptly throws the error above:

Traceback (most recent call last):
  File "file.py", line 35, in <module>
    model.fit_generator(keras_builder(), samples_per_epoch = 1, nb_epoch=3, verbose = 2, nb_worker = 1)
  ...
ValueError: Error when checking model target: expected dense_1 to have 2 dimensions, but got array with shape (1, 8038, 2)

Thank you!

Answer 1

There are a lot of things that do not add up.

I assume that you are trying to solve a sequential classification task, i.e. your data is shaped as (<batch size>, <sequence length>, <feature length>).

In your batch generator you create a batch consisting of one sequence of length 8038 and 463 features per sequence element. You create a matching Y batch to compare against, consisting of one sequence with 8038 elements, each of size 2.

Your problem is that Y does not match up with the output of the last layer. Your Y is 3-dimensional while the output of your model is only 2-dimensional: Y.shape = (1, 8038, 2) does not match dense_1.shape = (1,1). This explains the error message you get.

The solution to this: you need to enable return_sequences=True in the LSTM layer to return a sequence instead of only the last element (effectively removing the time-dimension). This would give an output shape of (1, 8038, 100) at the LSTM layer. Since the Dense layer is not able to handle sequential data you need to apply it to each sequence element individually which is done by wrapping it in a TimeDistributed wrapper. This then gives your model the output shape (1, 8038, 1).

Your model should look like this:

from keras.layers.wrappers import TimeDistributed

model = Sequential()
model.add(LSTM(100, input_dim=num_features, return_sequences=True))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))

This can be easily spotted when examining the summary of the model:

print(model.summary())