Add SVM to last layer

Question

What I did:

I implement the following model using of Keras:

train_X, test_X, train_Y, test_Y = train_test_split(X, Y, test_size=0.2, random_state=np.random.seed(7), shuffle=True)

train_X = np.reshape(train_X, (train_X.shape[0], 1, train_X.shape[1]))
test_X = np.reshape(test_X, (test_X.shape[0], 1, test_X.shape[1]))

inp = Input((train_X.shape[1], train_X.shape[2]))
lstm = LSTM(1, return_sequences=False)(inp)
output = Dense(train_Y.shape[1], activation='softmax')(lstm)

model = Model(inputs=inp, outputs=output)
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
model.fit(train_X, train_Y, validation_split=.20, epochs=2, batch_size=50)

What I want:

I want to add SVM to the last layer in my model but i dont know how? Any idea?

Answer 1

This should work for adding svm as last layer.

inp = Input((train_X.shape[1], train_X.shape[2]))
lstm = LSTM(1, return_sequences=False)(inp)
output = Dense(train_Y.shape[1], activation='softmax', W_regularizer=l2(0.01)))(lstm)

model = Model(inputs=inp, outputs=output)
model.compile(loss='hinge', optimizer='adam', metrics=['accuracy'])
model.fit(train_X, train_Y, validation_split=.20, epochs=2, batch_size=50)

Here I have used hinge as loss considering binary categorised target. But if it is more than that, then you can consider using categorical_hinge

Answer 2

Change softmax to linear and add kernel_regularizer=l2(1e-4) instead W_regularizer=l2(0.01) using keras 2.2.4. Use loss = categorical_hinge.