How to approximate the determinant with keras

Question

As an experiment I am building a keras model to approximate the determinant of a matrix. However, when I run it the loss goes down at every epoch and the validation loss goes up! For example:

8s - loss: 7573.9168 - val_loss: 21831.5428
Epoch 21/50
8s - loss: 7345.0197 - val_loss: 23594.8540
Epoch 22/50
13s - loss: 7087.7454 - val_loss: 24718.3967
Epoch 23/50
7s - loss: 6851.8714 - val_loss: 25624.8609
Epoch 24/50
6s - loss: 6637.8168 - val_loss: 26616.7835
Epoch 25/50
7s - loss: 6446.8898 - val_loss: 28856.9654
Epoch 26/50
7s - loss: 6255.7414 - val_loss: 30122.7924
Epoch 27/50
7s - loss: 6054.5280 - val_loss: 32458.5306
Epoch 28/50

Here is the complete code:

import numpy as np
import sys
from scipy.stats import pearsonr
from scipy.linalg import det
from sklearn.model_selection import train_test_split
from tqdm import tqdm
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import math
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from keras import backend as K

def baseline_model():
# create model
        model = Sequential()
        model.add(Dense(200, input_dim=n**2, kernel_initializer='normal', activation='relu'))
        model.add(Dense(1, input_dim=n**2))
        #        model.add(Dense(1, kernel_initializer='normal'))
        # Compile model
        model.compile(loss='mean_squared_error', optimizer='adam')
        return model


n = 15

print("Making the input data using seed 7", file=sys.stderr)
np.random.seed(7)
U = np.random.choice([0, 1], size=(n**2,n))
#U is a random orthogonal matrix
X =[]
Y =[]
# print(U)
for i in tqdm(range(100000)):
        I = np.random.choice(n**2, size = n)
        # Pick out the random rows and sort the rows of the matrix lexicographically.
        A = U[I][np.lexsort(np.rot90(U[I]))] 
        X.append(A.ravel())
        Y.append(det(A))

X = np.array(X)
Y = np.array(Y)

print("Data created")

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=32, verbose=2)))
pipeline = Pipeline(estimators)
X_train, X_test, y_train, y_test = train_test_split(X, Y,
                                                    train_size=0.75, test_size=0.25)
pipeline.fit(X_train, y_train, mlp__validation_split=0.3)

How can I stop it overfitting so badly?

---

Update 1

I tried adding more layers and L_2 regularization. However, it makes little or no difference.

def baseline_model():
# create model
        model = Sequential()
        model.add(Dense(n**2, input_dim=n**2, kernel_initializer='glorot_normal', activation='relu'))
        model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01)))
        model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01)))
        model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01)))
        model.add(Dense(1, kernel_initializer='glorot_normal'))
        # Compile model
        model.compile(loss='mean_squared_error', optimizer='adam')
        return model

I increased the number of epochs to 100 and it finishes with:

19s - loss: 788.9504 - val_loss: 18423.2807
Epoch 97/100
24s - loss: 760.2046 - val_loss: 18305.9273
Epoch 98/100
20s - loss: 806.0941 - val_loss: 18174.8706
Epoch 99/100
24s - loss: 780.0487 - val_loss: 18356.7482
Epoch 100/100
27s - loss: 749.2595 - val_loss: 18331.5859

Is it possible to approximate the determinant of a matrix using keras?

Answer 1

I tested your code and got the same result. But let's go into basic understanding of matrix determinant (DET). DET consists of n! products, so you cannot really approximate it with n*n weights in few layers of neural network. This requires number of weights that would not scale to n=15, since 15! is 1307674368000 terms for multiplication in the DET.