Why neural networks are failing in a simple classifcation case

Question

I have the code below where a simple rule based classification data set is formed:

# # Data preparation
data = data.frame(A = round(runif(100)), B = round(runif(100)), C = round(runif(100)))
# Y - is the classification output column
data$Y = ifelse((data$A == 1 & data$B == 1 & data$C == 0), 1, ifelse((data$A == 0 & data$B == 1 & data$C == 1), 1, ifelse((data$A == 0 & data$B ==0 & data$C == 0), 1, 0)))
# Shuffling the data set
data = data[sample(rownames(data)), ]

I have divided the data set into training and testing so that I can validate my results on the test set:

# # Divide into train and test
library(caret)
trainIndex = createDataPartition(data[, "Y"], p = .7, list = FALSE, times = 1) # for balanced sampling
train = data[trainIndex, ]
test = data[-trainIndex, ]

I have tried building a simple neural-net with the number of neurons in hidden layer is chosen by looping (as mentioned here)

# # Build a neural net
library(neuralnet)
for(alpha in 2:10)
{
    nHidden = round(nrow(train)/(alpha*(3+1)))
    nn = neuralnet(Y ~ A + B + C, train, linear.output = F, likelihood = T, err.fct = "ce", hidden = nHidden)

    # Calculate Mean Squared Error for Train and Test
    trainMSE = mean((round(nn$net.result[[1]]) - train$Y)^2)
    testPred = round(compute(nn,test[-length(ncol(test))])$net.result)
    testMSE = mean((testPred - test$Y)^2)

    print(paste("Train Error: " , round(trainMSE, 4), ", Test Error: ", round(testMSE, 4), ", #. Hidden = ", nHidden, sep = ""))
}

[1] "Train Error: 0, Test Error: 0.6, #. Hidden = 9"

[1] "Train Error: 0, Test Error: 0.6, #. Hidden = 6"

[1] "Train Error: 0, Test Error: 0.6, #. Hidden = 4"

[1] "Train Error: 0, Test Error: 0.6, #. Hidden = 4"

[1] "Train Error: 0.1429, Test Error: 0.8333, #. Hidden = 3"

[1] "Train Error: 0.1429, Test Error: 0.8333, #. Hidden = 2"

[1] "Train Error: 0.0857, Test Error: 0.6, #. Hidden = 2"

[1] "Train Error: 0.1429, Test Error: 0.8333, #. Hidden = 2"

[1] "Train Error: 0.0857, Test Error: 0.6, #. Hidden = 2"

It was giving poor over fitted results. But, when I have built a simple random forest on the same data set. I am getting the train and test errors as - 0

# # Build a Random Forest
trainRF = train
trainRF$Y = as.factor(trainRF$Y)
testRF = test

library(randomForest)
rf = randomForest(Y ~ ., data = trainRF, mtry = 2)

# Calculate Mean Squared Error for Train and Test
trainMSE = mean((round(rf$votes[,2]) - as.numeric(as.character(trainRF$Y)))^2)
testMSE = mean((round(predict(rf, testRF, type = "prob")[,2]) - as.numeric(as.character(testRF$Y)))^2)

print(paste("Train Error: " , round(trainMSE, 4), ", Test Error: ", round(testMSE, 4), sep = ""))

[1] "Train Error: 0, Test Error: 0"

Please help me in understanding why neural nets are failing in a simple case where random forest is working with 100-Percent Accuracy.

Note: I have used only one hidden layer (assuming one hidden layer is enough for such simple classification) and iterated on the number of neurons in the hidden layer.

Also, help me if my understanding of neural network parameters is wrong.

Complete code can be found here

Answer 1

A similar question has been hunting me for some time, so I tried understanding your data and problem and compared them to mine. In the end, though, it's just a small bug in this line:

testPred = round(compute(nn,test[-length(ncol(test))])$net.result)

You select B, C and Y for prediction, instead of A, B and C, because length(ncol(something)) will always return 1. You want only test[-ncol(test)].

> summary(test[-length(ncol(test))])

          B              C             Y            
 Min.   :0.00   Min.   :0.0   Min.   :0.0000000  
 1st Qu.:0.00   1st Qu.:0.0   1st Qu.:0.0000000  
 Median :0.00   Median :0.5   Median :0.0000000  
 Mean   :0.48   Mean   :0.5   Mean   :0.3766667  
 3rd Qu.:1.00   3rd Qu.:1.0   3rd Qu.:1.0000000  
 Max.   :1.00   Max.   :1.0   Max.   :1.0000000