Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

caret - random-forests not working: "Something is wrong; all the Accuracy metric values are missing:"

Related to these:

  1. getting this error in Caret
  2. https://github.com/topepo/caret/issues/160

I'm getting this error:

Something is wrong; all the Accuracy metric values are missing:
    Accuracy       Kappa    
 Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA  
 NA's   :5     NA's   :5    
Error in train.default(x, y, weights = w, ...) : Stopping
In addition: Warning message:
In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

The first link suggests that the levels of the response variable cannot be 0 and 1. This is not the case in my data:

R> str(test$y)
 Factor w/ 2 levels "No","Yes": 1 1 1 1 1 1 1 1 1 1 ...
R> levels(test$y)
[1] "No"  "Yes"

So, I'm not sure what's going on.

Ex. Data

test <- structure(list(y = structure(c(2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L
), .Label = c("No", "Yes"), class = "factor"), x1 = structure(c(6L, 
40L, 26L, 7L, 18L, 9L, 26L, 36L, 23L, 16L, 6L, 20L, 23L, 26L, 
41L, 20L, 31L, 7L, 2L, 2L, 18L, 2L, 12L, 9L, 40L, 40L, 14L, 8L, 
2L, 20L, 15L, 12L, 8L, 17L, 17L, 21L, 18L, 32L, 2L, 2L), .Label = c("Accommodation and Restaurant Services", 
"Admin/Support Services", "Agriculture", "Arts, Entertainment, and Rec.", 
"Construction: Heavy and Civil Engineering", "Construction: of Buildings", 
"Construction: Specialty Trade Contractors", "EDU Services", 
"Finance / Insurance", "Fishing, Hunting, Trapping", "Forestry & Logging", 
"Health Care and Social Assistance", "Information", "Management of Companies and Enterprises", 
"Manufacturing: Food/Bev/Textile", "Manufacturing: Metals/Machinery/Computers/Appliances", 
"Manufacturing: Wood/Paper/Chemical/Mineral", "Merchandise Trade", 
"Mining, Quarrying, and Oil and Gas Extraction", "Other Services (Blue Collar)", 
"Prof./Sci./Tech: Acct / Tax", "Prof./Sci./Tech: Advertising / Media", 
"Prof./Sci./Tech: Architecture / Eng.", "Prof./Sci./Tech: Computer Design", 
"Prof./Sci./Tech: Law", "Prof./Sci./Tech: Mgmt Consulting", "Prof./Sci./Tech: Other", 
"Prof./Sci./Tech: R&D", "Prof./Sci./Tech: Specialized Design", 
"Public Admin.", "Real Estate", "Retail Trade", "Support Agriculture", 
"Transportation", "Unknown", "Utilities", "Warehousing", "Waste Management & Remediation Services", 
"Wholesale Trade: Brokers", "Wholesale Trade: Durable Goods", 
"Wholesale Trade: NonDurable Goods"), class = "factor"), x2 = structure(c(36L, 
11L, 35L, 46L, 5L, 10L, 37L, 41L, 11L, 5L, 5L, 10L, 20L, 10L, 
5L, 5L, 45L, 20L, 11L, 10L, 18L, 35L, 5L, 6L, 41L, 5L, 44L, 36L, 
39L, 10L, 44L, 8L, 34L, 15L, 39L, 10L, 18L, 19L, 35L, 11L), .Label = c("AK", 
"AL", "AR", "AZ", "CA", "CO", "CT", "DC", "DE", "FL", "GA", "HI", 
"IA", "ID", "IL", "IN", "KS", "KY", "LA", "MA", "MD", "ME", "MI", 
"MN", "MO", "MS", "MT", "NC", "ND", "NE", "NH", "NJ", "NM", "NV", 
"NY", "OH", "OK", "OR", "PA", "RI", "SC", "SD", "TN", "TX", "UT", 
"VA", "VT", "WA", "WI", "WV", "WY"), class = "factor"), x3 = c(0.004714, 
0, 0.015551, 0.360246999999988, 5e-04, 0.035714, 0.357143, 0.00591043019290109, 
0.138889, 0.028846, 0.0075, 0.00051, 0.006329, 0.065789, 0.1125, 
0.003125, 0.003889, 0.000391, 0.011905, 0.004, 0, 0.00025, 0.005, 
0.076923, 0.149254, 0.0220719438793245, 0.360246999999988, 0.057692, 
0, 0.015625, 0.000714, 0, 0.001087, 0.006135, 0.003846, 0.066667, 
0.009091, 0, 0.360246999999988, 0.012821), x4 = c(3.69626899674553, 
0, 4.34824643385123, 4.22834902062364, 2.94001815500766, 3.27207378750001, 
4.61543448110941, 4.56919828334781, 4.32498170308737, 3.73719264270474, 
3.87511916546257, 1.70757017609794, 3.76499759928488, 3.7635028654676, 
4.15094055396548, 3.43949059038968, 3.70423633730879, 3.18864729599972, 
2.85186960072977, 2.37291200297011, 0, 2.69983772586725, 3.23829706787539, 
3.17695898058691, 4.32314893008404, 0, 4.64518638929519, 3.17405980772503, 
0, 2.5092025223311, 2.47856649559384, 0, 2.06818586174616, 4.08439751914115, 
3.50906804501716, 3.02160271602824, 2.71349054309394, 0, 4.6020708485543, 
2.79657433321043), x5 = c(472, 502, 506, 510, 497, 493, 515, 
542, 557, 465, 480, 369.618950156498, 518, 571, 512, 520, 464, 
578, 500, 526, 489.830047438596, 345, 664.964755505884, 546, 
505, 572, 540, 567, 473, 575, 558, 509.58218597766, 579, 616, 
561, 581, 291, 415.846613389669, 476, 442), x6 = c(374, 482, 
491, 540, 534, 493, 514, 570, 577, 485, 488, 627, 542, 529, 445, 
531, 456, 535, 381, 586, 474.392596434054, 484, 487.854513298151, 
518, 524, 582, 530, 571, 582.582737417662, 572, 592, 477, 585, 
594, 574, 609, 389, 581.722630168064, 550, 458), x7 = c(5.8e-05, 
0, 0.015551, 0.01, 0, 0, 0.0683816249999983, -0.00050051658067362, 
0.068194, 0.056615, 0, 0, 0.001097, 0, 0.0683816249999983, 0, 
0.002361, 0.000781, 0.021667, 0, 0, 0, 0, 0.001154, 0.001, -0.000657947357427473, 
0, 0, 0, 0, 0, 0, 0, 0.001479, 0.001269, 0.005333, 0.000455, 
0, 0, 0), x8 = c(14, 13, 53, 24, 8, 13, 13, 20, 17, 35, 19, 11, 
42, 15, 33, 1, 20, 6, 24, 3, 14, 3, 3, 17, 42, 8, 4, 0, 5, 4, 
10, 5, 8, 41, 31, 6, 2, 18, 7, 7), x9 = c(18, 2, 49, 19, 14, 
8, 7, 6, 7, 21, 19, 1, 34, 2, 24, 3, 30, 5, 3, 12, 9, 4, 2, 9, 
59, 15, 7, 0, 20, 1, 6, 13, 1, 64, 34, 18, 12, 0, 0, 6), x10 = c(48, 
68.8884165199473, 63, 54, 78, 80, 77.3502747403963, 74, 79, 71, 
76.7682937433346, 65.0624751538981, 63, 80, 41, 81.4257054732527, 
67, 78, 80, 73, 52.5390991618267, 60.8813703575155, 66, 72, 64, 
61.266324949851, 43.2207804060158, 80, 61.708917114202, 80, 75, 
73.3412226739437, 80, 78, 57, 78, 23, 30.321279640657, 69.1391208799255, 
60.9766796474371), x11 = c(4.62, 0.81, 1.98, 1.51, 1.51, 1.2, 
0.74, 1.2, 4.04, 2.06, 1.43, 1.51, 4.16, 0.81, 0.81, 1.82, 2.1, 
0.89, 0.73, 0.97, 20.49, 1.51, 1.51, 4.09, 1.33, 0.89, 1.59, 
1.43, 4.54, 1.51, 1.2, 1.04, 1.59, 2.57, 4.4, 1.28, 0.89, 17.94, 
1.29, 1.59), x12 = c(-3, -44.4574826440087, 1, 5, 2, 2, 39.0861520260711, 
14, 0, -6, 40.5638314058397, 22.0124501206663, 3, 12, 27, 7.55072978911628, 
5, -1, -12, 0, 14.5217398963732, -2.06782290930381, -13, 4, 1, 
39.251983622172, 0, 0, 33.2355632837177, 0, 6, 20.3416928763606, 
40.7136165846826, -2, 7, 0, 9, 0.622995283657772, -6.64967287401836, 
-3.6632790085156)), .Names = c("y", "x1", "x2", "x3", "x4", "x5", 
"x6", "x7", "x8", "x9", "x10", "x11", "x12"), row.names = c(59110L, 
266133L, 110275L, 271642L, 54361L, 54818L, 59197L, 94902L, 80531L, 
291L, 51460L, 228662L, 174960L, 27500L, 105584L, 132839L, 233895L, 
194802L, 123435L, 165332L, 318615L, 133731L, 256878L, 99780L, 
31551L, 106032L, 280841L, 130066L, 136252L, 29868L, 282962L, 
55762L, 312670L, 152593L, 50020L, 220877L, 13104L, 20888L, 319386L, 
229603L), class = "data.frame")

Code (updated):

Based on comments both here and on github/caret, I have updated the code. The non-parallel forest now works, but the parallel forests do not.

test$x7 <- NULL # remove low variance "dummy" variable 
                # based on comments on github (link above).

library(caret)
library(randomForest)
library(party) # conditional RF
library(kernlab)
library(parallel)
library(doParallel)

t_control <- trainControl(method= "repeatedcv", number= 10,
                          repeats= 1)
mtry_def <- floor(sqrt(ncol(test)))
t_grid <- expand.grid(mtry= c(mtry_def/2, mtry_def, 2 * mtry_def))


set.seed(14387)
## works without parallel (after removing options per @topepo):
rf1 <- train(y ~ ., data= test,
             method= "cforest", trControl= t_control,
             tuneGrid= t_grid) # remove verbose, importance, proximity

## doesn't work with parallel:
cl <- makeCluster(detectCores() - 1)
registerDoParallel(cl)
rf1 <- train(y ~ ., data= test,
             method= "cforest", trControl= t_control,
             tuneGrid= t_grid, allowParallel= TRUE) # same errors as prior to edit
rf2 <- train(y ~ ., data= test,
             method= "parRF", trControl= t_control, verbose= FALSE,
             tuneGrid= t_grid, allowParallel= TRUE, proximity= FALSE,
             importance= TRUE) # same errors as prior to edit

# moving from method= "parRF" --> method= "rf" does work:
rf3 <- train(y ~ ., data= test,
             method= "rf", trControl= t_control, verbose= FALSE,
             tuneGrid= t_grid, allowParallel= TRUE, proximity= FALSE,
             importance= TRUE)

stopCluster(cl) 

# defaults (ie-- outside caret) work
rf3a <- randomForest(y ~ ., data= test, mtry= 3, importance=TRUE)
rf3b <- cforest(y ~ ., data= test, controls= cforest_control(mtry= 3))

Sessioninfo:

# updated sessionInfo() -- AM running on a different computer
R version 3.2.2 (2015-08-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
 [1] stats4    grid      parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] kernlab_0.9-22      party_1.0-23        strucchange_1.5-1   sandwich_2.3-4      zoo_1.7-12          modeltools_0.2-21  
 [7] mvtnorm_1.0-3       randomForest_4.6-10 caret_6.0-52        ggplot2_1.0.1       lattice_0.20-33     doParallel_1.0.8   
[13] iterators_1.0.7     foreach_1.4.2      

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.1         compiler_3.2.2      nloptr_1.0.4        plyr_1.8.3          class_7.3-13        tools_3.2.2        
 [7] digest_0.6.8        lme4_1.1-9          nlme_3.1-122        gtable_0.1.2        mgcv_1.8-7          Matrix_1.2-2       
[13] brglm_0.5-9         SparseM_1.7         coin_1.1-0          proto_0.3-10        e1071_1.6-7         BradleyTerry2_1.0-6
[19] stringr_1.0.0       gtools_3.5.0        MatrixModels_0.4-1  nnet_7.3-11         survival_2.38-3     multcomp_1.4-1     
[25] TH.data_1.0-6       minqa_1.2.4         reshape2_1.4.1      car_2.1-0           magrittr_1.5        scales_0.3.0       
[31] codetools_0.2-14    MASS_7.3-43         splines_3.2.2       pbkrtest_0.4-2      colorspace_1.2-6    quantreg_5.19      
[37] stringi_0.5-5       munsell_0.4.2


#### original sessionInfo()
R> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
 [1] parallel  stats4    grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] doParallel_1.0.8    iterators_1.0.7     foreach_1.4.2       kernlab_0.9-22      party_1.0-23        strucchange_1.5-1  
 [7] sandwich_2.3-3      zoo_1.7-12          modeltools_0.2-21   mvtnorm_1.0-3       randomForest_4.6-10 caret_6.0-52       
[13] ggplot2_1.0.1       lattice_0.20-33    

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.1         compiler_3.2.2      nloptr_1.0.4        plyr_1.8.3          class_7.3-13        tools_3.2.2        
 [7] digest_0.6.8        lme4_1.1-9          gtable_0.1.2        nlme_3.1-121        mgcv_1.8-7          Matrix_1.2-2       
[13] SparseM_1.7         brglm_0.5-9         coin_1.1-0          proto_0.3-10        e1071_1.6-7         BradleyTerry2_1.0-6
[19] stringr_1.0.0       MatrixModels_0.4-1  gtools_3.5.0        nnet_7.3-10         survival_2.38-3     multcomp_1.4-1     
[25] TH.data_1.0-6       minqa_1.2.4         car_2.1-0           reshape2_1.4.1      magrittr_1.5        scales_0.3.0       
[31] codetools_0.2-14    splines_3.2.2       MASS_7.3-43         pbkrtest_0.4-2      colorspace_1.2-6    quantreg_5.19      
[37] stringi_0.5-5       munsell_0.4.2      

Any help would be greatly appreciated, thanks!!

like image 651
Alex W Avatar asked Oct 12 '15 19:10

Alex W


1 Answers

When I run the first cforest model, I can see that "In addition: There were 31 warnings (use warnings() to see them)". These say that

unused arguments (verbose = FALSE, proximity = FALSE, importance = TRUE)

These are arguments to the randomForest function and not cforest. Removing them removes the errors.

Update for the update:

This looks like confusion over the ... and where allowParallel can be invoked. When running the code for rf1, I get these warnings:

unused argument (allowParallel = TRUE)

Looking at ?train and ?cforest, neither has that argument; it is in trainControl.

Here is the confusing part: running rf3 with allowParallel as an argument to train does not generate an error. This is because cforest does not have the ellipses and randomForest does:

> names(formals(cforest))
[1] "formula"  "data"     "subset"   "weights"  "controls" "xtrafo"  
[7] "ytrafo"   "scores"   
> names(formals(randomForest:::randomForest.default))
 [1] "x"           "y"           "xtest"       "ytest"      
 [5] "ntree"       "mtry"        "replace"     "classwt"    
 [9] "cutoff"      "strata"      "sampsize"    "nodesize"   
[13] "maxnodes"    "importance"  "localImp"    "nPerm"      
[17] "proximity"   "oob.prox"    "norm.votes"  "do.trace"   
[21] "keep.forest" "corr.bias"   "keep.inbag"  "..."       

So, for rf1 there is no "bottomless pit" to send the inappropriate argument (allowParallel) but for rf3 there is a sequence of ... arguments and none of the functions ever have a terminal test to see if allowParallel is an inappropriate argument.

tl;dr

Pass allowParallel to trainControl and not train.

Max

like image 153
topepo Avatar answered Sep 21 '22 10:09

topepo