Calculating ROC/AUC for MaxEnt and BIOMOD

Question

Thanks very much Winchester for the kind help! I also saw the tutorial and that work for me! In the past two days I explored the output of both MaxEnt and BIOMOD, and I think I am still a little bit confused by the terms used within the two.

From Philips' code, it seems that he used the Sample points and backaground point to calculate ROC, while in BIOMOD, there is only prediction from the presence and pseudo absence points. which means, for the same dataset, I have the same number of presence/sample data, but different absence/background data for the two models, respectively. And when I recalculate the ROC, it is usually inconsistent with the values reported by the model themselves.

I think I still didnot get some of the point of model evaluation, concerning what is been evaluated and how to generate the evaluation dataset, ie. comfusion matrix, and which part of the data was selected as evaluation.

Thanks everybody for the kind reply! I am very sorry for the inconvenience. I appended a few more sentences to the post for BIOMOD to make it runable, as for MaxEnt, you can use the tutorial data.

Actually, the intend of my post is to find someone who have had the experience to work with both the presence/absence dataset and the presence-only dataset. I probably know how to deal with them separately , but not altogether.

I am using both MaxEnt and a few algorithms under BIOMOD for the distribution of my species, and I would like to plot the ROC/AUC in the same figue, anybody have done this before?

As far as I know, for MaxEnt, the ROC can be plotted using the ROCR and vcd library, which was given in the tutorial of MaxEnt by Philips:

   install.packages("ROCR", dependencies=TRUE)
   install.packages("vcd",  dependencies=TRUE)
   library(ROCR)
   library(vcd)
   library(boot)
   setwd("c:/maxent/tutorial/outputs")
   presence <- read.csv("bradypus_variegatus_samplePredictions.csv")
   background <- read.csv("bradypus_variegatus_backgroundPredictions.csv")
   pp <- presence$Logistic.prediction                # get the column of predictions
   testpp <- pp[presence$Test.or.train=="test"]       # select only test points
   trainpp <- pp[presence$Test.or.train=="train"]   # select only test points
   bb <- background$logistic
   combined <- c(testpp, bb)                             # combine into a single vector
   label <- c(rep(1,length(testpp)),rep(0,length(bb)))  # labels: 1=present, 0=random
   pred <- prediction(combined, label)                    # labeled predictions
   perf <- performance(pred, "tpr", "fpr")          # True / false positives, for ROC curve
   plot(perf, colorize=TRUE)                                # Show the ROC curve
   performance(pred, "auc")@y.values[[1]]            # Calculate the AUC

While for BIOMOD, they require presence/absence data, so I used 1000 pseudo.absence points, and there is no background. I found another script given by Thuiller himself:

library(BIOMOD)
library(PresenceAbsence)

data(Sp.Env)

Initial.State(Response=Sp.Env[,12:13], Explanatory=Sp.Env[,4:10], 
IndependentResponse=NULL, IndependentExplanatory=NULL)

Models(GAM = TRUE, NbRunEval = 1, DataSplit = 80,
   Yweights=NULL, Roc=TRUE, Optimized.Threshold.Roc=TRUE, Kappa=F, TSS=F, KeepPredIndependent = FALSE, VarImport=0,
   NbRepPA=0, strategy="circles", coor=CoorXY, distance=2, nb.absences=1000)


load("pred/Pred_Sp277")

    data=cbind(Sp.Env[,1], Sp.Env[,13], Pred_Sp277[,3,1,1]/1000)

    plotroc <- roc.plot.calculate(data)


    plot(plotroc$threshold, plotroc$sensitivity, type="l", col="blue ")

    lines(plotroc$threshold, plotroc$specificity)
    lines(plotroc$threshold, (plotroc$specificity+plotroc$sensitivity)/2, col="red")

Now, the problem is, how could I plot them altogether? I have tried both, they work well for both seperately, but exclusively. Maybe I need some one to help me understand the underling philosiphy of ROC.

Thanks in advance~

Marco

Answer 1

Ideally, if you are going to compare methods, you should probably generate predictions from MaxEnt and BIOMOD for each location of the testing portion of your data set (observed presences and absences). As Christian mentioned, pROC is a nice package, especially for comparing ROC curves. Although I don't have access to the data, I've generated a dummy data set which should illustrate plotting two roc curves and calculating the difference in AUC.

library(pROC)

#Create dummy data set for test observations
obs<-rep(0:1, each=50)
pred1<-c(runif(50,min=0,max=0.8),runif(50,min=0.3,max=0.6))
pred2<-c(runif(50,min=0,max=0.6),runif(50,min=0.4,max=0.9))

roc1<-roc(obs~pred1) # Calculate ROC for each method
roc2<-roc(obs~pred2) 

#Plot roc curves for each method

plot(roc1)
lines(roc2,col="red")

#Compare differences in area under ROC
roc.test(roc1,roc2,method="bootstrap",paired=TRUE)