CODE FOR 95% confidence intervals for sensitivity and specificity using confusion matrix

Question

I have a 5 variable data set called EYETESTS. The variables are MAD, SAD, RED, BLUE, LEVEL.

MAD, SAD, RED AND BLUE AND LEVEL are all factor variables with 2 factors that represent yes(1) or no(0).

Example:

MAD	SAD	RED	BLUE	LEVEL
0	0	0	1	1
0	1	1	0	0
1	0	0	1	0
0	1	0	0	0
0	0	1	0	0
1	0	0	0	1

I am trying to create a confusion matrix of MAD against LEVEL. My Reference variable is LEVEL. The other variables are all predictor/test variables.

Then a separate confusion matrix of SAD against LEVEL. Then a separate confusion matrix of RED against LEVEL. Then a separate confusion matrix of BLUE against LEVEL.

The issue that I am having trouble with is calculating the 95% Confidence Intervals for the sensitivity and specificity alongside the others.

I can get the output in the form I want using the caret library.

library(caret)
confusionMatrix(as.factor(SAD), as.factor(LEVEL))

This gives me the output I want in terms of sensitivity, specificity and accuracy but I want the 95% Confidence Intervals for the sensitivity and specificity.

Would be incredibly grateful for help with this. I have tried using the conf package and the epiR package but they do not give the confidence intervals for the sensitivity and specificity.

Answer 1

Sensitivity and specificity are proportions and their confidence intervals are binomial proportion intervals.
Here is a way with package binom, that gives 11 different confidence intervals for the binomial proportion.

df1 <- "MAD     SAD     RED     BLUE    LEVEL
0   0   0   1   1
0   1   1   0   0
1   0   0   1   0
0   1   0   0   0
0   0   1   0   0
1   0   0   0   1"
df1 <- read.table(text = df1, header = TRUE)

confusionMatrixCI <- function(x, ...) {
  y <- x$table
  Se <- binom::binom.confint(y[1,1], sum(y[,1]), ...)
  Sp <- binom::binom.confint(y[2,2], sum(y[,2]), ...)
  list(sensitivity = Se, specificity = Sp)
}

res <- caret::confusionMatrix(factor(df1$MAD), factor(df1$LEVEL), 
                              positive = "1")

ci95 <- confusionMatrixCI(res)
#> Warning in stats::prop.test(x[i], n[i]): Chi-squared approximation may be
#> incorrect

#> Warning in stats::prop.test(x[i], n[i]): Chi-squared approximation may be
#> incorrect
ci95$sensitivity
#>           method x n mean     lower     upper
#> 1  agresti-coull 3 4 0.75 0.2891407 0.9659139
#> 2     asymptotic 3 4 0.75 0.3256553 1.1743447
#> 3          bayes 3 4 0.70 0.3470720 0.9966562
#> 4        cloglog 3 4 0.75 0.1279469 0.9605486
#> 5          exact 3 4 0.75 0.1941204 0.9936905
#> 6          logit 3 4 0.75 0.2378398 0.9664886
#> 7         probit 3 4 0.75 0.2543493 0.9777762
#> 8        profile 3 4 0.75 0.2772218 0.9800582
#> 9            lrt 3 4 0.75 0.2775912 0.9837676
#> 10     prop.test 3 4 0.75 0.2194265 0.9868088
#> 11        wilson 3 4 0.75 0.3006418 0.9544127
ci95$specificity
#>           method x n mean        lower     upper
#> 1  agresti-coull 1 2  0.5  0.094531206 0.9054688
#> 2     asymptotic 1 2  0.5 -0.192951912 1.1929519
#> 3          bayes 1 2  0.5  0.060830276 0.9391697
#> 4        cloglog 1 2  0.5  0.005983088 0.9104101
#> 5          exact 1 2  0.5  0.012579117 0.9874209
#> 6          logit 1 2  0.5  0.058866787 0.9411332
#> 7         probit 1 2  0.5  0.041195981 0.9588040
#> 8        profile 1 2  0.5  0.038416621 0.9615834
#> 9            lrt 1 2  0.5  0.038100934 0.9618991
#> 10     prop.test 1 2  0.5  0.094531206 0.9054688
#> 11        wilson 1 2  0.5  0.094531206 0.9054688
ci95
#> $sensitivity
#>           method x n mean     lower     upper
#> 1  agresti-coull 3 4 0.75 0.2891407 0.9659139
#> 2     asymptotic 3 4 0.75 0.3256553 1.1743447
#> 3          bayes 3 4 0.70 0.3470720 0.9966562
#> 4        cloglog 3 4 0.75 0.1279469 0.9605486
#> 5          exact 3 4 0.75 0.1941204 0.9936905
#> 6          logit 3 4 0.75 0.2378398 0.9664886
#> 7         probit 3 4 0.75 0.2543493 0.9777762
#> 8        profile 3 4 0.75 0.2772218 0.9800582
#> 9            lrt 3 4 0.75 0.2775912 0.9837676
#> 10     prop.test 3 4 0.75 0.2194265 0.9868088
#> 11        wilson 3 4 0.75 0.3006418 0.9544127
#> 
#> $specificity
#>           method x n mean        lower     upper
#> 1  agresti-coull 1 2  0.5  0.094531206 0.9054688
#> 2     asymptotic 1 2  0.5 -0.192951912 1.1929519
#> 3          bayes 1 2  0.5  0.060830276 0.9391697
#> 4        cloglog 1 2  0.5  0.005983088 0.9104101
#> 5          exact 1 2  0.5  0.012579117 0.9874209
#> 6          logit 1 2  0.5  0.058866787 0.9411332
#> 7         probit 1 2  0.5  0.041195981 0.9588040
#> 8        profile 1 2  0.5  0.038416621 0.9615834
#> 9            lrt 1 2  0.5  0.038100934 0.9618991
#> 10     prop.test 1 2  0.5  0.094531206 0.9054688
#> 11        wilson 1 2  0.5  0.094531206 0.9054688

^{Created on 2023-01-08 with reprex v2.0.2}