choosing between algorithms

Question

I am sure there are lot of Software Testing Engineers, Algorithm Validation Engineers on Stackoverflow.Could someone please tell me how would one proceed in the following scenario.

Say we have a Mammogram and 5 different algorithms which take this mammogram as input and identify if there is Cancer in the patient. If 3 out of 5 algorithms say there is cancer in the patient and 2 say there is no cancer in the patient. Which algorithm should I believe. How should I proceed testing these algorithms. Is there any statistical concept used in such scenarios?

I was asked this question in an interview for Algorithm Validation Engineer position. I believe they were trying to see how I would think given such scenario. How should I have answered this?

Thanks for your time

-Sashi

Answer 1

You can't say anything having only this information. What if some of the algotithms reuse some other algorithms from those 5? Then they could be prone to the same defects.

Say A, B and C actually use the same sub-algorithm for preprocessing data and the latter gives suboptimal results on some specific image and so the preprocessed image causes the later phases to produce wrong results - it won't matter you have three algorithms saying the same.

You need more specific data on how the algorithms correlate and what statistical characteristics are knwon about there error rates to be able to perform any analysis.

Answer 2

This is actually pretty tough to answer. I'm sure each algorithm is good at picking up different types of input triggers. More than likely, you'll need some statistical analysis to determine what each algorithm will usually detect as cancer. Additionally, you may go so far as to do something like make a Bayesian model to describe/determine if a patient has cancer based off of the algorithmic results.

you may find that 3 algorithms consistently miss the specific type of cancer that the other two are moderately good at picking up. you may discover similar relationships that occur, like when algorithms 2, 3, and 5 say there is no cancer, algorithm 1 says there is, and algorithm 4 is inconclusive, that there are usually typically-benign spots of a certain shape and color intensity that should be analyzed but probably aren't cancer.