OpenCV 2.3.1: how to tell if haartraining is stuck or still working (on TINY example)

Question

I am using opencv_haartraining for the first time, using OpenCV 2.3.1 on Mac OS X Lion.

I am trying to train a very quick example. I am only using 23 positive examples and 45 negative examples. Yet opencv_haartraining has been using 100% of one core of my 2010 Macbook Air for at least 30 hours!

Here are the relevant files:

• The directory http://stanford.edu/~jonr1/haartraining_test_1/
• The vec file of positive samples http://stanford.edu/~jonr1/haartraining_test_1/vec_positive_samples/vec_positive_samples.vec
  • The bg (negative) examples http://stanford.edu/~jonr1/haartraining_test_1/bg_negative_examples.txt
  • The intermediate results produced so far http://stanford.edu/~jonr1/haartraining_test_1/results/

The vec file was produced by following this tutorial http://note.sonots.com/SciSoftware/haartraining.html , using that author's program mergevec to combine vec files individually produced by createsamples.

The output of opencv_haartraining was:

Data dir name: /Users/jon/Tabletop/haartraining_test_1/results
Vec file name: /Users/jon/Tabletop/haartraining_test_1/vec_positive_samples/vec_positive_samples.vec
BG  file name: /var/folders/85/96xv8qxx5ssc7ndg50s5lp480000gn/T/tmpZ2bASi.txt, is a vecfile: no
Num pos: 115
Num neg: 45
Num stages: 20
Num splits: 2 (tree as weak classifier)
Mem: 200 MB
Symmetric: TRUE
Min hit rate: 0.995000
Max false alarm rate: 0.500000
Weight trimming: 0.950000
Equal weights: FALSE
Mode: BASIC
Width: 20
Height: 20
Applied boosting algorithm: GAB
Error (valid only for Discrete and Real AdaBoost): misclass
Max number of splits in tree cascade: 0
Min number of positive samples per cluster: 500
Required leaf false alarm rate: 9.53674e-07

Tree Classifier
Stage
+---+
|  0|
+---+


Number of features used : 41910

Parent node: NULL

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 1
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-| 0.910420| 1.000000| 0.044444| 0.012500|
+----+----+-+---------+---------+---------+---------+
Stage training time: 2.00
Number of used features: 2

Parent node: NULL
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+
|  0|
+---+

   0


Parent node: 0

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.283019
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.965048| 1.000000| 1.000000| 0.018750|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-0.903213| 1.000000| 0.288889| 0.025000|
+----+----+-+---------+---------+---------+---------+
Stage training time: 3.00
Number of used features: 4

Parent node: 0
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+
|  0|  1|
+---+---+

   0---1


Parent node: 1

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.338346
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.961620| 1.000000| 1.000000| 0.043750|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-0.660077| 1.000000| 0.622222| 0.043750|
+----+----+-+---------+---------+---------+---------+
|   3| 88%|-| 0.142538| 1.000000| 0.044444| 0.012500|
+----+----+-+---------+---------+---------+---------+
Stage training time: 4.00
Number of used features: 6

Parent node: 1
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+
|  0|  1|  2|
+---+---+---+

   0---1---2


Parent node: 2

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.145631
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.975839| 1.000000| 0.777778| 0.025000|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-0.904803| 1.000000| 0.244444| 0.037500|
+----+----+-+---------+---------+---------+---------+
Stage training time: 3.00
Number of used features: 4

Parent node: 2
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+
|  0|  1|  2|  3|
+---+---+---+---+

   0---1---2---3


Parent node: 3

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.0293926
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.981092| 1.000000| 1.000000| 0.031250|
+----+----+-+---------+---------+---------+---------+
|   2| 91%|+|-0.820519| 1.000000| 0.333333| 0.031250|
+----+----+-+---------+---------+---------+---------+
Stage training time: 3.00
Number of used features: 4

Parent node: 3
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+
|  0|  1|  2|  3|  4|
+---+---+---+---+---+

   0---1---2---3---4


Parent node: 4

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.0244965
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.964250| 1.000000| 1.000000| 0.025000|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-1.801320| 1.000000| 1.000000| 0.025000|
+----+----+-+---------+---------+---------+---------+
|   3| 88%|-|-0.938272| 1.000000| 0.177778| 0.006250|
+----+----+-+---------+---------+---------+---------+
Stage training time: 4.00
Number of used features: 6

Parent node: 4
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|
+---+---+---+---+---+---+

   0---1---2---3---4---5


Parent node: 5

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.0100245
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.975839| 1.000000| 1.000000| 0.037500|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-0.109149| 1.000000| 0.133333| 0.037500|
+----+----+-+---------+---------+---------+---------+
Stage training time: 3.00
Number of used features: 4

Parent node: 5
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|  6|
+---+---+---+---+---+---+---+

   0---1---2---3---4---5---6


Parent node: 6

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.00587774
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.870814| 1.000000| 0.800000| 0.050000|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-0.437010| 1.000000| 0.200000| 0.050000|
+----+----+-+---------+---------+---------+---------+
Stage training time: 3.00
Number of used features: 4

Parent node: 6
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|  6|  7|
+---+---+---+---+---+---+---+---+

   0---1---2---3---4---5---6---7


Parent node: 7

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.00269655
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.825750| 1.000000| 1.000000| 0.087500|
+----+----+-+---------+---------+---------+---------+
|   2| 89%|+|-1.098274| 1.000000| 0.911111| 0.093750|
+----+----+-+---------+---------+---------+---------+
|   3| 99%|-|-0.387003| 1.000000| 0.222222| 0.050000|
+----+----+-+---------+---------+---------+---------+
Stage training time: 5.00
Number of used features: 6

Parent node: 7
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|  6|  7|  8|
+---+---+---+---+---+---+---+---+---+

   0---1---2---3---4---5---6---7---8


Parent node: 8

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.000656714
BACKGROUND PROCESSING TIME: 0.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.780975| 1.000000| 1.000000| 0.125000|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+|-1.143491| 1.000000| 0.866667| 0.125000|
+----+----+-+---------+---------+---------+---------+
|   3|100%|-|-1.267461| 1.000000| 0.355556| 0.037500|
+----+----+-+---------+---------+---------+---------+
Stage training time: 5.00
Number of used features: 6

Parent node: 8
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|  6|  7|  8|  9|
+---+---+---+---+---+---+---+---+---+---+

   0---1---2---3---4---5---6---7---8---9


Parent node: 9

*** 1 cluster ***
POS: 115 115 1.000000
NEG: 45 0.000245695
BACKGROUND PROCESSING TIME: 1.00
Precalculation time: 0.00
+----+----+-+---------+---------+---------+---------+
|  N |%SMP|F|  ST.THR |    HR   |    FA   | EXP. ERR|
+----+----+-+---------+---------+---------+---------+
|   1|100%|-|-0.982759| 1.000000| 1.000000| 0.006250|
+----+----+-+---------+---------+---------+---------+
|   2|100%|+| 0.017238| 1.000000| 0.000000| 0.000000|
+----+----+-+---------+---------+---------+---------+
Stage training time: 2.00
Number of used features: 4

Parent node: 9
Chosen number of splits: 0

Total number of splits: 0

Tree Classifier
Stage
+---+---+---+---+---+---+---+---+---+---+---+
|  0|  1|  2|  3|  4|  5|  6|  7|  8|  9| 10|
+---+---+---+---+---+---+---+---+---+---+---+

   0---1---2---3---4---5---6---7---8---9--10


Parent node: 10

*** 1 cluster ***
POS: 115 115 1.000000

All of this output was produced in the first 5 minutes of the run. After producing this output, it has proceeded to run with 100% of one core for 30 hours (so far) with no further output.

My question is: how can I tell if haartraining has crashed in this particular case, and more generally, does anyone know how to modify cvhaartraining.cpp so that it periodically outputs its status? Thanks a million!

(Related questions, both without answers:

• OpenCV 2.2 Haartraining Hanging
• OpenCV: How to know the progress status of HaarTraining

)

Answer 1

There's also similar thread on the OpenCV Yahoo tech group with a code by michael_p_horton for giving some additional feedback to determine whether the code entered an endless loop: tech.groups.yahoo.com/group/OpenCV/message/45080

To sum up the mentioned thread, there are two places where haartraining can go wild.

The first one is easy to catch by checking the output – you need the HR (Hit Rate) to increase and FA (False Alarm) to decrease. If this doesn't happen, the training entered an infinite loop.

However the issue you are experiencing, according to the maxenglander's answer, is an infinite loop in the icvGetHaarTrainingDataFromBG. To check for this you need to dig into the cvhaartraining.cpp code and add some debug output.

To quote the answer from Yahoo groups, you need to modify the icvGetHaarTrainingDataFromBG implementation in a following way (look for the cascade-eval() line and then add the CV_VERBOSE code):

icvGetAuxImages( &img, &sum, &tilted, &sqsum, normfactor );
if( cascade->eval( cascade, sumdata, tilteddata, *normfactor ) != 0.0F )
    break;

/* Display progress on negative image selection */
#ifdef CV_VERBOSE
if( thread_consumed_count % 1000 == 0 )
{
    fprintf( stderr, "%3d%%, %d negatives of %d required, %d images
    tested\r", (int) ( 100.0 * (i - first) / count ), (i-first), count,
    thread_consumed_count );
    fflush( stderr );
}
#endif /* CV_VERBOSE */

If this starts displaying messages like "0%, 0 negatives of 972 required, 10000000 images tested" you've entered an endless loop.

One final note – with the OpenCV 2.4 the relevant code is in the icvGetHaarTrainingData function.

Answer 2

From looking at haartraining/cvhaartraining.cpp, it looks like your process was at this line before you cancelled:

negcount = icvGetHaarTrainingDataFromBG( training_data, poscount, nneg,
    (CvIntHaarClassifier*) tcc, &false_alarm, bg_vecfile ? bgfilename : NULL );

You could drill down into functions that are called by icvGetHaarTrainingDataFromBG, add printf statements, and re-compile. I have been having my training runs hang as well (I'm using opencv_traincascade), and would like to see just where things are getting stuck. I may try sprinkling in printf statements into traincascade/traincascade.cpp to see why my runs are hanging.

Max