Caffe training without testing

Question

I am using Caffe to train AlexNet on a known image database. I am benchmarking and want to exclude a testing phase.

Here is the solver.prototxt for AlexNet:

net: "models/bvlc_alexnet/train_val.prototxt"
test_iter: 1000
test_interval: 1000
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
stepsize: 100000
display: 20
max_iter: 450000
momentum: 0.9
weight_decay: 0.0005
snapshot: 10000
snapshot_prefix: "models/bvlc_alexnet/caffe_alexnet_train"
solver_mode: GPU

While I have never found a definitive doc that detailed all of the prototxt options, comments within Caffe tutorials indicate this "test_interval" represents the number of iterations after which we test the trained network.

I figured that I might set it to zero to turn off testing. Nope.

F1124 14:42:54.691428 18772 solver.cpp:140] Check failed: param_.test_interval() > 0 (0 vs. 0)
*** Check failure stack trace: ***

So I set the test_interval to one million, but still of course, Caffe tests the network at iteration zero.

I1124 14:59:12.787899 18905 solver.cpp:340] Iteration 0, Testing net (#0)
I1124 14:59:15.698724 18905 solver.cpp:408]     Test net output #0: accuracy = 0.003

How do I turn testing off while training?

Answer 1

Caffe's documentation is somewhat scant on details. What I was finally told is this counterintuitive solution:

In your solver.prototxt, take the lines for test_iter and test_interval

test_iter: 1000
test_interval: 1000

and simply omit them. If you'd like to prevent the test at the beginning, you would add a line as @shai suggested:

test_initialization: false

Answer 2

You have a flag for that too. Add

test_initialization: false

To your 'solver.prototxt' and you are done ;)