HDF5-DIAG: Error detected in HDF5 (1.8.11)

Question

I am trying to load hdf5 in caffe, it is not working. I checked the paths and even able to view hdf file using viewer. Everything is ok but caffe cant seem to load.

I write the hdf5 using a python script like this, where X and labels are numpy arrays.

f = h5py.File("facialkp.hd5", "w")
f.create_dataset("data", data=X,  compression="gzip", compression_opts=4)
f.create_dataset("label", data=labels,  compression="gzip", compression_opts=4)

Here is the entire problem.

root@pbu-OptiPlex-740-Enhanced:/home/pbu/Desktop# ./facialkp_train.sh
I0119 19:55:50.779419  9905 caffe.cpp:103] Use CPU.
I0119 19:55:51.152626  9905 caffe.cpp:107] Starting Optimization
I0119 19:55:51.152817  9905 solver.cpp:32] Initializing solver from parameters: 
test_iter: 1000
test_interval: 1000
base_lr: 0.01
display: 1000
max_iter: 10000
lr_policy: "step"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 5000
snapshot: 10000
snapshot_prefix: "/home/pbu/Desktop/tmp"
solver_mode: CPU
net: "/home/pbu/Desktop/facialkp.prototxt"
I0119 19:55:51.152936  9905 solver.cpp:67] Creating training net from net file: /home/pbu/Desktop/facialkp.prototxt
I0119 19:55:51.153211  9905 net.cpp:39] Initializing net from parameters: 
name: "LogReg"
layers {
  top: "data"
  top: "label"
  name: "data"
  type: HDF5_DATA
  hdf5_data_param {
    source: "facialkp.hd5"
    batch_size: 10
  }
  include {
    phase: TRAIN
  }
}
layers {
  bottom: "data"
  top: "ip"
  name: "ip"
  type: INNER_PRODUCT
  inner_product_param {
    num_output: 30
  }
}
layers {
  bottom: "ip"
  bottom: "label"
  top: "loss"
  name: "loss"
  type: EUCLIDEAN_LOSS
}
state {
  phase: TRAIN
}
I0119 19:55:51.153457  9905 net.cpp:67] Creating Layer data
I0119 19:55:51.153486  9905 net.cpp:356] data -> data
I0119 19:55:51.153524  9905 net.cpp:356] data -> label
I0119 19:55:51.153560  9905 net.cpp:96] Setting up data
I0119 19:55:51.153580  9905 hdf5_data_layer.cpp:57] Loading filename from facialkp.hd5
I0119 19:55:51.171277  9905 hdf5_data_layer.cpp:69] Number of files: 19511
I0119 19:55:51.171358  9905 hdf5_data_layer.cpp:29] Loading HDF5 file�HDF
HDF5-DIAG: Error detected in HDF5 (1.8.11) thread 139901797366336:
  #000: ../../../src/H5F.c line 1586 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: ../../../src/H5F.c line 1275 in H5F_open(): unable to open file: time = Mon Jan 19 19:55:51 2015
, name = '�HDF', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #002: ../../../src/H5FD.c line 987 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: ../../../src/H5FDsec2.c line 343 in H5FD_sec2_open(): unable to open file: name = '�HDF', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibilty
    minor: Unable to open file
E0119 19:55:51.172713  9905 hdf5_data_layer.cpp:32] Failed opening HDF5 file�HDF
I0119 19:55:51.173208  9905 hdf5_data_layer.cpp:81] output data size: 10,0,0,0
I0119 19:55:51.173262  9905 net.cpp:103] Top shape: 10 0 0 0 (0)
I0119 19:55:51.173285  9905 net.cpp:103] Top shape: 10 0 0 0 (0)
I0119 19:55:51.173342  9905 net.cpp:67] Creating Layer ip
I0119 19:55:51.173368  9905 net.cpp:394] ip <- data
I0119 19:55:51.173400  9905 net.cpp:356] ip -> ip
I0119 19:55:51.173432  9905 net.cpp:96] Setting up ip
F0119 19:55:51.173466  9905 blob.cpp:72] Check failed: data_ 
*** Check failure stack trace: ***
    @     0x7f3d674c8daa  (unknown)
    @     0x7f3d674c8ce4  (unknown)
    @     0x7f3d674c86e6  (unknown)
    @     0x7f3d674cb687  (unknown)
    @           0x45ee6e  caffe::Blob<>::mutable_cpu_data()
    @           0x4b4d74  caffe::ConstantFiller<>::Fill()
    @           0x4bb6c9  caffe::InnerProductLayer<>::LayerSetUp()
    @           0x491431  caffe::Net<>::Init()
    @           0x492d3e  caffe::Net<>::Net()
    @           0x458b00  caffe::Solver<>::InitTrainNet()
    @           0x459db6  caffe::Solver<>::Init()
    @           0x459f16  caffe::Solver<>::Solver()
    @           0x419c40  caffe::GetSolver<>()
    @           0x416064  train()
    @           0x410a51  main
    @     0x7f3d630c8ec5  (unknown)
    @           0x414bb7  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)
root@pbu-OptiPlex-740-En

Answer 1

Solved :)

i created a text file, placed the path of real .hd5 file inside it. The caffe prototxt file points to text file and it worked :)

 hdf5_data_param {
   source: "train.txt"
   batch_size: 10
  }

train.txt contains line..

facialkp.hd5