caffe data layer example step by step

Question

I want to find a caffe python data layer example to learn. I know that Fast-RCNN has a python data layer, but it's rather complicated since I am not familiar with object detection.
So my question is, is there a python data layer example where I can learn how to define my own data preparation procedure?
For example, how to do define a python data layer do much more data augmentation (such as translation, rotation etc.) than caffe "ImageDataLayer".

Thank you very much

Answer 1

@Shai's answer is great. At the same time, I find another detailed example about python data layer in one PR of caffe-master. https://github.com/BVLC/caffe/pull/3471/files I hope this detailed example be helpful for anyone else.

Answer 2

You can use a "Python" layer: a layer implemented in python to feed data into your net. (See an example for adding a type: "Python" layer here).

import sys, os
sys.path.insert(0, os.environ['CAFFE_ROOT']+'/python')
import caffe
class myInputLayer(caffe.Layer):
  def setup(self,bottom,top):
    # read parameters from `self.param_str`
    ...
  def reshape(self,bottom,top):
    # no "bottom"s for input layer
    if len(bottom)>0:
      raise Exception('cannot have bottoms for input layer')
    # make sure you have the right number of "top"s
    if len(top)!= ...
       raise ...
    top[0].reshape( ... ) # reshape the outputs to the proper sizes
    
  def forward(self,bottom,top): 
    # do your magic here... feed **one** batch to `top`
    top[0].data[...] = one_batch_of_data


  def backward(self, top, propagate_down, bottom):
    # no back-prop for input layers
    pass

For more information on param_str see this thread.
You can find a sketch of a data loading layer with pre-fetch here.