Supervised learning(document classification) using deep learning techniques

Question

I was reading the papers on deep learning. Most of them refer to unsupervised learning.

They also say the neurons are pre-trained using unsupervised RBM network. Later they are fine tuned using Back propagation algorithm (supervised).

So can we solve supervised learning problems using deep learning??

I am trying to find out if deep learning can be applied for document classification problem. I know there are pretty good classifiers available. But my goal is to find out whether we can use deep learning for this purpose or not.

Answer 1

In short - yes, it can be, and is often used in supervised manner. Exactly as Ben J described - the "depth" of the architecture is used in the unsupervised manner in order to create very abstract representation of your data, which can be later on used as either preprocessing layer (no fine-tuning) or as an initialization for the neural network (with fine-tuning, Hinton-like). In particular you can use such approach for texts.

There is interesting very recent paper by Hinton regarding modeling the text documents with DBMs: http://www.cs.toronto.edu/~rsalakhu/papers/uai13.pdf

There are many resources available online, in particular pylearn library implements such approaches, including "classic" DBN (Deep Belief Network) http://deeplearning.net/tutorial/DBN.html

Furthermore, it is also possible to actually make classification using just stacked RBMs, this model is called "classification RBM". More details can be obtained here: http://machinelearning.org/archive/icml2008/papers/601.pdf