Which algorithms have been proposed to learn the architecture of a deep neural network?

Question

Yoshua Benhgio's Learning Deep Architectures for AI book mentions that

we should [...] strive to develop learning algorithms that use the data to determine the depth of the final architecture.

Would anyone know of any algorithms proposed thus far to achieve this?

This question is not about successful algorithms - in fact it seems there are none at the moment. The aim of this question is to aggregate every single algorithm that has ever been proposed so that anyone interested in the topic doesn't need to spend months finding them.

So far I have come across:

---

• The tiling algorithm
  • advantage: adds layers as well as units
  • caveat: only for learning Boolean functions, which aren't too relevant for applied problems.

---

• Genetic algorithms (courtesy of user vzn):
  • Finding Optimal Neural Network Architecture Using Genetic Algorithms
  • Using genetic algorithms to select architecture of a feedforward artificial neural network
  • advantage: learns continuous functions, ie relevant for applied problems
  • caveat: computationally very expensive

Answer 1

Arguably, the most used algorithm to learn the architecture of a neural network is Cascade Correlation, developed by Scott Fahlman and Christian Lebiere.

Here you can find the description, I also saw a C implementation somewhere before, but couldn't remember where.