Neural networks, how do they look in coding?

Question

Basically i know the concept of a neural network and what it is, but i can't figure out how it looks when you code it or how do you store the data, i went through many tutorials that i found on google but couldn't find any piece of code, just concepts and algorithms.

Can anyone give me a piece of code of a simple neural network something like "Hello World!"?

Answer 1

What you mainly need is an object representing a single neuron with the corrispective associations with other neurons (that represent synapses) and their weights.

A single neuron in a typical OOP language will be something like

class Synapse
{
  Neuron sending;
  Neuron receiving;
  float weight;
}

class Neuron
{
  ArrayList<Synapse> toSynapses;
  ArrayList<Synapse> fromSynapses;

  Function threshold;
}  

where threshold represents the function that is applied on weighted sum of inputs to see if the neuron activates itself and propagates the signal.

Of course then you will need the specific algorithm to feed-forward the net or back-propagate the learning that will operate on this data structure.

The simplest thing you could start implement would be a simple perceptron, you can find some infos here.

Answer 2

You said you're already familiar with neural networks, but since there are many different types of neural networks of differing complexity (convolutional, hebbian, kohonen maps, etc.), I'll go over a simple Feed-forward neural network again, just to make sure we're on the same page.

A basic neural network consists of the following things

1. Neurons
   1. Input Neuron(s)
   2. Hidden Neurons (optional)
   3. Output Neuron(s)
2. Links between Neurons (sometimes called synapses in analogy to biology)
3. An activation function

The Neurons have an activation value. When you evaluate a network, the input nodes' activation is set to the actual input. The links from the input nodes lead to nodes closer to the output, usually to one or more layers of hidden nodes. At each neuron, the input activation is processed using an activation function. Different activation functions can be used, and sometimes they even vary within the neurons of a single network.

The activation function processes the activation of the neuron into it's output. The early experiments usually used a simple threshold function (i.e. activation > 0.5 ? 1 : 0), nowadays a Sigmoid function is often used.

The output of the activation function is then propagated over the links to the next nodes. Each link has an associated weight it applies to its input.

Finally, the output of the network is extracted from the activation of the output neuron(s).

I've put together a very simple (and very verbose...) example here. It's written in Ruby and computes AND, which is about as simple as it gets.

A much trickier question is how to actually create a network that does something useful. The trivial network of the example was created manually, but that is infeasible with more complex problems. There are two approaches I am aware of, with the most common being backpropagation. Less used is neuroevolution, where the weights of the links are determined using a genetic algorithm.