Menu
I've been working with convnetjs for 1 year and now I want to move on to more powerful and fast libraries. I thought Tensorflow would be orders of magnitude faster than a JS library, so I wrote a simple neural network for both libraries and did some tests. It is a 3-5-5-1 neural network, trained on one single example for a certain number of epochs with SGD and RELU layers.
Tensorflow code:
import tensorflow as tf import numpy import time NUM_CORES = 1 # Choose how many cores to use. sess = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=NUM_CORES, intra_op_parallelism_threads=NUM_CORES)) # Parameters learning_rate = 0.001 training_epochs = 1000 batch_size = 1 # Network Parameters n_input = 3 # Data input n_hidden_1 = 5 # 1st layer num features n_hidden_2 = 5 # 2nd layer num features n_output = 1 # Data output # tf Graph input x = tf.placeholder("float", [None, n_input], "a") y = tf.placeholder("float", [None, n_output], "b") # Create model def multilayer_perceptron(_X, _weights, _biases): layer_1 = tf.nn.relu(tf.add(tf.matmul(_X, _weights['h1']), _biases['b1'])) #Hidden layer with RELU activation layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, _weights['h2']), _biases['b2'])) #Hidden layer with RELU activation return tf.matmul(layer_2, _weights['out']) + _biases['out'] # Store layers weight & bias weights = { 'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])), 'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])), 'out': tf.Variable(tf.random_normal([n_hidden_2, n_output])) } biases = { 'b1': tf.Variable(tf.random_normal([n_hidden_1])), 'b2': tf.Variable(tf.random_normal([n_hidden_2])), 'out': tf.Variable(tf.random_normal([n_output])) } # Construct model pred = multilayer_perceptron(x, weights, biases) # Define loss and optimizer cost = tf.reduce_sum(tf.nn.l2_loss(pred-y)) / batch_size # L2 loss optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost) # Adam Optimizer # Initializing the variables init = tf.initialize_all_variables() # Launch the graph sess.run(init) # Training Data train_X = numpy.asarray([[0.1,0.2,0.3]]) train_Y = numpy.asarray([[0.5]]) # Training cycle start = time.clock() for epoch in range(training_epochs): # Fit training using batch data sess.run(optimizer, feed_dict={x: train_X, y: train_Y}) end = time.clock() print end - start #2.5 seconds -> 400 epochs per second print "Optimization Finished!" JS code:
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8" /> <title>Regression example convnetjs</title> <script src="http://cs.stanford.edu/people/karpathy/convnetjs/build/convnet.js"></script> <script src="http://cs.stanford.edu/people/karpathy/convnetjs/build/util.js"></script> <script> var layer_defs, net, trainer; function start() { layer_defs = []; layer_defs.push({ type: 'input', out_sx: 1, out_sy: 1, out_depth: 3 }); layer_defs.push({ type: 'fc', num_neurons: 5, activation: 'relu' }); layer_defs.push({ type: 'fc', num_neurons: 5, activation: 'relu' }); layer_defs.push({ type: 'regression', num_neurons: 1 }); net = new convnetjs.Net(); net.makeLayers(layer_defs); trainer = new convnetjs.SGDTrainer(net, { learning_rate: 0.001, method: 'sgd', batch_size: 1, l2_decay: 0.001, l1_decay: 0.001 }); var start = performance.now(); for(var i = 0; i < 100000; i++) { var x = new convnetjs.Vol([0.1, 0.2, 0.3]); trainer.train(x, [0.5]); } var end = performance.now(); console.log(end-start); //3 seconds -> 33333 epochs per second var predicted_values = net.forward(x); console.log(predicted_values.w[0]); } </script> </head> <body> <button onclick="start()">Start</button> </body> </html> The results are that convnetjs trains for 100'000 epochs in 3 seconds, while Tensorflow trains for 1000 epochs in 2.5 seconds. Is this expected?
897
There could be many reasons why:
The data input is so small that most of the time is spent in just conversion between python and the C++ core, while the JS is just one language.
You are using only one core in Tensorflow while the JS could potentially leverage more than one
the JS library is able to create a highly optimized JIT version of the program.
The real benefits of Tensorflow will come when the distributed version will be public. Then the ability to run big networks on many nodes will be more important than the speed of a single node.
162
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
