what is the fastest way to calculate sigmoid?

Question

I am trying to implement a recurrent neural network where sigmoid is chosen to be the activation function.

My first prototype is written in python and I find sigmoid is somehow the bottleneck of the program, accounts for ~30% of the total running time.

# x is a fixed size vector here
def sigmoid(x):
    return numpy.reciprocal(1.0 + numpy.exp(-x))

So I tried another implementation

def sigmoid(x):
    y = numpy.exp(x)
    return y/(1+y)

and surprisingly found it is 50% faster than the first approach.

I also tried a third approach

def sigmoid(x):
    return (1.0+numpy.tanh(x/2.0))/2.0

, which is slightly slower than the first approach.

Later I tested all 3 implementations in C++. The first two approach has barely any difference, and tanh is slightly (~5%) faster. Why does this happen? I am thinking numpy was written in C++.

Answer 1

Based on this it seems your best bet is scipy.special.expit