I've been profiling an application all day long and, having optimized a couple bits of code, I'm left with this on my todo list. It's the activation function for a neural network, which gets called over a 100 million times. According to dotTrace, it amounts to about 60% of the overall function time.
How would you optimize this?
public static float Sigmoid(double value) { return (float) (1.0 / (1.0 + Math.Pow(Math.E, -value))); }
Optimization is a program transformation technique, which tries to improve the code by making it consume less resources (i.e. CPU, Memory) and deliver high speed. In optimization, high-level general programming constructs are replaced by very efficient low-level programming codes.
Mathematical optimization is the process of maximizing or minimizing an objective function by finding the best available values across a set of inputs. Some variation of optimization is required for all deep learning models to function, whether using supervised or unsupervised learning.
Try:
public static float Sigmoid(double value) { return 1.0f / (1.0f + (float) Math.Exp(-value)); }
EDIT: I did a quick benchmark. On my machine, the above code is about 43% faster than your method, and this mathematically-equivalent code is the teeniest bit faster (46% faster than the original):
public static float Sigmoid(double value) { float k = Math.Exp(value); return k / (1.0f + k); }
EDIT 2: I'm not sure how much overhead C# functions have, but if you #include <math.h>
in your source code, you should be able to use this, which uses a float-exp function. It might be a little faster.
public static float Sigmoid(double value) { float k = expf((float) value); return k / (1.0f + k); }
Also if you're doing millions of calls, the function-calling overhead might be a problem. Try making an inline function and see if that's any help.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With