Does using more memory decrease performance?

Question

Particularly does memoization decrease performance somewhat? Is performance increase linear?

I have a function that calls some complex math function 200,000,000 times. Without memoization (saving values/caching) it takes 1m. If I save the values-about 5,000,000 unique entries-it still takes 30s. The values are doubles, I am using my own hash function and the hash table size is about 20,000,000 (to make calculating the hash values a little bit easier).

But the complex math function is still only being run 5,000,000 times (I even checked with a counter). Why is it not running at roughly 2.5% of the speed 5,000,000/200,000,000?

Before I was using no large data structures, and now I am using a double array of size 20,000,000 just to clarify. I don't know if that'll make a difference.

Answer 1

The answer to all performance questions is "benchmark it and find out". Always. So your real-world runtime results are your answer - empirically, that's how it behaves. You can find out the why using something like valgrind's callgrind/cachegrind tools.

Now, I'm going to ignore the fact you talk about hashing - I think you're aware that if the cost of the hash computation is a significant fraction of the cost of running the function body, memoization isn't going to help. So imagine the hash has zero cost for the purposes of the below.

That all said, one of the biggest factors in the performance of CPU-intensive code is the cache hit rate. This is whether your processor, when it looks for information, has to go out to RAM to fetch it; if it's already hot in the cache, the access latency is thousands of times lower and the CPU gets its work done more quickly (I'm simplifying a bit, because not all memory hits cause a pipeline stall, but this is the gist of it).

So, although "using more memory" doesn't directly correlate to a decrease in performance (I mean, the act of using it does, but I'm assuming you're not talking about how much it costs to allocate objects here), when you have a wider swath of RAM in which things you need could lie, the odds of getting a cache hit are lower and that can severely lower the runtime speed of your code.

Memoization is only a win when your function takes a nontrivial amount of time to execute. That's usually the case, but it is a trade-off, and even under ideal circumstances, memoizing ten function calls down to five will never give you the 50% theoretical speed-up.

This counterintuitive behavior ("but Borealid, I'm doing more work, how can it be faster?!") is a prime example of why you should always double-check to see that an "optimization" you put in place actually boosts performance. Premature optimization is the root of all evil.

Answer 2

More memory usage will often significantly decrease performance - the main issue comes from the fact that large data structures will not fit in your processor cache and will take much longer to access.

On modern processors, you will often find that redoing a maths calculation from scratch is actually much quicker than getting a result from memory. Memory access is basically the bottleneck, not CPU.

Also if you are memoizing values then be aware of the potential overhead of hashing and lookup functions. In particular, if you don't implement your hash function correctly and have lots of hash collisions, then you could be suffering from very expensive lookups.