I have a simulator that runs through three states:
What the heck! During standard testing, CPU usage dropped from 100% down to 20%, and the total run took about 30 times longer than normal (130secs vs 4.2secs).
When Callgrind
revealed nothing suspicious, my head buzzed as I was on the precipice of rolling back to the last commit, losing all bug-fixes.
Discouraged, I walked into the server room during a run and noticed nasty grinding sounds, verified later to be caused by writes to Mysql sockets in /proc/PID/fd!!! It turned out that Mysql code, several layers deep in Stage 2., was causing problems.
Lessons Learned
Fix I will introduce thread-local-storage IOSentinels and asserts() on ReadAllowed() and WriteAllowed() to ensure that Stage 2 threads will never do any IO.
Anyone have any luck with attaching/writing a benchmarking framework with googletest?
Unfortunately, all my googletests passed this time. Had I stepped away for a bit and come back without noticing the run-time, this would have been a disastrous commit, and possibly much harder to fix.
I would like googletest to fail if a run takes >2 or 3 times the last runtime: this last part is tricky because for very quick runs, system state can cause something to take twice as long but still be ok. But for a long simulation run/test, I don't expect runtimes to change by a great deal (>50% would be unusual).
I am open to suggestions here, but it would be nice to have a low-maintenance check that would work with automated testing so it will be obvious if the system suddenly got slow, even if all the outputs appear to be ok.
Benchmarking is a process of measuring the performance of a company's products, services, or processes against those of another business considered to be the best in the industry, aka “best in class.” The point of benchmarking is to identify internal opportunities for improvement.
A Benchmark in Performance Testing is a metric or a point of reference against which software products or services can be compared to assess the quality measures. In other words, Benchmark means a set standard that helps to determine the quality of software product or service.
Right click on project and go to properties. Go to Configuration Properties->Debugging. In Command Arguments add --gtest_output="xml:\home\user\XML_Report. xml"
Some updates on this question (in 2016):
Here is a nice blog-post of Nick Brunn about his Hayai benchmarking framework. (2012)
Google published 'Benchmark' in 2014. This provides similar behaviour then Hayai above. As far as I understand, defining requirements is not possible. Again, the syntax is inspired by GoogleTest.
GoogleTest has this as an open feature on Github. There is a rudimentary implementation but it is not part of GoogleTest yet.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With