WebGL Benchmark - What kind of tests should I create?

Question

(I am not sure if this should be on https://softwareengineering.stackexchange.com/ or not, please comment if you think so)

I am about to create a benchmark for WebGL implementations for my bachelor thesis. I am unsure what kind of tests I should create and if I should only measure the frames per seconds of if I could get some other useful data to benchmark...

Currently I just thought about tests like that:

• 1 single colored object
• 1 multi colored object
• 1 textured object
• 1 textured object with blending
• 1 textured object with lightning
• 1 textured object with multiple lighting points
• 1 scene rendered to texture and use it on another object
• 1 model animation
• all tests with more objects: 50, 500, 5000
• changing shaders/programs per rendering (once, twice, multiple times)

This would result in 40 different tests, but I am not sure if these are significant tests for the performance.

A friend suggested to test on complex shader but as the shader are running on the graphic hardware there shouldn't be a difference to a java desktop application, right?

Anyway my feeling is, that the performance of the JavaScript is the main bottleneck in WebGL.

Update

I finally did my test. After a discussion about the way the benchmark should work, I created the following: http://martin.cyor.eu/benchmark

Answer 1

I agree with @timday that you should bias your investigation toward something "real", and as you suggested in a comment you might want to the story to be about deciding between a desktop or browser based app.

This is exactly what I'm working on right now. My client has a visualization application that currently runs on the Windows desktop. A typical scene for them has 500,000 triangles, lots of textures, and transparency. Currently their users are disinclined to install the viewer - they tend to work in corporate environments where sysadmins control what is installed on their computers. And several users would prefer to run the visualization on their iPads, where the viewer would not run anyways. So my client wants to know whether WebGL would solve their platform issues - nevermind that no browser yet officially supports WebGL, and that neither IE nor iPad have announced any kind of support.

Bear in mind that any benchmarks you do are relatively meaningless, because you are measuring a moving target. Browser makers are working hard to implement WebGL, and they are updating their beta releases frequently. Not only are they working on implementing WebGL conformantly, but they have to worry about browser security issues and overall pipeline flow. This video talks about some of the issues (and give you an idea of what to look into). As well, performance may vary depending on your OS and your graphics hardware.

As you pointed out, once WebGL is running in the graphics hardware it should run as fast as a desktop app. Your benchmarks should try to confirm that, and then you should try to measure where performance is lost as a result of being in a browser. My feeling is that Javascript per se is not the bottleneck, just because there's not that much Javascript to execute (and it's pretty fast these days). However, as described towards the end of the aforementioned video, there may be inefficiencies that arise in the Javascript-C++ binding, request validation, flow control and what not. On the other hand, the browser makers (at least Google) are working hard on ironing out those kinks.

One of the things I've noticed is not not framerate/performance problems (in my current test, I can render 500,000 textured triangles at 30fps), but that framerates don't seem to be very consistent, and that frames seem to be dropped from time to time. I suspect, but don't know whether this has to do with the relatively simple setInterval() way or running animations in Javascript. (Mozilla's mozRequestAnimationFrame might be a way of better handling this).

Although I don't know how helpful any of the above is for your thesis, it seems to me you have a rich topic and you should do more than concoct simple benchmarks. Maybe you should start off on some benchmarks, compare browser vs. desktop performance, and then try to examine best practices for not only deciding between browser and desktop, but also for writing WebGL apps.

There are quite a few WebGL frameworks out there as well. I tried a couple, and was very impressed -- there is a lot to learn from them. Depending on your interests and thesis requirements, you may be interested in benchmarking these as well.

Whatever way you go, I suspect that there is a large community of would-be WebGL adopters that will be starving for the kind of information you're going to be researching.