OpenGL: create complex and smoothed polygons

Question

In my OpenGL project, I want to dynamically create smoothed polygons, similiar like this one:

The problem relies mainly in the smoothing process. My procedure up to this point, is firstly to create a VBO with randomly placed vertices.

Then, in my fragment shader, (I'm using the programmable function pipeline) there should happen the smoothing process, or in other words, created the curves out of the previously defined "lines" between the vertices.

And exactly here is the problem: I am not very familiar with thoose complex mathematical algorithms, which would examine, if a point is inside the "smoothed polygon" or not.

Answer 1

First up, you can't really do it in the fragment shader. The fragment shader is limited to setting the final(ish) color of a "pixel" (which is basically, but not exactly, an actual pixel) before it gets written to the screen. It can't create new points on a curve.

This page gives a nice overview of the different algorithms for creating smooth curves.

The general approach is to break a couple of points into multiple points using a geometry shader, and then render them just like a normal polygon. But I don't know the details. Try a google search for bezier geometry shader for example.

Wait, I lie. I found a program here that does it in the fragment shader.