OpenGL GLSL Cel Shading and Outline algorithm

Question

I have successfully implemented a cel shader using glsl. But my problem is about outlining everything and every example of opengl on the internet is pretty old, doesn't match current technology, so slow techniques etc... So I couldn't find anyway to do it in glsl. I thought about some ideas like colorizing least intensive areas black but there is not even enough explanation about gl_Vertex, gl_Position and eye positions. So, since every technique is too old, I have finally decided to ask someone about this outline problem.

Answer 1

I would use some kind of edge detection filter on the zbuffer (could potentially be combined with edge detection on the color buffer) and then modulate the framebuffer with the (inverted) result. One of the most common edge detection filters is the sobel operator:

http://en.wikipedia.org/wiki/Sobel_operator

EDIT: Another cheap way of doing this is to before you render the ordinary mesh draw a black copy of the mesh with cull facing inverted and the vertices slightly moved in the direction of the normal. This is probably what they did in XIII, I'm also pretty sure this is how it was done in Jet Set Radio.

EDIT2: If you want the thickness of the outline to be constant regardless of the distance to the viewer, you should scale the amount you move the vertices with this distance.

Answer 2

Have you looked at glPolygonOffset?

I've used it to draw outlines on top of filled polygons.

http://www.khronos.org/opengles/sdk/docs/man/xhtml/glPolygonOffset.xml