glLineStipple deprecated in OpenGL 3.1

Question

glLineStipple has been deprecated in the latest OpenGL APIs. What is it replaced with? If not replaced, how can I get a similar effect? (I don't want to use a compatibility profile of course...)

Answer 1

Sorry, it hasn't been replaced with anything. The first idea coming to my mind for emulating it would be the geometry shader. You feed the geometry shader with a line, compute its screen space length and based on that you generate a variable number of sub lines between its start and end vertex.

---

EDIT: Perhaps you could also use a 1D texture with the alpha (or red) channel encoding the pattern as 0.0 (no line) or 1.0 (line) and then have the lines texture coordinate go from 0 to 1 and in the fragment chader you make a simple alpha test, discarding fragments with alpha below some threshold. You can facilitate the geometry shader to generate your line texCoords, as otherwise you need different vertices for every line. This way you can also make the texCoord dependent on the screen space length of the line.

The whole thing get's more difficult if you draw triangles (using polygon mode GL_LINE). Then you have to do the triangle-line transformation yourself in the geometry shader, putting in triangles and putting out lines (that could also be a reason for deprecating polygon mode in the future, if it hasn't already).

---

EDIT: Although I believe this question abandomned, I have made a simple shader triple for the second approach. It's just a minimal solution, feel free to add custom features yourself. I haven't tested it because I lack the neccessary hardware, but you should get the point:

uniform mat4 modelViewProj;

layout(location=0) in vec4 vertex;

void main()
{
    gl_Position = modelViewProj * vertex;
}

The vertex shader is a simple pass through.

layout(lines) in;
layout(line_strip, max_vertices=2) out;

uniform vec2 screenSize;
uniform float patternSize;

noperspective out float texCoord;

void main()
{
    vec2 winPos0 = screenSize.xy * gl_in[0].gl_Position.xy / gl_in[0].gl_Position.w;
    vec2 winPos1 = screenSize.xy * gl_in[1].gl_Position.xy / gl_in[1].gl_Position.w;
    gl_Position = gl_in[0].gl_Position;
    texCoord = 0.0;
    EmitVertex();
    gl_Position = gl_in[1].gl_Position;
    texCoord = 0.5 * length(winPos1-winPos0) / patternSize;
    EmitVertex();
}

In the geometry shader we take a line and compute its screen space length in pixels. We then devide this by the size of the stipple pattern texture, which would be factor*16 when emulating a call to glLineStipple(factor, pattern). This is taken as 1D texture coordinate of the second line end point.

Note that this texture coordinate has to be interpolated linearly (noperspective interpolation specifier). The usual perpective-correct interpolation would cause the stipple pattern to "squeeze together" on farther away parts of the line, whereas we are explicitly working with screen-space values.

uniform sampler1D pattern;
uniform vec4 lineColor;

noperspective in float texCoord;

layout(location=0) out vec4 color;

void main()
{
    if(texture(pattern, texCoord).r < 0.5)
        discard;
    color = lineColor;
}

The fragment shader now just performs a simple alpha test using the value from the pattern texture, which contains a 1 for line and a 0 for no line. So to emulate the fixed function stipple you would have a 16 pixel 1-component 1D texture instead of a 16bit pattern. Don't forget to set the pattern's wrapping mode to GL_REPEAT, about the filtering mode I'm not that sure, but I suppose GL_NEAREST would be a good idea.

But as said earlier, if you want to render triangles using glPolygonMode, it won't work this way. Instead you have to adapt the geometry shader to accept triangles and generate 3 lines for each triangle.

---

EDIT: In fact OpenGL 3's direct support for integer operations in shaders allows us to completely drop this whole 1D-texture approach and work straight-forward with an actual bit-pattern. Thus the geometry shader is slightly changed to put out the actual screen-size pattern coordinate, without normalization:

texCoord = 0.5 * length(winPos1-winPos0);

In the fragment shader we then just take a bit pattern as unsigned integer (though 32-bit in contrast to glLineStipple's 16-bit value) and the stretch factor of the pattern and just take the texture coordinate (well, no texture anymore actually, but nevermind) modulo 32 to get it's position on the pattern (those explicit uints are annoying, but my GLSL compiler says implicit conversions between int and uint are evil):

uniform uint pattern;
uniform float factor;

...
uint bit = uint(round(linePos/factor)) & 31U;
if((pattern & (1U<<bit)) == 0U)
    discard;

Answer 2

To answer this question, we've to investigate first, what glLineStipple actually does.

See the image, where the quad at the left is drawn by 4 separated line segments using the primitive type GL_LINES.
The circle at the right is drawn by a consecutive polygon line, using the primitive type GL_LINE_STRIP.

When using line segments, the stipple pattern started at each segment. The pattern is restarted at each primitive.
When using a line strip, then the stipple pattern is applied seamless to the entire polygon. A pattern seamlessly continuous beyond vertex coordinates.
Be aware that the length of the pattern is stretched at the diagonals. This is possibly the key to the implementation.

For separate line segments, this is not very complicated at all, but for line strips things get a bit more complicated. The length of the line cannot be calculated in the shader program, without knowing all the primitives of the line. Even if all the primitives would be known (e.g. SSBO), then the calculation would have to be done in a loop.
See also Dashed lines with OpenGL core profile.

Anyway, it is not necessary to implement a geometry shader. The trick is to know the start of the line segment in the fragment shader. This easy by using a flat interpolation qualifier.

The vertex shader has to pass the normalized device coordinate to the fragment shader. Once with default interpolation and once with no (flat) interpolation. This causes that in the fragment shade, the first input parameter contains the NDC coordinate of the actual position on the line and the later the NDC coordinate of the start of the line.

#version 330

layout (location = 0) in vec3 inPos;

flat out vec3 startPos;
out vec3 vertPos;

uniform mat4 u_mvp;

void main()
{
    vec4 pos    = u_mvp * vec4(inPos, 1.0);
    gl_Position = pos;
    vertPos     = pos.xyz / pos.w;
    startPos    = vertPos;
}

Additionally the varying inputs, the fragment shader has uniform variables. u_resolution contains the width and the height of the viewport. u_factor and u_pattern are the multiplier and the 16 bit pattern according to the parameters of glLineStipple.

So the length of the line from the start to the actual fragment can be calculated:

vec2  dir  = (vertPos.xy-startPos.xy) * u_resolution/2.0;
float dist = length(dir);

And fragment on the gap can be discarded, by the discard command.

uint bit = uint(round(dist / u_factor)) & 15U;
if ((u_pattern & (1U<<bit)) == 0U)
    discard; 

Fragment shader:

#version 330

flat in vec3 startPos;
in vec3 vertPos;

out vec4 fragColor;

uniform vec2  u_resolution;
uniform uint  u_pattern;
uniform float u_factor;

void main()
{
    vec2  dir  = (vertPos.xy-startPos.xy) * u_resolution/2.0;
    float dist = length(dir);

    uint bit = uint(round(dist / u_factor)) & 15U;
    if ((u_pattern & (1U<<bit)) == 0U)
        discard; 
    fragColor = vec4(1.0);
}

This implementation is much easier and shorter, then using geometry shaders. The flat interpolation qualifier is supported since GLSL 1.30 and GLSL ES 3.00. In this version geometry shaders are not supported.
See the line rendering which was generated with the above shader.

The shader gives a proper result line segments, but fails for line strips, since the stipple pattern is restarted at each vertex coordinate.
The issue can't even be solved by a geometry shader. This part of the question remains still unresolved.

For the following simple demo program I've used the GLFW API for creating a window, GLEW for loading OpenGL and GLM -OpenGL Mathematics for the math. I don't provide the code for the function CreateProgram, which just creates a program object, from the vertex shader and fragment shader source code:

#include <vector>
#include <string>
#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>
#include <glm/gtc/type_ptr.hpp>
#include <gl/gl_glew.h>
#include <GLFW/glfw3.h>

std::string vertShader = R"(
#version 330

layout (location = 0) in vec3 inPos;

flat out vec3 startPos;
out vec3 vertPos;

uniform mat4 u_mvp;

void main()
{
    vec4 pos    = u_mvp * vec4(inPos, 1.0);
    gl_Position = pos;
    vertPos     = pos.xyz / pos.w;
    startPos    = vertPos;
}
)";

std::string fragShader = R"(
#version 330

flat in vec3 startPos;
in vec3 vertPos;

out vec4 fragColor;

uniform vec2  u_resolution;
uniform uint  u_pattern;
uniform float u_factor;

void main()
{
    vec2  dir  = (vertPos.xy-startPos.xy) * u_resolution/2.0;
    float dist = length(dir);

    uint bit = uint(round(dist / u_factor)) & 15U;
    if ((u_pattern & (1U<<bit)) == 0U)
        discard; 
    fragColor = vec4(1.0);
}
)";

GLuint CreateVAO(std::vector<glm::vec3> &varray)
{
    GLuint bo[2], vao;
    glGenBuffers(2, bo);
    glGenVertexArrays(1, &vao);
    glBindVertexArray(vao);
    glEnableVertexAttribArray(0); 
    glBindBuffer(GL_ARRAY_BUFFER, bo[0] );
    glBufferData(GL_ARRAY_BUFFER, varray.size()*sizeof(*varray.data()), varray.data(), GL_STATIC_DRAW);
    glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 0, 0); 

    return vao;
}

int main(void)
{
    if ( glfwInit() == 0 )
        return 0;
    GLFWwindow *window = glfwCreateWindow( 800, 600, "GLFW OGL window", nullptr, nullptr );
    if ( window == nullptr )
        return 0;
    glfwMakeContextCurrent(window);

    glewExperimental = true;
    if ( glewInit() != GLEW_OK )
        return 0;

    GLuint program    = CreateProgram(vertShader, fragShader);
    GLint loc_mvp     = glGetUniformLocation(program, "u_mvp");
    GLint loc_res     = glGetUniformLocation(program, "u_resolution");
    GLint loc_pattern = glGetUniformLocation(program, "u_pattern");
    GLint loc_factor  = glGetUniformLocation(program, "u_factor");

    glUseProgram(program);

    GLushort pattern = 0x18ff;
    GLfloat  factor  = 2.0f;
    glUniform1ui(loc_pattern, pattern);
    glUniform1f(loc_factor, factor);
    //glLineStipple(2.0, pattern);
    //glEnable(GL_LINE_STIPPLE);

    glm::vec3 p0(-1.0f, -1.0f, 0.0f);
    glm::vec3 p1(1.0f, -1.0f, 0.0f);
    glm::vec3 p2(1.0f, 1.0f, 0.0f);
    glm::vec3 p3(-1.0f, 1.0f, 0.0f);
    std::vector<glm::vec3> varray1{ p0, p1, p1, p2, p2, p3, p3, p0 };
    GLuint vao1 = CreateVAO(varray1);

    std::vector<glm::vec3> varray2;
    for (size_t u=0; u <= 360; u += 8)
    {
        double a = u*M_PI/180.0;
        double c = cos(a), s = sin(a);
        varray2.emplace_back(glm::vec3((float)c, (float)s, 0.0f));
    }
    GLuint vao2 = CreateVAO(varray2);

    glm::mat4(project);
    int vpSize[2]{0, 0};
    while (!glfwWindowShouldClose(window))
    {
        int w, h;
        glfwGetFramebufferSize(window, &w, &h);
        if (w != vpSize[0] ||  h != vpSize[1])
        {
            vpSize[0] = w; vpSize[1] = h;
            glViewport(0, 0, vpSize[0], vpSize[1]);
            float aspect = (float)w/(float)h;
            project = glm::ortho(-aspect, aspect, -1.0f, 1.0f, -10.0f, 10.0f);
            glUniform2f(loc_res, (float)w, (float)h);
        }

        glClear(GL_COLOR_BUFFER_BIT);

        glm::mat4 modelview1( 1.0f );
        modelview1 = glm::translate(modelview1, glm::vec3(-0.6f, 0.0f, 0.0f) );
        modelview1 = glm::scale(modelview1, glm::vec3(0.5f, 0.5f, 1.0f) );
        glm::mat4 mvp1 = project * modelview1;

        glUniformMatrix4fv(loc_mvp, 1, GL_FALSE, glm::value_ptr(mvp1));
        glBindVertexArray(vao1);
        glDrawArrays(GL_LINES, 0, (GLsizei)varray1.size());

        glm::mat4 modelview2( 1.0f );
        modelview2 = glm::translate(modelview2, glm::vec3(0.6f, 0.0f, 0.0f) );
        modelview2 = glm::scale(modelview2, glm::vec3(0.5f, 0.5f, 1.0f) );
        glm::mat4 mvp2 = project * modelview2;

        glUniformMatrix4fv(loc_mvp, 1, GL_FALSE, glm::value_ptr(mvp2));
        glBindVertexArray(vao2);
        glDrawArrays(GL_LINE_STRIP, 0, (GLsizei)varray2.size());

        glfwSwapBuffers(window);
        glfwPollEvents();
    }
    glfwTerminate();

    return 0;
}

---

_{See also
Dashed line in OpenGL3?
OpenGL ES - Dashed Lines}