Simulating palette swaps with OpenGL Shaders (in LibGDX)

Question

I'm trying to use LibGDX to make a retro-style little game, and I'd like to let the players to choose the colors of several characters, so I thought about loading png indexed images and then updating palettes programatically... How wrong I was ^^U

It seems that color pallettes are something of the past, and also seems that the best option to achieve a similar result is by using Shaders.

Here is an image explaining what I'm trying right now:

My intention is to use 2 images. One of them, pixel_guy.png is a png image with only 6 colors (those colors are its original palette). The other image, colortable.png, would be a 6x6 pixel png that contains 6 palettes of 6 colors each (each row is a different palette). The colors from the first row of pixels of colortable.png would match the colors used in pixel_guy.png, that would be the first/original palette, and the other rows would be palettes 2 to 6. What I try to achieve is to use colortable's first palette to index pixelguy colors and then change the palette by sending a number (from 2 to 6) to the shader.

After doing some research, I found a post in gamedev stackexchange, and apparently it was what I was looking for, so I tried to test it.

I created the Vertex and Fragment Shaders and loaded my textures (the one whose palette I wanted to swap, and the one that contained several palettes for that), but the output is an unexpected white image.

My Vertex Shader:

attribute vec4 a_position;
attribute vec4 a_color;
attribute vec2 a_texCoord0;

uniform mat4 u_projTrans;

varying vec4 v_color;
varying vec2 v_texCoords;

void main() {
    v_color = a_color;
    v_texCoords = a_texCoord0;
    gl_Position = u_projTrans * a_position;
}

My Fragment Shader:

    // Fragment shader
// Thanks to Zack The Human https://gamedev.stackexchange.com/questions/43294/creating-a-retro-style-palette-swapping-effect-in-opengl/

uniform sampler2D texture; // Texture to which we'll apply our shader? (should its name be u_texture?)
uniform sampler2D colorTable; // Color table with 6x6 pixels (6 palettes of 6 colors each)
uniform float paletteIndex; // Index that tells the shader which palette to use (passed here from Java)

void main()
{
        vec2 pos = gl_TexCoord[0].xy;
        vec4 color = texture2D(texture, pos);
        vec2 index = vec2(color.r + paletteIndex, 0);
        vec4 indexedColor = texture2D(colorTable, index);
        gl_FragColor = indexedColor;      
}

The code I used to make the texture binding and pass the palette number to the shader:

package com.test.shaderstest;

import com.badlogic.gdx.ApplicationAdapter;
import com.badlogic.gdx.Gdx;
import com.badlogic.gdx.graphics.GL20;
import com.badlogic.gdx.graphics.Texture;
import com.badlogic.gdx.graphics.g2d.SpriteBatch;
import com.badlogic.gdx.graphics.glutils.ShaderProgram;

public class ShadersTestMain extends ApplicationAdapter {
    SpriteBatch batch;

    Texture imgPixelGuy;
    Texture colorTable;

    private ShaderProgram shader;
    private String shaderVertIndexPalette, shaderFragIndexPalette;

    @Override
    public void create () {
        batch = new SpriteBatch();

        imgPixelGuy = new Texture("pixel_guy.png"); // Texture to which we'll apply our shader
        colorTable =  new Texture("colortable.png"); // Color table with 6x6 pixels (6 palettes of 6 colors each)

        shaderVertIndexPalette = Gdx.files.internal("shaders/indexpalette.vert").readString();
        shaderFragIndexPalette = Gdx.files.internal("shaders/indexpalette.frag").readString();

        ShaderProgram.pedantic = false; // important since we aren't using some uniforms and attributes that SpriteBatch expects

        shader = new ShaderProgram(shaderVertIndexPalette, shaderFragIndexPalette);
        if(!shader.isCompiled()) {
            System.out.println("Problem compiling shader :(");
        }
        else{
            batch.setShader(shader);
            System.out.println("Shader applied :)");
        }

        shader.begin();
        shader.setUniformi("colorTable", 1); // Set an uniform called "colorTable" with index 1
        shader.setUniformf("paletteIndex", 2.0f); // Set a float uniform called "paletteIndex" with a value 2.0f, to select the 2nd palette 
        shader.end();

        colorTable.bind(1); // We bind the texture colorTable to the uniform with index 1 called "colorTable"
    }

    @Override
    public void render () {
        Gdx.gl.glClearColor(0.3f, 0.3f, 0.3f, 1);
        Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);
        batch.begin();
        batch.draw(imgPixelGuy, 0, 0); // Draw the image with the shader applied
        batch.end();
    }
}

I Don't know if that's the correct way to pass a float value to an uniform of the fragment. Not sure either about how the code snippet I tried to use works.

Edit: I tried the changes suggested by TenFour04 and they worked flawlessly :)

Here is the pre-processed sprite

I have updated the repository with the changes, (Java code here, Fragment Shader here), in case somebody is interested it can be downloaded here: https://bitbucket.org/hcito/libgdxshadertest

Edit 2: I've just added to the repository the last optimization that TenFour04 adviced (to pass the palette index to each sprite within the R channel calling sprite.setColor() method), and again it worked perfect :)

Answer 1

I noticed a few issues.

1) In your fragment shader, shouldn't vec2 index = vec2(color.r + paletteIndex, 0); be vec2 index = vec2(color.r, paletteIndex);, so the sprite texture tells you which row and the paletteIndex tells you which column of the color table to look at?

2) The palette index is being used as a texture coordinate, so it needs to be a number between 0 and 1. Set it like this:

int currentPalette = 2; //A number from 0 to (colorTable.getHeight() - 1)
float paletteIndex = (currentPalette + 0.5f) / colorTable.getHeight();
//The +0.5 is so you are sampling from the center of each texel in the texture

3) A call to shader.end happens inside batch.end so you need to set your uniforms on every frame, not in create. Probably a good idea to also bind your secondary texture each frame as well, in case you do any multitexturing later.

@Override
public void render () {
    Gdx.gl.glClearColor(0.3f, 0.3f, 0.3f, 1);
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT);

    colorTable.bind(1);

    //Must return active texture unit to default of 0 before batch.end 
    //because SpriteBatch does not automatically do this. It will bind the
    //sprite's texture to whatever the current active unit is, and assumes
    //the active unit is 0
    Gdx.gl.glActiveTexture(GL20.GL_TEXTURE0); 

    batch.begin(); //shader.begin is called internally by this line
    shader.setUniformi("colorTable", 1);
    shader.setUniformf("paletteIndex", paletteIndex);
    batch.draw(imgPixelGuy, 0, 0);
    batch.end(); //shader.end is called internally by this line

}

4) The GPU might do this for you anyway, but since you aren't using vertex colors, you can remove the two lines involving v_color from your vertex shader.

5) You probably also want transparent pixels to stay transparent. So I would change the last line of your fragment shader to preserve alpha:

gl_FragColor = vec4(indexedColor.rgb, color.a); 

6) This one is probably the reason you were getting all white. In your fragment shader, you should be using v_texCoords instead of gl_TexCoord[0].xy, which is something from desktop OpenGL and not used in OpenGL ES. So your fragment shader should be:

uniform sampler2D texture; // Texture to which we'll apply our shader? (should its name be u_texture?)
uniform sampler2D colorTable; // Color table with 6x6 pixels (6 palettes of 6 colors each)
uniform float paletteIndex; // Index that tells the shader which palette to use (passed here from Java)
varying vec2 v_texCoords;

void main()
{
    vec4 color = texture2D(texture, v_texCoords);
    vec2 index = vec2(color.r, paletteIndex);
    vec4 indexedColor = texture2D(colorTable, index);
    gl_FragColor = vec4(indexedColor.rgb, color.a); // This way we'll preserve alpha      
}

7) Your source sprite needs to be pre-processed to map to the columns of your palette lookup table. Your shader is pulling a coordinate from the red channel of the texture. Therefore you need to do a color replacement of each pixel color in your source image. You can do this by hand ahead of time. Here's an example:

Skin tone is index 2 out of the six columns (0-5). So just like we calculated the paletteIndex, we normalize it to the center of pixel: skinToneValue = (2+0.5) / 6 = 0.417. The 6 is for the six columns. Then in your drawing program, you need the appropriate value of red.

0.417 * 255 = 106, which is 6A in hex, so you want color #6A0000. Replace all the skin pixels with this color. And so on for the other hues.

---

Edit:

One more optimization is that you probably want to be able to batch all your sprites together. The way it is now, you will have to group all your sprites for each palette separately and call batch.end for each of them. You can avoid this by putting paletteIndex into the vertex color of each sprite, since we weren't using vertex color anyway.

So you could set it to any of the four color channels of the sprite, let's say the R channel. If using the Sprite class, you can call sprite.setColor(paletteIndex, 0,0,0); Otherwise call batch.setColor(paletteIndex,0,0,0); before calling batch.draw for each of the sprites.

The vertex shader will need to declare varying float paletteIndex; and set it like this:

paletteIndex = a_color.r;

The fragment shader will need to declare varying float paletteIndex; instead of uniform float paletteIndex;.

And of course you no longer should call shader.setUniformf("paletteIndex", paletteIndex);.