OpenGL multiple matrix transormations

Question

I have simple vertex shader for models

#version 330

layout(location = 0) in vec3 VertexPosition;
layout(location = 1) in vec3 VertexNormal;
layout(location = 2) in vec2 VertexUV;

out VS_GS_VERTEX
{
    vec3 vs_worldpos;
    vec3 vs_normal;
    vec2 VertexUV;
} vertex_out;

uniform mat4 modelMatrix;
uniform mat4 projectionMatrix;
uniform mat4 lookAtMatrix;

void main(void)
{
    mat4 MVP = projectionMatrix * lookAtMatrix * modelMatrix;

    gl_Position = MVP * vec4(VertexPosition, 1.0);
    gl_Normal = mat3(modelMatrix) * VertexNormal;

    vertex_out.vs_worldpos = gl_Position.xyz;
    vertex_out.vs_normal = gl_Normal;
    vertex_out.VertexUV = VertexUV;
}

and I pass there modelMatrix

  modelMat := MatrixMultiply(transMat, baseMat);
  modelMat := MatrixMultiply(modelMat, scaleMat);
  modelMat := MatrixMultiply(modelMat, rotMat);
  glUniformMatrix4fv(modelMatrix_loc, 1, false, @modelMat);

where transMat for positioning, scaleMat to enlarge whole picture and rotMat to rotate whole terrain. Everything fine while baseMat is coordinates of model on terrain

baseMat := CreateTranslationMatrix(AffineVectorMake(pos.x, pos.y, pos.z));

but when I trying also to enlarge model by own model's scale (model have zero based coordinates, for example roots of tree at (0, 0, 0))

  scMat := CreateScaleMatrix(AffineVectorMake(scale, scale, scale));
  trMat := CreateTranslationMatrix(AffineVectorMake(pos.x, pos.y, pos.z));

  baseMat := MatrixMultiply(scMat, trMat);

models not just enlarged, they also changing coordinates and not placed on terrain anymore. Is it possible to make one model matrix with model's scaling, rotation, translation, yet another translation, scaling and rotation? Or I should do something else? Matrix math from GLScene and seems works fine.

Edit: I can't use proper order with matrices by scaling first because I have this

The structure of terrain is tiles of 64x64 chunks, each have base coordinates. So to put it together I need first to translate each chunk to proper position and only then scale to enlarge image and rotate with mouse. Each model also have absolute position assigned to terrain. I should scale and rotate model (own model's modifiers) and put it to proper position on terrain. transMat (to put tiles and models at the center of screen), scaleMat and rotMat shared for both terrain and models. But local scMat and trMat are for enlarge model and put it to right place.

If I leave terrain working as is with rotMat * scaleMat * transMat but using models as rotMat * transMat * trMat * scaleMat * scMat I have this
a bit closer

Edit2: changed vertex shader to

uniform mat4 modelMatrix;
uniform mat4 MVP;

void main(void)
{
    vec4 modelPos = modelMatrix * vec4(VertexPosition, 1.0);
    gl_Position = MVP * modelPos;
    gl_Normal = mat3(modelMatrix) * VertexNormal;

    vertex_out.vs_worldpos = gl_Position.xyz;
    vertex_out.vs_normal = gl_Normal;
    vertex_out.VertexUV = VertexUV;
}

and passed

modelMatrix = modelRotMat * modelScaleMat;
mvpMatrix = projMat * lookAtMat * rotMat * scaleMat * modelTransMat * transNat;

now it works as expected, thanks for help.

Answer 1

You have to apply the scaling before the translation. The right order in which to apply matrices is

FinalMatrix = TranslationMatrix * RotationMatrix * ScaleMatrix;

(you may switch rotation and translation in specific cases, depending on what you want to rotate, but generally will be like this) So, if you want to combine more of them, you'll have to do it this way;

FinalMatrix = TranslationMatrix1 *TranslationMatrix2* RotationMatrix1 * RotationMatrix2 * ScaleMatrix1*ScaleMatrix2;

EDIT

So, you have models with a base position rotation and scale on your terrain. And you want to add other transforms to them from time to time. You'll need a lot of matrices. From the base coordinates on the terrain of you models, calculate TranslationMatrix2. Your CreateTranslationMatrix(AffineVectorMake(pos.x, pos.y, pos.z)) should be fine. The basic scale and rotation of your model on the terrain should also be in RotationMatrix2 and ScaleMatrix2. These three matrix hold the base scale position and orientation of each of your models on the terrain. Your final model matrix will be

ModelMatrix =  TranslationMatrix2 *RotationMatrix2 * ScaleMatrix2;

When you want to scale them or move them from that position, you can combine new transformations with those, with the formula. Let's say you want to apply a second scale, and new rotation and a new translation, whose matrices are ScaleMatrix1,RotationMatrix1, and TranslationMatrix1. Your final model matrix will be:

ModelMatrix =  TranslationMatrix1 *TranslationMatrix2*RotationMatrix1 * RotationMatrix2 * ScaleMatrix1*ScaleMatrix2;

EDIT2

If you want to keep that structure of your terrain, I think you have to scale the position of the models with the scale matrix used for the terrain. Then, from that position, compute the translation matrix of your model. Use the shared rotation matrix, and the model's own scale, like this

pos = scaleMat * pos;
trMat = CreateTranslationMatrix(AffineVectorMake(pos.x, pos.y, pos.z));
ModelMatrix = SharedRotation* scMat * trMat ;