Image manipulation and texture mapping using HTML5 Canvas?

Question

In a 3D engine I'm working on I've succesfully managed to draw a cube in 3D. The only method to fill the sides is using either a solid color or gradient as far as I'm concerned. To make things more exciting, I'd really love to implement texture mapping using a simple bitmap.

The point is that I can hardly find any articles or code samples on the subject of image manipulation in JavaScript. Moreover, image support in HTML5 canvas seems to be restricted to cropping.

How could I go about stretching a bitmap so that a rectangular bitmap can fill up a unregular cube face? In 2D, a projected square cube face is, due to perspective, not of a square shape, so I'll have to stretch it to make it fit in any quadrilateral.

Hopefully this image clarifies my point. The left face is now filled up with a white/black gradient. How could I fill it with a bitmap, after it has been texture-mapped?

Does anyone have any tips on perspective texture mapping (or image manipulation at all) using JavaScript and HTML5 Canvas?

Edit: I got it working, thanks to 6502!

It is, however, rather CPU intensive so I'd love to hear any optimization ideas.

Result using 6502's technique - Texture image used

Answer 1

I think you will never get an accurate result... I spent some time investigating how to do 3d graphics using canvas 2d context and I found it viable to do texture mapping gouraud shading by computing appropriate 2d gradients and matrices:

• Solid polygons are of course easy
• Gouraud filling is possible only on one component (i.e. you cannot have a triangle where every vertex is an arbitrary RGB filled with bilinear interpolation, but you can do that filling using for example three arbitrary shades of a single color)
• Linear texture mapping can be done using clipping and image drawing

I would implement perspective-correct texture mapping using mesh subdivision (like on PS1).

However I found many problems... for example image drawing with a matrix transform (needed for texture mapping) is quite inaccurate on chrome and IMO it's impossible to get a pixel-accurate result; in general there is no way to turn off antialiasing when drawing on a canvas and this means you will get visible see-through lines when subdividing in triangles. I also found multipass rendering working really bad on chrome (probably because of how hw-accellerated rendering is implemented).

In general this kind of rendering is surely a stress for web browsers and apparently these use cases (strange matrices for example) are not tested very well. I was even able to get Firefox crashing so bad that it took down the whole X susbsystem on my Ubuntu.

You can see the results of my efforts here or as a video here... IMO is surely impressing that this can be done in a browser without using 3D extensions, but I don't think current problems will be fixed in the future.

Anyway the basic idea used to draw an image so that the 4 corners ends up in specific pixels position is to draw two triangles, each of which will use bilinear interpolation.

In the following code I assume you have a picture object texture and 4 corners each of which is an object with fields x,y,u,v where x,y are pixel coordinates on the target canvas and u,v are pixel coordinates on texture:

function textureMap(ctx, texture, pts) {
    var tris = [[0, 1, 2], [2, 3, 0]]; // Split in two triangles
    for (var t=0; t<2; t++) {
        var pp = tris[t];
        var x0 = pts[pp[0]].x, x1 = pts[pp[1]].x, x2 = pts[pp[2]].x;
        var y0 = pts[pp[0]].y, y1 = pts[pp[1]].y, y2 = pts[pp[2]].y;
        var u0 = pts[pp[0]].u, u1 = pts[pp[1]].u, u2 = pts[pp[2]].u;
        var v0 = pts[pp[0]].v, v1 = pts[pp[1]].v, v2 = pts[pp[2]].v;

        // Set clipping area so that only pixels inside the triangle will
        // be affected by the image drawing operation
        ctx.save(); ctx.beginPath(); ctx.moveTo(x0, y0); ctx.lineTo(x1, y1);
        ctx.lineTo(x2, y2); ctx.closePath(); ctx.clip();

        // Compute matrix transform
        var delta = u0*v1 + v0*u2 + u1*v2 - v1*u2 - v0*u1 - u0*v2;
        var delta_a = x0*v1 + v0*x2 + x1*v2 - v1*x2 - v0*x1 - x0*v2;
        var delta_b = u0*x1 + x0*u2 + u1*x2 - x1*u2 - x0*u1 - u0*x2;
        var delta_c = u0*v1*x2 + v0*x1*u2 + x0*u1*v2 - x0*v1*u2
                      - v0*u1*x2 - u0*x1*v2;
        var delta_d = y0*v1 + v0*y2 + y1*v2 - v1*y2 - v0*y1 - y0*v2;
        var delta_e = u0*y1 + y0*u2 + u1*y2 - y1*u2 - y0*u1 - u0*y2;
        var delta_f = u0*v1*y2 + v0*y1*u2 + y0*u1*v2 - y0*v1*u2
                      - v0*u1*y2 - u0*y1*v2;

        // Draw the transformed image
        ctx.transform(delta_a/delta, delta_d/delta,
                      delta_b/delta, delta_e/delta,
                      delta_c/delta, delta_f/delta);
        ctx.drawImage(texture, 0, 0);
        ctx.restore();
    }
}

Those ugly strange formulas for all those "delta" variables are used to solve two linear systems of three equations in three unknowns using Cramer's method and Sarrus scheme for 3x3 determinants.

More specifically we are looking for the values of a, b, ... f so that the following equations are satisfied

a*u0 + b*v0 + c = x0
a*u1 + b*v1 + c = x1
a*u2 + b*v2 + c = x2

d*u0 + e*v0 + f = y0
d*u1 + e*v1 + f = y1
d*u2 + e*v2 + f = y2

delta is the determinant of the matrix

u0  v0  1
u1  v1  1
u2  v2  1

and for example delta_a is the determinant of the same matrix when you replace the first column with x0, x1, x2. With these you can compute a = delta_a / delta.