How can I clamp a rectangle inside the boundaries of an outer rotated rectangle

Question

I'm working on a WebGL (with 2d canvas fallback photo editor).

I have decided to incorporate rotation directly into the crop tool, in a fashion similar to the iOS 8 photo cropper. i.e. the size and position of the photograph changes dynamically as you rotate the photo, in order for the crop area to always be contained within the photograph itself.

However, I am struggling with some of the math.

I have two rectangles, the photo and the crop area.

Both are defined as:

var rect = {
     x : x,
     y : y,
     w : width,
     h : height
}

The rectangle that defines the photo itself also has a rotation property in radians, which of course describes the angle of the photo (in radians).

It should be noted that the x and y coordinates of the photo rectangle (not the crop rectangle) actually define the center of the photo, and not the top-left point. While this may seem strange, it makes calculations easier for us elsewhere, and should not effect this question. In the interests of completeness I am mentioning it.

The transform origin of the photo is always set as the center of the the croparea.

When the angle of the photo is 0 and the crop area is the same size as the photo, the rectangles align like so:

When I rotate the photo, a scale factor is applied to the photo rectangle to ensure that the croparea stays within the photos boundaries:

This is achieved by calculating the bounding box of the croparea and from this, working out the required scale factor to apply to the photo rectangle:

TC.UI.PhotoCropper.prototype._GetBoundingBox = function(w,h,rads){
    var c = Math.abs(Math.cos(rads));
    var s = Math.abs(Math.sin(rads));
    return({  w: h * s + w * c,  h: h * c + w * s });
}

var bbox = this._GetBoundingBox(crop.w, crop.h, photo.rotation);
var scale = Math.max(1, Math.max(bbox.w/photo.w, bbox.h/photo.h));

This currently works exactly as intended and the width of the photo rectangle is correctly scaled wherever the croparea (and the resulting rotation origin) happens to be.

Also, as this is a photo cropping tool, the position of the croparea can of course be modified. Please note that when modifying the croparea position we are moving the photo rectangle itself. (the croparea always stays centered as we feel this is far more natural with touch based devices, and still acceptable when used with a mouse... This is the same in both iOS and windows 8).

Therefore we currently clamp the x and y coordinates of the photo rectangle to the edges of the croparea like so:

var maxX = crop.x + (crop.w/2) - (photo.w/2);
var maxY = crop.y + (crop.h/2) - (photo.w/2);
var minX = crop.x - (crop.w/2) + (photo.w/2);
var minY = crop.y - (crop.h/2) + (photo.h/2);

photo.x = Math.min(minX, Math.max(maxX, photo.x));
photo.y = Math.min(minY, Math.max(maxY, photo.y));

And this, is where I am having trouble.

When the croparea is not centered within the photo, we end up with the croparea moving outside the bounds of the photo, like so:

Remember, the origin of the rotation is always exactly the center of the croparea

In the photo above you can see that the required width is correctly calculated and the photo gets scaled correctly in order to encompass the croparea.

However since our clamping function only checks the edges of the croparea, we end up with our croparea outside the photos boundaries.

We would like to modify our clamping function so that it takes into account the rotation of the photo rectangle. Therefore, the x and y coordinates (in the screenshot above) should be correctly clamped to ensure that the end result looks like this:

Unfortunately I have no idea where to begin with the math and could really use some help.

We currently use the same clamping function while both rotating and moving the photo rectangle, and would like to continue to do so if possible.

Answer 1

What's wrong?

Instead of clamping the photo coordinates in the original coordinate system, you need to clamp them in the rotated coordinate system. Using the coordinates of the crop area corners (when calculating minX, maxX, minY and maxY) in the rotated coordinate system makes no sense.

Clamping the photo coordinates

You need to construct an axis-aligned bounding box of the crop area in the rotated coordinate system which you can use for clamping. Like so:

Since we are rotating around the center of the crop area the center coordinates stay the same. However the width and the height will increase.

Notice that you already calculated these quantities to scale the photo appropriately. So in short, you can just replace the width crop.w and height crop.h used while scaling by those obtained when calculating the bounding box (bbox.w and bbox.h):

var maxX = crop.x + (bbox.w/2) - (photo.w/2);
var maxY = crop.y + (bbox.h/2) - (photo.w/2);
var minX = crop.x - (bbox.w/2) + (photo.w/2);
var minY = crop.y - (bbox.h/2) + (photo.h/2);

center = {
    x: Math.min(minX, Math.max(maxX, photo.x)),
    y: Math.min(minY, Math.max(maxY, photo.y))
};

Keep in mind that the coordinates obtained on the last two lines only work in the rotated coordinate system. If you need them in the original coordinate system you still have to rotate them back:

var sin = Math.sin(-rotation);
var cos = Math.cos(-rotation);

photo.x = crop.x + cos*(center.x - crop.x) - sin*(center.y - crop.y);
photo.y = crop.y + sin*(center.x - crop.x) + cos*(center.y - crop.y);

Answer 2

As DARK_DUCK said there are ways (non-trivial ways) to write a clamp function that solves your problem. However, when you say "clamp" I automatically assume that your solution must restrict users from reaching some invalid states which IMHO is not a good solution. Maybe a user will rotate the image passing through some invalid states just to reach a valid state.

I propose an alternative solution:

1. Store the last valid state with every change (rotation, translation, scale)
2. When the user finishes his change you either do nothing if the current state is valid or you restore last valid state.

The state will consists of image origin (or position), rotation and scale.
You can animate the transition from a invalid state to last valid state. While on a invalid state you can inform the user by making the crop area border red:

A valid state is when all 4 corners of the crop area are inside the image rectangle. The following functions will help you test this condition:

//rotate a point around a origin
function RotatePoint(point, angle, origin)
{
    var a = angle * Math.PI / 180.0;
    var sina = Math.sin(a);
    var cosa = Math.cos(a);

    return
    {
        x : origin.x + cosa * (point.x - origin.x) - sina * (point.y - origin.y),
        y : origin.y + sina * (point.x - origin.x) + cosa * (point.y - origin.y)
    }

}

//position of point relative to line defined by a and b
function PointPos(a, b, point)
{
    return Math.sign((b.x - a.x) * (point.y - a.y) - (b.y - a.y) * (point.x - a.x));
}

You have to:

1. Calculate the position of the corners of the image rectangle after applying rotations and translations

2. Check if crop rectangle corners are inside the image rectangle using second function provided above. A point x is inside the rectangle if

   PointPos(a,b,x) == -1 && PointPos(b,c,x) == -1 && PointPos(c,d,x) == -1 && PointPos(d,a,x) == -1;

Edit: Here is an example: JSFiddle

The two rectangles have different origins. The rotation origin of the outer (pictureArea) rectangle is the origin of the inner (cropArea).

Good luck!