Work out whether to turn clockwise or anticlockwise from two angles

Question

Im making a game in XNA. I have enemies and player. The enemies should turn gradually towards the player. They should work out whether they need to turn clockwise or anticlockwise, whichever is shorter.

I got the angle the enemy is currently facing and the angle it should be facing (the angle of the line between the enemy and the player) as radians by using Atan2.

I get some weird behavior though. Lets say in the scenario below. the enemy might turn all the way around in the wrong direction.

My code (below) keeps getting longer and I'm still having issues. This code is part of the enemy classes Update method. This must be a common problem to overcome in games. Is there some way of dealing with this?

            //this bit is just in case enemy has rotated more than 360 degrees (gets set back to 0)
            if (Math.Abs(_blocklist[0]._floor.Revolutions) >= 2)
            {
                _blocklist[0]._floor.Rotation = 0.0f;
            }

            //enemy rotation in radians          
            float blockroat = _blocklist[0]._floor.Rotation;
            // vector to player - vector to enemy
            _vectToPlayer = playerpos - _blocklist[0].Centre
            angletoplayer = (float)(Math.Atan2(_vectToPlayer.Y, _vectToPlayer.X));
            diff = blockroat - angletoplayer;

            if (diff < -Math.PI)
            {
                diff += (float) Math.PI;
                diff = -diff;
            }
            else if (diff > Math.PI)
            {
                diff -= (float)Math.PI;
                diff = -diff;
            }

            // if enemy angle if off by a certain amount
            if (Math.Abs(diff) >_maxturn)
            {
                if (diff < 0)
                {
                    //turn clockwise
                    _blocklist[0]._floor.Rotation += _maxturn;
                }
                else
                {
                     //turn anti clockwise
                    _blocklist[0]._floor.Rotation -= _maxturn;
                }
            }

---

UPDATE

I ended up using method 2 like this.. Works perfectly. Also it is a lot neater than my previous code

            //enemy rotation in radians from farseer (red line)
            float brot = _blocklist[0]._floor.Rotation + ((float)Math.PI/2);
            //vector from enemy to player (blue line)
            Vector2 _vectToPlayer = playerpos - _blocklist[0].Centre;
            //cross product of 2d vectors
            cross = (_vectToPlayer.X * (float)Math.Sin(brot)) - ((float)Math.Cos(brot) * _vectToPlayer.Y);

            //tolerance for how closely enemy must point towards player
            if (Math.Abs(cross) > 5)
            {
                if (cross > 0)
                {
                    //turn anticlockwise
                    _blocklist[0]._floor.Rotation -= _npcstats.maxturnspeed;
                }
                else
                {
                    //turn clockwise
                    _blocklist[0]._floor.Rotation += _npcstats.maxturnspeed;
                } 
            }

I think that my previous code was more or less doing exactly the suggested method 1. But I could not get it to work.. I put this down to the vagaries of farseers coordinate system + how it interacted with my own.

Answer 1

Technique #1:

You are using a convention that I'm not familiar with. In your convention, east is 0, north is -π/2, west is both π and -π, and south is π/2. All angles are between -π and π.

Normally the angle of a character facing east is zero, north is π/2, west is π, and due south is 3π/2. All angles are between 0 and 2π.

Let's assume the normal convention rather than your convention. Start by getting your red and blue vector angles correct in the normal convention; how you do that is up to you.

Subtract the angle of the red vector from both angles. Now we have the guy at the origin facing due east.

Now normalize the new blue angle; if it is smaller than 0, add 2π. If it is larger than 2π, subtract 2π. Do that until it is between 0 and 2π.

Now we have two angles; the angle of the new red vector is zero and the angle of the new blue vector is between 0 and 2π.

If the angle of the new blue vector is less than π then the character at the origin needs to turn towards its left. If it is greater than π then turn right.

Technique #2:

Take a non-zero point on your blue and red vectors, say (bx, by) and (rx, ry). Now compute bx * ry - by * rx. If it is positive, turn right, if it is negative, turn left. If it is zero then either they are facing directly towards or directly away; in that case you'll have to figure out which case you're in by some other means. (This is essentially Jacek's answer stated more directly.)

Answer 2

If you have both blue and red vectors as Vector3, you can do:

Vector3 crossProduct = Vector3.Cross(red, blue)
if (crossProduct.z > 0)
    // Turn Right
else
    // Turn Left