Why this code throws System.ExecutionEngineException

Question

Background: I am using DirectX 9.0 Managed Libraries to transform arrays of 3d points to 2d screen coordinates. For speed I use the UnsafeNativeMethods to do all the transformations.

The Problem: If my custom line clipping function is used my application dies without throwing any exceptions, it took me a while to figure out that it was throwing an uncatchable System.ExecutionEngineException. I have narrowed it down to happening because of the last two lines of my clipping function.

List<Vector3> verticesAfterClipping = new List<Vector3>;
public unsafe void ClipLine(Line lineToClip)
{
    this.verticesAfterClipping.Clear();

    // Clipping algorithm happens here... (this is psuedo-code of what it does)
    foreach(Vertex in lineToClip.Vertices)
    {
        bool thisIsClipped =   // Set to whether this vertex is clipped
        bool lastWasClipped =  // Set to whether last vertex was clipped

        if(thisIsClipped == false && lastWasClipped == true)
        {
            verticesAfterClipping.Add( /* intersection on clipping plane */ );
            verticesAfterClipping.Add( /* thisVertex */ );
        }
        else if (thisIsClipped == false && lastWasClipped == false)
        {
            verticesAfterClipping.Add( /* thisVertex */ );
        }
        else if (thisIsClipped == true && lastWasClipped == false)
        {
            verticesAfterClipping.Add(/* intersection on clipping plane */);
        }
    }

    // THIS IS WHERE BAD THINGS HAPPEN
    lineToClip.Vertices = new Vertex[verticesAfterClipping.Count];
    verticesAfterClipping.CopyTo(lineToClip.Vertices, 0);
}

When the verticesAfterClipping list is copied to the lineToClip vertices the lineToClip object is then passed to an UnsafeNativeMethod which transforms these vertices to 2d vertices. From everything I can see when I step through it in Debug mode it is working completely fine, until it just dies.

I simply cannot figure out what is wrong. Any help would be much appreciated.

Answer 1

The problem may not actually be occurring in the line that throws an exception. This may just be a symptom of something that happened earlier.

The System.ExecutionEngineException exception is thrown when the CLR detects that something has gone horribly wrong. This can happen some considerable time after the problem occurred. This is because the exception is usually a result of corruption of internal data structures - the CLR discovers that something has got into a state that makes no sense. It throws an uncatchable exception because it's not safe to proceed.

So you might have some code in some completely unrelated part of the system that corrupts something, but this only becomes apparent when this particular piece of code runs. The code you've shown might be just fine. (It also might not be...I don't see anything obvious wrong, but then I don't know the DX 9 managed libraries well. I can't see which feature of this method requires the unsafe keyword, for example.)

Unfortunately, this means you need to start casting the net a bit wider. Pretty much anything that uses either unsafe code, or COM interop is potentially suspect. This will be a long and tedious process, sadly. One way you might approach it is to try gradually simplifying the program: what's the smallest piece of code that can illustrate the problem? (E.g., if you put the code you've shown there into an application that contains nothing else except the simplest possible call to that method, does it still fail?)