Exception CallbackOnCollectedDelegate when creating tensorflow graph

Question

I try to build a little tensorflow application with TensorFlowSharp and sometimes I recieve this exception:

Managed Debugging Assistant 'CallbackOnCollectedDelegate'

For the function TensorFlowSharp!TensorFlow.TFBuffer+BufferReleaseFunc::Invoke

I tried to find out what it means but I didn't fully understand the explanations. This is the part of the code where the exception is thrown:

var graph = new TFGraph();
var model = File.ReadAllBytes(ModelsFile);
graph.Import(model, "");

Does somebody know what I should do to prevent this exception?

Bruno

Answer 1

I see only one good candidate for this bug, a delegate in Buffer.cs. But Miguel already fixed this bug on July 27th, the diff is here. So just make sure to update your copy. If you obtained it from Nuget then ensure you have at least version 1.30

Answer 2

I assume this is a bug in TensorflowSharp.

The error looks like a usually inconsistent access violation in the CLR code (occurs usually only under heavy load or on a random number of attempts). Citing from Microsoft docs:

The callbackOnCollectedDelegate managed debugging assistant (MDA) is activated if a delegate is marshaled from managed to unmanaged code as a function pointer and a callback is placed on that function pointer after the delegate has been garbage collected.

This type of error occurs when a delegate from which the function pointer was created and exposed to unmanaged code was garbage collected. When the unmanaged component tries to call on the function pointer, it generates an access violation. The failure appears random because it depends on when garbage collection occurs.

The resolution can be difficult, since once a delegate has been marshaled out as an unmanaged function pointer, the garbage collector cannot track its lifetime. Instead, it is required to keep a reference to the delegate for the lifetime of the unmanaged function pointer. In order to do this, the faulty delegate that was collected, has to be identified in TensorFlowShapr's code (or your code).

You can also enable the gcUnmanagedToManaged MDA to force a garbage collection before every callback into the runtime. This will remove the uncertainty introduced by the garbage collection by ensuring that a garbage collection always occurs before the callback. Once you know what delegate was collected, change your code to keep a reference to that delegate on the managed side for the lifetime of the marshaled unmanaged function pointer.

So, I guess it's best to report this to the maker of the library.