How to construct a tensorflow::Tensor from raw pointer data in C++

Question

I want to change the output tensor's underlying storage during an op.

I have a raw pointer(float*) of the new data. I want to set the output tensor to this new data before launch kernel and return, so that I can hijack this op.

However I am confusing with when should I delete the raw pointer, since the the tensor construct seems to be a shallow copy. I can only delete the raw pointer after all this tensor's usage finished. But how can I be notified this?

Answer 1

There is no public API for doing this inside the TensorFlow runtime, but it is possible to create a Tensor object from a raw pointer using the C API method TF_NewTensor(), which has the following signature:

// Return a new tensor that holds the bytes data[0,len-1].
//
// The data will be deallocated by a subsequent call to TF_DeleteTensor via:
//      (*deallocator)(data, len, deallocator_arg)
// Clients must provide a custom deallocator function so they can pass in
// memory managed by something like numpy.
extern TF_Tensor* TF_NewTensor(TF_DataType, const int64_t* dims, int num_dims,
                               void* data, size_t len,
                               void (*deallocator)(void* data, size_t len,
                                                   void* arg),
                               void* deallocator_arg);

Internally, this creates a reference-counted TensorBuffer object that takes ownership of the raw pointer. (Unfortunately, only the C API has friend access to create a tensorflow::Tensor from a TensorBuffer directly. This is an open issue.) The deallocator function is called with the values of data, len and dellocator_arg when the reference count drops to zero.

Answer 2

Unfortunately, this is too little information to give you a precise answer. Possibly, you're not even allowed to delete the pointer!

Imagine something like this:

float* gf = nullptr; // global pointer (just for illustration)

void calculate()
{
   float f;
   gf = &f;
   doSomething();
   gf = nullptr;
}

Same applies, if your pointer in question points to some class-static or global variable.

If you create your variable on the heap, then delete it when you know you don't need it any more, which can be handled quite locally (typically), such as in this example:

class C
{
   std::vector<float>values;

   C(size_t num) : values(num, 0.0f) { }
   ~C() { } // data deleted automatically with vector

   void function()
   {
        for(float& f : values)
        {
            gf = &f;
            doSomething();
        }
        gf = nullptr;
   }
};

Missing the explicit call to operator delete[]? Well, the vector handles this implicitly for me, so I don't have to bother. Even if you are forced to use raw pointers, you can avoid explict deletion by use of e. g. std::unique_ptr... Attention with the vector, though: the pointer might get invalid, if you add new elements to the vector or remove contained ones from it!

In my examples, I set the gf pointer explicitly to nullptr so that you can inspect when there is no float value in use - you'd have to check for while hijacking, of course... Be aware that in an multithreaded environment, you possibly might protect your pointer against race conditions.