C++: Multithreading and refcounted object

Question

I'm currently trying to pass a mono threaded program to multithread. This software do heavy usage of "refCounted" objects, which lead to some issues in multithread. I'm looking for some design pattern or something that might solve my problem.

The main problem is object deletion between thread, normally deletion only decrement the reference counting, and when refcount is equal to zero, then the object is deleted. This work well in monothread program, and allow some great performance improvement with copy of big object.

However, in multithread, two threads might want to delete the same object concurrently, as the object is protected by a mutex, only one thread delete the object and block the other one. But when it releases the mutex, then the other thread continue its execution with invalid (freed object), which lead to memory corruption.

Here is an example with this class RefCountedObject

class RefCountedObject
{
public:
RefCountedObject()
:   _refCount( new U32(1) )
{}

RefCountedObject( const RefCountedObject& obj )
:   _refCount( obj._refCount )
{
    ACE_Guard< ACE_Mutex > guard( _refCountMutex );
    ++(*_refCount);
}

~RefCountedObject()
{
    Destroy();
}

RefCountedObject& operator=( const RefCountedObject& obj )
{
    if( this != &obj )
    {
        Destroy();
        ACE_Guard< ACE_Mutex > guard( _refCountMutex );
        _refCount = obj._refCount;
        ++(*_refCount);
    }

    return *this;
}

private:
    void Destroy()
    {
        ACE_Guard< ACE_Mutex > guard( _refCountMutex );  // thread2 are waiting here
        --(*_refCount);         // This cause a free memory write by the thread2
        if( 0 == *_refCount )
            delete _refCount;
    }

private:
    mutable U32* _refCount;
    mutable ACE_Mutex _refCountMutex; // BAD: this mutex only protect the refCount pointer, not the refCount itself
};

Suppose that two threads want to delete the same RefCountedObject, both are in ~RefCountedObject and call Destroy(), the first thread has locked the mutex and the other one is waiting. After the deletion of the object by the first thread, the second will continue its execution and cause a free memory write.

Anyone has experience with a similar problem and found a solution ?

---

Thanks all for your help, I realize my mistake: The mutex is only protecting refCount pointer, not the refCount itself! I've created a RefCount class which is mutex protected. The mutex is now shared between all refCounted object.

Now all works fine.

Answer 1

If the count is part of the object then you have an inherent problem if one thread can be trying to increase the reference count whilst another is trying to remove the last reference. There needs to be an extra value on the ref count for each globally accessible pointer to the object, so you can always safely increase the ref count if you've got a pointer.

One option would be to use boost::shared_ptr (see the docs). You can use the free functions atomic_load, atomic_store, atomic_exchange and atomic_compare_exchange (which are conspicuously absent from the docs) to ensure suitable protection when accessing global pointers to shared objects. Once your thread has got a shared_ptr referring to a particular object you can use the normal non-atomic functions to access it.

Another option is to use Joe Seigh's atomic ref-counted pointer from his atomic_ptr_plus project

Answer 2

Surely each thread simply needs to manage the reference counts correctly... That is, if ThreadA and ThreadB are both working with Obj1 then BOTH ThreadA and ThreadB should own a reference to the object and BOTH should call release when they're done with the object.

In a single threaded application it's likely that you have a point where a reference counted object is created, you then do work on the object and eventually call release. In a multi-threaded program you would create the object and then pass it to your threads (however you do that). Before passing the object to the thread you should call AddRef() on your object to give the thread its own reference count. The thread that allocated the object can then call release as it's done with the object. The threads that are working with the object will then call release when they're done and when the last reference is released the object will be cleaned up.

Note that you dont want the code that's running on the threads themselves to call AddRef() on the object as you then have a race condition between the creating thread calling release on the object before the threads that you've dispatched to get a chance to run and call AddRef().