How to find a (segmentation fault) bug in C++ (pthread) multithread program on linux?

Question

I am doing debug for a (pthread) multithread C++ program on Linux.

It works well when thread number is small such as 1, 2,3.

When thread number is increased, I got SIGSEGV (segmentation fault , UNIX signal 11).

But, the error sometimes appear and sometimes disappear when I increase thread number above 4.

I used valgrind, I got

==29655== Process terminating with default action of signal 11 (SIGSEGV)

==29655== Access not within mapped region at address 0xFFFFFFFFFFFFFFF8

==29655== at 0x3AEB69CA3E: std::string::assign(std::string const&) (in /usr/lib64/libstdc++.so.6.0.8)

==29655== by 0x42A93C: bufferType::getSenderID(std::string&) const (boundedBuffer.hpp:29)

It seems that my code tried to read a memory which is not allocated. But, I cannot find any bugs in the function getSenderID(). It only return a string of a member data in Class bufferType. It has been initialized.

I used GDB and DDD (GDB GUI) to find the bug , which also points there but the error sometimes disappear so that in GDB, I cannot capture it with breakpoint.

Moreover, I also print out values of the function pointed by valgrind, but it is not helpful because multiple threads print out results with different orders and they interleave with each other. Each time I run the code, the print-output is different.

The bufferType is in a map, the map may have multiple entries. Each entry can be written by one thread and read by another thread at the same time. I have used pthread read/write lock to lock a pthread_rwlock_t. Now, there is no SIGSEGV but the program stops in some point without progress. I think this is a deadlock. But, one map entry can only be written by only one thread at one time point, why still have deadlock ?

Would you please recommend some methods to capture the bug so that I can find it no matter how many threads I use to run the code.

thanks

The code of boundedBuffer.hpp is as follows:

 class bufferType
 {
 private:

    string senderID;// who write the buffer

    string recvID; // who should read the buffer

    string arcID; // which arc is updated

    double price; // write node's price 

    double arcValue; // this arc flow value 

    bool   updateFlag ;

    double arcCost;


    int  arcFlowUpBound; 

    //boost::mutex  senderIDMutex; 

    //pthread_mutex_t  senderIDMutex; 

    pthread_rwlock_t       senderIDrwlock;

    pthread_rwlock_t    setUpdateFlaglock;

  public: 
   //typedef boost::mutex::scoped_lock lock;  // synchronous read / write 

   bufferType(){}

   void   getPrice(double& myPrice ) const {myPrice = price;}

   void   getArcValue(double& myArcValue ) const {myArcValue = arcValue;}

   void   setPrice(double& myPrice){price = myPrice;}

   void   setArcValue(double& myValue ){arcValue = myValue;}

   void   readBuffer(double& myPrice, double& myArcValue );

   void   writeBuffer(double& myPrice, double& myArcValue );

   void   getSenderID(string& myID) 

   {
       //boost::mutex::scoped_lock lock(senderIDMutex);
      //pthread_rwlock_rdlock(&senderIDrwlock); 
      cout << "senderID is " << senderID << endl ; 
      myID = senderID;
      //pthread_rwlock_unlock(&senderIDrwlock);
   }
//void   setSenderID(string& myID){ senderID = myID ;}

    void   setSenderID(string& myID)

    { 
        pthread_rwlock_wrlock(&senderIDrwlock); 

            senderID = myID ;

            pthread_rwlock_unlock(&senderIDrwlock);
    }

    void   getRecvID(string& myID) const {myID = recvID;}

    void   setRecvID(string& myID){ recvID = myID ;}

    void   getArcID(string& myID) const {myID  = arcID ;}

    void   setArcID(string& myID){arcID = myID ;}

    void   getUpdateFlag(bool& myFlag)
    {
            myFlag = updateFlag ; 

        if (updateFlag)

           updateFlag  = false; 
    }

//void   setUpdateFlag(bool myFlag){ updateFlag = myFlag ;}

    void   setUpdateFlag(bool myFlag)
    { 
        pthread_rwlock_wrlock(&setUpdateFlaglock);

        updateFlag = myFlag ;

         pthread_rwlock_unlock(&setUpdateFlaglock);

    }

   void   getArcCost(double& myc) const {myc = arcCost; }

   void   setArcCost(double& myc){ arcCost = myc ;}

   void   setArcFlowUpBound(int& myu){ arcFlowUpBound = myu ;}

   int    getArcFlowUpBound(){ return arcFlowUpBound ;}

   //double getLastPrice() const {return price; }

   } ;

From the code, you can see that I have tried to use read/write lock to assure invariant. Each entry in map has a buffer like this above. Now, I have got deadlock.

Answer 1

Access not within mapped region at address 0xFFFFFFFFFFFFFFF8

at 0x3AEB69CA3E: std::string::assign(std::string const&)

This would normally mean that you are assigning to a string* that was NULL, and then got decremented. Example:

#include <string>

int main()
{
  std::string *s = NULL;

  --s;
  s->assign("abc");
}

g++ -g t.cc && valgrind -q ./a.out

...
==20980== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==20980==  Access not within mapped region at address 0xFFFFFFFFFFFFFFF8
==20980==    at 0x4EDCBE6: std::string::assign(char const*, unsigned long)
==20980==    by 0x400659: main (/tmp/t.cc:8)

...

So show us the code in boundedBuffer.hpp (with line numbers), and think how that code could end up with a string pointer that points at -8.

Would you please recommend some methods to capture the bug so that I can find it no matter how many threads I use to run the code.

When thinking about multi-threaded programs, you must think about invariants. You should put assertions to confirm that your invariants do hold. You should think how they might be violated, and what violations would cause the post-mortem state you have observed.