Schedule error when calling multi-threaded C FFI with Haskell Callback function

Question

Below is a Haskell/C FFI code that is throwing schedule error at runtime (GHC 7.0.3, Mac OS 10.7, x86_64). I searched for explanation of the error but didn't find anything relevant.

C Code (mt.c):

#include <pthread.h>
#include <stdio.h>

typedef void(*FunctionPtr)(int);

/* This is our thread function.  It is like main(), but for a thread*/
void *threadFunc(void *arg)
{
  FunctionPtr fn;
  fn = (FunctionPtr) arg;
  fn(1); //call haskell function with a CInt argument to see if it works
}

void create_threads(FunctionPtr* fp, int numThreads )
{
  pthread_t pth[numThreads];  // array of pthreads
  int t;
  for (t=0; t < numThreads;){
    pthread_create(&pth[t],NULL,threadFunc,*(fp + t));
    t++;
  }

  printf("main waiting for all threads to terminate...\n");
  for (t=0; t < numThreads;t++){
    pthread_join(pth[t],NULL);
  }
}

Haskell code (t.hs) - it calls create_threads in mt.c above with Storable Vector of FunPtr to Haskell function f (after applying first three arguments to f):

{-# LANGUAGE BangPatterns #-}
import Control.Concurrent (forkIO, threadDelay, MVar, newEmptyMVar, putMVar, takeMVar)
import qualified Data.Vector.Storable.Mutable as MSV
import qualified Data.Vector.Storable as SV
import Control.Monad.Primitive (PrimState)
import Control.Monad (mapM, forM_)
import Foreign.Ptr (Ptr, FunPtr)
import Foreign.C.Types (CInt)


type Length = CInt

-- | f is a function that is called back by create_threads in mt.c
f :: MVar Int -> MSV.MVector (PrimState IO) CInt -> Length -> CInt -> IO ()
f m v l x = do
              !i <- takeMVar m
              case (i< fromIntegral l) of
                True -> MSV.unsafeWrite v i x >> print x >> putMVar m (i+1) 
                False -> return () -- overflow

-- a "wrapper" import gives us a converter for converting a Haskell function to a foreign function pointer
foreign import ccall "wrapper"
  wrap :: (CInt -> IO()) -> IO (FunPtr (CInt -> IO()))

foreign import ccall safe "create_threads"
  createThreads :: Ptr (FunPtr (CInt -> IO())) -> CInt -> IO()

main = do
  let threads = [1..4]
  m <- mapM (\x -> newEmptyMVar) $ threads
  -- intialize mvars with 0
  forM_ m $ \x -> putMVar x 0
  let l = 10
  -- intialize vectors of length 10 that will be filled by function f
  v <- mapM (\x -> MSV.new l) threads
  -- create a list of function pointers to partial function - the partial function is obtained by applying first three arguments to         function f
  lf <- mapM (\(x,y) -> wrap (f x y (fromIntegral l))) $ zip m v
  -- convert above function list to a storable vector of function pointers
  let fv = SV.fromList lf
  -- call createThreads with storable vector of function pointers, and number of threads - createThreads will spawn threads which will use  function pointers for callback
  SV.unsafeWith fv $ \x -> createThreads x (fromIntegral $ length threads)

Please ignore unsafe parts in the code - my objective here is to test callback using Haskell FFI with multi-threaded C code. When I compile it, and run it, I get the error below:

$ ghc -O2 t.hs mt.c -lpthread
[1 of 1] Compiling Main             ( t.hs, t.o )
Linking t ...
$ ./t
main waiting for all threads to terminate...
t: schedule: re-entered unsafely.
   Perhaps a 'foreign import unsafe' should be 'safe'?
$ uname -a
Darwin desktop.local 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug  9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.0.3

The schedule error happens only if I have the C threads call back haskell function f. I guess it is more likely there is a bug in my code, than there is a bug in one of the libraries or GHC. So, I will like to check here first for pointers on cause of the error.

Answer 1

In this case, the schedule error happened because the haskell code was compiled without -threaded option.

The haskell code is calling C function create_threads which spawns multiple threads for threadFunc C function. threadFunc calls back into Haskell function f. So, even though Haskell code is compiled without -threaded option, it still results in multiple C threads executing f.

It was a good catch by GHC Runtime Scheduler to detect this oversight of executing multiple threads without threaded runtime, and mark it as error. That is much better than a cryptic run-time crash. I realized the oversight when I checked rts/schedule.c code in GHC code base, and saw the comment below. It tipped me off about threaded runtime not being enabled:

// Check whether we have re-entered the RTS from Haskell without
 // going via suspendThread()/resumeThread (i.e. a 'safe' foreign
 // call).