Why does Rcpp function segfault only when called from R package but not when sourced directly via sourceCpp?

Question

I have a C++ function that works in R when sourced independently using Rcpp, but when I include it in a compiled R package, I keep receiving the following error: error: arma::memory::acquire(): out of memory. In this post, I have provided the C++ code containing the function Znew_gen2 that I want to use in my compiled R package. Using a working example, I can show that the Znew_gen2 function works when I source it independently (outside of an R package). However, when I call this same function from an example R package named HelpWithZnewgen2, the function no longer works and I get the error error: arma::memory::acquire(): out of memory. All of the code mentioned in this post can be found in the GitHub repo https://github.com/hheiling/myrepo_R.

My working example is provided here:

# Working Example:

library(bigmemory)
library(Matrix)

set.seed(1)
U = matrix(rnorm(3000), nrow=100, ncol=30)
Z = matrix(rnorm(15000), nrow = 500, ncol = 30)
group = rep(1:10, each = 50)
cols = c(1,11,21)
n = 500
q = 3
d = 10 
Znew = big.matrix(nrow = nrow(Z)*nrow(U), ncol = 6)
J_SpMat = Matrix(0, 9, 6, sparse = TRUE)
  sumy = 0
  sumx = 0
  zeros = 0
  for(i in 1:3){
    J_SpMat[ sumx + zeros + 1:(3 - (i-1)), sumy + 1:(3 - (i-1))] = diag((3 - (i-1)))
    sumy = sumy + (3 - (i-1))
    sumx = sumy
    zeros = zeros + i
  }

When I run the working example and call the Znew_gen2 function using sourceCPP, as shown below, the function runs with no errors.

library(Rcpp)

## Code to download the "Znew_gen2.cpp" file from the GitHub repo and 
## specify where you want the file to download to:
destination_file = "Source_Code_Znew_gen2.cpp" 
  # Can specify however you like, but must not have spaces in the filename
download.file(url = "https://raw.githubusercontent.com/hheiling/myrepo_R/master/Znew_gen2.cpp", 
              destfile = destination_file)
sourceCpp(file = destination_file)

# Calling the sourced `Znew_gen2` function:
Znew_gen2(U, Z, group, cols, n, q, d, Znew@address, J_SpMat)

## Output:

# First For Loop 
# Second For Loop 
# End of Function 

However, when I place this same Znew_gen2 function in an R package and call this function from the R package, I get the error: error: arma::memory::acquire(): out of memory. For illustrative purposes, I have created an R package named HelpWithZnewgen2 and call the Znew_gen2 Rcpp function using a wrapper function named Znew_gen2.Rfunction.

# Instructions to download the `HelpWithZnewgen2` package:
library(devtools)
library(remotes)
install_github("hheiling/myrepo_R", subdir = "HelpWithZnewgen2")

library(HelpWithZnewgen2)

# Calling the function from the compiled package:
Znew_gen2.Rfunction(U, Z, group, cols, n, q, d, Znew@address, J_SpMat)

The error received:

# error: arma::memory::acquire(): out of memory
# Error in Znew_gen2(U, Z, group, cols, n, q, d, pBigMat, J) : 
#   std::bad_alloc

In another setting, I have tried calling the Znew_gen2 function from within another function in an R package, and I get a similar error regarding memory allocation.

Because the code itself works when sourced outside of the R package, I suspect my issue is related to how my R package is set up. From searching online, I wouldn't be surprised if there is an issue with one or more of the following components: the lines in Znew_gen2.cpp before the "using namespace Rcpp" line, my Description file, or possibly some line I am missing in the Makevars file of the R package. Despite my suspicions (which may not be correct, since I'm relatively new to writing R packages), I haven't been able to address the issue. Therefore, I would greatly appreciate any suggestions on how to fix this problem.

Further details of the Znew_gen2 code (file Znew_gen2.cpp, https://github.com/hheiling/myrepo_R/blob/master/Znew_gen2.cpp) and components of the HelpWithZnewgen2 package are provided the Github repo https://github.com/hheiling/myrepo_R. Since I'm not sure which, if any, of these details are relevant for answering the question, they are not posted here.

All of the code above is provided in the file Stack Overflow Example.R https://github.com/hheiling/myrepo_R/blob/master/Stack%20Overflow%20Example.R.

Answer 1

I hinted in the comments that should try to decompose your problem into smaller ones. I just tried a little bit. Given the persistent

 error: arma::memory::acquire(): out of memory
 Error in Znew_gen2(U, Z, group, cols, n, q, d, Znew@address, J_SpMat) : 
   std::bad_alloc
 Execution halted

I added code to check your parameters, and indeed for

 Rcpp::Rcout << "n*nMC is " << n*nMC << ", J.ncols is " << J.n_cols << std::endl;

I see

 n*nMC is 50000, J.ncols is 25769803830

so currently your problem is not with a bigmemory object but rather the sparse matrix.

Edit a little while later: Turns out that the problem is likely your use of #define ARMA_64BIT_WORD which makes the matrix dimensions be outside of int and hence the appearance of these YUGE values. If I removed that, your code runs.

So lesson re-learned: make the problem smaller and smaller, here it meant decompose your use of a sparse matrix from the use of bigmemory matrix.