Fortran execute_command_line runtime error, depends on memory consumption

Question

I am getting runtime errors when trying to create a directory using the execute_command_line intrinsic in Fortran. The error occurs both with Ifort (18.0.3 20180410) and gfortran (4.8.5). Here is a minimal example that fails with whatever compile flags I use:

PROGRAM directory_test

    IMPLICIT NONE

    INTEGER :: cstat, estat, i, j
    CHARACTER(LEN=100) :: cmsg

    REAL, DIMENSION(:,:), ALLOCATABLE :: field
    INTEGER, PARAMETER :: fieldsize = 80000

    allocate(field(fieldsize,fieldsize))
    do j=1, fieldsize
        do i=1, fieldsize
            field(i,j) = real(i+j)
        end do
    end do

    call execute_command_line('mkdir -p newdir', WAIT=.true., EXITSTAT=estat, CMDSTAT=cstat, CMDMSG=cmsg)

    write(*,*) 'estat: ', estat
    write(*,*) 'cstat: ', cstat
    write(*,*) 'cmsg:  ', cmsg

END PROGRAM directory_test

Output ifort:
estat: 0
cstat: 124
cmsg: Invalid command supplied to EXECUTE_COMMAND_LINE

Output gfortran:
estat: -520880432
cstat: 1
cmsg: Termination status of the command-language interpreter cannot be obtained

Here is the catch: the program runs just fine as long as the array size is small enough. For me the threshold is about half of the physical memory used (adjust the value of "fieldsize" if you want to try the code). If the array is larger than that, the error occurs. If the array is smaller, the code executes without errors and the directory is created. The machines I used to test this all have 2 physical CPUs and 128GB-256GB of RAM.
What am I doing wrong?

OS: Linux, Opensuse 42.3
shell: bash
file system: Ext4

Edit: the issue is not exclusive to "execute_command_line()". Trying to do the same with "call system()" I get similar behaviour. The new directory is not created in cases where the original method fails with a runtime error. Additional tests on smaller/older dual-socket machines down to 48GB of available RAM yield the same results. Some fail earlier, some need almost the entire physical memory occupied by the program to fail. Unfortunately I don't have a single-socket machine to test right now.

Answer 1

So the way EXECUTE_COMMAND_LINE as well as the SYSTEM intrinsics are implemented on Linux is that first the process calls the fork() syscall, which creates a clone of the process. Then the child will call exec() which replaces that process with the new process to execute.

So even if the second forked process is very short-lived before it's replaced by the new exec()ed process, the system still needs sufficient virtual memory for both the original process and the forked child process. On Linux, if you have superuser access you can tune the limits (as a function of physical memory) with the various "overcommit" knobs.

If possible, one way to get around this problem is to call EXECUTE_COMMAND_LINE before you allocate a lot of memory, or after you have deallocated it.

Answer 2

You also posted this in the Intel Fortran for Linux forum. Replies indicated that your array allocation was using up all available virtual memory, leaving insufficient memory for EXECUTE_COMMAND_LINE or alternatives. Three different compilers all gave errors on your example - NAGfor was the most helpful with an "insufficient virtual memory" error.