Why is virtual memory allocated with malloc not released?

Question

Given this C program where the private virtual memory is printed:

#include <stdio.h>
#include <stdlib.h>

#define _PSTAT64 
#include <sys/param.h> 
#include <sys/pstat.h> 
#include <sys/unistd.h> 

void pstatvm() 
{ 
    struct pst_vm_status pst; 
    int idx, count; 
    long long shared_vm = 0; 
    long long shared_ram = 0; 
    long long private_vm = 0; 
    long long private_ram = 0; 

    pid_t  pid = getpid();
    idx=0; 
    count = pstat_getprocvm(&pst, sizeof(pst), (size_t)pid, idx); 
    while (count > 0) { 
    switch ((long)pst.pst_type) { 
        case PS_IO: break; 
                        /* Don't count IO space. It really is not RAM or swap. */ 
        default: 
            if (pst.pst_flags & PS_SHARED) { 
                shared_vm += (long long) pst.pst_length; 
                shared_ram += (long long)pst.pst_phys_pages; 
            } else { 
                private_vm += (long long) pst.pst_length; 
                private_ram += (long long)pst.pst_phys_pages; 
            } 
                    break; 
    } 

    idx++; 
    count = pstat_getprocvm(&pst, sizeof(pst), (size_t)pid, idx); 
    } 
    printf("%d\t\t", pid); 
    printf("%lldK\t\t", shared_vm*4); 
    printf("%lldK\t\t", shared_ram*4); 
    printf("%lldK\t\t", private_vm*4); 
    printf("%lldK\n", private_ram*4); 
} 



int main()
{
    void *p=NULL;
    int cont = 1;

    printf("pid\t\tshared_vm\tshared_ram\tprivate_vm\tprivate_ram\n"); 

     while(cont < 100)
    {
        p = malloc((cont*10)*1024*1024);
        if (p == NULL)
        {
            printf("exit\n");
            exit(1);
        }
        pstatvm();
        free(p); p=NULL;
        sleep(1);
        cont += 10;
    }

    printf ("\n\n");
    cont = 0;
    while(cont < 10)
    {
        sleep(100);
        pstatvm();
        ++cont;
    }

}

Why is the private_vm not released by the OS?

pid             shared_vm       shared_ram      private_vm      private_ram
8988            3436K           2432K           26880K          320K
8988            3436K           2432K           129280K         336K
8988            3436K           2432K           231680K         352K
8988            3436K           2432K           334080K         368K
8988            3436K           2432K           436480K         384K
8988            3436K           2432K           538880K         400K
8988            3436K           2432K           641280K         416K
8988            3436K           2432K           743680K         432K
8988            3436K           2432K           846080K         448K
8988            3436K           2432K           948480K         464K


8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K
8988            3436K           2432K           948480K         464K

From http://www.linuxforums.org/forum/programming-scripting/153369-solved-where-does-memory-allocated-malloc-actually-come.html:

In Linux/Unix, all application memory is virtual, and unless it is swapped out to disc it points to a location in physical memory. Each page can be separately addressed, so while your application heap (memory allocated with malloc()) appears to your application as contiguous, it may actually be spread out all over your physical RAM

Answer 1

If you want to allocate large regions of memory such that they are really freed when your application no longer needs them, then use the POSIX mmap and munmap functions directly, rather than relying on the behavior that sufficiently large malloc requests are translated to mmap.

That may be how the GNU C Library works on GNU/Linux, but it isn't a standard-defined requirement, and so the expectation isn't portable.

Evidently, your HP-UX system's malloc is retaining the underlying storage of even large allocations. However, note how the increments in VM size are just around 100 Mb. This shows that the allocator is re-using the previously freed memory, avoiding fragmentation. That is to say, when you free a 300 Mb block and then allocate a 400 Mb, it's not retaining the original 300 Mb and then allocating a brand new 400 Mb block; it realizes that the 300 Mb space that was previously freed can just be extended. There could be additional hidden strategies in the allocator which your program's allocation pattern doesn't demonstrate. All we know from your program is that when we make a single large allocation and then free it, the allocator is able to keep it around and extend it to meet an even larger allocation request. That doesn't prove that the allocator always keeps around large allocations that were freed; it could be an optimization just for this case (perhaps for the sake of supporting a repeated realloc).

Answer 2

malloc() and free() are not obligated to allocate / deallocate virtual memory. They cope with heap. And regardless to fact that malloc() uses sbrk() to enlarge heap if need, free() doesn't shrink heap back, so amount of allocated virtual memory remains the same as immediately after malloc().