Why is VSIZE so much larger for 64 bit Linux process?

Question

My basic question is why is the VSIZE for a 64 bit process so much larger than that of the exact same program compiled for 32 bit?

The following is the output of the /proc/<pid>/maps file for the 32 bit process.

00148000-00149000 r-xp 00000000 00:00 0               [vdso]
00149000-002d2000 r-xp 00000000 fd:02 8914142         /lib/libc-2.12.so
002d2000-002d3000 ---p 00189000 fd:02 8914142         /lib/libc-2.12.so
002d3000-002d5000 r--p 00189000 fd:02 8914142         /lib/libc-2.12.so
002d5000-002d6000 rw-p 0018b000 fd:02 8914142         /lib/libc-2.12.so
002d6000-002d9000 rw-p 00000000 00:00 0 
005c9000-005da000 r-xp 00000000 fd:02 17059392        /tmp/vsizetest/lib/libtesting.so
005da000-005db000 rw-p 00010000 fd:02 17059392        /tmp/vsizetest/lib/libtesting.so
005db000-0061b000 rw-p 00000000 00:00 0 
00661000-00689000 r-xp 00000000 fd:02 8917713         /lib/libm-2.12.so
00689000-0068a000 r--p 00027000 fd:02 8917713         /lib/libm-2.12.so
0068a000-0068b000 rw-p 00028000 fd:02 8917713         /lib/libm-2.12.so
00694000-006ab000 r-xp 00000000 fd:02 8917680         /lib/libpthread-2.12.so
006ab000-006ac000 r--p 00016000 fd:02 8917680         /lib/libpthread-2.12.so
006ac000-006ad000 rw-p 00017000 fd:02 8917680         /lib/libpthread-2.12.so
006ad000-006af000 rw-p 00000000 00:00 0 
006e5000-00703000 r-xp 00000000 fd:00 3150403         /lib/ld-2.12.so
00703000-00704000 r--p 0001d000 fd:00 3150403         /lib/ld-2.12.so
00704000-00705000 rw-p 0001e000 fd:00 3150403         /lib/ld-2.12.so
00983000-009a0000 r-xp 00000000 fd:02 8914997         /lib/libgcc_s-4.4.5-20110214.so.1
009a0000-009a1000 rw-p 0001d000 fd:02 8914997         /lib/libgcc_s-4.4.5-20110214.so.1
00ca5000-00d86000 r-xp 00000000 fd:02 6300601         /usr/lib/libstdc++.so.6.0.13
00d86000-00d8a000 r--p 000e0000 fd:02 6300601         /usr/lib/libstdc++.so.6.0.13
00d8a000-00d8c000 rw-p 000e4000 fd:02 6300601         /usr/lib/libstdc++.so.6.0.13
00d8c000-00d92000 rw-p 00000000 00:00 0 
08048000-08049000 r-xp 00000000 fd:02 21134666        /tmp/vsizetest/bin/testvsz
08049000-0804a000 rw-p 00000000 fd:02 21134666        /tmp/vsizetest/bin/testvsz
09b8d000-09bae000 rw-p 00000000 00:00 0               [heap]
f7796000-f779c000 rw-p 00000000 00:00 0 
ff998000-ff9ae000 rw-p 00000000 00:00 0               [stack]

Which results in a total VSIZE of 3656.

The following is the output of the /proc/<pid>/maps file for the 64 bit process.

00400000-00401000 r-xp 00000000 fd:02 21134667              /tmp/vsizetest/bin64/testvsz
00600000-00601000 rw-p 00000000 fd:02 21134667              /tmp/vsizetest/bin64/testvsz
02301000-02322000 rw-p 00000000 00:00 0                     [heap]
3b7c800000-3b7c820000 r-xp 00000000 fd:00 661349            /lib64/ld-2.12.so
3b7ca1f000-3b7ca20000 r--p 0001f000 fd:00 661349            /lib64/ld-2.12.so
3b7ca20000-3b7ca21000 rw-p 00020000 fd:00 661349            /lib64/ld-2.12.so
3b7ca21000-3b7ca22000 rw-p 00000000 00:00 0 
3b7cc00000-3b7cd86000 r-xp 00000000 fd:00 661350            /lib64/libc-2.12.so
3b7cd86000-3b7cf86000 ---p 00186000 fd:00 661350            /lib64/libc-2.12.so
3b7cf86000-3b7cf8a000 r--p 00186000 fd:00 661350            /lib64/libc-2.12.so
3b7cf8a000-3b7cf8b000 rw-p 0018a000 fd:00 661350            /lib64/libc-2.12.so
3b7cf8b000-3b7cf90000 rw-p 00000000 00:00 0 
3b7d000000-3b7d083000 r-xp 00000000 fd:00 661365            /lib64/libm-2.12.so
3b7d083000-3b7d282000 ---p 00083000 fd:00 661365            /lib64/libm-2.12.so
3b7d282000-3b7d283000 r--p 00082000 fd:00 661365            /lib64/libm-2.12.so
3b7d283000-3b7d284000 rw-p 00083000 fd:00 661365            /lib64/libm-2.12.so
3b7d800000-3b7d817000 r-xp 00000000 fd:00 661352            /lib64/libpthread-2.12.so
3b7d817000-3b7da16000 ---p 00017000 fd:00 661352            /lib64/libpthread-2.12.so
3b7da16000-3b7da17000 r--p 00016000 fd:00 661352            /lib64/libpthread-2.12.so
3b7da17000-3b7da18000 rw-p 00017000 fd:00 661352            /lib64/libpthread-2.12.so
3b7da18000-3b7da1c000 rw-p 00000000 00:00 0 
3b7e000000-3b7e007000 r-xp 00000000 fd:00 661361            /lib64/librt-2.12.so
3b7e007000-3b7e206000 ---p 00007000 fd:00 661361            /lib64/librt-2.12.so
3b7e206000-3b7e207000 r--p 00006000 fd:00 661361            /lib64/librt-2.12.so
3b7e207000-3b7e208000 rw-p 00007000 fd:00 661361            /lib64/librt-2.12.so
3b87000000-3b87016000 r-xp 00000000 fd:00 664219            /lib64/libgcc_s-4.4.6-20110824.so.1
3b87016000-3b87215000 ---p 00016000 fd:00 664219            /lib64/libgcc_s-4.4.6-20110824.so.1
3b87215000-3b87216000 rw-p 00015000 fd:00 664219            /lib64/libgcc_s-4.4.6-20110824.so.1
3d44c00000-3d44ce8000 r-xp 00000000 fd:00 3019214           /usr/lib64/libstdc++.so.6.0.13
3d44ce8000-3d44ee8000 ---p 000e8000 fd:00 3019214           /usr/lib64/libstdc++.so.6.0.13
3d44ee8000-3d44eef000 r--p 000e8000 fd:00 3019214           /usr/lib64/libstdc++.so.6.0.13
3d44eef000-3d44ef1000 rw-p 000ef000 fd:00 3019214           /usr/lib64/libstdc++.so.6.0.13
3d44ef1000-3d44f06000 rw-p 00000000 00:00 0 
7f30ab397000-7f30ab39c000 rw-p 00000000 00:00 0 
7f30ab39c000-7f30ab3ad000 r-xp 00000000 fd:02 21127804      /tmp/vsizetest/lib64/libtesting.so
7f30ab3ad000-7f30ab5ac000 ---p 00011000 fd:02 21127804      /tmp/vsizetest/lib64/libtesting.so
7f30ab5ac000-7f30ab5ad000 rw-p 00010000 fd:02 21127804      /tmp/vsizetest/lib64/libtesting.so
7f30ab5ad000-7f30ab5ee000 rw-p 00000000 00:00 0 
7f30ab606000-7f30ab609000 rw-p 00000000 00:00 0 
7fff69512000-7fff69528000 rw-p 00000000 00:00 0             [stack]
7fff695ff000-7fff69600000 r-xp 00000000 00:00 0             [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0     [vsyscall]

Which results in a VSIZE of 18480.

The major difference between the 2 maps are the following entries from the 64 bit data:

3b7cd86000-3b7cf86000 ---p 00186000 fd:00 661350             /lib64/libc-2.12.so
3b7d083000-3b7d282000 ---p 00083000 fd:00 661365             /lib64/libm-2.12.so
3b7d817000-3b7da16000 ---p 00017000 fd:00 661352             /lib64/libpthread-2.12.so
3b7e007000-3b7e206000 ---p 00007000 fd:00 661361             /lib64/librt-2.12.so
3b87016000-3b87215000 ---p 00016000 fd:00 664219             /lib64/libgcc_s-4.4.6-20110824.so.1
3d44ce8000-3d44ee8000 ---p 000e8000 fd:00 3019214            /usr/lib64/libstdc++.so.6.0.13
7f30ab3ad000-7f30ab5ac000 ---p 00011000 fd:02 21127804       /tmp/vsizetest/lib64/libtesting.so

Which account for 14316 of the 18480 VSIZE.

Other experimentation with other programs seems to show that in 64 bit you seem to get one of these private, non-readable, non-writeable, non-executable chunks of memory for each shared library that is used by the process, while in 32 bit there are hardly any of these chunks.

Does anyone know what these chunks of memory are?

Note: Based on some answers to a similar question, What these memory regions for, from a Linux process?, this is not a multi-threaded process and it is already compiled -fPIC.

Answer 1

The major VSIZE difference comes from how the PROT_NONE mappings (mode "---p") of the shared libraries are done in the case of the 32-bit and 64-bit versions.

These are exactly the mappings that you spotted as producing the difference.

In general for each shared library loaded we will have four mappings:

3b7cc00000-3b7cd86000 r-xp 00000000 fd:00 661350            /lib64/libc-2.12.so
3b7cd86000-3b7cf86000 ---p 00186000 fd:00 661350            /lib64/libc-2.12.so
3b7cf86000-3b7cf8a000 r--p 00186000 fd:00 661350            /lib64/libc-2.12.so
3b7cf8a000-3b7cf8b000 rw-p 0018a000 fd:00 661350            /lib64/libc-2.12.so

The first one is the code segment with executable permissions, the second the PROT_NONE (mode ---) mapping (pages may not be accessed), and the last two ones the data segment (read only part and read write).

The PROT_NONE has size MAXPAGESIZE and so it is created differently in the 32-bit and 64-bit versions. In the case of the 32-bit version it has 4KB size (MAXPAGESIZE for i386) and in the case of the 64-bit version 2MB (standard MAXPAGESIZE for x86_64 systems).

It should be noted that this memory is not actually consumed (it just consumes addresses of the address space) as noted here:

http://www.greenend.org.uk/rjk/tech/dataseg.html

"This extra doesn’t cost you any RAM or swap space, just address space within each process, which is in plentiful supply on 64-bit platforms. The underlying reason is to do with keeping libraries efficiently sharable, but the implementation is a little odd."

Just a last trick, I find easier to check memory mappings using the pmap utility than parses the maps file and produces a simpler to read output:

For basic info:

pmap <PID>

For extended info:

pmap -x <PID>