Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What's the max file mapping size in 64bits machine

I'm new to 64-bits architecture. Could you tell me what's MAX file size supported by file mapping in 64 bits linux machine. I want to open more than 20GB files by file mapping, is it available?

I write a sample code. But it causes Bus Error when I get the value of the pointer in GBSIZE offset:

unsigned char* pCur = pBegin + GBSIZE;
//pBegin is the pointer returned by mmap
printf("%c",*pCur); 

BTW, printf("%c",*pBegin ); works fine. and my address sizes : 38 bits physical, 48 bits virtual

Here is the full code:

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>

//#define FILEPATH "smallfile"
#define FILEPATH "bigfile"
#define GBSIZE (1024L*1024L*1024L)
#define TBSIZE (1024L*GBSIZE)
#define NUMSIZE  (20L * GBSIZE)
//#define NUMSIZE  (10)
#define FILESIZE (NUMINTS * sizeof(int))

int main(int argc, char *argv[])
{
    int i;
    int fd;
    unsigned char *pBegin;

    fd = open(FILEPATH, O_RDONLY);
        if (fd == -1) {
        perror("Error opening file for reading");
        exit(EXIT_FAILURE);
    }

    pBegin = mmap(0, NUMSIZE, PROT_READ, MAP_SHARED, fd, 0);
    if (pBegin == MAP_FAILED) {
        close(fd);
        perror("Error mmapping the file");
        exit(EXIT_FAILURE);
    }

    /** ERROR happens here!!! **/
    unsigned char* pCur = pBegin + GBSIZE;
    printf("%c",*pCur);

    if (munmap(pBegin, NUMSIZE) == -1) {
        perror("Error un-mmapping the file");
    }
    close(fd);
    return 0;
}
like image 783
Xuchao Zhang Avatar asked Jan 29 '10 02:01

Xuchao Zhang


People also ask

Are memory mapped files faster?

Accessing memory mapped files is faster than using direct read and write operations for two reasons. Firstly, a system call is orders of magnitude slower than a simple change to a program's local memory.

What is memory mapping a file?

A memory-mapped file contains the contents of a file in virtual memory. This mapping between a file and memory space enables an application, including multiple processes, to modify the file by reading and writing directly to the memory.

Are memory mapped files thread safe?

Yes. If one thread changes part of the data in the mapping, then all other threads immediately see that change.


2 Answers

Although pointers are 64-bit wide, most processors do not actually support virtual addresses using the full 64 bits. To see what size virtual addresses your processor supports, look in /proc/cpuinfo (48 bits is typical).

grep "address sizes" /proc/cpuinfo

Additionally, half of the virtual address space is used by the kernel and not available to userspace - leaving 47 bits in the current Linux implementation.

However, even taking this into account, you will still have plenty of room for a 20GB file. 47 bits in theory means a virtual address space of 128TB.

like image 190
caf Avatar answered Sep 28 '22 06:09

caf


From the mmap(2) man page:

   void *mmap(void *addr, size_t length, int prot, int flags,
              int fd, off_t offset);

length is a size_t, which on 64-bit machines is 64 bits in length. Therefore yes, you can theoretically map a 20GB file.

like image 27
Ignacio Vazquez-Abrams Avatar answered Sep 28 '22 06:09

Ignacio Vazquez-Abrams