How can we specify physical address for variable?

Question

Any suggestions/discussions are welcome!

The question is actually brief as title, but I'll explain why I need physical address.

---

Background:

These days I'm fascinated by cache and multi-core architecture, and now I'm quite curious how cache influence our programs, under the parallel environment.

In some CPU models (for example, my Intel Core Duo T5800), the L2 cache is shared among cores. So, if program A is accessing memory at physical address like

0x00000000, 0x20000000, 0x40000000...

and program B accessing data at

0x10000000, 0x30000000, 0x50000000...

Since these addresses share the same suffix, the related set in L2 cache will be flushed frequently. And we're expected to see two programs fighting with each other, reading data slowly from memory instead of cache, although, they are separated in different cores.

Then I want to verify the result in practice. In this experiment, I have to know the physical address instead of virtual address. But how can I cope with this?

---

The first attempt:

Eat a large space from heap, mask, and get the certain address.

My CPU has a L2 cache with size=2048KB and associativity=8, so physical addressess like 0x12340000, 0x12380000, 0x123c0000 will be related to the first set in L2 cache.

int HEAP[200000000]={0};
int *v[2];
int main(int argc, char **argv) {

    v[0] = (int*)(((unsigned)(HEAP)+0x3fffc) & 0xfffc0000);
    v[1] = (int*) ((unsigned)(v[0]) + 0x40000); 

    // one program pollute v[0], another polluting v[1]
}

Sadly, with the "help" of virtual memory, variable HEAP is not always continuous inside physical memory. v[0] and v[1] might be related to different cache sets.

---

The second attempt

access /proc/self/mem, and try to get memory information.

Hmm... seems that the results are still about virtual memory.

Answer 1

Your understanding of memory and these addresses is incomplete/incorrect. Essentially, what you're trying to test is futile.

In the context of user-mode processes, pretty much every single address you see is a virtual address. That is, an address that makes sense only in the context of that process. The OS manages the mapping of where this virtual memory space (unique to a process) maps to memory pages. These memory pages at any given time may map to pages that are paged-in (i.e. reside in physical RAM) - or they may be paged-out, and exist only in the swap file on disk.

So to address the Background example, those addresses are from two different processes - it means absolutely nothing to try and compare them. Whether or not their code is present in any of the caches depends on a number of things, including the cache-replacement strategy of the processor, the caching policies enabled by the OS, the number of other processes (including kernel-mode threads), etc.

In your first attempt, again you aren't going to get anywhere as far as actually testing CPU cache directly. First of all, your large buffer is not going to be on the heap. It is going to be part of the data section (specifically the .bss) of the executable. The heap is used for the malloc() family of memory allocations. Secondly, it doesn't really matter if you allocate some huge 1GB region, because although it is contiguous in the virtual address space of your process, it is up to the OS to allocate pages of virtual memory wherever it deems fit - which may not actually be contiguous. Again, you have pretty much no control over memory allocation from userspace. "Is there a way to allocate contiguous physical memory from userspace in linux?" The short answer is No.

/proc/$pid/maps isn't going to get you anywhere either. Yes there are plenty of addresses listed in there, but again, they are all in the virtual address space of process $pid. Some more information on these: How do I read from /proc/$pid/mem under Linux?