How this simple paging in ARMv8a works

Question

According to the ARM manual:

In the case of a 4kB granule, the hardware can use a 4-level look up process. The 48-bit address has nine address bits for each level translated (that is, 512 entries each), with the final 12 bits selecting a byte within the 4kB coming directly from the original address.

Bits [47:39] of the virtual address index into the 512 entry L0 table. Each of these table entries spans a 512GB range and points to an L1 table. Within that 512 entry L1 table, bits [38:30] are used as index to select an entry and each entry points to either a 1GB block or an L2 table.

Bits [29:21] index into a 512 entry L2 table and each entry points to a 2MB block or next table level. At the last level, bits [20:12] index into a 512 entry L2 table and each entry points to a 4kB block

This makes 100% sense for me. L0, L1, L2 tables and the final offset to arrive at a physical address.

However, look at this code: https://github.com/bztsrc/raspi3-tutorial/blob/abaf5a5b2bc1a9fdfe5a9d8191c061671555da3d/10_virtualmemory/mmu.c#L66, explained here:

Because we choose 4k as page size, and one translation entry is 8 bytes, that means we have 512 entries on each page. Therefore indeces 0..511 belong to the first page, 512..1023 to the second and so forth. With other words, the address of paging[0] equals to _end (first page), and paging[512] equals to _end + PAGESIZE (second page).

It looks like it's setting the L0, L1 and L2 as mentioned in the manual. So the first 512 entries would be the entries of the L0 table, the 513-1024 entries would be the L1 and 1025-1536 entries would be the L2 table.

However in the code it starts doing this like this:

paging[4*512+511]=(unsigned long)((unsigned char*)&_end+5*PAGESIZE) |   // physical address
        PT_PAGE |     // we have area in it mapped by pages
        PT_AF |       // accessed flag
        PT_KERNEL |   // privileged
        PT_ISH |      // inner shareable
        PT_MEM;       // normal memory

The index 4*512+511 = 2559 is way past the L2 table I imagined. I think I misunderstood something very wrong!

Should paging[0] and paging[511] span the first table (L0), then paging[512] and paging[2013] span the second table (L1) and paging[1024] and paging[2559] span the last table (L2)?

What about r<<21 and r*PAGESIZE, what do these mean?

Answer 1

There are two tables, which are pointed to by TTBR0 and TTBR1.

The first, TTBR0, points directly at &paging[0], and form the L0,L1,L2 page heirarchy:

Paging[0] points at &paging[512*2]
Paging[512*2] points at &paging[512*3]
Paging[512*3..512*3+511] contains page descriptors for physical memory at 0..200000.

Addintionally

Paging[512*2+1..512*2+511] contains large descriptors for physical memory at 400000..40000000

The second (kernel), TTBR1, points directly at &paging[512], forming a similar L0,L1,L2 heirarchy:

Paging[512+511] points at &paging[512*4]
Paging[512*4+511] points at &paging[512*5]
Paging[512*5] contains a descriptor for MMIO_BASE+0x201000.

The reason the second set is offset to the 511th descriptor of each table is to make it be at a very high address.

The virtual address decoding is selected by the translation control register's T1SZ; it is annotated as 3 Levels, or 39 bits of virtual addressing: 12 bits of offset and 27 bits of table indices (9 bits * 3 levels).

Address bits 63..40 traditionally must have the same value -- either all zeroes or all ones. This can be loosened in a control register, but regardless, bit 63 chooses which of the TTBR[01] will be used to select one of two L0 page table sets.

Traditionally, each process will have its own TTBR0, and the kernel will have a TTBR1 which is used for all processes [ thus needn't be changed ].