What is reference when it says L1 Cache Reference or Main Memory Reference

Question

So I am trying to learn performance metrics of various components of computer like L1 cache, L2 cache, main memory, ethernet, disk etc as below:

Latency Comparison Numbers
--------------------------
L1 cache **reference**                       0.5 ns
Branch mispredict                            5   ns
L2 cache **reference**                       7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory **reference**                  100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy            10,000   ns       10 us
Send 1 KB bytes over 1 Gbps network     10,000   ns       10 us
Read 4 KB randomly from SSD*           150,000   ns      150 us          ~1GB/sec SSD
Read 1 MB sequentially from memory     250,000   ns      250 us
Round trip within same datacenter      500,000   ns      500 us
Read 1 MB sequentially from SSD*     1,000,000   ns    1,000 us    1 ms  ~1GB/sec SSD, 4X memory
Disk seek                           10,000,000   ns   10,000 us   10 ms  20x datacenter roundtrip
Read 1 MB sequentially from 1 Gbps  10,000,000   ns   10,000 us   10 ms  40x memory, 10X SSD
Read 1 MB sequentially from disk    30,000,000   ns   30,000 us   30 ms 120x memory, 30X SSD
Send packet CA->Netherlands->CA    150,000,000   ns  150,000 us  150 ms

I don't think the reference mentioned above is for how much data is read in bits or bytes. But is actually about maybe accessing one address in cache or memory. Can someone please explain better what is this reference that's happening in 0.5 n/s ?

Answer 1

• This table is listing typical numbers for some representative system, as the actual values for a real example system would hardly be so "smooth numbers" but complicated sums over some non-even multiples of CPU and/or bus clock periods. We could find such a table in a textbook for educative use. This one apparently found its way into a general introduction into system designing¹ from some conference presentations Google AI's lead person, Jeff Dean held in back in 2009^3,4.

• The two presentation PDFs^3,4 do not give an explicit definition what exactly was meant by "reference" in those tables. Instead, the tables are presented to point out that the ability for "back-of-the-envelope calculations" is crucial for successful system design.

• The term "reference" likely means retrieving a piece of information from the corresponding level of memory if the requested value is maintained there, so that it doesn't have to be reloaded from a slower source:

  L1 cache <- L2 cache <- Main memory (RAM) <- Disk (e.g., swap)

  The upper-level sources (RAM, disk) can just be seen as a very rough sketch because here you will find lots of sub-levels and variants (type of mass device, internal cache on the disk's chipset, buses/bridges etc. etc.). The present numbers appear to be a conclusion of experiences at Google's data center. Therefore, let's assume they are based on some high-performance class hardware which was relevant in 2009 (or earlier). Today (2020), the numbers should not be taken literally but to demonstrate the orders of magnitude in the context of the corresponding values for other levels of data transfer.

• The label "branch mispredict" stands for all cases when a fetch operation from the next level is necessary, because a mispredicted branching decision is the most important reason for cases when such a fetch operation is critical w. r. t. latencies. In other cases, branch prediction infrastructure is supposed to trigger data fetch operations in time so all latencies beyond the low "reference" value are hidden behind pipeline operations.

---

¹ The URL you gave us in comment discussion

"Latency numbers every programmer should know" in: "The System Design Primer"

references the following sources:

²Jeff Dean: "Latency Numbers Every Programmer Should Know", 31 May 2012.

"Originally by Peter Norvig ("Teach Yourself Programming in Ten Years") with some updates from Brendan", 1 Jun 2012.

³Jeff Dean: "Designs, Lessons and Advice from Building Large Distributed Systems", 13 Oct 2009, page 24.

⁴Jeff Dean: "Software Engineering Advice from Building Large-Scale Distributed Systems", 17 Mar 2009, page 13.