Systems Research

The Role of Aligning Memory Addresses With Cachelines

Thu, 10 Oct 2024 09:17:21 +0000

There are three common reasons for aligning memory addresses with cachelines:

Improving Performance
Maintaining Atomicity
Preventing False Sharing

The third reason has been widely discussed online, so we will not cover it here. Instead, we will focus on the first two reasons.

Cross-Line Performance

Let’s start with the following program. The program first allocates an 8K memory space. It’s important to note that the starting address of the space provided by malloc is aligned to 8 or 16 bytes. We need to manually align it to 64 bytes for easier manipulation. The allocated space is 8K, which is the size of two pages. Regardless of the starting address, this space will contain a page boundary, which is a good alignment point and useful for testing. The subsequent operations simply involve repeatedly writing data and measuring the total time to calculate the average time per operation.

The program accepts an input parameter offset, which specifies the starting write position from buf_pageend+offset. Thus, this parameter controls the starting position of the write operation.

#include <stdio.h>
#include <time.h>
#include <string.h>
#include <stdlib.h>

#define BUF_SIZE 8192
#define ROUND 100000000UL
int main(int argc, char **argv)
{
 char *buf, *buf_newaddr, *buf_pageend;
 unsigned long i __attribute__((aligned(64)));
 unsigned long offset __attribute__((aligned(64)));
 struct timespec start={0,0}, end={0,0};
 double start_ns, end_ns;

 if (argc != 2) {
 printf("missing args\n");
 exit(-1);
 }

 offset = atoi(argv[1]);

again:
 buf = (void *)malloc(BUF_SIZE);
 /* Align to page boundary */
 buf_pageend = (void *)((unsigned long)(buf + 4095) & 0xfffffffffffff000UL);
 if (buf_pageend - buf < 1024) { // Ensure sufficient space before the pageend in case offset is negative
 goto again; // No free, continue allocation, otherwise the same memory block may be allocated
 }
 memset(buf, 0, BUF_SIZE); // Pre-allocate

 printf("&i = %lx, &offset=%lx\n", &i, &offset);
 clock_gettime(CLOCK_MONOTONIC, &start);
 for (i = 0; i < ROUND; i++) {
 *((volatile unsigned long *)(buf_pageend + offset)) = 0; // Write 8 bytes
 }
 clock_gettime(CLOCK_MONOTONIC, &end);
 start_ns = start.tv_sec * 1000000000 + start.tv_nsec;
 end_ns = end.tv_sec * 1000000000 + end.tv_nsec;
 printf("ns per store: %lf\n", (end_ns - start_ns)/ROUND);
}

Now that we’ve gone through the program, let’s start testing:

On my system, the cacheline size of the L1 cache is 64 bytes:

$ getconf LEVEL1_DCACHE_LINESIZE
64

When memory is loaded into the cache, it is loaded as a whole cacheline, aligned to a 64-byte boundary. It won’t load in smaller granularity. For example, if you access address 0x2, what actually gets loaded is the 64-byte block from 0x0 to 0x3F.

In our program, buf_pageend points to the page boundary, which is aligned to 4K and, therefore, also aligned to 64B (cacheline-aligned). So, when we write to the range [buf_pageend, buf_pageend+63], no matter where we write within this range, we are only accessing one cacheline, and the speed will be the fastest.

$ ./a.out 0; ./a.out 1; ./a.out 56
&i = 7ffccd268780, &offset=7ffccd2687c0
ns per store: 0.429188
&i = 7ffc0f704140, &offset=7ffc0f704180
ns per store: 0.436388
&i = 7ffce5fedbc0, &offset=7ffce5fedc00
ns per store: 0.434455

However, if we pass in the parameter 57, since we are writing 8 bytes, the write range will be [buf_pageend+57, buf_pageend+64]. Notice that the last byte overflows into the next cacheline, so the speed decreases. The same applies if you pass in values from 58 to 63.

$ ./a.out 57; ./a.out 58; ./a.out 63
&i = 7fff250ea3c0, &offset=7fff250ea400
ns per store: 0.847032
&i = 7ffe82952d40, &offset=7ffe82952d80
ns per store: 0.851936
&i = 7fff046c6f00, &offset=7fff046c6f40
ns per store: 0.844823

If you pass in 64, you’re back to reading and writing within a single cacheline ([buf_pageend+64, buf_pageend+127]), and the speed increases again.

$ ./a.out 64
&i = 7ffd12741400, &offset=7ffd12741440
ns per store: 0.438385

Why does cross-line access slow down? This is because writing once requires accessing two cachelines to complete the operation, effectively doubling the time.

If you not only cross cachelines but also cross pages, the speed will be even slower. For example, if the first 4 bytes of the 8-byte write fall at the end of one page and the last 4 bytes fall at the beginning of the next page, the two pages (called virtual pages) typically map to non-contiguous physical pages. The x86 architecture uses VIPT (Virtually Indexed, Physically Tagged), and when the CPU looks up a cacheline, it must resolve the virtual address to a physical address when matching the tag. Since our data crosses cachelines, it first resolves the virtual address and writes the first cacheline, then resolves the virtual address again to write the second cacheline. This address resolution process is slow, and if a TLB miss occurs, it becomes even slower. So, cross-page writes not only incur the penalty of crossing cachelines but also the penalty of crossing pages, resulting in a significant performance degradation.

$ ./a.out -1; ./a.out -7

&i = 7fff143b3400, &offset=7fff143b3440
ns per store: 10.084426
&i = 7ffcfb6bb8c0, &offset=7ffcfb6bb900
ns per store: 10.089472

Of course, if you pass in -8, the write will fall entirely within the previous cacheline, and the speed will be fast again.

Atomicity

We know that the MESI protocol in cache coherence ensures data consistency when multiple CPUs read and write to the same cacheline, meaning that a single cacheline’s reads and writes are inherently atomic. But how does cross-line access break atomicity? Let’s look at the following program.

The main process creates 80 child processes. All parent and child processes share a memory region. Forty of the child processes repeatedly write 8 bytes of all fs to an address, while the other forty child processes repeatedly write 8 bytes of all 0s. The main process repeatedly reads these 8 bytes. If atomicity is preserved, the main process should see either 8 bytes of all 0s or 8 bytes of all fs, but never any other value. If other values appear, atomicity has been violated.

The program accepts an input parameter offset, which specifies the starting write position from buf+offset, allowing us to control the starting position of the write operation.

#include <stdio.h>
#include <time.h>
#include <string.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/wait.h>

#define BUF_SIZE 1024
int main(int argc, char **argv) {
 char *buf;
 int pid, status;
 int count = 80;
 int i,j;
 int buf_realsize;
 int write_count = 10000000;
 int read_count = 30000000;
 unsigned long data;
 int offset;

 if (argc != 2) {
 printf("missing args\n");
 exit(-1);
 }

 offset = atoi(argv[1]);

 buf = mmap(NULL, BUF_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
 memset(buf, 0xdd, BUF_SIZE);
 buf = (void *)((unsigned long)(buf + 63) & 0xffffffffffffffc0);

 for (i = count; i; i--) {
 pid = fork();
 if (pid == 0) {
 int child_pid = getpid();
 if (child_pid % 2)
 data = 0xffffffffffffffff;
 else
 data = 0x0000000000000000;
 for (j = write_count; j; j--)
 *(volatile unsigned long *)(buf + offset) = data;
 exit(0);
 } else {
 // parent
 }
 }

 for (j = read_count; j; j--) {
 data = *(unsigned long *)(buf + offset);
 if (data != 0xffffffffffffffff && data != 0)
 printf("non-atomic: %016lx\n", data);
 }
 while ((pid = wait(&status)) > 0);
}

If you pass in offset values like 0, 1, ..., 56, the 8-byte writes will clearly fall within the range [buf, buf+63], i.e., within a single cacheline. There is no issue with atomicity.

$ ./a.out 0; ./a.out 10; ./a.out 56;

As we can see, there is no output, indicating that no non-atomic situations occurred.

However, if you pass in 57, 58, ..., 63, the 8-byte write will span two cachelines. There is no atomicity guarantee between writing to the first and second cachelines. As a result, the 80 child processes can interfere with each other. For example, if the offset is 60, half of the data is written to one cacheline and the other half to another. The main process might see a mix of data, like 0xffffffff00000000 or 0x00000000ffffffff, from two different processes. Of course, all 0s or all fs are also possible, but you wouldn’t be able to tell just by looking.

$ ./a.out 60;
non-atomic: ffffffff00000000
non-atomic: 00000000ffffffff
non-atomic: 00000000ffffffff
non-atomic: 00000000ffffffff
non-atomic: ffffffff00000000
non-atomic: 00000000ffffffff
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000
non-atomic: ffffffff00000000

To avoid such situations, you would need to use the lock prefix at the assembly level. However, the best solution is simply to ensure alignment.

Compiler Optimization

The code compiled with -O0 is unoptimized and contains many performance bottlenecks, such as instruction address alignment, branch prediction failures, pipeline bubbles, store-forwarding stalls, etc. These factors introduce significant overhead and obscure the performance differences of cross-line access. Therefore, the above program must be compiled with -O2 to eliminate these irrelevant performance bottlenecks.

Additionally, with -O2 optimizations, the compiler will unroll constant loops and perform dead code elimination, meaning intermediate assignment statements may be removed or reduced to a single assignment. Therefore, we write assignment statements in the following way:

*(volatile unsigned long *)(buf + offset) = xxx;

The volatile keyword tells the compiler not to optimize away this statement, even if the same value is written multiple times. Whatever I write, the compiler must assign exactly as specified.

Revisiting the Volatile Keyword in C

Thu, 10 Oct 2024 08:58:57 +0000

The Most Common Usage

When a variable is declared as volatile, it tells the compiler that even though the current code being compiled does not modify this variable, the corresponding memory data might be modified for other reasons. There are many possible reasons for this, such as the memory location corresponding to the variable being mapped to a peripheral port using the memory mapped I/O mechanism. Essentially, we are accessing a hardware register, and the value of that register can change independently of the program.

So, why do we need to tell the compiler about this? Because it ensures that during assembly code generation, every access to this variable will involve reading the memory location to get the most recent value. On the other hand, without volatile, the compiler might load the variable into a register for efficiency, and later accesses to the variable would read from the register without checking the memory again, causing the code to use stale data from the register. Commonly used functions like ioread encapsulate volatile operations to ensure the latest data is read. You can refer to build_mmio_read for specific definitions.

However, aside from this memory mapped I/O scenario and a few special cases, if a variable may be accessed concurrently by multiple processes, the volatile keyword should not be used to ensure that each process sees the latest value of the variable. The correct approach is to use locks to protect the variable. Once the lock is acquired, the protected variable only needs to be read from memory once and stored in a register, and subsequent accesses can use the register value, which is more efficient. Since the lock mechanism ensures no other process can modify the variable before we exit the critical section, the data in the register remains valid. Declaring the protected variable as volatile in this case would be unnecessary and could even hinder optimization within the critical section, forcing the compiler to read from memory every time, which is clearly inefficient.

Preventing Instruction Reordering

Another use case for volatile involves the READ_ONCE/WRITE_ONCE macros. Kernel comments mention that these macros prevent compiler reordering:

The compiler is also forbidden from reordering successive instances of
READ_ONCE and WRITE_ONCE

We’ll analyze the READ_ONCE macro in detail below.

From the kernel’s definition of this macro, its essence is that it uses the volatile keyword to modify the type of the variable. At first glance, it doesn’t seem like it could act as a compiler barrier to prevent compiler reordering.

#define READ_ONCE(x) \
({\
 compiletime_assert_rwonce_type(x); \
 __READ_ONCE(x); \
})
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))

So, let’s test it out.

First, a piece of C code:

int a, b;
int i, j;

void foo()
{
 a = i;
 b = j / 16;
}

Using gcc -O2 example.c -S generates the following assembly code:

movl j(%rip), %edx // read j
movl i(%rip), %eax // read i
testl %edx, %edx
movl %eax, a(%rip)
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, b(%rip)

Clearly, the read order of i and j is reversed compared to the C code. To prevent this optimization, let’s first try using a compiler barrier, barrier(), and see the effect.

#define barrier() __asm__ __volatile__("": : :"memory")

int a, b;
int i, j;

void foo()
{
 a = i;
 barrier();
 b = j / 16;
}

The assembly code now looks like this:

movl i(%rip), %eax // read i
movl %eax, a(%rip)
// Barrier here
movl j(%rip), %edx // read j
testl %edx, %edx
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, b(%rip)

Clearly, the barrier() works effectively. It informs the compiler that the code before the barrier and the code after the barrier belong to two different “worlds.” The instructions before the barrier cannot be reordered with those after, and vice versa. This means no instruction reordering can cross the barrier.

After witnessing the effect of the compiler barrier, let’s see if volatile can also act as a compiler barrier.

#define __READ_ONCE(x) (*(const volatile int *)&(x))

int a, b;
int i, j;

void foo()
{
 a = __READ_ONCE(i);
 b = __READ_ONCE(j) / 16;
}

The resulting assembly code is:

movl i(%rip), %eax // read i
movl j(%rip), %edx // read j
movl %eax, a(%rip) // write a
testl %edx, %edx
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, b(%rip) // write b

Now, the read order of i and j is preserved. However, notice that while volatile can ensure the order of some instructions, it is still not a compiler barrier. Using __READ_ONCE, we can only guarantee the read order of i and j, but not the write order of a and b. Theoretically, the compiler could generate the following code:

movl i(%rip), %eax // read i
movl j(%rip), %edx // read j
movl %eax, %ecx // temporarily store i in ecx
testl %edx, %edx
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, b(%rip) // write b
movl %ecx, a(%rip) // write i to a

Although this situation is difficult to construct, it’s theoretically possible.

The reason why the compiler enforces this behavior with volatile is due to the following C standard:

The least requirements on a conforming implementation are:
At sequence points, volatile objects are stable in the sense that previous accesses are complete and subsequent accesses have not yet occurred.
...
...
...
The following are the sequence points described in 5.1.2.3:
The end of a full expression: an initializer (6.7.8); the expression in an expression
statement (6.8.3); the controlling expression of a selection statement (if or switch)
(6.8.4); the controlling expression of a while or do statement (6.8.5); each of the
expressions of a for statement (6.8.5.3); the expression in a return statement
(6.8.6.4).

Here, the concept of a sequence point is introduced. In simple terms, a sequence point ensures that the effects of expressions before it are completed and that subsequent accesses have not yet started. In plain language, a sequence point acts as a boundary, and before accessing any volatile variables after the point, all prior accesses must be completed. According to this standard, a semicolon (;) is considered a sequence point, so the a = __READ_ONCE(i) and b = __READ_ONCE(j) / 16 statements are separated by a sequence point, ensuring that i is accessed before j.

However, the compiler only guarantees the read order of volatile variables relative to each other. The read order between non-volatile and volatile variables may still be reordered.

For example, if we remove the __READ_ONCE from j:

#define __READ_ONCE(x) (*(const volatile int *)&(x))

int a, b;
int i, j;

void foo()
{
 a = __READ_ONCE(i);
 b = j / 16;
}

The generated assembly would be:

movl j(%rip), %edx // read j
movl i(%rip), %eax // read i
testl %edx, %edx
movl %eax, a(%rip)
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, b(%rip)

As you can see, the read order of i and j has been reversed again.

At this point, we’ve thoroughly analyzed the role of READ_ONCE (i.e., volatile) in variable reads. It can ensure that variables are strictly read in the order specified by the code. Similarly, WRITE_ONCE ensures the write order of variables.

Now, what happens if READ_ONCE and WRITE_ONCE are used together? According to the C standard, the ordering guarantees are not limited to read-read or write-write operations, so read-write order will also be preserved. Let’s look at an example:

int a, b;
int i;

void foo()
{
 a = i / 16;
 b = 0;
}

The assembly code for this function is:

movl $0, b(%rip) // write b
movl i(%rip), %edx // read i
testl %edx, %edx
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, a(%rip) // write a

As you can see, the read of i, the write to a, and the write to b are completely out of order.

Now, let’s use volatile:

#define __READ_ONCE(x) (*(const volatile int *)&(x))
#define __WRITE_ONCE(x, val) do {*(volatile typeof(x) *)&(x) = (val);} while(0)

int a, b;
int i;

void foo()
{
 a = __READ_ONCE(i) / 16;
 __WRITE_ONCE(b, 0);
}

The resulting assembly code is:

movl i(%rip), %edx // read i
movl $0, b(%rip) // write b
testl %edx, %edx
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, a(%rip) // write a

Now, the read of i and the write to b follow the exact order of the C code, indicating that volatile has taken effect. But the write to a still occurs after the write to b. This is because the write to a was not marked as volatile. If we rewrite the code like this:

__WRITE_ONCE(a, __READ_ONCE(i) / 16);
__WRITE_ONCE(b, 0);

The generated assembly would be:

movl i(%rip), %edx // read i
testl %edx, %edx
leal 15(%rdx), %eax
cmovns %edx, %eax
sarl $4, %eax
movl %eax, a(%rip) // write a
movl $0, b(%rip) // write b

Now, the order perfectly matches the C code. However, this can be simplified further because a depends on i, and the compiler will ensure that i is read before a is written. Therefore, the first __READ_ONCE can be removed, and the following code will have the same effect:

__WRITE_ONCE(a, i / 16);
__WRITE_ONCE(b, 0);

Postscript

The role of volatile in preventing reordering is quite clear in Java, where the JVM behaves much like an operating system. In Java, after compilation to bytecode, instruction reordering can still occur, leading to compiler optimizations. Therefore, the volatile keyword in Java explicitly prevents these optimizations, which has become common knowledge among Java developers. However, in C, the behavior of volatile is somewhat more obscure.

The Principle of Workingset in the Linux Kernel

Thu, 25 Jul 2024 16:58:14 +0000

Overview of Basic Concepts

The concept of workingset was first introduced by Professor Peter Denning in 1968. The fundamental idea is that programs exhibit temporal locality in their memory usage. At time $t$, the workingset of a program is considered to be the set of pages accessed by the process during the interval $(t-\tau, t)$, denoted as $W(t, \tau)$, where $\tau$ is known as the workingset parameter, a user-defined value. The original paper suggested that memory not accessed within $\tau$ could be reclaimed through sampling.

In the Linux kernel, workingset refers to the inactive and active LRU (Least Recently Used) lists. For file pages read via the read system call, they are placed in the inactive LRU list upon the first read. If these pages are repeatedly read, they are promoted to the active LRU list. For pages mapped via mmap, the process is similar: they are initially placed in the inactive LRU list and may be activated to the active LRU list if deemed sufficiently active during memory reclamation.

When memory pressure triggers reclamation, pages are freed from the tail of the inactive list. If the inactive list becomes significantly smaller than the active list, pages from the active list are moved to the inactive list to balance their sizes.

Overall, the aging process of pages involves moving from the head of the active list to its tail, then to the head of the inactive list, and finally to the tail of the inactive list before being reclaimed.

This article focuses on the workingset for file pages, although the current kernel also considers anonymous pages, as the underlying principles are consistent.

Lifecycle Model of File Pages

Assuming the machine’s memory is divided into two parts: the active list and the inactive list. This assumption simplifies analysis. The sizes of these two parts are mutually adjustable, with the ratio of active:inactive not exceeding 3:1 according to kernel settings. If the active list grows too large, pages are moved to the inactive list to maintain the balance.

The kernel’s page lifecycle model can be summarized as follows:

File pages are placed at the head of the inactive list when read into memory.
Each new file page read causes a page at the tail of the inactive list to be reclaimed, making space for the newly inserted page, effectively moving the pages in the inactive list toward the tail by one position.
Pages with high access frequency, detected during reclamation or repeated reads via the read system call, are promoted to the active list, causing pages ahead of them in the inactive list to move one position towards the tail.
If the active list grows more than three times the size of the inactive list, pages from the tail of the active list are moved to the head of the inactive list to rebalance the ratio.

This simplified model describes the kernel’s page lifecycle. It also shows that pages in the inactive list are either activated or reclaimed, causing pages ahead of them to move towards the tail (aging). A page is reclaimed after moving toward the tail through len(inactive) positions.

Problem Introduction

Some file pages are accessed periodically. When first loaded into memory, they are placed at the head of the inactive list. As other file pages are read into inactive list and pages at the tail are reclaimed, these file pages are gradually pushed to the tail of the inactive list and eventually reclaimed. After being reclaimed for a while, they are accessed again, repeating the process of loading, moving, and reclaiming (referred to as periodically accessed pages).

These pages are not highly active; if their access frequency were high enough, they would have been promoted to the active list. The fact that they oscillate within the inactive list indicates moderate access frequency.

However, the active list only moves pages to the inactive list when it becomes too long. Consequently, the active list may contain long-unaccessed inactive pages. Since the active list is not yet long enough, these pages remain there, even though their access frequency is lower than that of the periodically accessed pages. We need a method to allow these inactive pages in the active list to compete fairly with periodically accessed pages, retaining genuinely valuable and frequently accessed pages.

Basic Idea of Workingset

Workingset is such a method. The inactive and active lists together form the workingset, aiming to retain pages accessed by processes in the workingset and prevent their reclamation.

A periodically accessed page moves from the head of the inactive list to its tail before being reclaimed, traveling through len(inactive) positions.

From the moment a page is reclaimed until it is accessed again, the inactive list continues to activate and reclaim pages. Suppose X pages are reclaimed and activated during this period.

If the page were initially placed X + len(inactive) positions from the tail, it would be accessed just as it reaches the tail, avoiding reclamation and preventing oscillation.

The key is that the inactive list is only len(inactive) long. What we need is to place the page X + len(inactive) positions from the tail.

Since the inactive list is preceded by the active list, we can insert the page at the X-th position from the end of the active list. Instead of inserting in the middle, we can simply place it at the head of the active list, achieving protection.

Additionally, this approach serves another purpose: once the page is added to the active list, it will compete with other active pages. As the length of the active list gradually increases, when it becomes necessary to age a batch of pages from the active list to the inactive list, the kernel will compare the access frequencies. This comparison ensures that genuinely active pages remain in the active list, while less active pages are moved to the inactive list for potential reclamation.

This is the basic idea of workingset.

Implementation of Workingset

By placing the page at the head of the active list, it moves through the active list to the tail of the inactive list, providing a maximum travel distance of len(active) + len(inactive). The required distance to prevent reclamation is X + len(inactive). As long as len(active) + len(inactive) is greater than X + len(inactive), placing the page at the head of the active list provides sufficient protection; otherwise, the page will eventually be reclaimed.

The inequality len(active) + len(inactive) > X + len(inactive) simplifies to len(active) > X. Just checking this inequality is sufficient. When a page is read into memory, if the inequality holds, it is placed at the head of the active list to avoid reclamation before the next access.

The remaining issue is calculating X. As previously mentioned, it represents the number of pages reclaimed and activated in the inactive list from the time a periodically accessed page is reclaimed until it is accessed again. We can set up a counter, incremented each time a page in the inactive list is reclaimed or activated.

Thus, when a periodically accessed page is reclaimed, we record the current value of this counter in its radix tree slot. When the page is read again, it is reinserted into the same radix tree slot. We retrieve the old counter value, subtract it from the current counter value to obtain X.

Then, we compare len(active) > X to decide whether to place the page at the head of the active list. If protection is feasible, the page is placed at the head of the active list; otherwise, it is placed in the inactive list.

Understanding the Implementation of qspinlock in the Linux Kernel

Thu, 18 Jul 2024 11:07:08 +0000

The qspinlock is implemented on top of the mcs spinlock. The MCS stands for Mellor, Crummey, and Scott, the surnames of the creators. The main idea is to have each spinner spin on its own per-cpu variable, thereby avoiding the constant cacheline bouncing between different CPUs. Cacheline bouncing occurs when all CPUs spin-wait on the same lock variable, causing them to repeatedly read this variable. When one CPU unlocks, this variable is modified, invalidating the cachelines of all other CPUs, which then have to re-read the variable. This results in a performance overhead. The MCS lock mitigates this by having each CPU spin on its own dedicated variable, thus avoiding contention on a single lock variable.

This article does not delve into the evolution of spinlocks but directly explains the final implementation of the qspinlock.

If the text in images is unclear, you can right-click and select “Open Image in New Tab” for a high-resolution view.

Data Structure Definition

The qspinlock structure is defined as follows:

typedef struct qspinlock {
 union {
 atomic_t val;

 /*
 * By using the whole 2nd least significant byte for the
 * pending bit, we can allow better optimization of the lock
 * acquisition for the pending bit holder.
 */
 struct {
 u8 locked;
 u8 pending;
 };
 struct {
 u16 locked_pending;
 u16 tail;
 };
 };
} arch_spinlock_t;

The memory layout is shown below. Note that a union type is used in the definition, so using the locked_pending variable allows simultaneous retrieval or modification of both locked and pending variables.

The CPU holding the lock is called the owner, and the waiting CPUs are termed spinners. These spinners can be further categorized based on their behavior into three types: pender, successor, and queuer. The term pender is a coined word to better illustrate the spinlock logic.

Locking and Unlocking Process Demonstration

The locking and unlocking process is essentially a state machine. We’ll demonstrate the state changes graphically.

Initially, in an unlocked state, the structure members have the values shown below:

0 Spinner

Locking

Now, let’s assume a CPU attempts to acquire the lock. Since no other CPUs are waiting, the state transitions as shown below. The spinlock->locked variable is set to 1, indicating that the lock is acquired. At this point, there is only an owner and no spinners.

Unlocking

In this scenario, if the owner releases the lock, the state transitions back to the unlocked state:

1 Spinner

Locking

Assume that while the owner holds the lock, another CPU attempts to acquire it. This CPU will have to wait, and we call this waiting CPU a pender. The pender sets the pending variable to 1 to indicate its presence and then spins-waits for the spinlock->locked variable to become 0. The dashed arrow in the diagram indicates the waiting relationship and the corresponding code.

It’s worth noting that the pending variable is exclusive to the pender. If the pender disappears, the variable reverts to 0.

Unlocking

In this case, if the owner releases the lock, two steps occur:

The owner sets the spinlock->locked variable to 0, indicating the lock is released.
The pender detects that spinlock->locked is 0 and atomically changes (locked=0, pending=1) to (locked=1, pending=0). Here, setting locked to 1 signifies that the pender becomes the new owner, and clearing pending to 0 indicates the pender’s disappearance (as it has become the owner). As mentioned earlier, the pending variable is an exclusive indicator for the pender.

The overall state transition for unlocking is shown below. The state within the dashed box on the right represents an intermediate state, where only the first step is completed. Once the second step is completed, the state transitions to the 0 spinners scenario:

2 Spinners

Locking

Let’s consider a scenario where, in addition to the pender, another CPU attempts to acquire the lock. This new CPU will spin-wait on both spinlock->{locked, pending} variables. We call this CPU a successor.

In this situation, the spinlock->tail variable stores the successor’s cpu id. The idx field in the diagram indicates the context (i.e. process context, softirq, hardirq, nmi) of this successor, but we can ignore it for our discussion. It’s enough to know that the tail variable “points to” the successor cpu. The arrow in the diagram indicates this pointing relationship starting from the tail to the successor cpu’s per-cpu mcs node structure. While the mcs node appears in this scenario, it hasn’t yet come into play. We will discuss it further later.

To summarize, in this situation, there are two spinners: the pender waiting for spinlock->locked to become 0, and the successor waiting for both spinlock->{locked, pending} to become 0.

Unlocking

If the owner releases the lock, the owner disappears, and the pender detects that spinlock->locked is 0. The pender then atomically changes (locked=0, pending=1) to (locked=1, pending=0), becoming the new owner and indicating the pender’s disappearance (now the owner). [This part is the same as the “1 spinner” scenario]

The unlocking state transition is shown below. You can see that after releasing the lock, the pender becomes the new owner, and the pender disappears. However, the successor is still there, waiting for both spinlock->{locked, pending} to become 0.

Next, if the owner releases the lock again, the spinlock->locked variable becomes 0, and finally, both spinlock->{locked, pending} variables become 0. The successor detects this change and atomically sets the spinlock->locked variable to 1, thereby acquiring the lock and becoming the new owner. The state transition is shown below:

At this point, you can observe a pattern: the pender and successor have equivalent statuses. Their next step is to become the owner, but they wait for different conditions. So why have two equivalent spinners? This is a small optimization, which we will discuss after explaining the overall process. For now, just focus on understanding the state transitions.

3 Spinners

Locking

Now, let’s make the scenario more complex. Suppose another CPU attempts to acquire the lock while the successor is waiting. This CPU will wait on its own per-cpu mcs node structure and is called a queuer.

Specifically, each CPU has its own dedicated mcs node structure, which includes a member called locked, denoted as mcs->locked. This variable initially has a value of 0, and the queuer spin-waits until its mcs->locked variable becomes 1. This way, it doesn’t repeatedly read the spinlock variable, which is already being read by the pender and successor. This reduces cache contention and bouncing.

The state transition for this scenario is shown below. The pender and successor each wait for their respective variables, while the queuer waits on its dedicated mcs node. Note that the spinlock->tail variable now points to the queuer, aligning with the term “tail”, which indicates the end of the waiting queue.

At this point, the three spinners are named pender, successor, and queuer, based on the variables they wait for. This naming convention is to distinguish them.

Unlocking

Let’s examine the state transitions during unlocking in this scenario.

In the first unlock, the owner disappears and the pender moves up (i.e. be promoted):

In the second unlock, the owner disappears, and the successor becomes the owner. When the successor moves up, it checks if there’s a queuer behind it. If there is, it promotes the queuer to the new successor. In this case, there is a queuer, so the successor sets the queuer’s mcs->locked to 1, allowing the queuer to finish waiting and become the new successor.

The state transition is shown below:

If we unlock once more, the successor moves up to become the owner, as shown in the diagram. Note the additional arrow on the right side:

4 Spinners

Let’s make the scenario even more complex by adding another CPU waiting for the lock, making it a second queuer. The queuer can have many instances, queued up one after another. As a queuer, it will spin-wait on its own mcs->locked variable, waiting to be promoted to successor.

Assume two queuers are named queuer1 and queuer2. When queuer1 moves up to successor, queuer2 becomes the first in line. When the successor (formerly queuer1) moves up to owner, queuer2 can be promoted to the new successor by the old successor (formerly queuer1). The successor uses the mcs structure’s next pointer to find and promote the next queuer.

The locking and unlocking in this scenario are shown in the bottom row of the diagram. Some intermediate states are omitted for brevity.

Generalization

From the state transition processes above, we can draw the following conclusions:

The pender and successor have equivalent statuses, both aiming to become the owner.
The first queuer in line will be promoted to the new successor by the current successor upon its departure.
Subsequent queuers patiently wait for the successor to promote them.
The pender appears only once at the beginning and never reappears after it disappears.

Based on these conclusions, we can summarize the state transitions for locking and unlocking processes:

Continuous Locking

unlocked
-> 1 owner
-> 1 owner, 1 pender
-> 1 owner, 1 pender, 1 successor
-> 1 owner, 1 pender, 1 successor, 1 queuer
-> 1 owner, 1 pender, 1 successor, 2 queuer
-> ...
-> 1 owner, 1 pender, 1 successor, N queuer

Continuous Unlocking

-> 1 owner, 1 pender, 1 successor, N queuer
-> 1 owner, 1 successor, N queuer
-> 1 owner, 1 successor, (N-1) queuer
-> 1 owner, 1 successor, (N-2) queuer
-> ...
-> 1 owner, 1 successor
-> 1 owner
-> unlocked

There are also some subtle variations to consider. For instance, if a spinner arrives between the owner releasing the lock and the pender acquiring it, what happens? I’ll leave that for the reader to ponder.

Why Have Both `pender` and `successor` Roles

This primarily stems from practical testing, which revealed that when there are two spinners, and one waits on the spinlock variable while the other waits on its mcs node variable, this can lead to an additional mcs node cacheline read. This results in a cold cache miss, which is more costly than the potential cache bouncing incurred by having both spinners wait on the spinlock. To optimize this, the pending flag and thus the pender role were introduced during the development of qspinlock.

Why Increment the CPU ID by 1 Before Storing it in `tail` in Actual Code Implementation

In actual code implementation, there are instances where it is necessary to determine whether a successor or queuer exists. This determination is made by checking if tail is 0. Therefore, we must encode the cpu id by incrementing it by 1 before storing it. Otherwise, if tail is 0, it would be unclear whether it means that successor and queuer don’t exist, or if it simply indicates that the cpu id of the successor or queuer is 0.

About

Thu, 29 Feb 2024 11:18:48 +0000

I am a programmer who enjoys systems research, nominally a Linux kernel developer. I hope to study some interesting things, including but not limited to operating systems, computer architecture, algorithms, mathematics, etc.

Systems Research

The Role of Aligning Memory Addresses With Cachelines

Cross-Line Performance

Atomicity

Compiler Optimization

Revisiting the Volatile Keyword in C

The Most Common Usage

Preventing Instruction Reordering

Postscript

The Principle of Workingset in the Linux Kernel

Overview of Basic Concepts

Lifecycle Model of File Pages

Problem Introduction

Basic Idea of Workingset

Implementation of Workingset

Understanding the Implementation of qspinlock in the Linux Kernel

Data Structure Definition

Locking and Unlocking Process Demonstration

0 Spinner

Locking

Unlocking

1 Spinner

Locking

Unlocking

2 Spinners

Locking

Unlocking

3 Spinners

Locking

Unlocking

4 Spinners

Generalization

Continuous Locking

Continuous Unlocking

Why Have Both pender and successor Roles

Why Increment the CPU ID by 1 Before Storing it in tail in Actual Code Implementation

About

Why Have Both `pender` and `successor` Roles

Why Increment the CPU ID by 1 Before Storing it in `tail` in Actual Code Implementation