# Lecture 15 - Race Detection

Source: https://wisconsin.twango.dev/sp26-cs537/lectures/lecture-15

# Lecture 15 - Race Detection

**Instructor:** Tej Chajed
**Topics:** Data races, race detection, ThreadSanitizer, happens-before relation, linearizability

> [!abstract]- Learning Outcomes
> - Define a data race
> - Compare race detection with other testing strategies
> - Interpret race detection output

---

## Finding Bugs in Concurrent Code

Three main strategies for finding bugs in concurrent programs:

### Static Analysis

- Concurrent code depends on context: what locks are held, what other threads are running
- Static analysis tends to produce **false positives**: reports a bug but there is not one

> [!note] Lecture Note
> **False positives** = reports a bug, but code is correct. **False negatives** = does not report a bug, but code is incorrect.

### Running Many Times

- Bug might not appear now, but after many months of running in production
- Linux kernel has torture tests for synchronization primitives: high cost of testing due to importance

### Exhaustive Search

- Run with a special scheduler that can force a specific interleaving
- **Deterministic**: can re-run when we find a bug
- **Exhaustive**: if there is a bug, we will find it
- **Very slow**

### Systematic Concurrency Testing

Lots of work on exploring thread interleaving schedules:

- CHESS
- Probabilistic testing
- DPOR (Dynamic Partial-Order Reduction)
- Also see loom and shuttle for Rust

---

## What Bugs Should We Look For?

- Test result failures
- Linearizability
- Internal assertion violations
- **Data races** (this lecture)

> [!note] Lecture Note
> Example with locks: T1 calls `lock()`, `unlock()`, `unlock()` while T2 calls `lock()`. If T2 calls `lock()` between T1's two `unlock()` calls, there is no mutual exclusion — this is a bug where the lock discipline is violated.

---

## Data Races

> **Data Race:** Simultaneous reads/writes to the same memory location, at least one of which is a write.

In C, a data race is always a bug (undefined behavior).

> [!note] Lecture Note
> read/write to the same location = data race. write/write to the same location = data race. Only read/read is safe.

### Example Bug

```c
int shared_counter = 0;

void *increment() {
  for (int i = 0; i < iters; i++) {
    shared_counter++;
  }
}

pthread_t t1, t2;
pthread_create(&t1, NULL, increment, NULL);
pthread_create(&t2, NULL, increment, NULL);
```

Both threads increment `shared_counter` without synchronization — this is a data race.

### Fix: Add a Lock

```c
int shared_counter = 0;
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;

void *increment() {
  for (int i = 0; i < iters; i++) {
    lock(&m);
    shared_counter++;
    unlock(&m);
  }
}
```

---

## ThreadSanitizer: Race Detection

### How It Works

Idea: hook into the compiler and instrument memory operations.

What needs to be instrumented:

- Memory reads/writes
- Lock operations
- Synchronization (e.g., signal/wait)

Compile with `clang -fsanitize=thread` to enable ThreadSanitizer.

### Race Condition Example 1

```c
// Global - we will fail to synchronize access to this
int shared_counter = 0;

// Thread function that increments the counter
void *increment_thread(void *arg) {
  int iterations = *(int *)arg;

  for (int i = 0; i < iterations; i++) {
    // BUG: this is a race when run concurrently
    shared_counter++;
  }

  return NULL;
}
```

### Race Condition Example 2

```c
int shared_data = 0;
int ready = 0;

void *writer_thread() {
  shared_data = 42;
  ready = 1;
  return NULL;
}

void *reader_thread() {
  // DATA RACE: Reading 'ready' and 'shared_data' without synchronization
  while (!ready) {
    // spin loop
  }
  printf("Reader shared_data: %d\n", shared_data);
  return NULL;
}
```

### Reading ThreadSanitizer Output

```text
Demo 1: Counter increment race
================
WARNING: ThreadSanitizer: data race (pid=24344)
  Write of size 4 at 0x0001008f4000 by thread T2:
    #0 increment_thread tsan_demo.c:20 (tsan_demo:arm64+0x100000690)

  Previous write of size 4 at 0x0001008f4000 by thread T1:
    #0 increment_thread tsan_demo.c:20 (tsan_demo:arm64+0x100000690)

  Location is global 'shared_counter' at 0x0001008f4000 (tsan_demo+0x100008000)

  Thread T2 (tid=41794395, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2f708)
    #1 main tsan_demo.c:54 (tsan_demo:arm64+0x1000007f4)

  Thread T1 (tid=41794394, finished) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2f708)
    #1 main tsan_demo.c:54 (tsan_demo:arm64+0x1000007f4)

SUMMARY: ThreadSanitizer: data race tsan_demo.c:20 in increment_thread
================
Expected: 20000, Got: 20000
```

Key parts of the output:

- **WARNING line**: identifies it as a data race
- **Write/Previous write**: shows the two conflicting accesses, with source file and line number
- **Location**: identifies the variable involved (`shared_counter`)
- **Thread creation**: shows where each thread was created
- **SUMMARY**: one-line summary with file, line, and function

---

## How ThreadSanitizer Detects Races

### Idea 1: Track Lock Sets

- For each memory location, record what locks are held when it is accessed
- It is safe to have simultaneous (unordered) access to the same address if both threads hold the same lock

> [!note] Lecture Note
> Example: Thread 1 does `lock(l)`, `read(p)`, `unlock(l)`. Thread 2 does `lock(l)`, `write(p)`, `unlock(l)`. Both hold lock `l` when accessing `p`, so this is safe — no race reported.

### Idea 2: Track Simultaneous Events

- When the program runs, there is one sequence of operations
- Want to know if events are unsynchronized and could happen in reverse order

### The Happens-Before Relation

Consider events E1 and E2 in the program (memory reads/writes to specific addresses):

- **E1 < E2** means E1 "happens before" E2
- If **E1 does not happen before E2** and **E2 does not happen before E1**, then E1 and E2 are **simultaneous**

> [!note] Lecture Note
> Example: `a = 1` then `x = a` — within the same thread, the first happens before the second (program order).

### Happens-Before Example

From "ThreadSanitizer — data race detection in practice":

| Thread T1 | Thread T2 | Thread T3 |
|-----------|-----------|-----------|
| S1        | S2        | S3        |
| Signal(H1)|           |           |
| S4        | Wait(H1)  |           |
|           | S5        |           |
|           | Signal(H2)|           |
|           | S6        | Wait(H2)  |
|           |           | S7        |

The paper uses S for **segments** — sequences of events from one thread.

- **S1 < S4** (same thread — program order)
- **S1 < S5** (happens-before arc: Signal_T1(H1) -> Wait_T2(H1))
- **S1 < S7** (happens-before is transitive)
- **S4 and S2 have no relation** (they are simultaneous)

> [!note] Lecture Note
> S1 < S4 is due to program order. S1 < S5 is due to signaling (Signal on H1 in T1 happens before Wait on H1 in T2).

### Putting It Together: Reporting a Bug

If E1 and E2 for the same memory location are **simultaneous**, at least one is a write, and they hold **no common lock**: report a data race.

Some sophisticated tracking is used to report *where* the data race is.

---

## False Positives in Race Detection

```c
// Thread1                    // Thread2
obj->UpdateMe();              mu.Lock();
mu.Lock();                    bool f = flag;
flag = true;                  mu.Unlock();
mu.Unlock();                  if (f) obj->UpdateMe();
```

No common locks protect `obj->UpdateMe()` — ThreadSanitizer reports a race.

But the code is actually correct: `flag` starts as `false`, so Thread2 only calls `obj->UpdateMe()` if `flag` is `true`, which means Thread1 has already finished its call.

> [!note] Lecture Note
> `flag = false` initially. Thread2 reads `flag` under the lock. If `flag` is still false, Thread2 never calls `UpdateMe()`. The race detector does not understand this application-level logic.

### Pure Happens-Before

Add additional happens-before arcs: **Unlock_T1(L) < Lock_T2(L)**

Reduces false positives, but might miss bugs (more false negatives).

---

## Issues with Race Detection

- **Slow**: ThreadSanitizer typically makes programs 3-5x slower
- **False positives**: benign races, synchronization the tool does not understand
- **False negatives**: does not try every possible execution

---

## Linearizability

Consider a map with `Get(k)` and `Put(k, v)` operations. If you access it concurrently, what do you expect?

> **Linearizability (informal):** Each operation appears to execute at one instant in time between invocation and return.
>
> **Linearizability (more precise):** For each trace, you can find a linearization (an ordering of operations) consistent with observed return values.

### Linearizability Testing

Tools: Jepsen, Porcupine

> [!note] Lecture Note
> In the Porcupine visualization, a `put('200')` operation spans a time range. A `get()` returning `'200'` is valid if the put appeared to execute before it. A `get()` returning `'0'` after the put completed would be invalid (the put must have taken effect by then).

---

## Concurrency Review

- Use threads to take advantage of multiple cores
- Correctness is challenging: many thread interleavings
- Locks and condition variables
- Bugs: atomicity violation, ordering problems, deadlocks
- Deadlock prevention: lock ordering, atomic lock acquisition, lock-free code

---

## Related

- **Navigation:** [[lecture-14|← Lecture 14]] · [[lecture-16|Lecture 16 →]]
- **Exam:** [[exams/midterm-2/index|Midterm 2]]
- **Course:** [[README|CS 537 Map of Content]]