wisconsin
Lecture 15 - Race Detection

Lecture 15 - Race Detection

Curated by Last modified: 7 min read

Instructor: Tej Chajed Topics: Data races, race detection, ThreadSanitizer, happens-before relation, linearizability

Finding Bugs in Concurrent Code

Three main strategies for finding bugs in concurrent programs:

Static Analysis

  • Concurrent code depends on context: what locks are held, what other threads are running
  • Static analysis tends to produce false positives: reports a bug but there is not one

Lecture Note

False positives = reports a bug, but code is correct. False negatives = does not report a bug, but code is incorrect.

Running Many Times

  • Bug might not appear now, but after many months of running in production
  • Linux kernel has torture tests for synchronization primitives: high cost of testing due to importance
  • Run with a special scheduler that can force a specific interleaving
  • Deterministic: can re-run when we find a bug
  • Exhaustive: if there is a bug, we will find it
  • Very slow

Systematic Concurrency Testing

Lots of work on exploring thread interleaving schedules:

  • CHESS
  • Probabilistic testing
  • DPOR (Dynamic Partial-Order Reduction)
  • Also see loom and shuttle for Rust

What Bugs Should We Look For?

  • Test result failures
  • Linearizability
  • Internal assertion violations
  • Data races (this lecture)

Lecture Note

Example with locks: T1 calls lock(), unlock(), unlock() while T2 calls lock(). If T2 calls lock() between T1’s two unlock() calls, there is no mutual exclusion — this is a bug where the lock discipline is violated.

Data Races

Data Race: Simultaneous reads/writes to the same memory location, at least one of which is a write.

In C, a data race is always a bug (undefined behavior).

Lecture Note

read/write to the same location = data race. write/write to the same location = data race. Only read/read is safe.

Example Bug

int shared_counter = 0;

void *increment() {
  for (int i = 0; i < iters; i++) {
    shared_counter++;
  }
}

pthread_t t1, t2;
pthread_create(&t1, NULL, increment, NULL);
pthread_create(&t2, NULL, increment, NULL);

Both threads increment shared_counter without synchronization — this is a data race.

Fix: Add a Lock

int shared_counter = 0;
pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;

void *increment() {
  for (int i = 0; i < iters; i++) {
    lock(&m);
    shared_counter++;
    unlock(&m);
  }
}

ThreadSanitizer: Race Detection

How It Works

Idea: hook into the compiler and instrument memory operations.

What needs to be instrumented:

  • Memory reads/writes
  • Lock operations
  • Synchronization (e.g., signal/wait)

Compile with clang -fsanitize=thread to enable ThreadSanitizer.

Race Condition Example 1

// Global - we will fail to synchronize access to this
int shared_counter = 0;

// Thread function that increments the counter
void *increment_thread(void *arg) {
  int iterations = *(int *)arg;

  for (int i = 0; i < iterations; i++) {
    // BUG: this is a race when run concurrently
    shared_counter++;
  }

  return NULL;
}

Race Condition Example 2

int shared_data = 0;
int ready = 0;

void *writer_thread() {
  shared_data = 42;
  ready = 1;
  return NULL;
}

void *reader_thread() {
  // DATA RACE: Reading 'ready' and 'shared_data' without synchronization
  while (!ready) {
    // spin loop
  }
  printf("Reader shared_data: %d\n", shared_data);
  return NULL;
}

Reading ThreadSanitizer Output

Demo 1: Counter increment race
================
WARNING: ThreadSanitizer: data race (pid=24344)
  Write of size 4 at 0x0001008f4000 by thread T2:
    #0 increment_thread tsan_demo.c:20 (tsan_demo:arm64+0x100000690)

  Previous write of size 4 at 0x0001008f4000 by thread T1:
    #0 increment_thread tsan_demo.c:20 (tsan_demo:arm64+0x100000690)

  Location is global 'shared_counter' at 0x0001008f4000 (tsan_demo+0x100008000)

  Thread T2 (tid=41794395, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2f708)
    #1 main tsan_demo.c:54 (tsan_demo:arm64+0x1000007f4)

  Thread T1 (tid=41794394, finished) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2f708)
    #1 main tsan_demo.c:54 (tsan_demo:arm64+0x1000007f4)

SUMMARY: ThreadSanitizer: data race tsan_demo.c:20 in increment_thread
================
Expected: 20000, Got: 20000

Key parts of the output:

  • WARNING line: identifies it as a data race
  • Write/Previous write: shows the two conflicting accesses, with source file and line number
  • Location: identifies the variable involved (shared_counter)
  • Thread creation: shows where each thread was created
  • SUMMARY: one-line summary with file, line, and function

How ThreadSanitizer Detects Races

Idea 1: Track Lock Sets

  • For each memory location, record what locks are held when it is accessed
  • It is safe to have simultaneous (unordered) access to the same address if both threads hold the same lock

Lecture Note

Example: Thread 1 does lock(l), read(p), unlock(l). Thread 2 does lock(l), write(p), unlock(l). Both hold lock l when accessing p, so this is safe — no race reported.

Idea 2: Track Simultaneous Events

  • When the program runs, there is one sequence of operations
  • Want to know if events are unsynchronized and could happen in reverse order

The Happens-Before Relation

Consider events E1 and E2 in the program (memory reads/writes to specific addresses):

  • E1 < E2 means E1 “happens before” E2
  • If E1 does not happen before E2 and E2 does not happen before E1, then E1 and E2 are simultaneous

Lecture Note

Example: a = 1 then x = a — within the same thread, the first happens before the second (program order).

Happens-Before Example

From “ThreadSanitizer — data race detection in practice”:

Thread T1Thread T2Thread T3
S1S2S3
Signal(H1)
S4Wait(H1)
S5
Signal(H2)
S6Wait(H2)
S7

The paper uses S for segments — sequences of events from one thread.

  • S1 < S4 (same thread — program order)
  • S1 < S5 (happens-before arc: Signal_T1(H1) → Wait_T2(H1))
  • S1 < S7 (happens-before is transitive)
  • S4 and S2 have no relation (they are simultaneous)

Lecture Note

S1 < S4 is due to program order. S1 < S5 is due to signaling (Signal on H1 in T1 happens before Wait on H1 in T2).

Putting It Together: Reporting a Bug

If E1 and E2 for the same memory location are simultaneous, at least one is a write, and they hold no common lock: report a data race.

Some sophisticated tracking is used to report where the data race is.

False Positives in Race Detection

// Thread1                    // Thread2
obj->UpdateMe();              mu.Lock();
mu.Lock();                    bool f = flag;
flag = true;                  mu.Unlock();
mu.Unlock();                  if (f) obj->UpdateMe();

No common locks protect obj->UpdateMe() — ThreadSanitizer reports a race.

But the code is actually correct: flag starts as false, so Thread2 only calls obj->UpdateMe() if flag is true, which means Thread1 has already finished its call.

Lecture Note

flag = false initially. Thread2 reads flag under the lock. If flag is still false, Thread2 never calls UpdateMe(). The race detector does not understand this application-level logic.

Pure Happens-Before

Add additional happens-before arcs: Unlock_T1(L) < Lock_T2(L)

Reduces false positives, but might miss bugs (more false negatives).

Issues with Race Detection

  • Slow: ThreadSanitizer typically makes programs 3-5x slower
  • False positives: benign races, synchronization the tool does not understand
  • False negatives: does not try every possible execution

Linearizability

Consider a map with Get(k) and Put(k, v) operations. If you access it concurrently, what do you expect?

Linearizability (informal): Each operation appears to execute at one instant in time between invocation and return.

Linearizability (more precise): For each trace, you can find a linearization (an ordering of operations) consistent with observed return values.

Linearizability Testing

Tools: Jepsen, Porcupine

Lecture Note

In the Porcupine visualization, a put('200') operation spans a time range. A get() returning '200' is valid if the put appeared to execute before it. A get() returning '0' after the put completed would be invalid (the put must have taken effect by then).

Concurrency Review

  • Use threads to take advantage of multiple cores
  • Correctness is challenging: many thread interleavings
  • Locks and condition variables
  • Bugs: atomicity violation, ordering problems, deadlocks
  • Deadlock prevention: lock ordering, atomic lock acquisition, lock-free code

Linked to this page