Skip to main content

C++20 Concurrency Features for Low-Latency C++

 

Important C++20 Concurrency Features for Low-Latency C++

C++20 added a useful set of concurrency primitives. For low-latency work the question is never just "does it exist?" but "what does it cost on the hot path, and does it ever block, allocate or make a syscall?" This guide covers the C++20 concurrency features that matter most, what they do under the hood, and where they help or hurt latency.

A reminder of the usual low-latency rules these features are judged against:

  • Avoid blocking on the hot path (a context switch costs microseconds, which is an eternity in this domain).
  • Avoid hidden allocation and hidden locks.
  • Keep shared data on separate cache lines, and minimize cache-line contention.
  • Pin threads and create them up front. Never create threads on the hot path.

Tier 1: Features You Should Know Cold

1. std::jthread and std::stop_token: safe threads and cooperative cancellation

std::jthread is a std::thread that joins automatically in its destructor (after requesting a stop), which fixes the classic std::terminate() on a joinable thread. It also integrates a stop token for cooperative cancellation.

std::jthread worker([](std::stop_token st) {
    while (!st.stop_requested()) {
        pollAndProcess();
    }
});

// worker.request_stop();   // optional: explicit stop
// Destructor: request_stop() then join()

Related types:

  • std::stop_source: the object that requests the stop.
  • std::stop_token: the lightweight handle a thread checks.
  • std::stop_callback: registers a function to run when a stop is requested (for example, to wake a blocked thread).
std::jthread t([&](std::stop_token st) {
    std::stop_callback cb(st, [&] {
        wakeFlag.store(true, std::memory_order_release);
        wakeFlag.notify_all();            // unblock a thread waiting on the flag
    });
    // ... blocking work that can be woken by the callback ...
});

Low-latency notes

  • stop_requested() is an atomic load, so it is cheap, but not free. In a very tight loop, check it every N iterations or once per batch.
  • The stop state is shared and heap-allocated when a stop_source is created. That's fine at startup, but not something to create per message.
  • jthread doesn't make thread creation cheaper. Create your threads at startup, pin them to cores (via native_handle() and pthread_setaffinity_np, or the platform equivalent), and keep them alive.
  • stop_callback runs on the thread that calls request_stop(), so keep callbacks short.

Interview takeaway: jthread = RAII join + built-in cooperative cancellation. It prevents the destroyed-while-joinable bug, but the cancellation is cooperative: the thread has to check the token.


2. std::atomic::wait, notify_one, notify_all: efficient blocking on an atomic

Before C++20, waiting for an atomic variable to change meant either spinning (burning a core) or pairing it with a mutex and condition variable (heavier). C++20 adds wait() / notify_*() directly on atomics, typically implemented with a futex on Linux (or the equivalent WaitOnAddress on Windows).

std::atomic<uint32_t> seq{0};

// Consumer: block until the value is no longer 0
seq.wait(0, std::memory_order_acquire);     // returns once seq != 0

// Producer
seq.store(1, std::memory_order_release);
seq.notify_one();                           // wake one waiter

A common low-latency pattern is a hybrid wait: spin briefly for the lowest latency, then fall back to blocking so you don't burn a core indefinitely.

void waitForChange(std::atomic<uint32_t>& a, uint32_t old) {
    for (int i = 0; i < 1000; ++i) {
        if (a.load(std::memory_order_acquire) != old) return;
        _mm_pause();                        // x86 PAUSE (no standard portable equivalent in C++20)
    }
    a.wait(old, std::memory_order_acquire); // then sleep
}

Pitfalls

  • wait(old) returns when the value differs from old. If the value changes and changes back between checks (an ABA pattern), a waiter can miss the change. Use a monotonically increasing sequence number to avoid this.
  • notify_*() on the producer side can cost a syscall when waiters exist. Implementations try to avoid it when nobody is waiting, but the cost is implementation-defined, so measure.
  • Waking a sleeping thread still costs microseconds of scheduler latency. If the absolute lowest latency matters, keep the consumer spinning on a pinned core.

Interview takeaway: atomic::wait is "efficient sleeping on a value", which is useful for idle or low-rate paths. The hot path should still spin.


3. std::atomic_ref: atomic operations on a plain object

std::atomic_ref<T> lets you perform atomic operations on an object that is not declared std::atomic. This helps when data must have a plain layout, such as arrays filled in a single-threaded phase and then updated concurrently, or structures in shared memory and memory-mapped regions.

uint64_t counters[8];                       // plain array, initialized normally

void hit(size_t i) {
    std::atomic_ref<uint64_t> ref(counters[i]);
    ref.fetch_add(1, std::memory_order_relaxed);
}

Rules to know:

  • The referenced object must satisfy std::atomic_ref<T>::required_alignment, which can be stricter than alignof(T) for some types (for example 64-bit values on 32-bit x86).
  • While any atomic_ref to an object exists, all concurrent accesses to it must go through atomic_refs. Mixing in plain reads or writes is a data race.
  • It is lock-free only when std::atomic_ref<T>::is_always_lock_free is true. Check it with a static_assert.
static_assert(std::atomic_ref<uint64_t>::is_always_lock_free);

Interview takeaway: atomic_ref gives atomic access without changing the type's layout, but you must guarantee no non-atomic concurrent access during its lifetime.


4. std::counting_semaphore and std::binary_semaphore

Semaphores are counters you can acquire() (decrement, blocking at zero) and release() (increment, waking a waiter). Unlike a mutex, any thread can release, so a semaphore is ideal for signaling between threads.

std::binary_semaphore ready{0};

// Producer thread
publish(data);
ready.release();            // signal

// Consumer thread
ready.acquire();            // block until signaled
consume();

Non-blocking and timed variants:

if (ready.try_acquire()) { /* got it, never blocks */ }
if (ready.try_acquire_for(std::chrono::microseconds(50))) { /* ... */ }

For simple "wake me when work arrives" signaling, a semaphore is lighter and simpler than a mutex + condition_variable + predicate, and it doesn't suffer from lost wakeups because the count remembers a release() that happened before the acquire().

Pitfalls

  • try_acquire() is allowed to fail spuriously.
  • The implementation is not specified: on Linux it is generally futex-based. Still, treat acquire() as a potential block and don't call it on a path that must never sleep.
  • counting_semaphore<N>::max() is a least upper bound. Don't release past it.

5. std::latch and std::barrier: group synchronization

std::latch is a single-use countdown. Threads call count_down(), and others wait() until it reaches zero.

std::latch ready(numWorkers);

// Each worker
initThreadLocalState();
pinToCore();
ready.count_down();          // "I'm ready"

// Main thread
ready.wait();                // block until every worker has signaled
startTrading();

std::barrier is reusable and organizes work into phases. All threads must arrive before any can continue, and an optional completion function runs once per phase.

std::barrier sync(numThreads, []() noexcept { swapBuffers(); });

// In each thread's loop
for (;;) {
    doPhaseWork();
    sync.arrive_and_wait();   // all threads synchronize here
}

Low-latency notes

  • Both are great for startup and shutdown coordination (every thread pinned and warmed up before the first message).
  • They block, so they generally don't belong on the per-message hot path.
  • A barrier makes every phase as slow as its slowest thread, which hurts tail latency.

Interview takeaway: latch = one-shot, barrier = reusable with phases and a completion function.


6. std::atomic_flag::test and std::atomic_flag with wait

C++20 added atomic_flag::test() (a read that doesn't modify the flag) plus wait / notify. The big use is writing a proper test-and-test-and-set (TTAS) spinlock, which avoids hammering the cache line with writes while waiting.

class SpinLock {
    std::atomic_flag flag;      // C++20: default-initialized to clear
public:
    void lock() {
        for (;;) {
            if (!flag.test_and_set(std::memory_order_acquire)) return;
            while (flag.test(std::memory_order_relaxed)) {   // read-only spin
                _mm_pause();
            }
        }
    }
    void unlock() { flag.clear(std::memory_order_release); }
};

Spinning on the read (test) keeps the cache line in the shared state across waiters, rather than bouncing it between cores with each test_and_set.

Pitfall: a spinlock is only appropriate when threads are pinned to dedicated cores and critical sections are tiny. If the lock holder is preempted by the OS, the waiters burn CPU until it runs again.


7. Atomic floating-point fetch_add / fetch_sub

std::atomic<float> and std::atomic<double> gained fetch_add and fetch_sub in C++20.

std::atomic<double> total{0.0};
total.fetch_add(price * qty, std::memory_order_relaxed);

Pitfall: x86 has no native atomic floating-point add, so this is usually implemented as a compare-exchange loop. Under contention it can retry repeatedly and become slow. Prefer per-thread accumulators (each on its own cache line) that you combine later, rather than one shared atomic double.


Tier 2: Useful in Specific Situations

8. std::atomic<std::shared_ptr<T>> and std::atomic<std::weak_ptr<T>>

C++20 finally provides a correct, standard way to atomically publish and read a shared_ptr.

std::atomic<std::shared_ptr<const Config>> g_config;

// Writer
g_config.store(std::make_shared<const Config>(newCfg));

// Reader
auto cfg = g_config.load();      // takes a shared reference
use(*cfg);

For latency, be careful: in mainstream implementations this is generally not lock-free (is_lock_free() is typically false), since the pointer and the control block have to be updated together. Loading also does reference-count traffic on a shared cache line. It is excellent for infrequently updated configuration, but a poor fit for a hot read path. For hot paths, prefer a raw std::atomic<T*> with a deliberate reclamation strategy (epoch-based, hazard pointers, or RCU-style double buffering).


9. Condition variables with stop tokens

std::condition_variable_any can wait with a stop_token, so a blocked thread can be woken by a stop request, which solves the "how do I shut down a thread that is waiting?" problem cleanly.

void worker(std::stop_token st) {
    std::unique_lock lock(m);
    while (cv.wait(lock, st, [] { return !queue.empty(); })) {
        process(queue.pop());
    }
    // wait() returned false: stop was requested
}

10. Memory model refinements

C++20 tightened parts of the memory model that were inconsistent or too strong:

  • The definition of release sequences was weakened to match what hardware and compilers actually do.
  • The rules for seq_cst fences were strengthened.
  • memory_order_consume is still effectively unusable in practice, and compilers treat it as acquire. Use acquire.
  • std::atomic<T> default construction now value-initializes (a plain std::atomic<int> x; starts as zero instead of being uninitialized).
std::atomic<int> x;     // C++20: holds 0

For low latency, the practical advice stays the same: use relaxed where you only need atomicity, release/acquire for publication, and seq_cst only where you truly need a single total order.


11. std::osyncstream: synchronized output (not for the hot path)

std::osyncstream buffers output and writes it atomically to avoid interleaved lines across threads.

std::osyncstream(std::cout) << "order " << id << " filled\n";

It is convenient for debugging, but it takes a lock and buffers on a shared sink, so keep it off the latency-critical path. Real systems log through a lock-free queue to a background thread.


12. Coroutines as a concurrency tool

C++20 coroutines enable cooperative, single-threaded concurrency (async state machines, generators, schedulers) with no thread switching.

Why they are interesting: a coroutine suspend/resume is much cheaper than a thread context switch, and you can write straight-line async code instead of callback chains.

Why to be cautious on the hot path: the coroutine frame is heap-allocated by default (the compiler may elide it, but it is not guaranteed), and resumption goes through an indirect call. You can customize allocation in the promise type (operator new), but it takes care. Treat coroutines as an architectural tool, and measure before putting one on the critical path.


C++20 Concurrency Features to Treat with Caution in Low-Latency Code

Feature Why to be careful
atomic<shared_ptr> / atomic<weak_ptr> Typically not lock-free, and does reference counting on shared cache lines. Fine for rarely changing config, wrong for hot reads.
atomic<double>::fetch_add Usually a CAS loop on x86, so it can degrade badly under contention. Use per-thread accumulators.
atomic::wait / notify A blocked thread pays scheduler wake-up latency. wait(old) can miss ABA-style changes. Use a sequence counter, and spin first on the hot path.
barrier / latch / semaphores They block. Fine for startup, shutdown and low-rate signaling, but not for the per-message path. The implementation is not specified.
stop_token checks in tight loops Each stop_requested() is an atomic load. Check once per batch rather than every iteration of a very hot loop.
stop_source creation Allocates shared state. Create it once at startup, not per task.
jthread creation Does not make thread creation cheap. Create threads up front and reuse them.
osyncstream Locks and buffers. Use a lock-free logger instead.
Coroutines Frame allocation and indirect resumption. Measure first, and customize allocation if used.

Summary Table

# Feature Low-latency relevance
1 jthread + stop_token RAII threads with cooperative cancellation
2 atomic::wait / notify Futex-backed sleeping on a value; hybrid spin-then-wait
3 atomic_ref Atomic access to plain-layout or shared-memory data
4 counting_semaphore / binary_semaphore Lightweight cross-thread signaling
5 latch / barrier Startup, shutdown and phase synchronization
6 atomic_flag::test Proper TTAS spinlocks
7 Atomic floating-point fetch_add Convenient, but a CAS loop under contention
8 atomic<shared_ptr> Safe config publication (not lock-free)
9 condition_variable_any + stop_token Clean shutdown of waiting threads
10 Memory model refinements Consistent semantics for acquire/release reasoning
11 osyncstream Debug output only
12 Coroutines Cooperative concurrency without thread switches

Final thought: the C++20 concurrency additions are mostly about safer and more convenient coordination (jthread, stop_token, latch, barrier, semaphores), not about making the hot path faster. The genuinely hot-path tools remain atomics, memory ordering, cache-line layout and thread pinning, and the new atomic::wait, atomic_ref and atomic_flag::test give you better building blocks for them. I would expect a strong candidate to be able to explain when each new primitive blocks, what it costs, and why the critical path should still avoid it.

Comments

Popular posts from this blog

FIX protocol interview questions : Part 1 - Admin/Session

Well, I just felt a need that key FAQs in FIX be documented, so that can be used for a variety of purposes, primarily during interviews to gauge a candidate. I have tried to be correct to the max, but do not guarantee it. This is what is best to my knowledge, and its good to cross check things with the FIX site and documentation available there. 1. What is buy side and what is sell side ? Sell side is the one who earns brokerage. Buy side is the one who pays brokerage. buy side (institutions) sell side (brokers/dealers) 2. What is the latest FIX version in market now? Its FIX 5.0 SP2. Something called FIXT.1.1 is also coming. 3. What is FIXML? FIXML is a structured, validated Extensible Markup Language (XML) derived grammar that is encapsulated within the standard FIX message. This format leaves the FIX session handling intact and minimizes the impact to existing implementations. As an XML-derived language, FIXML messages can be validated by standard parsers and take advantage of a fle...

what is a core dump

 A core dump (or core file ) is a file that captures the memory contents of a running process at a particular point in time, typically when the process crashes or encounters a serious error, such as a segmentation fault or illegal instruction. What is a core dump? It is essentially a snapshot of a process’s memory, including the call stack, memory allocations, and the state of the program at the time of the crash. It helps developers analyze the cause of a crash by examining the state of the program when the error occurred. Why does a core dump occur? A core dump typically happens when a process encounters a fatal error that causes it to terminate unexpectedly. This might include: Segmentation faults (segfaults): Trying to access memory that the process is not allowed to. Illegal instructions: Executing invalid machine instructions. Memory access violations: Trying to access memory in a way that is not allowed (e.g., reading from or writing to protected memory). Un...

std::unordered_map problems

 The std::unordered_map in C++ is a powerful container that provides fast access to key-value pairs with average constant-time complexity ( O(1) ) for lookups, inserts, and deletions. However, like any data structure, it has its potential issues, trade-offs, and limitations. Here are some common problems and limitations associated with std::unordered_map : 1. Unpredictable Iteration Order Problem: The order of elements in an unordered_map is not guaranteed. It depends on the internal hash table implementation and the distribution of hash values. Impact: This makes std::unordered_map unsuitable for use cases where the order of elements matters, such as when you need to traverse the elements in a specific order. Example: cpp Copy std::unordered_map< int , std::string> map; map[ 1 ] = "one" ; map[ 2 ] = "two" ; map[ 3 ] = "three" ; // The order in which elements are iterated is not guaranteed for ( auto & entry : map) { std::cout ...