Important C++20 Concurrency Features for Low-Latency C++
C++20 added a useful set of concurrency primitives. For low-latency work the question is never just "does it exist?" but "what does it cost on the hot path, and does it ever block, allocate or make a syscall?" This guide covers the C++20 concurrency features that matter most, what they do under the hood, and where they help or hurt latency.
A reminder of the usual low-latency rules these features are judged against:
- Avoid blocking on the hot path (a context switch costs microseconds, which is an eternity in this domain).
- Avoid hidden allocation and hidden locks.
- Keep shared data on separate cache lines, and minimize cache-line contention.
- Pin threads and create them up front. Never create threads on the hot path.
Tier 1: Features You Should Know Cold
1. std::jthread and std::stop_token: safe threads and cooperative cancellation
std::jthread is a std::thread that joins automatically in its destructor (after requesting a stop), which fixes the classic std::terminate() on a joinable thread. It also integrates a stop token for cooperative cancellation.
std::jthread worker([](std::stop_token st) {
while (!st.stop_requested()) {
pollAndProcess();
}
});
// worker.request_stop(); // optional: explicit stop
// Destructor: request_stop() then join()
Related types:
std::stop_source: the object that requests the stop.std::stop_token: the lightweight handle a thread checks.std::stop_callback: registers a function to run when a stop is requested (for example, to wake a blocked thread).
std::jthread t([&](std::stop_token st) {
std::stop_callback cb(st, [&] {
wakeFlag.store(true, std::memory_order_release);
wakeFlag.notify_all(); // unblock a thread waiting on the flag
});
// ... blocking work that can be woken by the callback ...
});
Low-latency notes
stop_requested()is an atomic load, so it is cheap, but not free. In a very tight loop, check it every N iterations or once per batch.- The stop state is shared and heap-allocated when a
stop_sourceis created. That's fine at startup, but not something to create per message. jthreaddoesn't make thread creation cheaper. Create your threads at startup, pin them to cores (vianative_handle()andpthread_setaffinity_np, or the platform equivalent), and keep them alive.stop_callbackruns on the thread that callsrequest_stop(), so keep callbacks short.
Interview takeaway:
jthread= RAII join + built-in cooperative cancellation. It prevents the destroyed-while-joinable bug, but the cancellation is cooperative: the thread has to check the token.
2. std::atomic::wait, notify_one, notify_all: efficient blocking on an atomic
Before C++20, waiting for an atomic variable to change meant either spinning (burning a core) or pairing it with a mutex and condition variable (heavier). C++20 adds wait() / notify_*() directly on atomics, typically implemented with a futex on Linux (or the equivalent WaitOnAddress on Windows).
std::atomic<uint32_t> seq{0};
// Consumer: block until the value is no longer 0
seq.wait(0, std::memory_order_acquire); // returns once seq != 0
// Producer
seq.store(1, std::memory_order_release);
seq.notify_one(); // wake one waiter
A common low-latency pattern is a hybrid wait: spin briefly for the lowest latency, then fall back to blocking so you don't burn a core indefinitely.
void waitForChange(std::atomic<uint32_t>& a, uint32_t old) {
for (int i = 0; i < 1000; ++i) {
if (a.load(std::memory_order_acquire) != old) return;
_mm_pause(); // x86 PAUSE (no standard portable equivalent in C++20)
}
a.wait(old, std::memory_order_acquire); // then sleep
}
Pitfalls
wait(old)returns when the value differs fromold. If the value changes and changes back between checks (an ABA pattern), a waiter can miss the change. Use a monotonically increasing sequence number to avoid this.notify_*()on the producer side can cost a syscall when waiters exist. Implementations try to avoid it when nobody is waiting, but the cost is implementation-defined, so measure.- Waking a sleeping thread still costs microseconds of scheduler latency. If the absolute lowest latency matters, keep the consumer spinning on a pinned core.
Interview takeaway:
atomic::waitis "efficient sleeping on a value", which is useful for idle or low-rate paths. The hot path should still spin.
3. std::atomic_ref: atomic operations on a plain object
std::atomic_ref<T> lets you perform atomic operations on an object that is not declared std::atomic. This helps when data must have a plain layout, such as arrays filled in a single-threaded phase and then updated concurrently, or structures in shared memory and memory-mapped regions.
uint64_t counters[8]; // plain array, initialized normally
void hit(size_t i) {
std::atomic_ref<uint64_t> ref(counters[i]);
ref.fetch_add(1, std::memory_order_relaxed);
}
Rules to know:
- The referenced object must satisfy
std::atomic_ref<T>::required_alignment, which can be stricter thanalignof(T)for some types (for example 64-bit values on 32-bit x86). - While any
atomic_refto an object exists, all concurrent accesses to it must go throughatomic_refs. Mixing in plain reads or writes is a data race. - It is lock-free only when
std::atomic_ref<T>::is_always_lock_freeis true. Check it with astatic_assert.
static_assert(std::atomic_ref<uint64_t>::is_always_lock_free);
Interview takeaway:
atomic_refgives atomic access without changing the type's layout, but you must guarantee no non-atomic concurrent access during its lifetime.
4. std::counting_semaphore and std::binary_semaphore
Semaphores are counters you can acquire() (decrement, blocking at zero) and release() (increment, waking a waiter). Unlike a mutex, any thread can release, so a semaphore is ideal for signaling between threads.
std::binary_semaphore ready{0};
// Producer thread
publish(data);
ready.release(); // signal
// Consumer thread
ready.acquire(); // block until signaled
consume();
Non-blocking and timed variants:
if (ready.try_acquire()) { /* got it, never blocks */ }
if (ready.try_acquire_for(std::chrono::microseconds(50))) { /* ... */ }
For simple "wake me when work arrives" signaling, a semaphore is lighter and simpler than a mutex + condition_variable + predicate, and it doesn't suffer from lost wakeups because the count remembers a release() that happened before the acquire().
Pitfalls
try_acquire()is allowed to fail spuriously.- The implementation is not specified: on Linux it is generally futex-based. Still, treat
acquire()as a potential block and don't call it on a path that must never sleep. counting_semaphore<N>::max()is a least upper bound. Don't release past it.
5. std::latch and std::barrier: group synchronization
std::latch is a single-use countdown. Threads call count_down(), and others wait() until it reaches zero.
std::latch ready(numWorkers);
// Each worker
initThreadLocalState();
pinToCore();
ready.count_down(); // "I'm ready"
// Main thread
ready.wait(); // block until every worker has signaled
startTrading();
std::barrier is reusable and organizes work into phases. All threads must arrive before any can continue, and an optional completion function runs once per phase.
std::barrier sync(numThreads, []() noexcept { swapBuffers(); });
// In each thread's loop
for (;;) {
doPhaseWork();
sync.arrive_and_wait(); // all threads synchronize here
}
Low-latency notes
- Both are great for startup and shutdown coordination (every thread pinned and warmed up before the first message).
- They block, so they generally don't belong on the per-message hot path.
- A barrier makes every phase as slow as its slowest thread, which hurts tail latency.
Interview takeaway:
latch= one-shot,barrier= reusable with phases and a completion function.
6. std::atomic_flag::test and std::atomic_flag with wait
C++20 added atomic_flag::test() (a read that doesn't modify the flag) plus wait / notify. The big use is writing a proper test-and-test-and-set (TTAS) spinlock, which avoids hammering the cache line with writes while waiting.
class SpinLock {
std::atomic_flag flag; // C++20: default-initialized to clear
public:
void lock() {
for (;;) {
if (!flag.test_and_set(std::memory_order_acquire)) return;
while (flag.test(std::memory_order_relaxed)) { // read-only spin
_mm_pause();
}
}
}
void unlock() { flag.clear(std::memory_order_release); }
};
Spinning on the read (test) keeps the cache line in the shared state across waiters, rather than bouncing it between cores with each test_and_set.
Pitfall: a spinlock is only appropriate when threads are pinned to dedicated cores and critical sections are tiny. If the lock holder is preempted by the OS, the waiters burn CPU until it runs again.
7. Atomic floating-point fetch_add / fetch_sub
std::atomic<float> and std::atomic<double> gained fetch_add and fetch_sub in C++20.
std::atomic<double> total{0.0};
total.fetch_add(price * qty, std::memory_order_relaxed);
Pitfall: x86 has no native atomic floating-point add, so this is usually implemented as a compare-exchange loop. Under contention it can retry repeatedly and become slow. Prefer per-thread accumulators (each on its own cache line) that you combine later, rather than one shared atomic double.
Tier 2: Useful in Specific Situations
8. std::atomic<std::shared_ptr<T>> and std::atomic<std::weak_ptr<T>>
C++20 finally provides a correct, standard way to atomically publish and read a shared_ptr.
std::atomic<std::shared_ptr<const Config>> g_config;
// Writer
g_config.store(std::make_shared<const Config>(newCfg));
// Reader
auto cfg = g_config.load(); // takes a shared reference
use(*cfg);
For latency, be careful: in mainstream implementations this is generally not lock-free (is_lock_free() is typically false), since the pointer and the control block have to be updated together. Loading also does reference-count traffic on a shared cache line. It is excellent for infrequently updated configuration, but a poor fit for a hot read path. For hot paths, prefer a raw std::atomic<T*> with a deliberate reclamation strategy (epoch-based, hazard pointers, or RCU-style double buffering).
9. Condition variables with stop tokens
std::condition_variable_any can wait with a stop_token, so a blocked thread can be woken by a stop request, which solves the "how do I shut down a thread that is waiting?" problem cleanly.
void worker(std::stop_token st) {
std::unique_lock lock(m);
while (cv.wait(lock, st, [] { return !queue.empty(); })) {
process(queue.pop());
}
// wait() returned false: stop was requested
}
10. Memory model refinements
C++20 tightened parts of the memory model that were inconsistent or too strong:
- The definition of release sequences was weakened to match what hardware and compilers actually do.
- The rules for
seq_cstfences were strengthened. memory_order_consumeis still effectively unusable in practice, and compilers treat it asacquire. Useacquire.std::atomic<T>default construction now value-initializes (a plainstd::atomic<int> x;starts as zero instead of being uninitialized).
std::atomic<int> x; // C++20: holds 0
For low latency, the practical advice stays the same: use relaxed where you only need atomicity, release/acquire for publication, and seq_cst only where you truly need a single total order.
11. std::osyncstream: synchronized output (not for the hot path)
std::osyncstream buffers output and writes it atomically to avoid interleaved lines across threads.
std::osyncstream(std::cout) << "order " << id << " filled\n";
It is convenient for debugging, but it takes a lock and buffers on a shared sink, so keep it off the latency-critical path. Real systems log through a lock-free queue to a background thread.
12. Coroutines as a concurrency tool
C++20 coroutines enable cooperative, single-threaded concurrency (async state machines, generators, schedulers) with no thread switching.
Why they are interesting: a coroutine suspend/resume is much cheaper than a thread context switch, and you can write straight-line async code instead of callback chains.
Why to be cautious on the hot path: the coroutine frame is heap-allocated by default (the compiler may elide it, but it is not guaranteed), and resumption goes through an indirect call. You can customize allocation in the promise type (operator new), but it takes care. Treat coroutines as an architectural tool, and measure before putting one on the critical path.
C++20 Concurrency Features to Treat with Caution in Low-Latency Code
| Feature | Why to be careful |
|---|---|
atomic<shared_ptr> / atomic<weak_ptr> |
Typically not lock-free, and does reference counting on shared cache lines. Fine for rarely changing config, wrong for hot reads. |
atomic<double>::fetch_add |
Usually a CAS loop on x86, so it can degrade badly under contention. Use per-thread accumulators. |
atomic::wait / notify |
A blocked thread pays scheduler wake-up latency. wait(old) can miss ABA-style changes. Use a sequence counter, and spin first on the hot path. |
barrier / latch / semaphores |
They block. Fine for startup, shutdown and low-rate signaling, but not for the per-message path. The implementation is not specified. |
stop_token checks in tight loops |
Each stop_requested() is an atomic load. Check once per batch rather than every iteration of a very hot loop. |
stop_source creation |
Allocates shared state. Create it once at startup, not per task. |
jthread creation |
Does not make thread creation cheap. Create threads up front and reuse them. |
osyncstream |
Locks and buffers. Use a lock-free logger instead. |
| Coroutines | Frame allocation and indirect resumption. Measure first, and customize allocation if used. |
Summary Table
| # | Feature | Low-latency relevance |
|---|---|---|
| 1 | jthread + stop_token |
RAII threads with cooperative cancellation |
| 2 | atomic::wait / notify |
Futex-backed sleeping on a value; hybrid spin-then-wait |
| 3 | atomic_ref |
Atomic access to plain-layout or shared-memory data |
| 4 | counting_semaphore / binary_semaphore |
Lightweight cross-thread signaling |
| 5 | latch / barrier |
Startup, shutdown and phase synchronization |
| 6 | atomic_flag::test |
Proper TTAS spinlocks |
| 7 | Atomic floating-point fetch_add |
Convenient, but a CAS loop under contention |
| 8 | atomic<shared_ptr> |
Safe config publication (not lock-free) |
| 9 | condition_variable_any + stop_token |
Clean shutdown of waiting threads |
| 10 | Memory model refinements | Consistent semantics for acquire/release reasoning |
| 11 | osyncstream |
Debug output only |
| 12 | Coroutines | Cooperative concurrency without thread switches |
Final thought: the C++20 concurrency additions are mostly about safer and more convenient coordination (
jthread,stop_token,latch,barrier, semaphores), not about making the hot path faster. The genuinely hot-path tools remain atomics, memory ordering, cache-line layout and thread pinning, and the newatomic::wait,atomic_refandatomic_flag::testgive you better building blocks for them. I would expect a strong candidate to be able to explain when each new primitive blocks, what it costs, and why the critical path should still avoid it.
Comments
Post a Comment