C++: Concurrency Pt 2 - Sharing

Pt 1 ended with threads that never touched the same data. This one is about what happens when they do.

Mutexes solve two problems. Both are “threads sharing memory goes wrong”, but they go wrong in different ways.

Problem 1: Interleaving Breaks Compound Operations

int balance = 100;

// two threads, each runs:
balance = balance - 10;

mov  eax, [balance]   ; 1. load
sub  eax, 10          ; 2. subtract  (register-local, invisible to others)
mov  [balance], eax   ; 3. store

std::mutex m;

m.lock();
balance = balance - 10;    // nobody can interleave here
m.unlock();

Problem 2: Without Synchronisation, Writes Are Not Published

Config cfg;        // plain struct
bool ready = false;

// Thread A                    // Thread B
cfg.port = 8080;               if (ready)
cfg.host = "x";                    use(cfg);   // may see garbage
ready = true;

// Thread A                    // Thread B
m.lock();                      m.lock();
cfg.port = 8080;               if (ready)
cfg.host = "x";                    use(cfg);   // guaranteed complete
ready = true;                  m.unlock();
m.unlock();

A mutex makes a region of code exclusive, and transfers its memory effects wholesale to the next locker. Exclusion handles the interleaving, the acquire/release edge handles the publication.

You need both, and both sides must lock. One thread using the mutex while another bypasses it gives you neither.

Anatomy of a Mutex

On Linux, std::mutex bottoms out in pthread_mutex_lock, which has two paths:

lock():
  CAS 0 -> 1 on a userspace word
    success:  done. ~nanoseconds, kernel never involved
    failure:  futex(WAIT) syscall, kernel sleeps the thread
              on a queue keyed by the word's address

unlock():
  release store 0, and if waiters exist: futex(WAKE) syscall

The word is ordinary memory. In the uncontended case the kernel never learns the mutex exists: one atomic RMW and you are in the critical section.

futex(WAIT) is where the thread actually blocks. It does not spin and does not hold its timeslice. The kernel takes it off the run queue and parks it on a wait queue hashed by the address of that word, and the scheduler will not consider it again until something issues a futex(WAKE) on the same address.

unlock() is what issues it, and for a mutex it wakes one waiter. Only one thread can take the lock next, so waking the rest would just have them fail their CAS and sleep again.

The two paths are about three orders of magnitude apart. Uncontended is one atomic RMW, call it 20ns. Contended is two syscalls, a context switch off the core and back, and the cache damage of being descheduled in between: tens of microseconds. The mutex is not the expensive part, the contention is.

Spinlocks vs Mutexes

A spinlock takes the other option on contention: instead of sleeping, keep retrying.

std::atomic<bool> flag{false};

while (flag.exchange(true, std::memory_order_acquire))
    ;                      // spin until we win it
// critical section
flag.store(false, std::memory_order_release);

No syscall is involved and nobody leaves the run queue.

The failure mode is preemption while holding the lock: the holder gets descheduled mid-critical-section, and every spinner burns its whole timeslice waiting on a thread that isn’t running. Userspace has no say in scheduling, which is what makes userspace spinlocks risky. Kernel spinlocks avoid this mostly because the kernel disables preemption while one is held.

So spinning only wins when two conditions both hold.

1. The critical section is shorter than the sleep round trip.

Spinning wastes (wait time × a core); sleeping costs a roughly fixed ~2 syscalls + 2 context switches, call it single-digit microseconds. If the lock is held for tens of nanoseconds, you’d be paying microseconds of machinery to avoid nanoseconds of waiting. Long or unpredictable hold times flip it.

2. The holder is actually running while you spin.

In practice: pinned threads, no oversubscription, no preemption or page faults mid-section. Then the wait is bounded by the critical section length and condition 1’s math holds. Break it and the wait becomes “until the scheduler reschedules the holder” — the disaster above.

glibc already spins briefly before falling back to futex(WAIT), so a plain std::mutex gets you most of this anyway.

Mutex RAII Wrappers

Everything above used raw m.lock() and m.unlock(), which you should not write in real code.

m.lock();
process(data);      // throws? early return?
m.unlock();         // never runs

Any exit path that skips the unlock leaves the mutex locked forever, and the next thread to lock it deadlocks. Early returns, break and exceptions all do this, and the compiler does not warn about any of them.

The fix is to lock in a constructor and unlock in a destructor, so leaving the scope releases the mutex however you leave it.

lock_guard

Nothing else. There is no unlock(), no owns_lock() and no release(), and no move constructor either, since the deleted copies suppress it (see the suppression matrix). The template parameter is any BasicLockable, which is anything with lock() and unlock().

unique_lock

All of it is paid for by one bool. Because the destructor asks that flag instead of unlocking unconditionally, a unique_lock can be empty, deferred, moved from, or unlocked early and still destruct correctly.

The one to watch is release(). It does not unlock, it only severs the association, so whoever called it now owes an unlock() by hand. unlock() and release() sound similar and do close to opposite things.

scoped_lock

scoped_lock: variadic, deadlock-free multi-lock, the C++17 default instead of lock_guard. Like lock_guard it has no unlock(), no owns_lock() and no move, and given a single argument the two produce the same code.

lock_guard is not deprecated and is not going away, but it survives for ABI stability and the code that already uses it, not because it does anything scoped_lock cannot.

One caveat comes from the mutex list being a parameter pack: std::scoped_lock lk; with no arguments compiles and locks nothing, where std::lock_guard lk; does not compile at all. A refactor that drops the mutex argument fails loudly with one and silently with the other.

The argument order differs from the other two, since the mutexes are a parameter pack and the tag has nowhere else to go:

std::lock_guard  g {m, std::adopt_lock};        // tag last
std::scoped_lock lk{std::adopt_lock, m1, m2};   // tag first

Multiple Locks and Deadlock

One mutex protects one invariant. Some operations span two at the same time:

struct Account {
    std::mutex m;
    int bal;
};

void transfer(Account& from, Account& to, int amt);

Locking only from.m does not protect to.bal. Locking one, updating, unlocking, then locking the other exposes the in-between state, where the money is missing from both accounts or present in both, to anyone summing balances. The operation is atomic over both accounts or it is wrong, so both mutexes have to be held at once. This is not exotic; it turns up the moment one operation touches two independently locked things.

The naive version:

void transfer(Account& from, Account& to, int amt) {
    from.m.lock();
    to.m.lock();
    // ... update both ...
    to.m.unlock();
    from.m.unlock();
}

Now run it from two threads:

Thread A: transfer(alice, bob, 10)    locks alice.m, wants bob.m
Thread B: transfer(bob, alice, 5)     locks bob.m,   wants alice.m

A holds alice and waits for bob, B holds bob and waits for alice. Neither releases what it holds, so both wait forever. This is a permanent hang rather than a slowdown, and it only fires when the interleaving lands just so, which means it passes tests and shows up in production.

Both threads ran the same correct-looking code. The bug is not in either one, it is in the pair: each locks its own from first, and the two calls disagree about which account that is.

Lock Ordering

The deadlock needed the two threads to take the locks in opposite orders, so forbid that. Pick one global order and have every thread acquire in it, regardless of which account is from:

void transfer(Account& from, Account& to, int amt) {
    Account* first  = &from;
    Account* second = &to;
    if (second < first) std::swap(first, second);   // order by address

    first->m.lock();
    second->m.lock();
    from.bal -= amt;      // from/to for the logic,
    to.bal   += amt;      // first/second only for locking
    second->m.unlock();
    first->m.unlock();
}

Both threads now take whichever account has the lower address first. One of them wins, finishes and releases, and the other proceeds.

std::lock and scoped_lock

Ordering by hand is clunky, and sometimes impossible when the locks are acquired in call frames that cannot see each other.

std::lock is a free function, not a type:

template <class L1, class L2, class... Ln>
void lock(L1&, L2&, Ln&...);

It takes two or more lockables and returns only once every one of them is locked. It owns nothing and releases nothing, so the unlocking is still your problem.

It gets there without a fixed order. It blocks on one of them, calls try_lock() on the rest, and if any of those refuse it releases everything it has taken and restarts from the one that refused. A thread running std::lock therefore never blocks while holding a lock, which is the hold-and-wait condition gone, so argument order stops mattering.

std::scoped_lock is that mechanism plus RAII. Its constructor calls std::lock on the pack, and its destructor unlocks all of them:

void transfer(Account& from, Account& to, int amt) {
    std::scoped_lock g{from.m, to.m};   // any order, deadlock-free, RAII
    from.bal -= amt;
    to.bal   += amt;
}

This is what adopt_lock was for. Before C++17 you called std::lock yourself and then adopted each mutex into its own guard to get the release half back. scoped_lock fuses the two steps into one type.

Two Closing Rules

Never call unknown code under a lock

void add(std::string s) {
    std::scoped_lock g{m};
    lines_.push_back(std::move(s));
    for (auto& cb : callbacks_) cb(s);   // unknown code, lock held
}

cb is code you do not control, and three separate failures hide in that one line.

Re-lock deadlock. The callback, directly or five frames down, calls back into a method that locks m. Same thread, same mutex, second lock, and std::mutex is not recursive, so that is UB. Nobody wrote a recursive call; it emerged from composition.

Lock-order inversion at a distance. The callback locks some other mutex n, while elsewhere a thread holding n calls something that locks m. That is the transfer deadlock again, except no single file shows both orders, and ordering discipline cannot see through a std::function.

Unbounded hold time. The callback does I/O, allocates, or blocks, and every thread needing m queues behind it.

The root is the same in all three: holding a lock puts you in a restricted context, and opaque code smuggles arbitrary behaviour into it. Allocation counts as opaque, since malloc can take internal locks and syscalls of its own.

The fix is mechanical. Shrink the critical section to the data mutation, snapshot what the callbacks need, release, then call:

void add(std::string s) {
    std::vector<Callback> snap;
    {
        std::scoped_lock g{m};
        lines_.push_back(std::move(s));
        snap = callbacks_;          // copy under lock
    }
    for (auto& cb : snap) cb(s);    // lock released
}

The price is that callbacks now run against possibly stale state, so one removed concurrently may still fire once. Usually worth paying, but it is a real trade rather than a free fix.

A mutex protects an invariant, not a region of code

The beginner model is that the lock makes this code safe. What it actually guards is a statement about data that is allowed to be temporarily false while the lock is held: size_ equals the number of live nodes, the freelist contains only unused slots, the balances sum to the total. Inside the critical section you break the invariant and restore it, and the unlock publishes a state where it holds again.

Everything practical falls out of that reframing.

• What to lock: every access to the data the invariant spans, not just the dangerous-looking writes. A racy read of size_ is still UB.
• Granularity has a floor: one invariant spanning two accounts needs both locks held at once. You cannot lock finer than your invariants.
• Granularity has a direction: one mutex per invariant, not per field. One giant lock serialises threads that never conflict, while per-field locks under a single invariant are wrong rather than merely slow. Start coarse, and split along invariant boundaries only once contention is measured.
• What to document: the useful comment is never “lock m before calling this”, it is “m protects x, y, and the invariant relating them”.

Pt 3 covers WAITING: condition variables, and the unique_lock they require.

Appendix: unique_lock, Sketched

template <class Mutex>
class unique_lock {
    Mutex* m_   = nullptr;    // pointer, not reference: must be reseatable and nullable
    bool  owns_ = false;
public:
    unique_lock() noexcept = default;

    explicit unique_lock(Mutex& m) : m_(&m), owns_(true) { m_->lock(); }
    unique_lock(Mutex& m, std::defer_lock_t) noexcept : m_(&m) {}
    unique_lock(Mutex& m, std::try_to_lock_t) : m_(&m), owns_(m.try_lock()) {}
    unique_lock(Mutex& m, std::adopt_lock_t)  : m_(&m), owns_(true) {}

    unique_lock(unique_lock&& o) noexcept
        : m_(std::exchange(o.m_, nullptr)),
          owns_(std::exchange(o.owns_, false)) {}

    unique_lock& operator=(unique_lock&& o) noexcept {
        if (owns_) m_->unlock();
        m_    = std::exchange(o.m_, nullptr);
        owns_ = std::exchange(o.owns_, false);
        return *this;
    }

    unique_lock(const unique_lock&) = delete;
    unique_lock& operator=(const unique_lock&) = delete;

    ~unique_lock() { if (owns_) m_->unlock(); }

    void lock()     { m_->lock(); owns_ = true; }      // real one throws if !m_ or owns_
    bool try_lock() { owns_ = m_->try_lock(); return owns_; }
    void unlock()   { m_->unlock(); owns_ = false; }   // real one throws if !owns_

    bool owns_lock() const noexcept { return owns_; }
    explicit operator bool() const noexcept { return owns_; }
    Mutex* release() noexcept {
        owns_ = false;
        return std::exchange(m_, nullptr);
    }
};

Appendix: The Four Conditions for Deadlock