nondeterminism in concurrent programs
A single-threaded program is deterministic in a comforting way: give it the same input and it does exactly the same thing every time, step for step. Concurrent programs lose that guarantee. Because the operating system decides moment to moment which thread runs when - and that depends on timing, load, and other programs on the machine - the same concurrent program with the same input can take a different path on each run. This is nondeterminism: the outcome is not fixed by the input alone, but also by an interleaving you neither see nor control.
Here is the mechanism. The threads' operations can be interleaved in many possible orders, and which order actually happens is decided by the scheduler based on factors outside your program - how busy the CPU is, when an interrupt arrives, what else is running. So run the program twice and the threads may interleave differently each time. If the code is correct under every interleaving, this nondeterminism is harmless - you just get the right answer by different routes. But if some rare interleaving is buggy (a race condition), then the bug appears only when that particular interleaving happens to occur, which might be one run in a million, and only under conditions you cannot easily recreate.
Why this is the deep reason concurrency bugs are so painful: they are not reliably reproducible. A race may pass every test, run perfectly in development, then fail once in production under heavy load - and when you try to reproduce it, your debugging changes the timing (adding a breakpoint or a print slows a thread down) so the harmful interleaving no longer happens and the bug seems to vanish. That is a heisenbug: observing it changes it. The honest consequence is that you cannot test concurrency bugs away by running the program a few times and seeing no failure - absence of a crash is not proof of correctness. You have to reason about all interleavings, eliminate shared mutable state or protect it, and lean on tools like a thread sanitizer that detect data races directly rather than waiting for them to manifest.
Run a program with an unsynchronized shared counter incremented by four threads. One run prints 4000000, the next prints 3998732, the next 3999981 - different each time, all wrong, because the threads interleaved differently on each run and lost different numbers of updates.
Same code, same input, different results each run - the signature of a concurrency bug.
'It ran fine 100 times' does NOT mean the code is correct - a rare interleaving may simply not have occurred yet. Worse, the act of observing (a debugger, a print, a sleep) can hide the bug by changing timing. Reason about interleavings and use a thread sanitizer instead of trusting lucky runs.