Atomic Counters in Python

Our primary mechanism for managing state in Python is through communication over queues. However, there are a few other options for managing state. Here we’ll look at using the threading module and the multiprocessing module for atomic counters accessed by multiple threads.

import threading
from multiprocessing import Value, Process

def increment(counter, num_increments):
    for _ in range(num_increments):
        with counter.get_lock():
            counter.value += 1

def main():
    # We'll use a Value object to represent our (always-positive) counter.
    ops = Value('i', 0)

    # We'll start 50 processes that each increment the counter exactly 1000 times.
    processes = []
    for _ in range(50):
        p = Process(target=increment, args=(ops, 1000))
        processes.append(p)
        p.start()

    # Wait until all the processes are done.
    for p in processes:
        p.join()

    # Here no processes are writing to 'ops', so it's safe to read the value.
    print("ops:", ops.value)

if __name__ == "__main__":
    main()

In this Python version, we’re using the multiprocessing module instead of goroutines. The Value object from multiprocessing is used as a shared memory location that can be safely accessed by multiple processes.

We create a Value object to represent our counter. This object is thread-safe and process-safe, so it can be safely incremented by multiple processes.

We then create 50 separate processes, each of which will increment the counter 1000 times. We use a separate function increment for this, which takes the counter and the number of increments as arguments.

After starting all processes, we wait for them to finish using the join method.

Finally, we print the final value of the counter.

To run the program:

$ python atomic_counters.py
ops: 50000

We expect to get exactly 50,000 operations. If we had used a regular Python integer and incremented it with ops += 1, we’d likely get a different number, changing between runs, because the processes would interfere with each other. The Value object ensures that the increments are atomic.

Next, we’ll look at locks, another tool for managing state in concurrent programs.