Worker Pools in Elixir

Our example demonstrates how to implement a worker pool using Elixir’s processes and message passing.

defmodule WorkerPool do
  def worker(id) do
    receive do
      {:job, job_id} ->
        IO.puts("worker #{id} started  job #{job_id}")
        :timer.sleep(1000)
        IO.puts("worker #{id} finished job #{job_id}")
        send(self(), {:result, job_id * 2})
        worker(id)
      :stop -> :ok
    end
  end

  def main do
    num_jobs = 5
    num_workers = 3

    workers = Enum.map(1..num_workers, fn id ->
      spawn(fn -> worker(id) end)
    end)

    Enum.each(1..num_jobs, fn job ->
      worker = Enum.random(workers)
      send(worker, {:job, job})
    end)

    results = Enum.map(1..num_jobs, fn _ ->
      receive do
        {:result, value} -> value
      end
    end)

    IO.inspect(results)

    Enum.each(workers, fn worker ->
      send(worker, :stop)
    end)
  end
end

WorkerPool.main()

In this example, we create a worker pool using Elixir’s processes. Here’s how it works:

1. We define a worker function that receives jobs and processes them. It simulates an expensive task by sleeping for a second.

2. In the main function, we spawn a number of worker processes.

3. We send jobs to random workers using Elixir’s message passing.

4. We collect the results of all jobs.

5. Finally, we stop all worker processes.

The program demonstrates concurrent execution of jobs. Despite having 5 jobs that each take about a second to complete, the total execution time will be around 2 seconds because we have 3 workers operating concurrently.

To run the program, save it in a file (e.g., worker_pool.exs) and execute it with:

$ elixir worker_pool.exs

The output will show jobs being executed by various workers concurrently:

worker 1 started  job 1
worker 2 started  job 2
worker 3 started  job 3
worker 1 finished job 1
worker 1 started  job 4
worker 2 finished job 2
worker 2 started  job 5
worker 3 finished job 3
worker 1 finished job 4
worker 2 finished job 5
[2, 4, 6, 8, 10]

This example showcases Elixir’s powerful concurrency model, which is based on the actor model and is well-suited for building concurrent and distributed systems.