Worker Pools in Erlang

In this example, we’ll look at how to implement a worker pool using Erlang processes and message passing.

-module(worker_pool).
-export([start/0]).

% Here's the worker function, of which we'll run several
% concurrent instances. These workers will receive
% work on their process mailbox and send the corresponding
% results to the main process. We'll sleep a second per job to
% simulate an expensive task.
worker(Id, Parent) ->
    receive
        {job, J} ->
            io:format("worker ~p started  job ~p~n", [Id, J]),
            timer:sleep(1000),
            io:format("worker ~p finished job ~p~n", [Id, J]),
            Parent ! {result, J * 2},
            worker(Id, Parent);
        stop -> ok
    end.

start() ->
    % In order to use our pool of workers we need to spawn them
    % and send them work. We'll use a list comprehension to create
    % 3 workers.
    NumJobs = 5,
    Workers = [spawn(fun() -> worker(W, self()) end) || W <- lists:seq(1, 3)],

    % Here we send 5 jobs to the workers using round-robin distribution
    [lists:nth(J rem 3 + 1, Workers) ! {job, J} || J <- lists:seq(1, NumJobs)],

    % Collect all the results of the work
    collect_results(NumJobs),

    % Stop all workers
    [Worker ! stop || Worker <- Workers].

collect_results(0) -> ok;
collect_results(N) ->
    receive
        {result, _} -> collect_results(N - 1)
    end.

Our running program shows the 5 jobs being executed by various workers. The program only takes about 2 seconds despite doing about 5 seconds of total work because there are 3 workers operating concurrently.

To run the program, save it as worker_pool.erl and use the Erlang shell:

$ erl
1> c(worker_pool).
{ok,worker_pool}
2> worker_pool:start().
worker 1 started  job 1
worker 2 started  job 2
worker 3 started  job 3
worker 1 finished job 1
worker 1 started  job 4
worker 2 finished job 2
worker 2 started  job 5
worker 3 finished job 3
worker 1 finished job 4
worker 2 finished job 5
ok

In this Erlang version:

We define a worker/2 function that receives jobs and sends results back to the parent process.
The start/0 function sets up the worker pool and distributes jobs.
We use Erlang’s built-in concurrency primitives: processes and message passing.
Instead of channels, we use process mailboxes for communication.
We use spawn/1 to create worker processes, similar to goroutines.
The collect_results/1 function gathers results from workers.

This example demonstrates how to implement a worker pool pattern in Erlang, showcasing the language’s powerful concurrency features.

Tickers in Erlang

Waitgroups in Erlang

Learn X By Example

Title here

Worker Pools in Erlang