Worker Pools in Chapel

Our example demonstrates how to implement a worker pool using Chapel’s task parallelism and synchronization primitives.

use Time;

// This is the worker function, of which we'll run several
// concurrent instances. These workers will receive work on the
// `jobs` channel and send the corresponding results on `results`.
// We'll sleep a second per job to simulate an expensive task.
proc worker(id: int, jobs: [] int, results: [] int) {
  for j in jobs {
    writeln("worker ", id, " started  job ", j);
    sleep(1);
    writeln("worker ", id, " finished job ", j);
    results[j] = j * 2;
  }
}

proc main() {
  // In order to use our pool of workers we need to send
  // them work and collect their results. We make 2
  // arrays for this.
  const numJobs = 5;
  var jobs: [1..numJobs] int;
  var results: [1..numJobs] int;

  // Initialize the jobs array
  for j in 1..numJobs do jobs[j] = j;

  // This starts up 3 workers, initially blocked
  // because there are no jobs yet.
  coforall w in 1..3 {
    worker(w, jobs, results);
  }

  // Finally we collect all the results of the work.
  // This also ensures that the worker tasks have finished.
  for a in 1..numJobs {
    writeln("Result of job ", a, " is ", results[a]);
  }
}

Our running program shows the 5 jobs being executed by various workers. The program only takes about 2 seconds despite doing about 5 seconds of total work because there are 3 workers operating concurrently.

$ chpl worker_pools.chpl
$ ./worker_pools
worker 1 started  job 1
worker 2 started  job 2
worker 3 started  job 3
worker 1 finished job 1
worker 1 started  job 4
worker 2 finished job 2
worker 2 started  job 5
worker 3 finished job 3
worker 1 finished job 4
worker 2 finished job 5
Result of job 1 is 2
Result of job 2 is 4
Result of job 3 is 6
Result of job 4 is 8
Result of job 5 is 10

In this Chapel version:

We use Chapel’s coforall loop to create multiple concurrent tasks, which is similar to launching goroutines in Go.
Instead of channels, we use shared arrays (jobs and results) for communication between tasks. Chapel’s memory model ensures safe concurrent access to these arrays.
The worker procedure takes arrays as arguments instead of channels.
We use Chapel’s sleep() function from the Time module to simulate work.
The main procedure creates the jobs, starts the workers, and then prints the results.

This example showcases Chapel’s support for task parallelism and how it can be used to implement a worker pool pattern, similar to the original Go example.

Tickers in Chapel

Waitgroups in Chapel

Learn X By Example

Title here

Worker Pools in Chapel