Worker Pools in Rust In this example we’ll look at how to implement a worker pool using threads and channels.
use std ::thread ;
use std ::sync ::mpsc ;
use std ::time ::Duration ;
// Here's the worker, of which we'll run several
// concurrent instances. These workers will receive
// work on the `jobs` channel and send the corresponding
// results on `results`. We'll sleep a second per job to
// simulate an expensive task.
fn worker ( id : i32 , jobs : mpsc ::Receiver < i32 > , results : mpsc ::Sender < i32 > ) {
for job in jobs {
println! ( "worker {} started job {} " , id , job );
thread ::sleep ( Duration ::from_secs ( 1 ));
println! ( "worker {} finished job {} " , id , job );
results . send ( job * 2 ). unwrap ();
}
}
fn main () {
// In order to use our pool of workers we need to send
// them work and collect their results. We make 2
// channels for this.
const NUM_JOBS : i32 = 5 ;
let ( jobs_sender , jobs_receiver ) = mpsc ::channel ();
let ( results_sender , results_receiver ) = mpsc ::channel ();
// This starts up 3 workers, initially blocked
// because there are no jobs yet.
for w in 1 ..= 3 {
let jobs = jobs_receiver . clone ();
let results = results_sender . clone ();
thread ::spawn ( move || {
worker ( w , jobs , results );
});
}
// Here we send 5 `jobs` and then `close` that
// channel to indicate that's all the work we have.
for j in 1 ..= NUM_JOBS {
jobs_sender . send ( j ). unwrap ();
}
drop ( jobs_sender );
// Finally we collect all the results of the work.
// This also ensures that the worker threads have
// finished.
for _ in 1 ..= NUM_JOBS {
results_receiver . recv (). unwrap ();
}
}
Our running program shows the 5 jobs being executed by various workers. The program only takes about 2 seconds despite doing about 5 seconds of total work because there are 3 workers operating concurrently.
$ cargo run
Compiling worker-pools v0.1.0 (/path/to/worker-pools)
Finished dev [unoptimized + debuginfo] target(s) in 1.21s
Running `target/debug/worker-pools`
worker 1 started job 1
worker 3 started job 3
worker 2 started job 2
worker 1 finished job 1
worker 1 started job 4
worker 3 finished job 3
worker 2 finished job 2
worker 2 started job 5
worker 1 finished job 4
worker 2 finished job 5
real 0m2.007s
user 0m0.004s
sys 0m0.000s
In this Rust version, we use std::thread
for concurrency and std::sync::mpsc
for channel-based communication between threads. The overall structure and functionality remain similar to the original example, with workers processing jobs concurrently and sending results back to the main thread.