Worker Pools in Rust
In this example we’ll look at how to implement a worker pool using threads and channels.
use std::thread;
use std::sync::mpsc;
use std::time::Duration;
// Here's the worker, of which we'll run several
// concurrent instances. These workers will receive
// work on the `jobs` channel and send the corresponding
// results on `results`. We'll sleep a second per job to
// simulate an expensive task.
fn worker(id: i32, jobs: mpsc::Receiver<i32>, results: mpsc::Sender<i32>) {
for job in jobs {
println!("worker {} started job {}", id, job);
thread::sleep(Duration::from_secs(1));
println!("worker {} finished job {}", id, job);
results.send(job * 2).unwrap();
}
}
fn main() {
// In order to use our pool of workers we need to send
// them work and collect their results. We make 2
// channels for this.
const NUM_JOBS: i32 = 5;
let (jobs_sender, jobs_receiver) = mpsc::channel();
let (results_sender, results_receiver) = mpsc::channel();
// This starts up 3 workers, initially blocked
// because there are no jobs yet.
for w in 1..=3 {
let jobs = jobs_receiver.clone();
let results = results_sender.clone();
thread::spawn(move || {
worker(w, jobs, results);
});
}
// Here we send 5 `jobs` and then `close` that
// channel to indicate that's all the work we have.
for j in 1..=NUM_JOBS {
jobs_sender.send(j).unwrap();
}
drop(jobs_sender);
// Finally we collect all the results of the work.
// This also ensures that the worker threads have
// finished.
for _ in 1..=NUM_JOBS {
results_receiver.recv().unwrap();
}
}Our running program shows the 5 jobs being executed by various workers. The program only takes about 2 seconds despite doing about 5 seconds of total work because there are 3 workers operating concurrently.
$ cargo run
Compiling worker-pools v0.1.0 (/path/to/worker-pools)
Finished dev [unoptimized + debuginfo] target(s) in 1.21s
Running `target/debug/worker-pools`
worker 1 started job 1
worker 3 started job 3
worker 2 started job 2
worker 1 finished job 1
worker 1 started job 4
worker 3 finished job 3
worker 2 finished job 2
worker 2 started job 5
worker 1 finished job 4
worker 2 finished job 5
real 0m2.007s
user 0m0.004s
sys 0m0.000sIn this Rust version, we use std::thread for concurrency and std::sync::mpsc for channel-based communication between threads. The overall structure and functionality remain similar to the original example, with workers processing jobs concurrently and sending results back to the main thread.
Comments powered by Disqus