Worker Pools in Julia

Our example demonstrates how to implement a worker pool using tasks and channels in Julia.

using Printf

# This is our worker function. We'll run several concurrent instances of it.
# These workers will receive work on the `jobs` channel and send the corresponding
# results on `results`. We'll sleep for a second per job to simulate an expensive task.
function worker(id, jobs, results)
    for j in jobs
        @printf("worker %d started  job %d\n", id, j)
        sleep(1)
        @printf("worker %d finished job %d\n", id, j)
        put!(results, j * 2)
    end
end

function main()
    # To use our pool of workers, we need to send them work and collect their results.
    # We create 2 channels for this.
    num_jobs = 5
    jobs = Channel{Int}(num_jobs)
    results = Channel{Int}(num_jobs)

    # This starts up 3 workers, initially blocked because there are no jobs yet.
    for w in 1:3
        @async worker(w, jobs, results)
    end

    # Here we send 5 jobs and then close that channel to indicate that's all the work we have.
    for j in 1:num_jobs
        put!(jobs, j)
    end
    close(jobs)

    # Finally we collect all the results of the work.
    # This also ensures that the worker tasks have finished.
    for a in 1:num_jobs
        take!(results)
    end
end

main()

Our program shows the 5 jobs being executed by various workers. The program only takes about 2 seconds despite doing about 5 seconds of total work because there are 3 workers operating concurrently.

To run the program, save it as worker_pools.jl and use:

$ julia worker_pools.jl
worker 1 started  job 1
worker 3 started  job 3
worker 2 started  job 2
worker 1 finished job 1
worker 1 started  job 4
worker 3 finished job 3
worker 2 finished job 2
worker 2 started  job 5
worker 1 finished job 4
worker 2 finished job 5

In this Julia version:

We use the @async macro to create concurrent tasks, which are similar to goroutines.
We use Channel to create buffered channels for communication between tasks.
The put! and take! functions are used to send and receive values on channels, respectively.
We use close to close a channel when we’re done sending values.
The @printf macro is used for formatted printing, similar to fmt.Println in the original example.

This implementation maintains the core concepts of the original example, demonstrating concurrent workers processing jobs and returning results using channels for communication.

Tickers in Julia

Waitgroups in Julia

Learn X By Example

Title here

Worker Pools in Julia