Worker Pools in Lua

Our example demonstrates how to implement a worker pool using Lua’s coroutines and channels.

local lanes = require("lanes").configure()

-- Here's the worker function, of which we'll run several
-- concurrent instances. These workers will receive
-- work on the `jobs` channel and send the corresponding
-- results on `results`. We'll sleep a second per job to
-- simulate an expensive task.
local function worker(id, jobs, results)
    while true do
        local job = jobs:receive()
        if job == nil then
            break
        end
        print("worker " .. id .. " started  job " .. job)
        lanes.sleep(1)
        print("worker " .. id .. " finished job " .. job)
        results:push(job * 2)
    end
end

local function main()
    -- In order to use our pool of workers we need to send
    -- them work and collect their results. We make 2
    -- channels for this.
    local numJobs = 5
    local jobs = lanes.linda()
    local results = lanes.linda()

    -- This starts up 3 workers, initially blocked
    -- because there are no jobs yet.
    local workers = {}
    for w = 1, 3 do
        workers[w] = lanes.gen("*", worker)(w, jobs, results)
    end

    -- Here we send 5 `jobs` and then `close` that
    -- channel to indicate that's all the work we have.
    for j = 1, numJobs do
        jobs:send(nil, j)
    end
    for _ = 1, 3 do
        jobs:send(nil, nil)
    end

    -- Finally we collect all the results of the work.
    -- This also ensures that the worker coroutines have
    -- finished.
    for _ = 1, numJobs do
        local _, result = results:receive()
        print("Received result:", result)
    end

    -- Wait for all workers to finish
    for _, worker in ipairs(workers) do
        worker:join()
    end
end

main()

This Lua implementation uses the lanes library to provide concurrency features similar to Go’s goroutines and channels. Here’s a breakdown of the changes:

We use lanes.linda() to create channel-like structures for job distribution and result collection.
Instead of goroutines, we use Lua coroutines created with lanes.gen().
The worker function is modified to use jobs:receive() for receiving jobs and results:push() for sending results.
In the main function, we create worker coroutines using lanes.gen().
Job distribution is done using jobs:send(), and we send nil to signal the end of jobs.
Result collection is done using results:receive().
We use worker:join() to wait for all workers to finish, which is similar to using a WaitGroup in Go.

When you run this program, it will show the 5 jobs being executed by various workers. The program should take about 2 seconds to complete, despite doing about 5 seconds of total work, because there are 3 workers operating concurrently.

Note that you’ll need to have the lanes library installed to run this code. You can install it using LuaRocks:

luarocks install lanes

This example demonstrates how to implement concurrent worker pools in Lua, showcasing parallel execution of tasks and inter-coroutine communication.

Tickers in Lua

Waitgroups in Lua

Learn X By Example

Title here

Worker Pools in Lua