Worker Pools in Lisp

(defpackage :worker-pools
  (:use :cl)
  (:import-from :bordeaux-threads
                :make-thread
                :make-condition-variable
                :condition-wait
                :condition-notify)
  (:import-from :cl-chan
                :make-channel
                :send
                :recv
                :close-channel))

(in-package :worker-pools)

(defun worker (id jobs results)
  "Worker function that processes jobs and sends results"
  (loop for j = (recv jobs)
        while j do
          (format t "worker ~A started  job ~A~%" id j)
          (sleep 1) ; Simulate an expensive task
          (format t "worker ~A finished job ~A~%" id j)
          (send results (* j 2))))

(defun main ()
  (let* ((num-jobs 5)
         (jobs (make-channel :buffer-size num-jobs))
         (results (make-channel :buffer-size num-jobs)))
    
    ; Start 3 worker threads
    (dotimes (w 3)
      (make-thread (lambda () (worker (1+ w) jobs results))))
    
    ; Send jobs and close the jobs channel when done
    (dotimes (j num-jobs)
      (send jobs (1+ j)))
    (close-channel jobs)
    
    ; Collect the results
    (dotimes (a num-jobs)
      (recv results))))

(main)

In this example, we’ll look at how to implement a worker pool using threads and channels in Common Lisp.

Here’s the worker function, of which we’ll run several concurrent instances. These workers will receive work on the jobs channel and send the corresponding results on results. We’ll sleep a second per job to simulate an expensive task.

We use the bordeaux-threads library for thread management and cl-chan for channel operations, which provide similar functionality to Go’s goroutines and channels.

In the main function:

1. We create two channels: jobs and results, both with a buffer size equal to the number of jobs.

2. We start 3 worker threads using make-thread. Each worker runs independently and processes jobs from the jobs channel.

3. We send 5 jobs to the jobs channel and then close it to indicate that’s all the work we have.

4. Finally, we collect all the results of the work from the results channel. This also ensures that the worker threads have finished.

When you run this program, it will show the 5 jobs being executed by various workers. The program should take about 2 seconds to complete despite doing about 5 seconds of total work because there are 3 workers operating concurrently.

Note that Common Lisp doesn’t have built-in constructs exactly equivalent to Go’s goroutines and channels. We’re using threads and a channel library to achieve similar functionality. The exact behavior and performance characteristics might differ from the Go version.