Worker Pools

Fan out work across N goroutines reading from a shared jobs channel to bound concurrency.

A worker pool bounds concurrency by fanning N worker goroutines out over a shared jobs channel. Workers pull jobs until the channel closes, results flow back on a results channel, and a sync.WaitGroup ensures clean shutdown.

Create a jobs channel and a results channel. Launch 3 workers each reading from jobs. Send 5 jobs, close the channel so workers know when to stop, then collect all results.

package main
 
import (
    "fmt"
    "sync"
    "time"
)
 
func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
    defer wg.Done()
    for j := range jobs {
        fmt.Printf("worker %d started job %d\n", id, j)
        time.Sleep(time.Second) // simulate work
        fmt.Printf("worker %d finished job %d\n", id, j)
        results <- j * 2
    }
}
 
func main() {
    const numWorkers = 3
    const numJobs = 5
 
    jobs := make(chan int, numJobs)
    results := make(chan int, numJobs)
 
    var wg sync.WaitGroup
    for w := 1; w <= numWorkers; w++ {
        wg.Add(1)
        go worker(w, jobs, results, &wg)
    }
 
    for j := 1; j <= numJobs; j++ {
        jobs <- j
    }
    close(jobs) // signal workers to stop after draining
 
    // close results once all workers are done
    go func() {
        wg.Wait()
        close(results)
    }()
 
    for r := range results {
        fmt.Println("result:", r)
    }
}

For pure fan-out with no results collection, skip the results channel and close when the WaitGroup finishes:

var wg sync.WaitGroup
for w := 0; w < numWorkers; w++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for job := range jobs {
            process(job)
        }
    }()
}
wg.Wait()

In production

The worker pool pattern bounds concurrency for CPU-bound or rate-limited work. Size the pool to the bottleneck: CPU-bound work should use runtime.GOMAXPROCS(0) workers; I/O-bound work (HTTP, DB calls) should be tuned empirically with load testing. A pool of 100 goroutines hitting a database with max_connections = 20 will queue at the DB level anyway - size the pool to match external resource limits, not to "be safe with more threads."

← PreviousTickers Next →WaitGroups

Enjoyed this? Get more essays on software craft delivered to your inbox.

Subscribe free