Go by Example

Worker Pools

Fan out work across N goroutines reading from a shared jobs channel to bound concurrency.

A worker pool runs a fixed number of goroutines that take jobs from one channel. This bounds how many jobs run concurrently.

Start two workers, send them four jobs, and close the jobs channel so they can finish after the queue is empty.

package main
 
import (
    "fmt"
    "sync"
)
 
func worker(jobs <-chan int, wg *sync.WaitGroup) {
    defer wg.Done()
    for job := range jobs {
        fmt.Println("processed", job)
    }
}
 
func main() {
    jobs := make(chan int)
    var wg sync.WaitGroup
    for i := 0; i < 2; i++ {
        wg.Add(1)
        go worker(jobs, &wg)
    }
    for job := 1; job <= 4; job++ {
        jobs <- job
    }
    close(jobs)
    wg.Wait()
}

In production

Choose pool size from the constrained resource and expected load. For database work, the useful limit may be the connection pool; for CPU-bound work, start near the number of available processors and measure. A worker pool does not by itself limit queued jobs, so bound the jobs channel or the upstream queue too.

Enjoyed this? Get more software craft delivered to your inbox.