The Problem with Unbounded Goroutines

When processing a batch of independent tasks in Go—such as calling an external API or transforming records—running them sequentially in a for loop is too slow.

On the other hand, spawning a goroutine per item (go process(item)) without a limit quickly spikes memory and exhausts file descriptors or database connections. We need a fixed-size worker pool (Fan-Out) that feeds results into a single channel (Fan-In).

Avoiding the Channel Closure Deadlock

The most common gotcha when combining worker pools with a results channel is deciding when to close results:

  • Close results too early and active workers panic with send on closed channel.
  • Never close results and the consumer for result := range results deadlocks once all jobs finish.

The clean fix is to track active workers with a sync.WaitGroup and close results inside a dedicated goroutine right after wg.Wait() unblocks.

Implementation

package main

import (
	"fmt"
	"sync"
	"time"
)

func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
	defer wg.Done()
	for j := range jobs {
		fmt.Printf("Worker %d processing job %d\n", id, j)
		time.Sleep(100 * time.Millisecond)
		results <- j * 2
	}
}

func main() {
	const numJobs = 10
	const numWorkers = 3

	jobs := make(chan int, numJobs)
	results := make(chan int, numJobs)

	var wg sync.WaitGroup

	// 1. Fan-out: start a fixed number of workers
	for w := 1; w <= numWorkers; w++ {
		wg.Add(1)
		go worker(w, jobs, results, &wg)
	}

	// 2. Enqueue jobs and close the jobs channel
	for j := 1; j <= numJobs; j++ {
		jobs <- j
	}
	close(jobs)

	// 3. Close results only after all workers have exited
	go func() {
		wg.Wait()
		close(results)
	}()

	// 4. Fan-in: drain results until the channel is closed
	for result := range results {
		fmt.Printf("Result: %d\n", result)
	}
}

Closing jobs tells each worker’s for j := range jobs loop to exit once the queue is empty. Once every worker calls wg.Done(), the background goroutine closes results, allowing the main loop to finish cleanly.