A few days ago, we started seeing some weird panics in our production environment. The strange thing was that it was only happening on our message consumers, and the logs were not very clear about what was going on. After some investigation, we found the root cause: we were not recovering from panics inside our goroutines. We already had a panic recover on the main goroutine, but we were not aware that each goroutine needs their own panic recover.

The problem

In our application, we have a process that reads messages from a queue and spawns a new goroutine for each message. This is a common pattern to process messages concurrently. The problem is that if one of the messages causes a panic, the entire application would crash. We were missing a recover statement in our goroutine, so the panic was not being handled.

Here is a simplified version of our code:

package main

import (
	"fmt"
	"time"
)

func main() {
	messages := []string{"message 1", "message 2", "message 3 that will panic", "message 4"}

	for _, msg := range messages {
		go func(msg string) {
			processMessage(msg)
		}(msg)
	}

	// wait for all goroutines to finish
	time.Sleep(1 * time.Second)
}

func processMessage(msg string) {
	fmt.Println("Processing message:", msg)
	if msg == "message 3 that will panic" {
		panic("something went wrong")
	}
}

If you run this code, you will see that the application panics and crashes when it tries to process the third message. The other messages will not be processed.

The solution

The solution is to add a defer function to our goroutine that will recover from the panic. This way, we can log the error and the application will not crash.

Here is the updated code:

package main

import (
	"fmt"
	"time"
)

func main() {
	messages := []string{"message 1", "message 2", "message 3 that will panic", "message 4"}

	for _, msg := range messages {
		go func(msg string) {
			defer func() {
				if r := recover(); r != nil {
					fmt.Println("Recovered from panic:", r)
				}
			}()
			processMessage(msg)
		}(msg)
	}

	// wait for all goroutines to finish
	time.Sleep(1 * time.Second)
}

func processMessage(msg string) {
	fmt.Println("Processing message:", msg)
	if msg == "message 3 that will panic" {
		panic("something went wrong")
	}
}

Now, if you run this code, you will see that the application does not crash. It will process all the messages, and when it encounters the panic, it will recover from it and log the error.

Conclusion

It is very important to always have a recover statement when you are using goroutines. This will prevent your application from crashing and will allow you to handle errors gracefully. It is a good practice to wrap the logic of your goroutine in a function that has a defer with a recover.