Errgroup

2019-02-27

The errgroup package provides a way to run a group of goroutines. It’s similar to the well known WaitGroup but allows for easy and correct cancellation.

Here’s an example:

package main

import (
	"context"
	"log"
	"math/rand"
	"time"

	"golang.org/x/sync/errgroup"
)

func runParallel(ctx context.Context, parallel int, files []string) error {
	// channel that limits the number of goroutines in flight
	limit := make(chan struct{}, parallel)
	defer func() {
		close(limit)
	}()

	g, ctx := errgroup.WithContext(ctx)
	for _, fname := range files {
		fname := fname
		limit <- struct{}{}
		g.Go(func() error {
			defer func() {
				<-limit
			}()
			log.Printf("Processing %v\n", fname)
			timeout := time.Duration(rand.Int63n(10) + 3)
			select {
			case <-time.Tick(timeout * time.Second):
				log.Printf("Successfully processed %v\n", fname)
			case <-ctx.Done():
				log.Printf("Cancelled %v\n", fname)
				return ctx.Err()
			}
			return nil
		})
	}
	if err := g.Wait(); err != nil {
		return err
	}
	return nil
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	go func() {
		time.Sleep(2 * time.Second)
		log.Printf("Cancelling context")
		cancel()
	}()
	err := runParallel(ctx, 5, []string{"a", "b", "c", "d",
                                        "e", "f", "g", "h", "i", "j"})
	log.Printf("Error: %v", err)
}

This runs 10 goroutines which wait for 3 seconds plus some random number of seconds. After 2 seconds the context is cancelled(). This cancellation is passed on to the goroutines by the Group type. They are all then cancelled and won’t wait time running.

The output of running this is as follows:

2019/02/27 18:44:21 Processing file e
2019/02/27 18:44:21 Processing file b
2019/02/27 18:44:21 Processing file a
2019/02/27 18:44:21 Processing file c
2019/02/27 18:44:21 Processing file d
2019/02/27 18:44:23 Cancelling context
2019/02/27 18:44:23 Cancelled file e
2019/02/27 18:44:23 Processing file f
2019/02/27 18:44:23 Cancelled file c
2019/02/27 18:44:23 Cancelled file f
2019/02/27 18:44:23 Processing file h
2019/02/27 18:44:23 Cancelled file h
2019/02/27 18:44:23 Processing file i
2019/02/27 18:44:23 Cancelled file i
2019/02/27 18:44:23 Cancelled file a
2019/02/27 18:44:23 Processing file g
2019/02/27 18:44:23 Cancelled file g
2019/02/27 18:44:23 Processing file j
2019/02/27 18:44:23 Cancelled file j
2019/02/27 18:44:23 Cancelled file d
2019/02/27 18:44:23 Cancelled file b
2019/02/27 18:44:23 Error: context canceled