Improve append allocations (in loops)

Avatar of the author Willem Schots
28 Jul, 2023
~2 min.
RSS

Is append causing more allocs/op then you expected in your benchmarks?

There is an optimization you can use when you know the (approximate) number of elements that you’re going to append to a slice: Create the initial slice with enough capacity.

You can do this using make with 3 parameters, the last parameter will be the capacity of the slice.

s := make([]string, 3, 12)

Will create a slice of strings called s with length of 3 and a capacity of 12.

So why can this improve performance when calling append?

Every time append is called, it checks if there is enough capacity in the backing array of the original slice.

If the array of the original slice does not have enough capacity, append will allocate a new backing array and copy the relevant elements.

However, when the original slice has enough capacity, append will use its existing backing array. No new backing array is allocated.

A common situation where this can make a big difference is when append is called in a loop. This can be seen in the following snippet, which prints out a message every time a new backing array is allocated.

Run the snippet with both versions of s to see the difference.

main.go
package main

import (
	"fmt"
)

func main() {
	// Use the following line to create s with enough capacity.
	//s := make([]int, 0, 100)
	s := make([]int, 0)

	prevCap := cap(s)
	count := 0
	for i := 0; i < 100; i++ {
		s = append(s, i)

		// Check if a new backing array was allocated by checking
		// if the capacity changed.
		c := cap(s)
		if prevCap != c {
			fmt.Printf("%d: new array with cap %d\n", i, c)
			prevCap = c
			count++
		}
	}

	fmt.Printf("%d new arrays allocated", count)
}

Make privacy claims you can back up

Join 800+ subscribers.
One email every few weeks.

I share what I learn building privacy-friendly systems. From confidential computing to Verbond and the occasional Go tip or trick. You can unsubscribe at any time.

A photo of Willem

Written by Willem Schots

I help teams prove what happens to their users' data, and I'm building Verbond, which shows where data can go before third-party code runs.

This site started as a place to learn Go. Most posts are now about privacy-friendly and verifiable systems, with the occasional Go post in between.

Want to talk about your system? Book a 30-minute call, or follow me on Bluesky and LinkedIn.