Golang string format using slice values

Question

Here I am trying to create a query string for my API from a slice containing strings.

ie. where={"node_name":"node1","node_name":"node_2"}

import (
   "fmt"
   "strings"
)

func main() {
    nodes := []string{"node1", "node2"}
    var query string
    for _, n := range nodes {
        query += fmt.Sprintf("\"node_name\":\"%s\",", n)
    }
    query = strings.TrimRight(query, ",")
    final := fmt.Sprintf("where={%s}", query)
    fmt.Println(final)
}

Here is goplayground link.

What is the best way to get the result?

Answer 1

Your solution uses way too many allocations due to string concatenations.

We'll create some alternative, faster and/or more elegant solutions. Note that the below solutions do not check if node values contain the quotation mark " character. If they would, those would have to be escaped somehow (else the result would be an invalid query string).

The complete, runnable code can be found on the Go Playground. The complete testing / benchmarking code can also be found on the Go Playground, but it is not runnable, save both to your Go workspace (e.g. $GOPATH/src/query/query.go and $GOPATH/src/query/query_test.go) and run it with go test -bench ..

Also be sure to check out this related question: How to efficiently concatenate strings in Go?

Alternatives

Genesis

Your logic can be captured by the following function:

func buildOriginal(nodes []string) string {
    var query string
    for _, n := range nodes {
        query += fmt.Sprintf("\"node_name\":\"%s\",", n)
    }
    query = strings.TrimRight(query, ",")
    return fmt.Sprintf("where={%s}", query)
}

Using bytes.Buffer

Much better would be to use a single buffer, e.g. bytes.Buffer, build the query in that, and convert it to string at the end:

func buildBuffer(nodes []string) string {
    buf := &bytes.Buffer{}
    buf.WriteString("where={")
    for i, v := range nodes {
        if i > 0 {
            buf.WriteByte(',')
        }
        buf.WriteString(`"node_name":"`)
        buf.WriteString(v)
        buf.WriteByte('"')
    }
    buf.WriteByte('}')
    return buf.String()
}

Using it:

nodes := []string{"node1", "node2"}
fmt.Println(buildBuffer(nodes))

Output:

where={"node_name":"node1","node_name":"node2"}

bytes.Buffer improved

bytes.Buffer will still do some reallocations, although much less than your original solution.

However, we can still reduce the allocations to 1, if we pass a big-enough byte slice when creating the bytes.Buffer using bytes.NewBuffer(). We can calculate the required size prior:

func buildBuffer2(nodes []string) string {
    size := 8 + len(nodes)*15
    for _, v := range nodes {
        size += len(v)
    }
    buf := bytes.NewBuffer(make([]byte, 0, size))
    buf.WriteString("where={")
    for i, v := range nodes {
        if i > 0 {
            buf.WriteByte(',')
        }
        buf.WriteString(`"node_name":"`)
        buf.WriteString(v)
        buf.WriteByte('"')
    }
    buf.WriteByte('}')
    return buf.String()
}

Note that in size calculation 8 is the size of the string where={} and 15 is the size of the string "node_name":"",.

Using text/template

We can also create a text template, and use the text/template package to execute it, efficiently generating the result:

var t = template.Must(template.New("").Parse(templ))

func buildTemplate(nodes []string) string {
    size := 8 + len(nodes)*15
    for _, v := range nodes {
        size += len(v)
    }
    buf := bytes.NewBuffer(make([]byte, 0, size))
    if err := t.Execute(buf, nodes); err != nil {
        log.Fatal(err) // Handle error
    }
    return buf.String()
}

const templ = `where={
{{- range $idx, $n := . -}}
    {{if ne $idx 0}},{{end}}"node_name":"{{$n}}"
{{- end -}}
}`

Using strings.Join()

This solution is interesting due to its simplicity. We can use strings.Join() to join the nodes with the static text ","node_name":" in between, proper prefix and postfix applied.

An important thing to note: strings.Join() uses the builtin copy() function with a single preallocated []byte buffer, so it's very fast! "As a special case, it (the copy() function) also will copy bytes from a string to a slice of bytes."

func buildJoin(nodes []string) string {
    if len(nodes) == 0 {
        return "where={}"
    }
    return `where={"node_name":"` + strings.Join(nodes, `","node_name":"`) + `"}`
}

Benchmark results

We'll benchmark with the following nodes value:

var nodes = []string{"n1", "node2", "nodethree", "fourthNode",
    "n1", "node2", "nodethree", "fourthNode",
    "n1", "node2", "nodethree", "fourthNode",
    "n1", "node2", "nodethree", "fourthNode",
    "n1", "node2", "nodethree", "fourthNode",
}

And the benchmarking code looks like this:

func BenchmarkOriginal(b *testing.B) {
    for i := 0; i < b.N; i++ {
        buildOriginal(nodes)
    }
}

func BenchmarkBuffer(b *testing.B) {
    for i := 0; i < b.N; i++ {
        buildBuffer(nodes)
    }
}

// ... All the other benchmarking functions look the same

And now the results:

BenchmarkOriginal-4               200000             10572 ns/op
BenchmarkBuffer-4                 500000              2914 ns/op
BenchmarkBuffer2-4               1000000              2024 ns/op
BenchmarkBufferTemplate-4          30000             77634 ns/op
BenchmarkJoin-4                  2000000               830 ns/op

Some unsurprising facts: buildBuffer() is 3.6 times faster than buildOriginal(), and buildBuffer2() (with pre-calculated size) is about 30% faster than buildBuffer() because it does not need to reallocate (and copy over) the internal buffer.

Some surprising facts: buildJoin() is extremely fast, even beats buildBuffer2() by 2.4 times (due to only using a []byte and copy()). buildTemplate() on the other hand proved quite slow: 7 times slower than buildOriginal(). The main reason for this is because it uses (has to use) reflection under the hood.