Golang Http Get Request very slow

Question

I have a very strange problem with a simple HTTP Get Request in Golang.

Every request in Golang to https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook needs about 6-8 seconds (!)

If same request fired in Chrome, with Postman or with Powershell it needs less than a second.

Does somebody has a clue why this happens?

My Code:

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    client := &http.Client{}

    req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)

    response, err := client.Do(req)
    if err != nil && response == nil {
        log.Fatalf("Error on request. %v", err)
    }
    defer response.Body.Close()

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        log.Fatalf("Couldn't get response body. %v", err)
    }

    fmt.Print(string(body))
}

Answer 1

The site you are trying to access is behind the Akamai CDN:

$ dig www.alltron.ch 
...
www.alltron.ch.         152     IN      CNAME   competec.botmanager.edgekey.net.
competec.botmanager.edgekey.net. 7052 IN CNAME  e9179.f.akamaiedge.net.
e9179.f.akamaiedge.net. 162     IN      A       2.20.176.40

Akamai offers its customers a detection of web clients which are not browsers so that the customers can keep bots away or slowing bots down.

As can be seen from Strange CURL issue with a particular website SSL certificate and Scraping attempts getting 403 error this kind of detection mainly cares about having a Accept-Language header, having a Connection header with the value Keep-Alive and having a User-Agent which matches Mozilla/....

This means the following code changes result in an immediate response:

req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)
req.Header.Set("Connection","Keep-Alive")
req.Header.Set("Accept-Language","en-US")
req.Header.Set("User-Agent","Mozilla/5.0")

Still, the site obviously does not like bots and you should adhere to these wishes and not stress the site too much (like doing lots of information scraping). And, the bot detection done by Akamai might change without notice, i.e. even if this code fixes the problem now it might no longer work in the future. Such changes will be especially true if many clients bypass the bot detection.