I would like to reduce the access time to my table on my Golang RestApi.
I have a Go restAPI with an enpoint that requires access to a bigtable database for each code specified at the request body. My design of the key access is "provider|client|productCode|date|.." (and other 4 parameters).
For each productCode of the request body I'm making a async call to read the row with a specified key.
On my test I requested 1000 products codes and the obtained times on my local machine are around of 800~900ms. The times on a cloud machine are around 550~450 ms.
I'm using the bigtable package to access to the bigTable and I'm using the following code:
package main
import (
"log"
"time"
"golang.org/x/net/context"
"google.golang.org/cloud/bigtable"
)
func main() {
start2 := time.Now()
listPKs := []string{"PROV|CLI|6030|20160714|8|30301.30302|ES", "PROV|CLI|44103|20160714|8|30301.30302|ES", "PROV|CLI|1454871|20160714|8|30301.30302|ES"}
providerRS := getBDresponse(listPKs, 50000)
if providerRS != nil {
///do something
}
elapsed2 := time.Since(start2)
log.Printf("Time access BT: %s", elapsed2)
}
func getBDresponse(listPKs []string, timeoutMiliseconds int) []string {
resp := make(chan string)
table := myClient.Client.Open(TABLE_NAME)
//async call to BT
for _, key := range listPKs {
go asyncDoUniqueCall(key, resp, myContext, table)
}
//get all responses
providerRS := getResponses(resp, timeoutMiliseconds, len(listPKs))
return providerRS
}
func asyncDoUniqueCall(pk string, ch chan string, ctx *context.Context, table *bigtable.Table) {
ch <- GetRowValue(pk, ctx, table)
}
func getResponses(resp chan string, timeoutMiliseconds int, totalNumRQ int) []string {
var ret []string
for j := 0; j < totalNumRQ; j++ {
select {
case rowResponse := <-resp: //catch the response
if rowResponse != "" {
ret = append(ret, rowResponse)
}
case <-time.After(time.Duration(timeoutMiliseconds) * time.Millisecond): // timeout control
return nil
}
}
return ret
}
//GetRowValue open the table and get the row indexed by pk and returns the stored message
func GetRowValue(pk string, ctx *context.Context, table *bigtable.Table) string {
var response string
r, err := table.ReadRow(*ctx, pk)
if err != nil {
return ""
}
if len(r) > 0 {
row := r[COLUMN_FAMILY_NAME]
numCol := len(row)
response = string(row[0].Value)
}
return response
}
I tried this example too and the times are not better:
Am I using the gorutines and chanels correctyl? Is the way how I'm accessing to the BT correct? Maybe the key design is not correct?
I think the problem is that you are using a 1 dimension chan
:
resp := make(chan string)
You should try to enlarge its dimension by doing:
resp := make(chan string,len(listPKs))
That's because if your async function finishes before the info in the chan
is read than the chan
will be blocking.
A side note: spawning go routines without performing any check could be a possible source of problems.
I'd suggest you to implement a working queue in order to optimise your task.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With