Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Reduce access time to Bigtable with Golang

Tags:

go

bigtable

I would like to reduce the access time to my table on my Golang RestApi.

I have a Go restAPI with an enpoint that requires access to a bigtable database for each code specified at the request body. My design of the key access is "provider|client|productCode|date|.." (and other 4 parameters).

For each productCode of the request body I'm making a async call to read the row with a specified key.
On my test I requested 1000 products codes and the obtained times on my local machine are around of 800~900ms. The times on a cloud machine are around 550~450 ms.

I'm using the bigtable package to access to the bigTable and I'm using the following code:

package main

import (
    "log"
    "time"

    "golang.org/x/net/context"
    "google.golang.org/cloud/bigtable"
)

func main() {
    start2 := time.Now()

    listPKs := []string{"PROV|CLI|6030|20160714|8|30301.30302|ES", "PROV|CLI|44103|20160714|8|30301.30302|ES", "PROV|CLI|1454871|20160714|8|30301.30302|ES"}

    providerRS := getBDresponse(listPKs, 50000)
    if providerRS != nil {
        ///do something
    }
    elapsed2 := time.Since(start2)
    log.Printf("Time access BT: %s", elapsed2)
}

func getBDresponse(listPKs []string, timeoutMiliseconds int) []string {

    resp := make(chan string)
    table := myClient.Client.Open(TABLE_NAME)
    //async call to BT
    for _, key := range listPKs {
        go asyncDoUniqueCall(key, resp, myContext, table)
    }
    //get all responses
    providerRS := getResponses(resp, timeoutMiliseconds, len(listPKs))
    return providerRS
}

func asyncDoUniqueCall(pk string, ch chan string, ctx *context.Context, table *bigtable.Table) {
    ch <- GetRowValue(pk, ctx, table)
}

func getResponses(resp chan string, timeoutMiliseconds int, totalNumRQ int) []string {
    var ret []string

    for j := 0; j < totalNumRQ; j++ {
        select {
        case rowResponse := <-resp: //catch the response
            if rowResponse != "" {
                ret = append(ret, rowResponse)
            }
        case <-time.After(time.Duration(timeoutMiliseconds) * time.Millisecond): // timeout control 
            return nil
        }
    }
    return ret
}

//GetRowValue open the table and get the row indexed by pk and returns the stored message 
func GetRowValue(pk string, ctx *context.Context, table *bigtable.Table) string {

    var response string

    r, err := table.ReadRow(*ctx, pk)
    if err != nil {
        return ""
    }
    if len(r) > 0 {
        row := r[COLUMN_FAMILY_NAME]
        numCol := len(row)

        response = string(row[0].Value)
    }
    return response
}

I tried this example too and the times are not better:

Am I using the gorutines and chanels correctyl? Is the way how I'm accessing to the BT correct? Maybe the key design is not correct?

like image 261
Jose Antonio Diaz Gonzalez Avatar asked Jul 22 '16 10:07

Jose Antonio Diaz Gonzalez


1 Answers

I think the problem is that you are using a 1 dimension chan:

resp := make(chan string) 

You should try to enlarge its dimension by doing:

resp := make(chan string,len(listPKs))

That's because if your async function finishes before the info in the chan is read than the chan will be blocking.

A side note: spawning go routines without performing any check could be a possible source of problems.

I'd suggest you to implement a working queue in order to optimise your task.

like image 123
Bestbug Avatar answered Sep 23 '22 06:09

Bestbug