Google App Engine Golang datastore.Query.GetAll not working locally

Question

I am new to Google App Engine and meet some problems with datastore.

I wrote a test GaeDatastore_test.go to test datastore.Query.GetAll method see below

package persist

import (
    "fmt"
    "testing"

    "appengine/aetest"
    "appengine/datastore"
)

type Mock struct {
    Name string
}

func TestAll(t *testing.T) {
    ctx, _ := aetest.NewContext(nil)
    defer ctx.Close()
    d := &Mock{
        "hello",
    }
    fmt.Println(datastore.Put(ctx, datastore.NewIncompleteKey(ctx, "test", nil), d))
    fmt.Println(datastore.Put(ctx, datastore.NewIncompleteKey(ctx, "test", nil), d))
    fmt.Println(datastore.Put(ctx, datastore.NewIncompleteKey(ctx, "test", nil), d))
    q := datastore.NewQuery("test")
    var ms []Mock
    q.GetAll(ctx, &ms)
    fmt.Printf("%#v", ms)
}

When I run the test file using

goapp test

it does not return the 3 entities I have stored. I could see key returned after calling datastore.Put and I could use datastore.Get to retrieve them using the key. But the 'ms' is always nil.

Then I tried to initialise ms using

make([]Mock,10)

But it does not make much difference and the data still not coming through.

I checked query.go source file . Internally it uses loadEntity which is the same method used by Get / GetMulti and use append to push elements to the slice. It is quite weird that the ms is nil.

The client is connecting to a Python server simulating GAE env. Could some one help?

Answer 1

I believe it to be because you are doing a cross group query and the write has not been applied to the data store yet. The dev server simulates production-like write visibility. When you are not in a transaction or entity group, queries will not show results right away.

From the docs,

The write operation returns immediately after the Commit phase and the Apply phase then takes place asynchronously.

and,

... Apply is rolled forward to completion when one of the following occurs:

Periodic Datastore sweeps check for uncompleted Commit jobs and apply them. Certain operations (get, put, delete, and ancestor queries) that use the affected entity group cause any changes that have been committed but not yet applied to be completed before proceeding with the new operation.

and,

Because Datastore gets and ancestor queries apply any outstanding modifications before executing, these operations always see a consistent view of all previous successful transactions. This means that a get operation (looking up an updated entity by its key) is guaranteed to see the latest version of that entity.

For the source of these quotes and much more detail on writes and data visibility rules see: https://developers.google.com/appengine/docs/go/datastore/#Go_Datastore_writes_and_data_visibility

Here is a modified version of your code that performs a Get on the returned key after each put. A Get() forces that key to be applied. This ensure that your GetAll query will return all your test entities.

package persist

import (
        "fmt"
        "testing"

        "appengine/aetest"
        "appengine/datastore"
)

type Mock struct {
        Name string
}

func TestAll(t *testing.T) {
        ctx, _ := aetest.NewContext(nil)
        defer ctx.Close()
        d := &Mock{
                "hello",
        }

        for i := 0; i < 3; i++ {
                k, err := datastore.Put(ctx, datastore.NewIncompleteKey(ctx, "test", nil), d)
                fmt.Println(k, err)
                datastore.Get(ctx, k, nil)
        }
        q := datastore.NewQuery("test")
        var ms []Mock
        q.GetAll(ctx, &ms)
        fmt.Printf("%#v", ms)
}