App engine push task always returns 404 in test

Question

I've got a push task queue in a Go App Engine application. When we try to enqueue tasks in testing for whatever reason the tasks always return 404.

Our app.yaml:

runtime: go
api_version: go1.9

handlers:
 - url: /worker/.*
   script: _go_app
   login: admin
 - url: /.*
   script: _go_app

The actual task invocation:

func Handler(w http.ResponseWriter, r *http.Request) {
    ctx := appengine.NewContext(r)
    t := taskqueue.NewPOSTTask("/worker", map[string][]string{"key": {"val"}})
    _, err := taskqueue.Add(ctx, t, "")
    if err != nil {
        log.Errorf(ctx, "Failed to add task");
    }
    fmt.Fprintf(w, "Success");
}

A still-incomplete handler, but it exists!

func Worker(w http.ResponseWriter, r *http.Request) {
    ctx := appengine.NewContext(r)
    log.Infof(ctx, "Worker succeeded")
}

and finally, proof that we actually added the path to our router:

func init() {
    http.HandleFunc("/", Handler)
    http.HandleFunc("/worker", Worker)
}

When we actually run tests, we always get the following logging output:

INFO     2018-05-03 09:51:11,794 module.py:846] default: "POST /worker HTTP/1.1" 404 19
WARNING  2018-05-03 09:51:11,794 taskqueue_stub.py:2149] Task � failed to execute. This task will retry in 0.100 seconds
INFO     2018-05-03 09:51:11,897 module.py:846] default: "POST /worker HTTP/1.1" 404 19
WARNING  2018-05-03 09:51:11,897 taskqueue_stub.py:2149] Task � failed to execute. This task will retry in 0.200 seconds
INFO     2018-05-03 09:51:12,101 module.py:846] default: "POST /worker HTTP/1.1" 404 19
WARNING  2018-05-03 09:51:12,101 taskqueue_stub.py:2149] Task � failed to execute. This task will retry in 0.400 seconds

Note that the /worker endpoint returns 302 when I try to ping it via an API client like Paw, so the route seems to have been configured correctly. The 404 only arises when I try to run things in a test.

Why is this returning 404? I've tried running tests around the example push queue in their documentation have run into the same issue there - is there some sort of missing configuration flag I'm failing to pass to goapp?

I've pushed up a GitHub repo with a minimal replicable example here

Answer 1

The order for running goapp is top-bottom but you need to be specific in your app.yaml. In your case this will works:

package main

import (
    "fmt"
    "net/http"

    "google.golang.org/appengine"
    "google.golang.org/appengine/log"
    "google.golang.org/appengine/taskqueue"
)

func Handler(w http.ResponseWriter, r *http.Request) {
    ctx := appengine.NewContext(r)
    t := taskqueue.NewPOSTTask("/worker", map[string][]string{"key": {"val"}})
    _, err := taskqueue.Add(ctx, t, "")
    if err != nil {
        log.Errorf(ctx, "Failed to add task")
    }
    fmt.Fprintf(w, "Success")
}

func Worker(w http.ResponseWriter, r *http.Request) {
    ctx := appengine.NewContext(r)
    log.Infof(ctx, "Worker succeeded")
}

func init() {
    http.HandleFunc("/", Handler)
    http.HandleFunc("/worker", Worker)
}

For this you need to map url like:

runtime: go
api_version: go1.9

handlers:
 - url: /worker
   script: _go_app
   login: admin
 - url: /.*
   script: _go_app

The result is:

See that worker is running two times. This is ocurring because GET /favicon.ico is entering in GET /.* mapping. So this is only details for you!

UPDATE (05/14/2018): In your test you use aetest.NewInstance(), that runs dev_appserver.py in ioutil.TempDir("", "appengine-aetest"), that writes your own main.go and app.yaml. See above in instance_vm.go:

i.appDir, err = ioutil.TempDir("", "appengine-aetest")
if err != nil {
    return err
}
defer func() {
    if err != nil {
        os.RemoveAll(i.appDir)
    }
}()
err = os.Mkdir(filepath.Join(i.appDir, "app"), 0755)
if err != nil {
    return err
}
err = ioutil.WriteFile(filepath.Join(i.appDir, "app", "app.yaml"), []byte(i.appYAML()), 0644)
  if err != nil {
    return err
}
err = ioutil.WriteFile(filepath.Join(i.appDir, "app", "stubapp.go"), []byte(appSource), 0644)
if err != nil {
    return err
}

//... others codes

const appYAMLTemplate = `
application: %s
version: 1
runtime: go
api_version: go1
vm: true
handlers:
- url: /.*
  script: _go_app
`

const appSource = `
package main
import "google.golang.org/appengine"
func main() { appengine.Main() }
`

So you need to create your own server-instance. This is a way:

//out buffer
var out bytes.Buffer
//start server
c := exec.Command("goapp", "serve") //default port=8080, adminPort=8000
c.Stdout = &out
c.Stderr = &out
c.Start()
defer c.Process.Kill()
//delay to wait server is completed
time.Sleep(10 * time.Second)

//... others codes

//quit server
quitReq, err := http.NewRequest("GET", "http://localhost:8000/quit", nil)
_, err := client.Do(quitReq)
if err != nil {
    fmt.Errorf("GET /quit handler error: %v", err)
}

To test your Handler function do:

//create request (testing Handler func)
req, err := http.NewRequest("GET", "http://localhost:8080/", nil)
if err != nil {
    t.Fatal(err.Error())
}
//do GET
client := http.DefaultClient
resp, err := client.Do(req)
if err != nil {
    t.Error(err)
}
defer resp.Body.Close()
//delay to wait for the worker to execute
time.Sleep(10 * time.Second)

To retrieve the result and test it:

//read response
b, _ := ioutil.ReadAll(resp.Body)
resp_content := string(b)
//checking
if !strings.Contains(resp_content, "Handler Success") {
    t.Errorf("Handler not working")
}
//log server content
logserver := out.String()
if !strings.Contains(logserver, "Worker succeeded") {
    t.Errorf("Worker not working")
}
//log response
t.Logf(logserver)

The result is:

UPDATE: Link for Github: https://github.com/ag-studies/go-appengine-sample

Hope this help!!

Answer 2

Your handlers pattern is /worker/.* but you are issuing tasks to /worker.

You should do 1 of the following:

1. Either change pattern to /worker.* or just /worker
2. Or issue request(s) to /worker/ or /worker/some-task-name.