golang ReverseProxy not working

Question

I don't know why this reverse proxy is not working. I've seen several examples and I can't find anything wrong with it.

package main

import (
    "log"
    "net/url"
    "net/http"
    "net/http/httputil"
)

func report(r *http.Request){
  log.Print("URL: " + r.URL.Path)
  log.Print("Scheme: " + r.URL.Scheme)
  log.Print("Host: " + r.URL.Host)
  //r.URL.Scheme = "http"
  //r.URL.Host = "stackoverflow.com"

  //r.Header.Set("Host", "stackoverflow.com")
  //log.Print("Header Host: " + r.Header.Get("Host"))
}

func main() {
  proxy := httputil.NewSingleHostReverseProxy( &url.URL{Scheme:"http",Host:"myrealserver.com"})
    proxy.Director = report
    // http.Handle("/", proxy)
    error := http.ListenAndServe("mylocalhost.com:8080", proxy)
    if(error != nil) {
        log.Fatal(error)
    }
}

It logs:

2014/04/18 21:32:50 URL: /arg/es
2014/04/18 21:32:50 Scheme:
2014/04/18 21:32:50 Host:
2014/04/18 21:32:50 http: proxy error: unsupported protocol scheme ""

2014/04/18 21:32:51 URL: /favicon.ico
2014/04/18 21:32:51 Scheme:
2014/04/18 21:32:51 Host:
2014/04/18 21:32:51 http: proxy error: unsupported protocol scheme ""

If I uncomment the line that redefines the Schema the error message becomes:

2014/04/18 21:38:05 http: proxy error: http: no Host in request URL

If I uncomment the line that redefines the host also, then the target server becomes stackoverflow.com (I mean, it never uses "myrealserver.com").

If I ask for mylocalhost.com:8080/somepath (or even /) then I get a 404 from Stackoverflow, no matter if stackoverflow.com/somepath exists or not. It says:

Couldn't find mylocalhost.com:8080
The Q&A site mylocalhost.com:8080 doesn't seem to exist... yet

It does not translate the Host header automatically.

If then I uncomment the line that sets (and the other one that prints) the Header "Host". Then I can read "stackoverflow.com" in the log, but I still get the same 404 page reporting that I am trying to access "mylocalhost.com".

I'm using go1.2.1 linux/amd64

How is it that I am supposed to make the program work as a proxy?

Answer 1

Thanks to Alex from Golang-nuts, I have the answer now.

This is what Alex said:

Just need to set http.Request.Host [and scheme] in the Director to get this working: http://play.golang.org/p/I17ZSM6LQb

If you read the source for SingleHostReverseProxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go#L61), it sets its own Director which you are overriding. So you need to reimplement what it already does plus the extra Host change.

Anyway, that didn't solve de Header part of the problem: the target server was still receiving "localhost:8080" as the HTTP Host name, so I did it without the ReverseProxy package, just with http and a RoundTripper, plus a helper function that copies all the headers:

package main

import (
    "flag"
    "fmt"
    "os"
    "log"
    "net/http"
    "io/ioutil"
)

var target *string

func main() {
  target = flag.String("target", "http://stackoverflow.com", "target URL for reverse proxy")
  flag.Parse()
  http.HandleFunc("/", report)
  log.Fatal(http.ListenAndServe("127.0.0.1:8080", nil))
}

func report(w http.ResponseWriter, r *http.Request){

  uri := *target+r.RequestURI

  fmt.Println(r.Method + ": " + uri)

  if r.Method == "POST" {
    body, err := ioutil.ReadAll(r.Body)
    fatal(err)
    fmt.Printf("Body: %v\n", string(body));
  }

  rr, err := http.NewRequest(r.Method, uri, r.Body)
  fatal(err)
  copyHeader(r.Header, &rr.Header)

  // Create a client and query the target
  var transport http.Transport
  resp, err := transport.RoundTrip(rr)
  fatal(err)

  fmt.Printf("Resp-Headers: %v\n", resp.Header);

  defer resp.Body.Close()
  body, err := ioutil.ReadAll(resp.Body)
  fatal(err)

  dH := w.Header()
  copyHeader(resp.Header, &dH)
  dH.Add("Requested-Host", rr.Host)

  w.Write(body)
}

func fatal(err error) {
  if err != nil {
    log.Fatal(err)
    os.Exit(1)
  }
}

func copyHeader(source http.Header, dest *http.Header){
  for n, v := range source {
      for _, vv := range v {
          dest.Add(n, vv)
      }
  }
}

Now I'm able to see StackOverflow or any other site how it's supposed to be.
I'm still working on POST calls, though, so this is a work in progress.

Answer 2

A little late to the party, but ReverseProxy isn't broken, it's just a little confusing because it doesn't work how you'd expect (at the least, I expected it to work the way you did, so that makes two of us).

From the docs:

// For server requests Host specifies the host on which the
// URL is sought. Per RFC 2616, this is either the value of
// the "Host" header or the host name given in the URL itself.
// It may be of the form "host:port". For international domain
// names, Host may be in Punycode or Unicode form. Use
// golang.org/x/net/idna to convert it to either format if
// needed.
//
// For client requests Host optionally overrides the Host
// header to send. If empty, the Request.Write method uses
// the value of URL.Host. Host may contain an international
// domain name.
Host string

Since under the hood ReverseProxy is using this Request to make a client request (after ReverseProxy.Director optionally modifies it), if the Host is set it will override the Host header. This will always be set, because as the first part of the doc comment states "For server requests, Host specifies the host on which the URL is sought".

So in addition to Sebastián's answer, you also need to set req.Host. For example, to proxy to example.com:

proxy := ReverseProxy{
    Director: func(req *http.Request) {
        req.URL.Scheme = "http"
        req.URL.Host = "example.com"
        req.Host = "example.com"
    }
}

Alternatively you can set req.Host to "" and it will use the value of req.URL.Host.

I figured this out by reading: https://github.com/golang/go/issues/14413