Breakpoints in vscode (Win 10) "unverified" and not hit when remote debugging Go app in Linux Docker container (Hyper-V)

Question

I am developing a Go (1.12.0) api-server using vscode (1.34.0 insider and stable builds) for Windows 10. The source code is located on the Windows machine in the defined %GOPATH%. Delve (dlv.exe - version 1.2.0) is also installed in %GOPATH%\bin and %GOPATH%\bin is also in the Windows %PATH%.

The go app is then built in a Docker (Docker Desktop Version 2.0.0.3 (31259)) container with docker-compose (as some other services such as Database and web-server are running in other containers). The final go app binary is then copied to an Alpine-Linux container together with the delve executable and the delve server is started in headless mode. No source code is copied over to the Alpine-Linux container, just the binaries.

I have not managed to get remote debugging set up properly in vscode with this set-up. The debugger does start my app but any breakpoints immediately turn grey and become "unverified". They are also not hit when the app (api server) runs.

Debugging with this set-up works perfectly (debugger starts, breakpoints can be set and are hit) when using the Goland IDE from Jetbrains for remote debugging.

I have tried to find a solution over the last couple of days, finding forum posts with similar problems for Chrome debugging, Node.js debugging etc. and this post on Go, specifically:

Remote debugging - Unverified breakpoint

I also found this sample config:

https://github.com/lukehoban/webapp-go/blob/debugging/.vscode/launch.json

I think the main problem I had is that nowhere could I find any examples of how to properly set the path of the program in launch.json for the remote debug config on a Windows machine (nor could I find any documentation referring to this). The source code is only on the Windows machine in GOPATH, not in the final container the app and delve run in (and this, again, works fine with Goland debugger)

My project path/directory structure (simplified):

%GOPATH%\github.com\myuser\project_dir\
   .vscode\
      launch.json
   cmd\
      my_api\
          main.go
      another_app\
          main.go
   package1\
      package1.go (this is where I am setting the breakpoint, this package is imported in cmd\my_api\main.go)
   Dockerfile
   ... (.gitignore, GoPkg etc.)

My current launch.json config (see some variants I tried below)

launch.json - remote config

{
         "name": "RemoteDockerAPI",
         "type": "go",
         "request": "launch",
         "mode": "remote",
         "program": "${workspaceFolder}/cmd/my_api",
         "env": {},
         "args": [],
         "remotePath": "/my_api",
         "port": 40400, // Port 
         "host": "127.0.0.1", // Docker IP
         "showLog": true
}

Notes: project_folder/cmd/my_api is where the main.go for the api server is located. However, some of the packages for this app are located under the project folder directly, i.e. project_folder/package1/package1.go

I have tried with just

"program": "${workspaceFolder}",

and

"program": "${workspaceFolder}\\cmd\\my_api",

and

"program": "${workspaceFolder}/cmd/my_api",

and

"program": "${workspaceFolder}\\cmd\\my_api\\main.go",

and

"program": "${workspaceFolder}/cmd/my_api/main.go",

I also tried to change this (with no discern-able change):

"remotePath": "/",

without success.

My multi-stage Dockerfile for building the app and launching delve in headless mode:

FROM golang:1.11.6-alpine3.9 AS builder

RUN wget -O /usr/bin/dep 'https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64' \
    && chmod +x /usr/bin/dep

# For debugging: conmpile Delve
RUN apk add --no-cache git
RUN go get github.com/derekparker/delve/cmd/dlv

# Copy Code and build it:
WORKDIR $GOPATH/src/github.com/myuser/my_api/
COPY Gopkg.toml Gopkg.lock ./
RUN dep ensure --vendor-only
COPY . ./

# Compile with necessary flags for delve
RUN CGO_ENABLED=0 GOOS=linux go build -gcflags "all=-N -l" -a -installsuffix nocgo -o /my_api ./cmd/my_api

FROM alpine:3.9 AS runtime-base

# DEBUGGING: Allow delve to run on Alpine based containers.
RUN apk add --no-cache libc6-compat

# App container
FROM runtime-base

WORKDIR /
# Copy certificates
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Copy app
COPY --from=builder /my_api ./
# Copy delve
COPY --from=builder /go/bin/dlv /

# 40400 for delve
EXPOSE 40400

CMD ["/dlv","--listen=:40400","--headless=true","--api-version=2", "exec", "./my_api"]

I set a breakpoint in one of the packages (not in project_dir/cmd/my_api/main.go, but for example in project_dir/package1/package1.go). This turns grey and "unverified" as soon as I launch the debugger in vscode and the breakpoint is not hit.

I may be doing something wrong on a really simple level, but I can't seem to figure out what.

UPDATE I have finally found the debugger log and see this:

From client: setBreakpoints({"source":{"name":"package1.go","path":"c:\\Users\\myuser\\go\\src\\github.com\\githubaccount\\project_dir\\package1\\package1.go"},"lines":[165,170],"breakpoints":[{"line":165},{"line":170}],"sourceModified":false})
SetBreakPointsRequest
All cleared
Creating on: C:\Users\myuser\go\src\github.com\githubaccount\project_dir\package1\package1.go (C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go) :165
Creating on: C:\Users\myuser\go\src\github.com\githubaccount\project_dir\package1\package1.go (C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go) :170
Error on CreateBreakpoint: could not find C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go:165
Error on CreateBreakpoint: could not find C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go:170

I am not sure it helps me really figure out the problem with vscode though. Could this be a bug? I found a reference to an older MacOS bug:

https://github.com/Microsoft/vscode-go/issues/1859

and

https://github.com/go-delve/delve/issues/1282

but these are old and fixed(?).

With the exception of the windows path separators ("\") being converted to Unix-style path separators "/" by vscode(?), the path is correct, the file exists and the lines to set the breakpoints on are in the middle of the file (and correct)...

In vscode, if I CTRL-click on the path shown in "could not find.." (made into a link by vscode), it takes me right to the file (so vscode can find/see it no problem).

This connects no problem on the Windows host (but only if called from within this directory):

%GOPATH%\...\project_dir\cmd\my_api\dlv debug -l 127.0.0.1:40400

and

(dlv) funcs any_function_in_package1 

finds this function, so the (source) code seems to be visible to delve.

Setting the breakpoint I want in Delve directly also works:

(dlv) break C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go:170
Breakpoint 1 set at 0xad8dd8 for github.com/githubaccount/project_dir/package1.(*Struct).Function() C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go:170

Windows-style path also works the same way on the command line:

(dlv) break C:\Users\myuser\go\src\github.com\githubaccount\project_dir\package1\package1.go:170
Breakpoint 1 set at 0xad8dd8 for github.com/githubaccount/project_dir/package1.(*Struct).Function() C:/Users/myuser/go/src/github.com/githubaccount/project_dir/package1/package1.go:170

Is this a Windows/Unix path-style issue in vscode? Any suggestions?

UPDATE 2 Just found this bug report from late 2018, which seems to describe a similar problem between Delve and vscode:

https://github.com/bazelbuild/rules_go/issues/1844

However, as I posted above, Delve seems to have no problem being passed absolute (Windows) paths for breakpoints in my case, so I am unsure the above bug applies to this situation? Also, why would it work in Delve directly and not via vscode? Or could it be a Windows path / Unix path issue after all?

Thank you for any help!

Answer 1

I have finally found a bug report on this, so it appears to be an open bug for now. Leaving this here for anybody searching as this bug was not easy to find (for me at least).