Force to close SSH client session

Question

I've written a SSH client to connect to network devices and I set a timeout through "select" once the running command takes more than 25 seconds. I noticed a few of devices that they have another IOS it can not drop the SSH session w/ them through Close() method once the timeout has been triggered and it causes goroutinge leaking. I need to keep up the client and disconnect the session to be ready for next command. looks the goroutine doesn't terminate for ever at that time! Do you guys have any idea?

    go func() {
       r <- s.Run(cmd)
    }()

    select {
       case err := <-r:
         return err
       case <-time.After(time.Duration(timeout) * time.Second):
         s.Close()
         return fmt.Errorf("timeout after %d seconds", timeout)
    }

Through the heap profiling I saw the below: 2.77GB 99.44% 99.44% 2.77GB 99.44% bytes.makeSlice

     0     0% 99.44%     2.77GB 99.44%  bytes.(*Buffer).ReadFrom

     0     0% 99.44%     2.77GB 99.44%  golang.org/x/crypto/ssh.(*Session).start.func1

     0     0% 99.44%     2.77GB 99.44%  golang.org/x/crypto/ssh.(*Session).stdout.func1

     0     0% 99.44%     2.77GB 99.44%  io.Copy

     0     0% 99.44%     2.77GB 99.44%  io.copyBuffer

     0     0% 99.44%     2.78GB 99.93%  runtime.goexit

ROUTINE ======================== runtime.goexit in /usr/local/go/src/runtime/asm_amd64.s

     0     2.78GB (flat, cum) 99.93% of Total

     .          .   1993:   RET

     .          .   1994:

     .          .   1995:// The top-most function running on a goroutine

     .          .   1996:// returns to goexit+PCQuantum.

     .          .   1997:TEXT runtime·goexit(SB),NOSPLIT,$0-0

     .     2.78GB   1998:   BYTE    $0x90   // NOP

     .          .   1999:   CALL    runtime·goexit1(SB) // does not return

     .          .   2000:   // traceback from goexit1 must hit code range of goexit

     .          .   2001:   BYTE    $0x90   // NOP

     .          .   2002:


     .          .   2003:TEXT runtime·prefetcht0(SB),NOSPLIT,$0-8

Answer 1

Channel r blocks the Go routine from returning, as it is not being emptied. I've written an adapted version of your code and inserted a Wait group to demonstrate the problem:

func main() {
    var wg sync.WaitGroup // This is only added for demonstration purposes
    s := new(clientSession)

    r := make(chan error)

    go func(s *clientSession) {
        wg.Add(1)
        r <- s.Run()
        wg.Done() // Will only be called after s.Run() is able to return
    }(s)

    fmt.Println("Client has been opened")

    select {
    case err := <-r:
        fmt.Println(err)
    case <-time.After(1 * time.Second):
        s.Close()
        fmt.Println("Timed out, closing")
    }

    wg.Wait() // Waits until wg.Done() is called.

    fmt.Println("Main finished successfully")
}

Go playground seems to terminate the program, so I've created a gist with the complete run-able code. When we run incorrect.go:

$ go run incorrect.go
Client has been opened
Timed out, closing
fatal error: all goroutines are asleep - deadlock!
....

That's because our code is deadlocked on the wg.Wait() line. This demonstrated that wg.Done() in the Go routine was never reached.

As the comments point out, a buffered channel can help out here. But only if you don't care about the error anymore, after calling s.Close()

r := make(chan error, 1)

buffered.go runs correctly, but the error is lost:

$ go run buffered.go
Client has been opened
Timed out, closing
Main finished successfully

Another option would be to drain the channel exactly 1 time:

select {
    case err := <-r:
        fmt.Println(err)
    case <-time.After(1 * time.Second):
        s.Close()
        fmt.Println("Timed out, closing")
        fmt.Println(<-r)
    }

Or by wrapping select in a for loop (without buffered channel):

X:
    for {
        select {
        case err := <-r:
            fmt.Println(err)
            break X // because we are in main(). Normally `return err`
        case <-time.After(1 * time.Second):
            s.Close()
            fmt.Println("Timed out, closing")
        }
    }

When we run drain.go we see the error also printed:

$ go run incorrect.go
Client has been opened
Timed out, closing
Run() closed
Main finished successfully

In real world, one would be running multiple Go routines. So you will want to use some counters on the for loop or further utilize Wait group functionality.