I've written a SSH client to connect to network devices and I set a timeout through "select" once the running command takes more than 25 seconds. I noticed a few of devices that they have another IOS it can not drop the SSH session w/ them through Close() method once the timeout has been triggered and it causes goroutinge leaking. I need to keep up the client and disconnect the session to be ready for next command. looks the goroutine doesn't terminate for ever at that time! Do you guys have any idea?
go func() {
r <- s.Run(cmd)
}()
select {
case err := <-r:
return err
case <-time.After(time.Duration(timeout) * time.Second):
s.Close()
return fmt.Errorf("timeout after %d seconds", timeout)
}
Through the heap profiling I saw the below: 2.77GB 99.44% 99.44% 2.77GB 99.44% bytes.makeSlice
0 0% 99.44% 2.77GB 99.44% bytes.(*Buffer).ReadFrom
0 0% 99.44% 2.77GB 99.44% golang.org/x/crypto/ssh.(*Session).start.func1
0 0% 99.44% 2.77GB 99.44% golang.org/x/crypto/ssh.(*Session).stdout.func1
0 0% 99.44% 2.77GB 99.44% io.Copy
0 0% 99.44% 2.77GB 99.44% io.copyBuffer
0 0% 99.44% 2.78GB 99.93% runtime.goexit
ROUTINE ======================== runtime.goexit in /usr/local/go/src/runtime/asm_amd64.s
0 2.78GB (flat, cum) 99.93% of Total
. . 1993: RET
. . 1994:
. . 1995:// The top-most function running on a goroutine
. . 1996:// returns to goexit+PCQuantum.
. . 1997:TEXT runtime·goexit(SB),NOSPLIT,$0-0
. 2.78GB 1998: BYTE $0x90 // NOP
. . 1999: CALL runtime·goexit1(SB) // does not return
. . 2000: // traceback from goexit1 must hit code range of goexit
. . 2001: BYTE $0x90 // NOP
. . 2002:
. . 2003:TEXT runtime·prefetcht0(SB),NOSPLIT,$0-8
The first way to disconnect from an SSH session is with the exit command. Issue this command on the remote terminal that you are currently logged in to. The second way to disconnect from an SSH session is with the logout command.
To end the Putty session, type the logout command such as exit or logout. This command might vary between servers. You can close the session by using the Close button.
Terminal the SSH Tunnel To terminate the ssh tunnel, run ps aux | grep ssh , search for the correct tunnel and PID, and then run kill 12345 replacing 12345 with the PID on your machine.
Using the WHO Command The first command you can use to show active SSH connections is the who command. The who command is used to show who is currently logged in to the system. It allows us to view the connected users and the source IP addresses. To use the who command, simply enter who without any parameters.
Channel r
blocks the Go routine from returning, as it is not being emptied. I've written an adapted version of your code and inserted a Wait group to demonstrate the problem:
func main() {
var wg sync.WaitGroup // This is only added for demonstration purposes
s := new(clientSession)
r := make(chan error)
go func(s *clientSession) {
wg.Add(1)
r <- s.Run()
wg.Done() // Will only be called after s.Run() is able to return
}(s)
fmt.Println("Client has been opened")
select {
case err := <-r:
fmt.Println(err)
case <-time.After(1 * time.Second):
s.Close()
fmt.Println("Timed out, closing")
}
wg.Wait() // Waits until wg.Done() is called.
fmt.Println("Main finished successfully")
}
Go playground seems to terminate the program, so I've created a gist with the complete run-able code. When we run incorrect.go
:
$ go run incorrect.go
Client has been opened
Timed out, closing
fatal error: all goroutines are asleep - deadlock!
....
That's because our code is deadlocked on the wg.Wait()
line. This demonstrated that wg.Done()
in the Go routine was never reached.
As the comments point out, a buffered channel can help out here. But only if you don't care about the error anymore, after calling s.Close()
r := make(chan error, 1)
buffered.go
runs correctly, but the error is lost:
$ go run buffered.go
Client has been opened
Timed out, closing
Main finished successfully
Another option would be to drain the channel exactly 1 time:
select {
case err := <-r:
fmt.Println(err)
case <-time.After(1 * time.Second):
s.Close()
fmt.Println("Timed out, closing")
fmt.Println(<-r)
}
Or by wrapping select
in a for
loop (without buffered channel):
X:
for {
select {
case err := <-r:
fmt.Println(err)
break X // because we are in main(). Normally `return err`
case <-time.After(1 * time.Second):
s.Close()
fmt.Println("Timed out, closing")
}
}
When we run drain.go
we see the error also printed:
$ go run incorrect.go
Client has been opened
Timed out, closing
Run() closed
Main finished successfully
In real world, one would be running multiple Go routines. So you will want to use some counters on the for
loop or further utilize Wait group functionality.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With