Investigation on ssh connection timeout

Introduction

This month I was assigned to a new project. I have to do load test for one part of the system that handles web services. I wrote little client framework for sending hl7v2 web service messages in jython, and I am using grinder as a load testing framework. Run time is java virtual machine, version 7.
We have testing environment consisting of VMware instances, and I am using Red Hat 64 bit VMWare instance for producing hl7v2 traffic using grinder scripts.

Issue

As some test runs last for hours, I have a requirement of ssh idle connection lasting for 24 hours. When I left putty session in idle state, I got putty error with text
"Server unexpectedly closed network connection".

Investigation in tester mode

I first asked colleague that installed VMware instance for help. He replied that he has no time right know for the investigation.
As I am using putty as a ssh client, I browsed putty connection properties. I know about keep alive TCP option, so I set Enable TCP keep alive options, along with Seconds between keep alive set to 5 seconds.
I started another ssh session, but after some time amount I got the same error.
I asked another colleague for advice, and he pointed me at the ssh server configuration in file /etc/ssh/sshd_config. I set following properties
TCPKeepAlive yes
ClientAliveInterval 5 (same as in putty)
And then I restarted sshd process (ssh server).
Same error again.
I examined putty event log. There were following messages that caught my attention


2012-09-28 16:17:51 Initiating key re-exchange (timeout)
2012-09-28 16:17:51 Server unexpectedly closed network connection

Time stamps were exactly one hour after the start of a session. I Googled first message, I was pointed to following putty property
Connection;SSH;Kex
Max minutes before rekey
I set that value from 60 to 1, and after one minute, I replicated my issue message.
I set max minutes before rekey to 0. I started new putty session in order to verify is this solution of my problem. Session was still up in the morning. And that satisfy my requirement for testing environment.

Conclusion

With this post I wanted to point out how tester should behave when encounters an issue. Your responsibility as a tester is to find proper solution, not a 'dirty hack'. Ask for help, your are not alone. And your colleagues job description is to help you. Also, I would like to point out that this is solution that works in my context. For example, if there where a lot of traffic data through putty session, this solution would not work because I did not set putty parameter 'Max data before rekey' to zero value (unlimited).

Labels: ,