Situation: Within our system, the connection with the LOG database was lost. The Adeptia GUI was able to show all processes, but no logs at all.
Below are some of our findings and few of the exceptions that are pointed out below:
Kernel Error: Failed to check the connection: java.net.SocketTimeoutException: Read timed out 2012-07-26 08:42:29 java.io.IOException: The client has been closed.
Cause: This seems to happen due to intermittent network issues. In above scenario, connection to JMX MBean server is lost due to socket time out exception. This issue shall not occur in an error free network.
After analyzing the Webrunner log files we have found the below exceptions:
a) java.net.SocketException: Connection reset
(b) java.rmi.ConnectIOException: error during JRMP connection establishment
(c) java.net.SocketTimeoutException: Read timed out
Cause: These issues seem to happen due to intermittent network issues. This shall not occur in an error free network.
(d) com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 85,740,486 milliseconds ago. The last packet sent successfully to the server was 85,740,486 milliseconds ago. is longer than the server configured value of 'wait_timeout'.
Recommendation: You should consider either expiring and/or testing connection validity before using in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
(e) java.net.SocketException: Broken pipe
Recommendation: It happens due to presence of stale connection objects in database connection pool in Adeptia product. This could happen when you have configured a short wait time out value at database level which tends to close the connection at database side but it is still active in database connection pool in Adeptia product. To resolve this issue, we have now added two configurable properties in database connection pool in Adeptia product i.e. “Test On Borrow” and “Validation Query”. If “Test On Borrow” property is enabled, then it will validate the connection object (based on specified “Validation Query”) before borrowing it from connection pool. If the object fails to validate, it will be dropped from the pool, and we will attempt to borrow another one. This will ensure that no stale connection objects are borrowed from database connection pool in Adeptia product.
PS: Above properties are available in v5.3 SP1. You can configure above database connection pool properties according to your need.
Also, MySQL server has a default timeout period after which it terminates the idle connections. You can also consider increasing the time limit by setting the wait_timeout variable. This variable specifies the number of seconds database server waits for activity on a non-interactive connection before closing it.