Monday, July 13, 2009

java.sql.SQLException: Io exception might be related to firewall or load balancer issues

You might see the following exception in the java SystemOut logs of WebSphere,

Caused by: java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA00
10E: SQL State = null, Error Code = 17,002
... 150 more
---- Begin backtrace for nested exception
java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA0010E: SQL St
ate = null, Error Code = 17,002
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java(Compiled Code))
at java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled Code))
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:501)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:544)
at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:788)
at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:765)
at com.ibm.ejs.j2c.poolmanager.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1431)
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1182)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))

If you see those exceptions it might be due to some network related, either the firewall ( FSWM - Firewall Services Module ) or your load balancer dropping connections or packets causing the exception to be thrown on the logs in the WebSphere with the strange message plus your connections might get lost and might also see an application related error. This exception might be seen when there was some change going on the FSWM or loadbalancer like acl updates where it's trying to reload and applying them, it usually affects the new connections existing pooled connections seems to work ok without problems.

Also if you are using oracle you might see the following error message in the oracle listener sqlnet.log,

TNS-12637: Packet receive failed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12537: TNS:connection closed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12547: TNS:lost contact
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12535: TNS:operation timed out
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12637: Packet receive failed

Here is the explanation from the oracle documentation,

TNS-12637 Packet receive failed

Cause: A process was unable to receive a packet from another process. Possible causes include:
The other process was terminated.
The machine on which the other process is running went down.
Some other communications error occurred.

To rectify it either update the software/firmware to the latest version in your FWSM or load balancer , check for any network bottlenecks, check load average of the machines and take necessary actions. If you are continously seeing the error then there is a possibility that there might be some kind of DoS attack on the db listener.

No comments: