Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Sunday, September 6, 2009

Slowing down your server process does have some benefits

Although lot people are finding various ways to speed up their application servers to improve performance, there are also benefits to slow down your server process especially when testing it to find problems that you won't see in normal scenarios. As you might have seen in your experience that 90% of the time things work well in a normal scenario when there the load on the server is normal, db response throughput is normal, network throughput is optimal , etc. But all of a sudden things start to break when something gets backed up like db fails to respond in an expected time leading to number of concurrent requests due to browser refresh by the user might break the applications due to deadlock or other concurrent related problems. Due to the availability of powerful multiprocessor severs nowadays , requests are executed in terms of milliseconds , hence it would be difficult to catch concurrency related problems unless you recreate the scenario where the requests are taking more time to process. Although one might say this can be simulated by doing a load test on the server, i agree but if there is a problem it would be a overwhelming task and extremely difficult for someone to debug given the amount of requests and objects created during the test, especially if there is a memory leak. Hence the same effect can be achieved by slowing down your server process with just using few concurrent requests which makes things easy to debug and trace.

For (e.g) check this below jsp lrumap.jsp, where it creates a simple LRUMap object, puts it into the session and then serializes it. You will not find any problems with the jsp as long as it runs quickly (usually takes about 10ms is a decent desktop server) with multiple concurrent requests, but if the code slows down due to I/O or some other reason , you will end up with the java.utl.ConcurrentModificationException if the code is executing the serialization part so.writeObject(lmap); and adding of the element lmap.put(lobj[i],new Integer(i)); at the same time even though the LRUMap is synchronizedm, the reason being the serialization code uses an iterator to iterate through the elements in which the iterator is not synchronized which caused the exception. Unless you make the server process to run slow you will never end up catching it until you see it in production during high load conditions where it will become extremely difficult to debug.



How to slow down a process ?


The trick is that you basically limit the CPU usage of the process which will slow down the process as it won't get enough CPU cycles to execute. There is a open source program in Linux CPU Limit can be used for this purpose or you can write a simple shell script where you can send STOP and CONT signal in a loop using the kill command. Although the later is not efficient but it works, but the CPU limit program is much precise even though it'd doing the same but through a C program.

1) Download and install the CPU Limit program.
2) If your System Administrator don't allow then use the shell script,

#!/bin/bash
while [ 1 ]
do
kill -STOP $1
usleep 100000
kill -CONT $1
done

Note using usleep can make the sleep in microseconds where as sleep will sleep in seconds which is not suitable for process executing transactions in milliseconds.

3) find the process id of the process you wanted to slow down.

4) Execute cpulimit -p -l

(e.g) cpulimit -p 11502 -l 10 - this command will allocate max of 10% CPU to the process 11502.

Saturday, August 22, 2009

WebSphere Java process hangs and freezes

We recently had an issue where the websphere Java process got hung and freezes in 4 servers almost at the same time, where 3 server nodes are part of a cluster and the other one is a standalone. Restarting of websphere AppServer fixed the issue. This issue was still puzzling as to why all the servers got hung at the same time and even the one that is not part of the cluster got hung as well. We did some investigation and found the commonality among all these servers is that all the websphere installation directory is nfs mounted on a NAS (Network Attached storage) device. We suspected that either nfs mount or the NAS might have had problems as there was no better explanation for all the server to go down at the same time. We checked the OS /var/log/messages file and found these nfs service messages happened around the same time the server went down ,

Aug 20 04:09:45 appserver01 kernel: nfs: server nasserver01 OK
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:53 appserver01 kernel: nfs: server nasserver01 OK

These messages seems to be related to nfs timeout. As there were no problem with the NAS device itself , it was clear that nfs service was timing out might have caused the issue. We changed the nfs to use the TCP and nfs version 3 which is more reliable instead of UDP with some additional tuning parameters. Once remounting with new parameters the problem didn't happen so far. Here are the new setting for the nfs mount over TCP.

/etc/fstab:

nasserver01:/app/WebSphere /mnt/WebSphere (rw,noatime,hard,intr,tcp,nfsvers=3,retrans=5,rsize=8192,wsize=8192,timeo=14,addr=10.10.1.20)

In case if the problem still exists after the tuning , nfsstat or tcpdump traces can be used to analyze the problem.

Thursday, August 20, 2009

Is Java really "Write once, run anywhere" ?

As many of us generally know, Java is popular for it's cross-platform portability "Write once, run anywhere", but i wanted to give it a test and see if it's truly one. I tried to run WebSphere 7 Application Server itself using Sun JRE 1.6.0 instead of IBM J9 VM which is bundled with AppServer and see if it works with cross vendor JVM on the same platform. I had to make couple of changes, the startServer.sh script adds IBM JVM specific arugments ( -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M )which i had to remove , changed the WAS_HOME/java to point to Sun's JAVA_HOME and set the environment variables as set in setupCmdline.sh and startServer.sh and ran the process from the cmdline with the huge list of arguments with the one i got from the process string when it's started from the startServer.sh script.

The AppServer failed to start with the following exception,

java.lang.NoClassDefFoundError: com/ibm/wsspi/buffermgmt/WsByteBufferPoolManager
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)

and

[20:48:03:436 PDT] 0000000a ServerCollabo E WSVR0121E: An exception occurred getting a socket for port 38930 on hostname waslinux with an IP address of 192.168.1.10.

I added the the AppServer/plugins/* directory to the classpath and even to -Djava.ext.dirs, but this time the JVM just exits and getting terminated without writing anything into the logs. I tried different things to find the actual root of the failure by enabling verbose:jni traces, strace and using jdb, but couldn't find anything useful and ultimately gave up. Note the same method of starting websphere using IBM J9 VM from the cmdline seems to work. Both Sun JDK and Open JDK just terminates for no reason.

I also ran apache tomcat server on both IBM's and Sun JVM and it seems to run fine without problems.

It seems like WebSphere AppServer java code is not compatible to run in other vendor JVM's, hence the promise of Java hasn't come true at least in this case where it's just "write once , run anywhere as long as you stick to the same JVM vendor that you used to develop and test " :) .


*If anyone had tried and been successful please comment on my post, I would really like to run and see as i can use some of the tools like jvisualvm, jmap, jps, jstack , etc which gets bundled with Sun JDK and not with IBM.

Monday, August 17, 2009

Problem running startxwin.bat in Cygwin/X on windows

Nowadays more and more graphical tools like jconsole , jvisualvm are shipped with java and several other tools like tda-Thread Dump Analyzer , IBM HeapAnalyzer , etc are available for download to analyze and debug problems, hence it seems like you need to have some kind of graphical terminal like Xserver or VNC to manage your environment. Since most of the production environments are UNIX/Linux based and more commonly connected through windows desktop, you might need to have a windows based Xserver or VNC client. VNC Servers are not very common in the enterprise as they deemed to be insecure compared to X and needed additional installation and configuration while X comes with the OS. One such windows based open source free XServer is Cygwin/X which i decided to give it a try and ran into some problems before i made it to work, hence i wanted to write about it to avoid the same problem by someone else. The installer is little different where it lists you different packages of Cygwin along with Cygwin/X packages so you just need to select X11 if that's what you needed.



Once installed you just have to goto the c:\cygwin\bin in a cmd prompt and run startxwin.bat ,which should start the Xserver, but i my case the batch file was giving some issues,

startxwin.bat - Starting on Windows NT/2000/XP/2003'c:\cygwin\bin\run' is not recognized as an internal or external command,operable program or batch file.'c:\cygwin\bin\run' is not recognized as an internal or external command,operable program or batch file.

So in order to make it work i had to change the one line in the startxwin.bat file as shown below,

SET RUN=%CYGWIN_ROOT%\bin\run -p /usr/bin

replace with,

SET RUN=run -p /usr/bin

Now you should be able to connect with your favorite SSH client like putty or secureCRT with X11 Forwarding.



Monday, August 3, 2009

How to find JVM is 32bit or 64bit ?

You might think it's something simple like typing java -version to find , yes it is for most Java versions, except for Sun JDK 32-bit version doesn't explicitly say which is causing confusion to many users which kind of prompted me to write this article.

If you run java -version in Sun HotSpot 1.6 VM , you will see the below output showing nothing about the bit on the 32-bit JVM where as in 64-bit JVM it explicitly says. so based on this you can almost be kind of sure it's a 32-bit JVM when no bit information shows up in the output.

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)

or
java -version -server from the jdk directory

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)

Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

Whereas on the other hand IBM JDK versions seems to explicitly say what type of JVM it is for both 32-bit as well as 64-bit.

Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)

Also when you try to run with 64-bit model on the 32-bit JVM it might complain as below,

java -d64 -server -version
Running a 64-bit JVM is not supported on this platform.

Also you can look at the lib directory of the jre and see if there is any subdirectory ends with 64 like /opt/jdk1.6.0_11/jre/lib/amd64/ confirming that it's a 64-bit jvm. On a running jvm run Linux command: lsof -p pid of java and see where the libjvm.so process are loaded, which might indicate the type of JVM.

Note there is no such thing as WebSphere Application Server 64-bit or 32-bit version , basically it depends on what kinf of JVM (32-bit 0r 64-bit) you are running on.

Sunday, August 2, 2009

Maximum heap size limit of java is smaller than you think

You might think that on a 32-bit OS, a process should be able to address address 2^32 = 4Gb of address space, however in practice some of the address space is used by the OS kernel and so is not available to the process. So there are limitations to how much a process can address and it can vary depending on the platform and the versions of JDK. There are programs like IBM Heap Analyzer requires large amount of memory while analyzing heap dumps taken from JVM with setting of max heap size of -Xmx 1400M(1.4GB) or more and while analyzing, your jvm might very well run out of memory, hence you might needed a 64-bit OS running 64-bit JVM to even analyze a heap dump taken in a 32-bit JVM. Here below is the table lists the max heap size a JVM can take in the respective platform and the different versions.


tr style="HEIGHT: 135pt">

S.No

JVM Version

OS

Bit

Xmx Max Heap Size

Error When JVM not able to allocate memory

1

Java(TM) SE Runtime Environment (build 1.6.0-b105) / Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)

Windows XP

32 bit

1612M

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

2


Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2)
Classic VM (build 1.4.2, J2RE 1.4.2 IBM Windows 32 build cn1420-20040626 (JIT enabled: jitc))

Windows XP

32 bit

1635M

[ Unable to allocate an initial java heap of 1724907520 bytes. ]
[ **Out of memory, aborting** ]
[ ]
[ *** panic: JVMST016: Cannot allocate memory for initial java heap ]

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

3

Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)

Windows XP

32 bit

1813M

JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 1814M requested
Could not create the Java virtual machine.

4

java version "1.6.0_13"Java(TM) SE Runtime Environment (build 1.6.0_13-b03)Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)

Linux

32 bit

2678M

Error occurred during initialization of VMCould not reserve enough space for object heapCould not create the Java virtual machine.

JIT Compilation of Java code won't happen before 10000 invocations of the same code block

With the default settings of the Java 1.6 HotSpot VM running in a -server mode don't expect your java code to be compiled before 10000 invocations of same code block or the method. This may not be suitable for servers where your server load is less and might take a long time to reach 10000 invocations or where you have an environment with multiple cells and the cell gets flipped every day before reaching the threshold or the servers are getting restarted frequently. In that case you might loose the performance, where the bytecode not being compiled at all. So based on your requirements you can set a lower value of compile threshold from 10000 to get the code compiled faster and boost performance within a short period of time after the restart of the server.

Set the -XX:CompileThreshold=1500 to a lower value in the JVM option and see how your code performs and tune accordingly. Note it's not advisable to set the value below 1000 or -Xcomp (always force JIT compile) as the JVM won't even have enough profiling information before it can generate optimized code and might actually performance degradation rather than improvement. Also note running the JVM with -Xint (interpreted mode) is not advisable in production systems which severely affect the performance unless you are running in a debug code or isolating problems related to JIT.

set -XX:+PrintCompilation to see when the methods are getting complied in the jvm stdout.

Saturday, July 25, 2009

VisualVM remote application monitoring through SSH proxy

As many of you might already know Java VisualVM an all in one Java troubleshooting tool is available as a JDK tool in Sun JDK distributions starting from JDK 6 update 7 and Apple's Java for Mac OS X 10.5 Update 4. Also VisualVM a stand alone separately downloadable tool is also available at visualvm.dev.java.net. It's a visual tool enhances the capability of performance and monitoring analysis for Java SE platform for both production and development environments. You can find more detailed information in the VisualVM product site.

But our topic is how to make visualvm work through SSH Tunnel specifically in production systems where it's a normal practice to allow only SSH access on port 22 for the users or administrators, hence visualvm will not work with the default settings as it uses port 1099 to communicate to remote host via jstatd. For some reason, VisualVM doesn't seem to work through SSH Port Forwarding or SSH Proxy which seems to be a big blocker for many people which wish yo use this tool in this similar kind of setup.

The only way i came up so far to make the VisualVM work is to use to use a system proxifier like WideCap (Free) or ProxyCap (Shareware) and was able to connect to remotely a running java process over ssh proxy. Here are the steps detailing on how to configure ,

1) Download WideCap and install.
2)Open WideCap control panel and click on new proxy , provide hostname as localhost, port
8333, and select the socks version of your SSH that your client and server uses.





3)Add a new application and point to the location of java.exe and create new rule to get triggered when the java is started.




4) Enable WideCap.
5) run putty or any ssh client as SSH proxy daemon. (e.g) putty -D 8333 username@server or ssh -D 8333 user@host
6) Start VisualVM and add remote host and now you should be able to see the java process running on the remote server and you should be able to connect and monitor.



On a cautious note that widecap is not very consistent and it crashes outlook (at least for me ) and IE sometimes but on the other hand proxycap seem to work fine along with other standard windows applications without problems.

Also if you run into problems make sure all the instructions are followed in this link

Wednesday, July 22, 2009

How to find the websphere version of your installation ?

You might think that it's something trivial that everyone might know but unfortunately my blog gets hits looking for the command to find the version of the websphere almost everyday , hence i wrote this article and also learnt that you can write blog articles about things that are trivial as well and might get hits as good or ever better number of hits compared to non-trivial articles with less amount of time spent on writing one :) .

To find the version of WebSphere Application Sever,
  • You can run versionInfo command to generate a report from data extracted from XML files in the properties/version folder. The report includes a list of changed components and installed or uninstalled maintenance packages. you can find more about the differnet
On Windows : run %WAS_HOME%\bin\versionInfo.bat (e.g) c:\WebSphere\AppServer\versionInfo.bat

On Unix : run $WAS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/AppServer/bin/versionInfo.sh

Sample Report generated by the versionInfo.bat command:

D:\WebSphere\AppServer\bin>versionInfo.bat
WVER0010I: Copyright (c) IBM Corporation 2002; All rights reserved.
WVER0011I: WebSphere Application Server Release 5.1
WVER0012I: VersionInfo reporter version 1.15.2.1, dated 4/28/04

--------------------------------------------------------------------------------
IBM WebSphere Application Server Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 2009-07-20T20:24:56-07:00

Installation
--------------------------------------------------------------------------------
Product Directory D:\WebSphere\AppServer
Version Directory ${product.dir}\properties\version
DTD Directory ${version.dir}\dtd
Log Directory D:\WebSphere\AppServer\logs\update
Backup Directory ${version.dir}\backup
TMP Directory C:\DOCUME~1\rkannan\LOCALS~1\Temp

Installation Platform
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1

Technology List
--------------------------------------------------------------------------------
BASE installed
JDK installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1.1.14
ID BASE
Build Level cf140715.02
Build Date 04/19/2007

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere SDK
Version 1.4.1
ID JDK
Build Level ${build.level}
Build Date ${dateStamp}

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

To find the version of WebSphere Portal Server,
  • You can run WPVersionInfo.sh to gather version information of the websphere portal server installation.
On Windows : run %WPS_HOME%\bin\WPVersionInfo.bat (e.g) c:\WebSphere\PortalServer\bin\WPVersionInfo.bat

On Unix : run $WPS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/PortalServer/bin/WPVersionInfo.sh

Sample report of the command:

[root@server101 bin]# ./WPVersionInfo.sh
WVER0010I: Copyright IBM Corporation 2002, 2008; All rights reserved.
WVER0011I: WebSphere Portal
WVER0012I: VersionInfo reporter version 1.10.1.1, dated 20080530_1701
--------------------------------------------------------------------------------
IBM WebSphere Portal Product Installation Status Report
--------------------------------------------------------------------------------
... ...
Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Portal MultiPlatform
Version6.1.0.0
ID MP
Build Levelwp610_494_01 2008-06-19
Build Date 06/19/2008
Installed Product
... ...
--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------


To find the version of IBM JDK installed along with WebSphere Application Server,

  • you can run java -fullversion to find the jdk version details
On Windows : run %WAS_HOME%\java\bin\java -fullversion (e.g) c:\WebSphere\AppServer\java\bin\java -fullversion

On Unix : run $WAS_HOME/java/bin/java -fullversion (e.g) /opt/WebSphere/AppServer/java/bin/java -fullversion

Sample output of the command:

D:\WebSphere\AppServer\java\bin>java -fullversion
java full version "J2RE 1.4.2 IBM Windows 32 build cn1420-20040626"



Monday, July 20, 2009

How to find the version of websphere, java and portal that you are running ?

You might think that it's something trivial that everyone might know but unfortunately my blog gets hits looking for the command to find the version of the websphere almost everyday , hence i wrote this article and also learnt that you can write blog articles about things that are trivial as well and might get hits as good or ever better number of hits compared to non-trivial articles with less amount of time spent on writing one :) .

To find the version of WebSphere Application Sever,
  • You can run versionInfo command to generate a report from data extracted from XML files in the properties/version folder. The report includes a list of changed components and installed or uninstalled maintenance packages. you can find more about the differnet
On Windows : run %WAS_HOME%\bin\versionInfo.bat (e.g) c:\WebSphere\AppServer\versionInfo.bat

On Unix : run $WAS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/AppServer/bin/versionInfo.sh

Sample Report generated by the versionInfo.bat command:

D:\WebSphere\AppServer\bin>versionInfo.bat
WVER0010I: Copyright (c) IBM Corporation 2002; All rights reserved.
WVER0011I: WebSphere Application Server Release 5.1
WVER0012I: VersionInfo reporter version 1.15.2.1, dated 4/28/04

--------------------------------------------------------------------------------
IBM WebSphere Application Server Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 2009-07-20T20:24:56-07:00

Installation
--------------------------------------------------------------------------------
Product Directory D:\WebSphere\AppServer
Version Directory ${product.dir}\properties\version
DTD Directory ${version.dir}\dtd
Log Directory D:\WebSphere\AppServer\logs\update
Backup Directory ${version.dir}\backup
TMP Directory C:\DOCUME~1\rkannan\LOCALS~1\Temp

Installation Platform
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1

Technology List
--------------------------------------------------------------------------------
BASE installed
JDK installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1.1.14
ID BASE
Build Level cf140715.02
Build Date 04/19/2007

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere SDK
Version 1.4.1
ID JDK
Build Level ${build.level}
Build Date ${dateStamp}

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

To find the version of WebSphere Portal Server,
  • You can run WPVersionInfo.sh to gather version information of the websphere portal server installation.
On Windows : run %WPS_HOME%\bin\WPVersionInfo.bat (e.g) c:\WebSphere\PortalServer\bin\WPVersionInfo.bat

On Unix : run $WPS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/PortalServer/bin/WPVersionInfo.sh

Sample report of the command:

[root@server101 bin]# ./WPVersionInfo.sh
WVER0010I: Copyright IBM Corporation 2002, 2008; All rights reserved.
WVER0011I: WebSphere Portal
WVER0012I: VersionInfo reporter version 1.10.1.1, dated 20080530_1701
--------------------------------------------------------------------------------
IBM WebSphere Portal Product Installation Status Report
--------------------------------------------------------------------------------
... ...
Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Portal MultiPlatform
Version6.1.0.0
ID MP
Build Levelwp610_494_01 2008-06-19
Build Date 06/19/2008
Installed Product
... ...
--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------


To find the version of IBM JDK installed along with WebSphere Application Server,

  • you can run java -fullversion to find the jdk version details
On Windows : run %WAS_HOME%\java\bin\java -fullversion (e.g) c:\WebSphere\AppServer\java\bin\java -fullversion

On Unix : run $WAS_HOME/java/bin/java -fullversion (e.g) /opt/WebSphere/AppServer/java/bin/java -fullversion

Sample output of the command:

D:\WebSphere\AppServer\java\bin>java -fullversion
java full version "J2RE 1.4.2 IBM Windows 32 build cn1420-20040626"



Monday, July 13, 2009

java.sql.SQLException: Io exception might be related to firewall or load balancer issues

You might see the following exception in the java SystemOut logs of WebSphere,

Caused by: java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA00
10E: SQL State = null, Error Code = 17,002
... 150 more
---- Begin backtrace for nested exception
java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA0010E: SQL St
ate = null, Error Code = 17,002
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java(Compiled Code))
at java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled Code))
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:501)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:544)
at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:788)
at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:765)
at com.ibm.ejs.j2c.poolmanager.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1431)
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1182)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))

If you see those exceptions it might be due to some network related, either the firewall ( FSWM - Firewall Services Module ) or your load balancer dropping connections or packets causing the exception to be thrown on the logs in the WebSphere with the strange message plus your connections might get lost and might also see an application related error. This exception might be seen when there was some change going on the FSWM or loadbalancer like acl updates where it's trying to reload and applying them, it usually affects the new connections existing pooled connections seems to work ok without problems.

Also if you are using oracle you might see the following error message in the oracle listener sqlnet.log,

TNS-12637: Packet receive failed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12537: TNS:connection closed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12547: TNS:lost contact
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12535: TNS:operation timed out
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12637: Packet receive failed

Here is the explanation from the oracle documentation,

TNS-12637 Packet receive failed

Cause: A process was unable to receive a packet from another process. Possible causes include:
The other process was terminated.
The machine on which the other process is running went down.
Some other communications error occurred.

To rectify it either update the software/firmware to the latest version in your FWSM or load balancer , check for any network bottlenecks, check load average of the machines and take necessary actions. If you are continously seeing the error then there is a possibility that there might be some kind of DoS attack on the db listener.

Saturday, July 11, 2009

Synchronization issues related to Java collections when accessed concurrently

Being a java developer or an application server administrator interested in learning how the code works like me :) , you must have seen the javadoc http://java.sun.com/javase/6/docs/api/java/util/HashMap.html warning: Note that this implementation is not synchronized. If multiple threads access a hash map concurrently, and at least one of the threads modifies the map structurally, it must be synchronized externally. (A structural modification is any operation that adds or deletes one or more mappings; merely changing the value associated with a key that an instance already contains is not a structural modification.) This is typically accomplished by synchronizing on some object that naturally encapsulates the map. If no such object exists, the map should be "wrapped" using the Collections.synchronizedMap method. This is best done at creation time, to prevent accidental unsynchronized access to the map.

So here are some things that might happen depends on what kind of Map you are using like LRUMap or TreeMap etc. I have seen these behaviors in runtime where the Map is not synchronized but concurrently accessed, and though of publishing it in the blog which might help others to debug similar issues or correlate the symptoms to the failure they might have seen but not having a clue.



1) LRUMap Memory leak : If you are using a LRUMap and multiple threads are putting and getting values concurrently, some kind of memory leak happens and the object grows beyond the specified maximum size. Every time you access the LRUMap either by get or put, the object you are getting or putting had to be moved to the MRU(Most recently used) place within the Map. Also when the LRUMap is full it has to remove the LRU object to provide room for the new object. I believe this is accomplished by some kind of a link list data structure within the map. Hence there is a lot of overhead during the process and there is quite a chance to get the link list corrupted when multiples threads are get/put’ing concurrently particularly when the map is full.

I noticed the “size()” as returned by the object sometimes gets higher than the maximum size and sometimes show less than the maximum size when it’s supposedly be full depending on how and where the corruption happens within the link list .Further accessing the object after it’s corrupted, the “size()” is starting to get reduced every time a further corruption happens and leads into negative size (-1…-20) and keep increasing in the negative side. Note the “size()” that I am referring is the size returned by the counter that the object is using to track and not the actual object size. so I took a heap dump and found that any object that is added after Is never discarded at all , I guess because of the logic that size never gets bigger than the maximum size after corruption.It looks like one the object is corrupted it’s starting to leak and then increasing the overall size(byte size) and the behavior in erroneous.
I am attaching the lru.jsp that I used to simulate so that you can try and see how it behaves. The below picture shows how the object java.math.BigDecimal grows in consecutive heapdumps even though the max size is set to 2.


Note that this behavior happened on WebSphere 5.1 Application Server on JDK 1.4.2 not sure what might be the behavior on other version of JDK or AppServer. Please post in my comment if you are seeing any different behaviors for any other types of Map.

2) TreeMap High Cpu Utilization: This behavior happened particularly when the interator iterating through the treemap collection is not synchronized, where one thread is changing (add or delete) the element and then other iterating through it concurrently. I didn't do much analysis but it looks like the underlying data structure of the TreeMap some how gets corrupted and causes a circular reference between two elements causing the interator to go into an infinite loop which caues the CPU to spike and eventually the ApplicationServer will become unresponsive. You probably want to kill the process to reocver from this state. Here is the thread stacktrace that you would see if take a thread dump, Attaching the tree.jsp for you to test and see.

at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1570)
at java.util.TreeMap$NavigableSubMap$SubMapKeyIterator.next(TreeMap.java:1629)
at org.apache.jsp.tree2_jsp._jspService(tree2_jsp.java:120)

Also note many programmers may forget to synchronize on the collection when iterating eventhough the collection is synchronized. Hence, it is imperative that the user manually synchronize on the returned collection when iterating over it to prevent any non-deterministic behavior.

Collection c = Collections.synchronizedCollection(myCollection);
...
synchronized(c) {
Iterator i = c.iterator(); // Must be in the synchronized block
while (i.hasNext())
foo(i.next());
}

Sunday, December 30, 2007

Java Class finder with a UNIX command

Being a Java developer or Java based Application server administrator like WebSphere or WebLogic , i am sure you should have come across this exception " Error: java.lang.ClassNotFoundException " which basically implies that the JVM is unable to find a definition for your class. It is usually resolved by adding the jar file that contains the class to the classpath or packaging the jar file along with your EAR or WAR file or placing the jar file in the right locations so that your Application Server classloader can find and load the class . So inorder to indentify the .jar file where the class file is present , the following simple UNIX command will help us to indentify the jar file and take appropiate actions to resolve the issue .

LINUX:

find PATH -name '*.jar' -o -name '*.war' -o -name '*.ear' -type f |xargs -i bash -c "jar -tvf {}| tr / . | grep CLASSNAME && echo {}"

(e.g)
find /usr/IBM/WebSphere/AppServer/ -name '*.jar' -o -name '*.war' -o -name '*.ear' -type f | xargs -i bash -c "jar -tvf {}| tr / . | grep org.apache.xerces.dom.DocumentImpl.class && echo {}"


Output:

11160 Fri Feb 20 14:00:34 PST 2004 org.apache.xerces.dom.DocumentImpl.class
/usr/IBM/WebSphere/AppServer/archive_lib/xerces-2.6.2.jar
11160 Fri Oct 06 10:12:24 PDT 2006 org.apache.xerces.dom.DocumentImpl.class
/usr/IBM/WebSphere/AppServer/updateinstaller/update.jar


The above command will recursively search all the jar , EAR and WAR files under the /usr/IBM/WebSphere/AppServer/ for the class name org.apache.xerces.dom.DocumentImpl.class and output the location of the .jar file that contains the class.

Although this command is useful only in unix environment but there are other java based utilities to find the same like Jar Class Finder , Java class finder, etc , but i find this command simple to use without having to copy addtional jar files to your systems.