java.util.ConcurrentModificationException during serialization of a synchronized LRUMap

As many of you might now already that LRUMap is not synchronized and is not thread-safe. If you wish to use this map from multiple threads concurrently, you must use appropriate synchronization. The simplest approach is to wrap this map using Collections.synchronizedMap(Map). But even after using this method lrumap=Collections.synchronizedMap(new LRUMap(10)) , particularly when serializing the Map if multiple threads are using it it throws the java.util.ConcurrentModificationException as shown below even though it's synchronized.

It looks like when serializing the Map attribute the map is iterated through a SequencedHashMap$OrderedIterator object to get each entry from the map and then serialize it. Even the LRUMap in this case is a synchronized Map but the iterator isn't, hence it is imperative that the user manually synchronize on the map when iterating over any of its collection views, not sure if this was missed in the serialization code of LRUMap. But if you use synchronized HashMap this problem doesn't seem to happen. Be cautious of using this LRUMap like adding to a HttpSession as an session attribute where serialization happens in a high availability cluster when session bounces between nodes and that you might run into the problem and might loose some session data. Note i was using commons-collections-2.1.jar, i didn't check if this was fixed in the latest releases.

java.util.ConcurrentModificationException
at org.apache.commons.collections.SequencedHashMap$OrderedIterator.next(Unknown Source)
at org.apache.jsp.serializemap_jsp._jspService(serializemap_jsp.java:97)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Jul 31, 2009 3:16:52 PM org.apache.catalina.core.StandardWrapperValve invoke

Poll: Which is the popular J2EE Application Server ?

Poll results for the Poll: Which is the popular J2EE Application Server ? , held between March 6th 2009 to Aug 6th 2009 and voted my about 90 voters.






You can still continue to vote though:

Which is the popular J2EE Application Server ?

WebSphere Portal Websites Gallery

Here are some of the websites created using WebSphere Portal. You can find all sorts of companies ranging from telecom, retail, government, etc.

corporate.homedepot.com	www.alcatel-lucent.com
www.usda.gov	www.ncaa.org
kb.palm.com	Fifth Third Bank
www.3m.com	portal.honeywell.com
www.gapinc.com	Air France Cargo
www.alltel.com	www.labcorp.com

Also you can search "/wps/portal" keyword search in google to all the websites provided they haven't changed the context root.

Verify firewall port assignments using netcat during WebSphere installation planning

During production installation of WebSphere, particularly where multiple node cluster is involved you will be overwhelmed with how many firewall changes need to be made across different vlans and open ports to make sure all the nodes, dmgr, websevers, databases, & ldap can communicate with each others without problems. Also at the same time make sure you only open the ports that are needed for security reasons. ACL firewall rules would become complex due to the different number of ports involved in WebSphere , by default on a single standalone node installation you can see below that there are about 18 ports or so involved,

Port Name Port
-------------------------------------------------------------------
BOOTSTRAP_ADDRESS 2809
SOAP_CONNECTOR_ADDRESS 8880
ORB_LISTENER_ADDRESS 9100
SAS_SSL_SERVERAUTH_LISTENER_ADDRESS 9401
CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS 9403
CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS 9402
WC_adminhost 9060
WC_defaulthost 9080
DCS_UNICAST_ADDRESS 9353
WC_adminhost_secure 9043
WC_defaulthost_secure 9443
SIP_DEFAULTHOST 5060
SIP_DEFAULTHOST_SECURE 5061
SIB_ENDPOINT_ADDRESS 7276
SIB_ENDPOINT_SECURE_ADDRESS 7286
SIB_MQ_ENDPOINT_ADDRESS 5558
SIB_MQ_ENDPOINT_SECURE_ADDRESS 5578
IPC_CONNECTOR_ADDRESS 9633

So in order to make sure all firewall port assignments are proper before installing you need to connect to the listening port and check from the client. Since you don't have websphere actually installed those ports won't be listening and makes it difficult to check and verify. In order to verify you can use the netcat or nc utility in unix or linux. Basically with nc or netcat utility you can listen on any partucular port TCP or UDP and make connections from the client and verify that these port communication is opened through the firewall and resolve your network issues quickly.

Use netcat or nc to listen on port 9080,
(e.g) nc -l 9080

and then you can connect to the port from another client machine using nc or any other tool like telnet to see if it accepts connection,

(e.g) nc 192.169.1.1 9080
you can type on the stdin which will be transferred and displayed across each machine, form that you can know that ports are opened.

Also use nc to port scan as well, (e.g) nc -v -z 192.168.1.1 9080-9084
the output will showing what ports are opened as below,
hostname [192.168.1.1] 9080 (?) open

How to find JVM is 32bit or 64bit ?

You might think it's something simple like typing java -version to find , yes it is for most Java versions, except for Sun JDK 32-bit version doesn't explicitly say which is causing confusion to many users which kind of prompted me to write this article.

If you run java -version in Sun HotSpot 1.6 VM , you will see the below output showing nothing about the bit on the 32-bit JVM where as in 64-bit JVM it explicitly says. so based on this you can almost be kind of sure it's a 32-bit JVM when no bit information shows up in the output.

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)

or
java -version -server from the jdk directory

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)

Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

Whereas on the other hand IBM JDK versions seems to explicitly say what type of JVM it is for both 32-bit as well as 64-bit.

Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)

Also when you try to run with 64-bit model on the 32-bit JVM it might complain as below,

java -d64 -server -version
Running a 64-bit JVM is not supported on this platform.

Also you can look at the lib directory of the jre and see if there is any subdirectory ends with 64 like /opt/jdk1.6.0_11/jre/lib/amd64/ confirming that it's a 64-bit jvm. On a running jvm run Linux command: lsof -p pid of java and see where the libjvm.so process are loaded, which might indicate the type of JVM.

Note there is no such thing as WebSphere Application Server 64-bit or 32-bit version , basically it depends on what kinf of JVM (32-bit 0r 64-bit) you are running on.

Maximum heap size limit of java is smaller than you think

You might think that on a 32-bit OS, a process should be able to address address 2^32 = 4Gb of address space, however in practice some of the address space is used by the OS kernel and so is not available to the process. So there are limitations to how much a process can address and it can vary depending on the platform and the versions of JDK. There are programs like IBM Heap Analyzer requires large amount of memory while analyzing heap dumps taken from JVM with setting of max heap size of -Xmx 1400M(1.4GB) or more and while analyzing, your jvm might very well run out of memory, hence you might needed a 64-bit OS running 64-bit JVM to even analyze a heap dump taken in a 32-bit JVM. Here below is the table lists the max heap size a JVM can take in the respective platform and the different versions.

tr style="HEIGHT: 135pt">

S.No	JVM Version	OS	Bit	Xmx Max Heap Size	Error When JVM not able to allocate memory
1	Java(TM) SE Runtime Environment (build 1.6.0-b105) / Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)	Windows XP	32 bit	1612M	Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine.
2	Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2) Classic VM (build 1.4.2, J2RE 1.4.2 IBM Windows 32 build cn1420-20040626 (JIT enabled: jitc))	Windows XP	32 bit	1635M	[ Unable to allocate an initial java heap of 1724907520 bytes. ] [ **Out of memory, aborting** ] [ ] [ *** panic: JVMST016: Cannot allocate memory for initial java heap ] This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
3	Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)	Windows XP	32 bit	1813M	JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 1814M requested Could not create the Java virtual machine.
4	java version "1.6.0_13"Java(TM) SE Runtime Environment (build 1.6.0_13-b03)Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)	Linux	32 bit	2678M	Error occurred during initialization of VMCould not reserve enough space for object heapCould not create the Java virtual machine.

JIT Compilation of Java code won't happen before 10000 invocations of the same code block

With the default settings of the Java 1.6 HotSpot VM running in a -server mode don't expect your java code to be compiled before 10000 invocations of same code block or the method. This may not be suitable for servers where your server load is less and might take a long time to reach 10000 invocations or where you have an environment with multiple cells and the cell gets flipped every day before reaching the threshold or the servers are getting restarted frequently. In that case you might loose the performance, where the bytecode not being compiled at all. So based on your requirements you can set a lower value of compile threshold from 10000 to get the code compiled faster and boost performance within a short period of time after the restart of the server.

Set the -XX:CompileThreshold=1500 to a lower value in the JVM option and see how your code performs and tune accordingly. Note it's not advisable to set the value below 1000 or -Xcomp (always force JIT compile) as the JVM won't even have enough profiling information before it can generate optimized code and might actually performance degradation rather than improvement. Also note running the JVM with -Xint (interpreted mode) is not advisable in production systems which severely affect the performance unless you are running in a debug code or isolating problems related to JIT.

set -XX:+PrintCompilation to see when the methods are getting complied in the jvm stdout.