IBM WebSphere Portal in Real-World Cloud Computing

Since IBM and Amazon announced the avalability of WebSphere Portal Server and Lotus Web Content Management Standard Edition on the Amazon EC2 Web Service, I used to wonder whether there are any real-world customers using it. But now it seems from the news, that an investment firm Quintana Capital Group converted its Web site and IBM WebSphere-powered portal to Amazon's EC2. Here is the URL http://www.qeplp.com/wps/portal/ of their portal powered by IBM WebSphere Portal on the Amazon EC2.


Also you can run a traceroute to confirm that it's hosted in EC2.

>tracert www.qeplp.com
100 ms 98 ms 97 ms ec2-174-129-234-118.compute-1.amazonaws.com [174.129.234.118]

WebSphere Java process hangs and freezes

We recently had an issue where the websphere Java process got hung and freezes in 4 servers almost at the same time, where 3 server nodes are part of a cluster and the other one is a standalone. Restarting of websphere AppServer fixed the issue. This issue was still puzzling as to why all the servers got hung at the same time and even the one that is not part of the cluster got hung as well. We did some investigation and found the commonality among all these servers is that all the websphere installation directory is nfs mounted on a NAS (Network Attached storage) device. We suspected that either nfs mount or the NAS might have had problems as there was no better explanation for all the server to go down at the same time. We checked the OS /var/log/messages file and found these nfs service messages happened around the same time the server went down ,

Aug 20 04:09:45 appserver01 kernel: nfs: server nasserver01 OK
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:53 appserver01 kernel: nfs: server nasserver01 OK

These messages seems to be related to nfs timeout. As there were no problem with the NAS device itself , it was clear that nfs service was timing out might have caused the issue. We changed the nfs to use the TCP and nfs version 3 which is more reliable instead of UDP with some additional tuning parameters. Once remounting with new parameters the problem didn't happen so far. Here are the new setting for the nfs mount over TCP.

/etc/fstab:

nasserver01:/app/WebSphere /mnt/WebSphere (rw,noatime,hard,intr,tcp,nfsvers=3,retrans=5,rsize=8192,wsize=8192,timeo=14,addr=10.10.1.20)

In case if the problem still exists after the tuning , nfsstat or tcpdump traces can be used to analyze the problem.

Is Java really "Write once, run anywhere" ?

As many of us generally know, Java is popular for it's cross-platform portability "Write once, run anywhere", but i wanted to give it a test and see if it's truly one. I tried to run WebSphere 7 Application Server itself using Sun JRE 1.6.0 instead of IBM J9 VM which is bundled with AppServer and see if it works with cross vendor JVM on the same platform. I had to make couple of changes, the startServer.sh script adds IBM JVM specific arugments ( -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M )which i had to remove , changed the WAS_HOME/java to point to Sun's JAVA_HOME and set the environment variables as set in setupCmdline.sh and startServer.sh and ran the process from the cmdline with the huge list of arguments with the one i got from the process string when it's started from the startServer.sh script.

The AppServer failed to start with the following exception,

java.lang.NoClassDefFoundError: com/ibm/wsspi/buffermgmt/WsByteBufferPoolManager
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)

and

[20:48:03:436 PDT] 0000000a ServerCollabo E WSVR0121E: An exception occurred getting a socket for port 38930 on hostname waslinux with an IP address of 192.168.1.10.

I added the the AppServer/plugins/* directory to the classpath and even to -Djava.ext.dirs, but this time the JVM just exits and getting terminated without writing anything into the logs. I tried different things to find the actual root of the failure by enabling verbose:jni traces, strace and using jdb, but couldn't find anything useful and ultimately gave up. Note the same method of starting websphere using IBM J9 VM from the cmdline seems to work. Both Sun JDK and Open JDK just terminates for no reason.

I also ran apache tomcat server on both IBM's and Sun JVM and it seems to run fine without problems.

It seems like WebSphere AppServer java code is not compatible to run in other vendor JVM's, hence the promise of Java hasn't come true at least in this case where it's just "write once , run anywhere as long as you stick to the same JVM vendor that you used to develop and test " :) .


*If anyone had tried and been successful please comment on my post, I would really like to run and see as i can use some of the tools like jvisualvm, jmap, jps, jstack , etc which gets bundled with Sun JDK and not with IBM.

Problem running startxwin.bat in Cygwin/X on windows

Nowadays more and more graphical tools like jconsole , jvisualvm are shipped with java and several other tools like tda-Thread Dump Analyzer , IBM HeapAnalyzer , etc are available for download to analyze and debug problems, hence it seems like you need to have some kind of graphical terminal like Xserver or VNC to manage your environment. Since most of the production environments are UNIX/Linux based and more commonly connected through windows desktop, you might need to have a windows based Xserver or VNC client. VNC Servers are not very common in the enterprise as they deemed to be insecure compared to X and needed additional installation and configuration while X comes with the OS. One such windows based open source free XServer is Cygwin/X which i decided to give it a try and ran into some problems before i made it to work, hence i wanted to write about it to avoid the same problem by someone else. The installer is little different where it lists you different packages of Cygwin along with Cygwin/X packages so you just need to select X11 if that's what you needed.

Once installed you just have to goto the c:\cygwin\bin in a cmd prompt and run startxwin.bat ,which should start the Xserver, but i my case the batch file was giving some issues,

startxwin.bat - Starting on Windows NT/2000/XP/2003'c:\cygwin\bin\run' is not recognized as an internal or external command,operable program or batch file.'c:\cygwin\bin\run' is not recognized as an internal or external command,operable program or batch file.

So in order to make it work i had to change the one line in the startxwin.bat file as shown below,

SET RUN=%CYGWIN_ROOT%\bin\run -p /usr/bin

replace with,

SET RUN=run -p /usr/bin

Now you should be able to connect with your favorite SSH client like putty or secureCRT with X11 Forwarding.

WebSphere Education videos on youtube.com

I found this WebSphere Education channel on youtube.com, thought i should publish it to my blog. The channel doesn't seem to be more active as i can see videos that were posted like 6 month old and not anything new , not sure if IBM had changed it's policy in publishing it to youtube.com. You can find some useful videos ranging on different topics in IBM WebSphere product family like Administration of IBM WebSphere Process Server, IBM WebSphere Application Server V6.1 Administration, IBM WebSphere Commerce Installation and Configuration and so on. You can also subscribe to the channel so that whenever a new education session is posted you will be notified.

Using OpenSSL tool to check SSL certificates for expiration dates

I am sure many of the Application or System administrators might have encoutered the issue of SSL certificates getting expired in the middle of the day causing application outages. The part of the reason is that the application server environments are getting complex day by day in terms of number of systems it's interacting over SSL like LDAP, WebServices, WebSever plugin, Siebel and even database connectivity in highly secured environments. It further complicates as these are disparate systems maintained by different groups in the enterprise having different expiration dates and different formats of keystore and truststore. So unless you have good enterprise wide policy of checking SSL expiration dates well ahead, there is a very good chance that you might run into application failures.

Being an Websphere Application Administrator, you will be the first point of contact when application goes down, even the SSL certs expired in one of the system that it's intreacting with like the LDAP wich is used for security authentication. In order to diagnose , identify and notify the repsective system owner, openSSL tool comes handy in such a way that you just need to know the hostname and the port where the services SSL port is listening on and you will be able to find the expiration dates without having to know the password for the keystore or the format or on how to access and view the keystore. The tool usually gets installed by default in unix systems and in windows you can download and install it.


Steps to check the SSL Certificate expiratio using openSSL tool:

1) openssl s_client -connect hostname:port > cert - this command will get the certificate and redirect it to the file.
2) openssl x509 -in cert -noout -enddate - show the expiration date of the downloaded certificate.


(e.g) To check the expiration for www14.software.ibm.com webserver host

$openssl s_client -connect www14.software.ibm.com:443 > cert

Loading 'screen' into random state - done
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0

$openssl x509 -in cert -noout -enddate

notAfter=Dec 8 13:00:22 2009 GMT

Note not only this works with webserver it will work with any service with SSL turned on.

You can also check other information about the certificates without redirecting to the file. Also you can check my other article for SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore as well.

$openssl s_client -connect www14.software.ibm.com:443

Loading 'screen' into random state - done
CONNECTED(000006DC)
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/C=US/ST=New York/L=Armonk/O=IBM/CN=www14.software.ibm.com
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
1 s:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC1TCCAj6gAwIBAgIDCiE+MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDgxMTA3MTMwMDIyWhcNMDkxMjA4MTMwMDIy
WjBgMQswCQYDVQQGEwJVUzERMA8GA1UECBMITmV3IFlvcmsxDzANBgNVBAcTBkFy
bW9uazEMMAoGA1UEChMDSUJNMR8wHQYDVQQDExZ3d3cxNC5zb2Z0d2FyZS5pYm0u
Y29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC4tUCL4Ar8Sx8pNEbRb+XQ
7CMa1fgh1k2p0zucagZeX/C8e3DMefU5UH3pJKjYZoYte6PWB92jwtw673Rj0Oq6
oDBljHbdtYo0H58C6QqSkjMLFEKyNBHOAZ1SGws11H5WCz+xiOZ1c5Xq11Jjl/BM
nu6PKbfKUUrBwIdTfK/heQIDAQABo4GuMIGrMA4GA1UdDwEB/wQEAwIE8DAdBgNV
HQ4EFgQUuB0DNKiqXdoH87tWzq+71OYK7FUwOgYDVR0fBDMwMTAvoC2gK4YpaHR0
cDovL2NybC5nZW90cnVzdC5jb20vY3Jscy9zZWN1cmVjYS5jcmwwHwYDVR0jBBgw
FoAUSOZo+SvSspXXR9gjIBBPM5iQn9QwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMA0GCSqGSIb3DQEBBQUAA4GBAExSdcrIMiwQqDOAelP4C6QiUeXgis2g
8ePc17lrt2NjRe1aRoSmMYq2U7Rc3203L3WFQ/SOFzfUSKWwBjx98IPFGOuKTMza
DYe1xknf+jjg8IwzhAtcg9Y06jabJBwxMXebSH2lazqFSD8ztBedBjLM7/zQ/Ttz
a7e4t98R0/fX
-----END CERTIFICATE-----
subject=/C=US/ST=New York/L=Armonk/O=IBM/CN=www14.software.ibm.com
issuer=/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
No client certificate CA names sent
---
SSL handshake has read 1688 bytes and written 322 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 00038028A696989D0973A59DEF091113DCCB885C585858584A80A84C0000298D
Session-ID-ctx:
Master-Key: 6A694E8816CE422DB3AD280BEC469ACBDBE1EB7BF116C5E3C600A1A68CC71B7864AE39D8A59CC1F07263C1AB1664238D
Key-Arg : None
Start Time: 1249945678
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)

java.util.ConcurrentModificationException during serialization of a synchronized LRUMap

As many of you might now already that LRUMap is not synchronized and is not thread-safe. If you wish to use this map from multiple threads concurrently, you must use appropriate synchronization. The simplest approach is to wrap this map using Collections.synchronizedMap(Map). But even after using this method lrumap=Collections.synchronizedMap(new LRUMap(10)) , particularly when serializing the Map if multiple threads are using it it throws the java.util.ConcurrentModificationException as shown below even though it's synchronized.

It looks like when serializing the Map attribute the map is iterated through a SequencedHashMap$OrderedIterator object to get each entry from the map and then serialize it. Even the LRUMap in this case is a synchronized Map but the iterator isn't, hence it is imperative that the user manually synchronize on the map when iterating over any of its collection views, not sure if this was missed in the serialization code of LRUMap. But if you use synchronized HashMap this problem doesn't seem to happen. Be cautious of using this LRUMap like adding to a HttpSession as an session attribute where serialization happens in a high availability cluster when session bounces between nodes and that you might run into the problem and might loose some session data. Note i was using commons-collections-2.1.jar, i didn't check if this was fixed in the latest releases.

java.util.ConcurrentModificationException
at org.apache.commons.collections.SequencedHashMap$OrderedIterator.next(Unknown Source)
at org.apache.jsp.serializemap_jsp._jspService(serializemap_jsp.java:97)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Jul 31, 2009 3:16:52 PM org.apache.catalina.core.StandardWrapperValve invoke

Poll: Which is the popular J2EE Application Server ?

Poll results for the Poll: Which is the popular J2EE Application Server ? , held between March 6th 2009 to Aug 6th 2009 and voted my about 90 voters.






You can still continue to vote though:

Which is the popular J2EE Application Server ?

WebSphere Portal Websites Gallery

Here are some of the websites created using WebSphere Portal. You can find all sorts of companies ranging from telecom, retail, government, etc.

corporate.homedepot.com	www.alcatel-lucent.com
www.usda.gov	www.ncaa.org
kb.palm.com	Fifth Third Bank
www.3m.com	portal.honeywell.com
www.gapinc.com	Air France Cargo
www.alltel.com	www.labcorp.com

Also you can search "/wps/portal" keyword search in google to all the websites provided they haven't changed the context root.

Verify firewall port assignments using netcat during WebSphere installation planning

During production installation of WebSphere, particularly where multiple node cluster is involved you will be overwhelmed with how many firewall changes need to be made across different vlans and open ports to make sure all the nodes, dmgr, websevers, databases, & ldap can communicate with each others without problems. Also at the same time make sure you only open the ports that are needed for security reasons. ACL firewall rules would become complex due to the different number of ports involved in WebSphere , by default on a single standalone node installation you can see below that there are about 18 ports or so involved,

Port Name Port
-------------------------------------------------------------------
BOOTSTRAP_ADDRESS 2809
SOAP_CONNECTOR_ADDRESS 8880
ORB_LISTENER_ADDRESS 9100
SAS_SSL_SERVERAUTH_LISTENER_ADDRESS 9401
CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS 9403
CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS 9402
WC_adminhost 9060
WC_defaulthost 9080
DCS_UNICAST_ADDRESS 9353
WC_adminhost_secure 9043
WC_defaulthost_secure 9443
SIP_DEFAULTHOST 5060
SIP_DEFAULTHOST_SECURE 5061
SIB_ENDPOINT_ADDRESS 7276
SIB_ENDPOINT_SECURE_ADDRESS 7286
SIB_MQ_ENDPOINT_ADDRESS 5558
SIB_MQ_ENDPOINT_SECURE_ADDRESS 5578
IPC_CONNECTOR_ADDRESS 9633

So in order to make sure all firewall port assignments are proper before installing you need to connect to the listening port and check from the client. Since you don't have websphere actually installed those ports won't be listening and makes it difficult to check and verify. In order to verify you can use the netcat or nc utility in unix or linux. Basically with nc or netcat utility you can listen on any partucular port TCP or UDP and make connections from the client and verify that these port communication is opened through the firewall and resolve your network issues quickly.

Use netcat or nc to listen on port 9080,
(e.g) nc -l 9080

and then you can connect to the port from another client machine using nc or any other tool like telnet to see if it accepts connection,

(e.g) nc 192.169.1.1 9080
you can type on the stdin which will be transferred and displayed across each machine, form that you can know that ports are opened.

Also use nc to port scan as well, (e.g) nc -v -z 192.168.1.1 9080-9084
the output will showing what ports are opened as below,
hostname [192.168.1.1] 9080 (?) open

How to find JVM is 32bit or 64bit ?

You might think it's something simple like typing java -version to find , yes it is for most Java versions, except for Sun JDK 32-bit version doesn't explicitly say which is causing confusion to many users which kind of prompted me to write this article.

If you run java -version in Sun HotSpot 1.6 VM , you will see the below output showing nothing about the bit on the 32-bit JVM where as in 64-bit JVM it explicitly says. so based on this you can almost be kind of sure it's a 32-bit JVM when no bit information shows up in the output.

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)

or
java -version -server from the jdk directory

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)

Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

Whereas on the other hand IBM JDK versions seems to explicitly say what type of JVM it is for both 32-bit as well as 64-bit.

Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)

Also when you try to run with 64-bit model on the 32-bit JVM it might complain as below,

java -d64 -server -version
Running a 64-bit JVM is not supported on this platform.

Also you can look at the lib directory of the jre and see if there is any subdirectory ends with 64 like /opt/jdk1.6.0_11/jre/lib/amd64/ confirming that it's a 64-bit jvm. On a running jvm run Linux command: lsof -p pid of java and see where the libjvm.so process are loaded, which might indicate the type of JVM.

Note there is no such thing as WebSphere Application Server 64-bit or 32-bit version , basically it depends on what kinf of JVM (32-bit 0r 64-bit) you are running on.

Maximum heap size limit of java is smaller than you think

You might think that on a 32-bit OS, a process should be able to address address 2^32 = 4Gb of address space, however in practice some of the address space is used by the OS kernel and so is not available to the process. So there are limitations to how much a process can address and it can vary depending on the platform and the versions of JDK. There are programs like IBM Heap Analyzer requires large amount of memory while analyzing heap dumps taken from JVM with setting of max heap size of -Xmx 1400M(1.4GB) or more and while analyzing, your jvm might very well run out of memory, hence you might needed a 64-bit OS running 64-bit JVM to even analyze a heap dump taken in a 32-bit JVM. Here below is the table lists the max heap size a JVM can take in the respective platform and the different versions.

tr style="HEIGHT: 135pt">

S.No	JVM Version	OS	Bit	Xmx Max Heap Size	Error When JVM not able to allocate memory
1	Java(TM) SE Runtime Environment (build 1.6.0-b105) / Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)	Windows XP	32 bit	1612M	Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine.
2	Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2) Classic VM (build 1.4.2, J2RE 1.4.2 IBM Windows 32 build cn1420-20040626 (JIT enabled: jitc))	Windows XP	32 bit	1635M	[ Unable to allocate an initial java heap of 1724907520 bytes. ] [ **Out of memory, aborting** ] [ ] [ *** panic: JVMST016: Cannot allocate memory for initial java heap ] This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
3	Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)	Windows XP	32 bit	1813M	JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 1814M requested Could not create the Java virtual machine.
4	java version "1.6.0_13"Java(TM) SE Runtime Environment (build 1.6.0_13-b03)Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)	Linux	32 bit	2678M	Error occurred during initialization of VMCould not reserve enough space for object heapCould not create the Java virtual machine.

JIT Compilation of Java code won't happen before 10000 invocations of the same code block

With the default settings of the Java 1.6 HotSpot VM running in a -server mode don't expect your java code to be compiled before 10000 invocations of same code block or the method. This may not be suitable for servers where your server load is less and might take a long time to reach 10000 invocations or where you have an environment with multiple cells and the cell gets flipped every day before reaching the threshold or the servers are getting restarted frequently. In that case you might loose the performance, where the bytecode not being compiled at all. So based on your requirements you can set a lower value of compile threshold from 10000 to get the code compiled faster and boost performance within a short period of time after the restart of the server.

Set the -XX:CompileThreshold=1500 to a lower value in the JVM option and see how your code performs and tune accordingly. Note it's not advisable to set the value below 1000 or -Xcomp (always force JIT compile) as the JVM won't even have enough profiling information before it can generate optimized code and might actually performance degradation rather than improvement. Also note running the JVM with -Xint (interpreted mode) is not advisable in production systems which severely affect the performance unless you are running in a debug code or isolating problems related to JIT.

set -XX:+PrintCompilation to see when the methods are getting complied in the jvm stdout.