IBM WebSphere Commerce Websites Gallery

Here are some of the shopping websites created using IBM WebSphere Commerce Suite. You can find all sorts of top brand companies ranging from Electronics retailers, Garments retailers, government, etc.

www.hallmark.com	www.sonystyle.com
">www.abercrombie.com	www.basspro.com
shop.usps.com	www.lee.com
www.homedepot.ca	www.lenscrafters.com

Also you can search "/wcs/stores" keyword search in google to list all the websites developed using IBM Commerce Suite, provided they haven't changed the default context root.

Tips to get most out of IBM Support

I am sure many of you might already worked with IBM support to resolve problems related to the product , but here are some tips that you can follow to effectively reduce the problem resolution time and ease the process.

1) If you suspect if the problem is product related, start by opening a service request or PMR (Problem Management Record) either through online or by calling the IBM Support 1-800-IBM-SERV. You might need a IBM customer number.

2) Start with a simple description of the problem when you open a PMR.

3) Make sure you provide a clear description in such a way it represents the problem area so that your PMR gets assigned to the right team as there are several teams with different specialization within the same product , so that you will save time by getting the problem assigned to the correct team. If you know the team before hand then specify the team queue in the PMR.

4) Specify mode of communication in the PMR either by email or phone. If you think the problem is complex and need some explanation to do, then specify phone call back so that IBM support representative can call back and understand the problem, if you think the problem something trivial a little description in the PMR would make them understand then specify email.

5) Collect MustGather documents either manually or using http://www-01.ibm.com/software/support/isa/ IBM Support Assistant tool and then attach the the logs to PMR which will help in faster resolution to the PMR.

6) If possible, try to create a reproducible test case preferably outside to your custom application, so that IBM can use this test case to test it in the lab and provide a solution, Also it will also help to isolate that the problem is not related to your company's custom code or the environment.

7) Please use commonly used terminologies when updating the PMR or talking to IBM Support representative instead of the terminologies that you might use within your IT department as it might confuse the person handling the PMR as different companies use different ways to describe a problem. Clarify things if and when needed.

8) Specify the severity and priority in the PMR, Be reasonable and provide a business justification, set your expectations like when you need the next update and when the problem should be resolved, so that they can work towards it.

9) Please update the PMR with any new findings and updates as and when you come to know or also when you need progress updates from IBM.

10) Understand the escalation procedure and use it to escalate when things are not moving as reasonably expected.

11) Be friendly and appreciate the IBM Support representative work when problem gets resolved.

12) Follow the instructions from IBM even though it doesn't make sense sometimes , otherwise it would become a bottleneck to proceed to the next step in debugging the problem.

13) Besides doing all the above you can also self serve simultaneously by searching support knowledge base, infocenter , forums , etc, as there might be problems with an already known resolution.

Poll: On what OS your WebSphere AppServer is installed ?

Poll results for : On what OS your WebSphere AppServer is installed ? , held between Aug 16th 2009 to Sep 16th 2009 and voted my about 86 voters.





You can still continue to vote though:

Poll: On what OS your WebSphere AppServer is installed ?

IBM WebSphere related twitters

Here are some of the IBM WebSphere related twitters having interesting tweets on various topics related to IBM WebSphere Product Family.

IBM_WAS IBM WAS SupportIBM WebSphere Application Server support news, updates, and information.
IBM_Commerce IBM Commerce SupportIBM WebSphere Commerce support news, updates, and information.
IBM_WPS IBM WPS SupportIBM WebSphere Process Server support news, updates, and information.
IBM_Monitor IBM Monitor SupportIBM WebSphere Business Monitor Support page.
IBM_Modeler IBM Modeler SupportIBM WebSphere Business Modeler support.
WebSphereBUZZ WebSphere Buzz News.
WebSpherePortal WebSphere PortalYour location for interesting links associated with IBM WebSphere Portal.
WebSphereHelp WebSphere Help, Tips & Tricks.

Book on WebSphere Application Server 7.0 Administration Guide from Packt Publishing

WebSphere Application Server 7.0 Administration Guide from Packt Publishing written by Steven Robinson is one of the few books available on WebSphere 7 Administration. This book is very useful for Websphere 7 Administrators from mid-level to advanced skills to manage Websphere Application Server 7.0. This book covers all the topics from installation, configuration, monitoring to product maintenance. This book has a lot of well written examples with step by step instructions including screenshots of major aspects of server configuration. The chapters are easy to read and you should be able to complete the entire book in few weeks and become a complete WebSphere 7.0 Administration Expert. The book also talk about WebSphere Messaging in chapter 6 in detail on how to configure using the default or WebSphere MQ provider.I would definitely recommend this book. The book is available in both eBook and as a regular printed book. More details about the book can be found in publishers website

Slowing down your server process does have some benefits

Although lot people are finding various ways to speed up their application servers to improve performance, there are also benefits to slow down your server process especially when testing it to find problems that you won't see in normal scenarios. As you might have seen in your experience that 90% of the time things work well in a normal scenario when there the load on the server is normal, db response throughput is normal, network throughput is optimal , etc. But all of a sudden things start to break when something gets backed up like db fails to respond in an expected time leading to number of concurrent requests due to browser refresh by the user might break the applications due to deadlock or other concurrent related problems. Due to the availability of powerful multiprocessor severs nowadays , requests are executed in terms of milliseconds , hence it would be difficult to catch concurrency related problems unless you recreate the scenario where the requests are taking more time to process. Although one might say this can be simulated by doing a load test on the server, i agree but if there is a problem it would be a overwhelming task and extremely difficult for someone to debug given the amount of requests and objects created during the test, especially if there is a memory leak. Hence the same effect can be achieved by slowing down your server process with just using few concurrent requests which makes things easy to debug and trace.

For (e.g) check this below jsp lrumap.jsp, where it creates a simple LRUMap object, puts it into the session and then serializes it. You will not find any problems with the jsp as long as it runs quickly (usually takes about 10ms is a decent desktop server) with multiple concurrent requests, but if the code slows down due to I/O or some other reason , you will end up with the java.utl.ConcurrentModificationException if the code is executing the serialization part so.writeObject(lmap); and adding of the element lmap.put(lobj[i],new Integer(i)); at the same time even though the LRUMap is synchronizedm, the reason being the serialization code uses an iterator to iterate through the elements in which the iterator is not synchronized which caused the exception. Unless you make the server process to run slow you will never end up catching it until you see it in production during high load conditions where it will become extremely difficult to debug.



How to slow down a process ?

The trick is that you basically limit the CPU usage of the process which will slow down the process as it won't get enough CPU cycles to execute. There is a open source program in Linux CPU Limit can be used for this purpose or you can write a simple shell script where you can send STOP and CONT signal in a loop using the kill command. Although the later is not efficient but it works, but the CPU limit program is much precise even though it'd doing the same but through a C program.

1) Download and install the CPU Limit program.
2) If your System Administrator don't allow then use the shell script,

#!/bin/bash
while [ 1 ]
do
kill -STOP $1
usleep 100000
kill -CONT $1
done

Note using usleep can make the sleep in microseconds where as sleep will sleep in seconds which is not suitable for process executing transactions in milliseconds.

3) find the process id of the process you wanted to slow down.

4) Execute cpulimit -p -l

(e.g) cpulimit -p 11502 -l 10 - this command will allocate max of 10% CPU to the process 11502.

Running your own copy of production server instance in your desktop

Although the title seems little strange and scary I will explain to you in this article why it's needed, it's advantages and how you can setup easily without much changes. Being an Administrator and Support Engineer throughout my career i haven't seen an single I.T department having an exact replica of the production and testing/staging environment, there is always some difference between those environments like the most common is the amount of production data is not same as that of staging data which changes the equation of how the application behaves. There are situations where production support engineers won't even have access to staging/testing environments because it's either managed by a different vendor, or because of SOX compliance or due to various security reasons. Being an administrator , the application is kind of a black box compared to the developers where you might not have had much chance to play with it and understand how it really works, most of them you will come know from the word of mouth from the developers which sometimes may not be true in terms of how it's behaving in a production as they are aware only in terms how it worked in their development environment which is usually a standalone server compared to a usual cluster environment in production. During outages, you are more focused on bringing the server backup up instead of finding what caused the issue and most of the time it's too late or you don't have enough time to collect the data. Hence it is necessary in my opinion to have your own production copy , so that you can turn on traces , understand how the application works, understand the symptoms if some services breaks and find ways to fix it, so that when the real problem comes you will be prepared for it and tackle the problem quickly with little or no down time. Although many will disagree and you might want to explain the benefits which out weighs the problems.

You are basically going to run the server with a system proxifier like proxychains and tunneling all connections through the SSH proxy from your dekstop with ease without much modifications due to firewall restrictions. Here are some suggested steps,

1) Tar up your production server installation directory and untar it up in your desktop.

2) Make sure you create any directories or symbolic links that the server references that is not
included as a part of installation directory, like log and config directories.

3) Download and Install system proxifier proxychain .

4) Configure /etc/proxychains.conf to point to the socks server. (e.g) socks5 127.0.0.1 9050

5) If there are DNS name resolution issues follow the instructions Performing UDP tunneling through an SSH connection to setup local dns proxy server or enable proxy_dns property in /etc/proxychains.conf

6) Run a SSH proxy from your desktop , (e.g) ssh -D 9050 user@prod-servername.

7) start the server with proxychains or modify the startup script to include proxychains (e.g) proxychain /root/Desktop/jdk1.6.0_16//bin/java -Djava.util.logging.config.file=/opt/apache-tomcat-6.0.20/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/apache-tomcat-6.0.20/endorsed -classpath :/opt/apache-tomcat-6.0.20/bin/bootstrap.jar -Dcatalina.base=/opt/apache-tomcat-6.0.20 -Dcatalina.home=/opt/apache-tomcat-6.0.20 -Djava.io.tmpdir=/opt/apache-tomcat-6.0.20/temp org.apache.catalina.startup.Bootstrap start
you can get the process string by doing ps -auxww after you normaly started the server. Now your server should be able to start and able to connect through the SSH proxy and proxychains output will show how the connections are made ,
(e.g)
ProxyChains-3.1 (http://proxychains.sf.net)
S-chain-<>-127.0.0.1:9050-<--timeout INFO: Mon Aug 31 22:37:41 PDT 2009: PlatformDetector detected platform: tomcat
S-chain-<>-127.0.0.1:9050-<>--10.23.23.11:1521--<>

8) website's domain/virtualhost mapping to your local ip to access the website in /etc/hosts. (e.g) www.example.com 127.0.0.1

IBM WebSphere Portal in Real-World Cloud Computing

Since IBM and Amazon announced the avalability of WebSphere Portal Server and Lotus Web Content Management Standard Edition on the Amazon EC2 Web Service, I used to wonder whether there are any real-world customers using it. But now it seems from the news, that an investment firm Quintana Capital Group converted its Web site and IBM WebSphere-powered portal to Amazon's EC2. Here is the URL http://www.qeplp.com/wps/portal/ of their portal powered by IBM WebSphere Portal on the Amazon EC2.


Also you can run a traceroute to confirm that it's hosted in EC2.

>tracert www.qeplp.com
100 ms 98 ms 97 ms ec2-174-129-234-118.compute-1.amazonaws.com [174.129.234.118]

WebSphere Java process hangs and freezes

We recently had an issue where the websphere Java process got hung and freezes in 4 servers almost at the same time, where 3 server nodes are part of a cluster and the other one is a standalone. Restarting of websphere AppServer fixed the issue. This issue was still puzzling as to why all the servers got hung at the same time and even the one that is not part of the cluster got hung as well. We did some investigation and found the commonality among all these servers is that all the websphere installation directory is nfs mounted on a NAS (Network Attached storage) device. We suspected that either nfs mount or the NAS might have had problems as there was no better explanation for all the server to go down at the same time. We checked the OS /var/log/messages file and found these nfs service messages happened around the same time the server went down ,

Aug 20 04:09:45 appserver01 kernel: nfs: server nasserver01 OK
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:53 appserver01 kernel: nfs: server nasserver01 OK

These messages seems to be related to nfs timeout. As there were no problem with the NAS device itself , it was clear that nfs service was timing out might have caused the issue. We changed the nfs to use the TCP and nfs version 3 which is more reliable instead of UDP with some additional tuning parameters. Once remounting with new parameters the problem didn't happen so far. Here are the new setting for the nfs mount over TCP.

/etc/fstab:

nasserver01:/app/WebSphere /mnt/WebSphere (rw,noatime,hard,intr,tcp,nfsvers=3,retrans=5,rsize=8192,wsize=8192,timeo=14,addr=10.10.1.20)

In case if the problem still exists after the tuning , nfsstat or tcpdump traces can be used to analyze the problem.

Is Java really "Write once, run anywhere" ?

As many of us generally know, Java is popular for it's cross-platform portability "Write once, run anywhere", but i wanted to give it a test and see if it's truly one. I tried to run WebSphere 7 Application Server itself using Sun JRE 1.6.0 instead of IBM J9 VM which is bundled with AppServer and see if it works with cross vendor JVM on the same platform. I had to make couple of changes, the startServer.sh script adds IBM JVM specific arugments ( -Xshareclasses:name=webspherev70_%g,groupAccess,nonFatal -Xscmx50M )which i had to remove , changed the WAS_HOME/java to point to Sun's JAVA_HOME and set the environment variables as set in setupCmdline.sh and startServer.sh and ran the process from the cmdline with the huge list of arguments with the one i got from the process string when it's started from the startServer.sh script.

The AppServer failed to start with the following exception,

java.lang.NoClassDefFoundError: com/ibm/wsspi/buffermgmt/WsByteBufferPoolManager
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)

and

[20:48:03:436 PDT] 0000000a ServerCollabo E WSVR0121E: An exception occurred getting a socket for port 38930 on hostname waslinux with an IP address of 192.168.1.10.

I added the the AppServer/plugins/* directory to the classpath and even to -Djava.ext.dirs, but this time the JVM just exits and getting terminated without writing anything into the logs. I tried different things to find the actual root of the failure by enabling verbose:jni traces, strace and using jdb, but couldn't find anything useful and ultimately gave up. Note the same method of starting websphere using IBM J9 VM from the cmdline seems to work. Both Sun JDK and Open JDK just terminates for no reason.

I also ran apache tomcat server on both IBM's and Sun JVM and it seems to run fine without problems.

It seems like WebSphere AppServer java code is not compatible to run in other vendor JVM's, hence the promise of Java hasn't come true at least in this case where it's just "write once , run anywhere as long as you stick to the same JVM vendor that you used to develop and test " :) .


*If anyone had tried and been successful please comment on my post, I would really like to run and see as i can use some of the tools like jvisualvm, jmap, jps, jstack , etc which gets bundled with Sun JDK and not with IBM.

Problem running startxwin.bat in Cygwin/X on windows

Nowadays more and more graphical tools like jconsole , jvisualvm are shipped with java and several other tools like tda-Thread Dump Analyzer , IBM HeapAnalyzer , etc are available for download to analyze and debug problems, hence it seems like you need to have some kind of graphical terminal like Xserver or VNC to manage your environment. Since most of the production environments are UNIX/Linux based and more commonly connected through windows desktop, you might need to have a windows based Xserver or VNC client. VNC Servers are not very common in the enterprise as they deemed to be insecure compared to X and needed additional installation and configuration while X comes with the OS. One such windows based open source free XServer is Cygwin/X which i decided to give it a try and ran into some problems before i made it to work, hence i wanted to write about it to avoid the same problem by someone else. The installer is little different where it lists you different packages of Cygwin along with Cygwin/X packages so you just need to select X11 if that's what you needed.

Once installed you just have to goto the c:\cygwin\bin in a cmd prompt and run startxwin.bat ,which should start the Xserver, but i my case the batch file was giving some issues,

startxwin.bat - Starting on Windows NT/2000/XP/2003'c:\cygwin\bin\run' is not recognized as an internal or external command,operable program or batch file.'c:\cygwin\bin\run' is not recognized as an internal or external command,operable program or batch file.

So in order to make it work i had to change the one line in the startxwin.bat file as shown below,

SET RUN=%CYGWIN_ROOT%\bin\run -p /usr/bin

replace with,

SET RUN=run -p /usr/bin

Now you should be able to connect with your favorite SSH client like putty or secureCRT with X11 Forwarding.

WebSphere Education videos on youtube.com

I found this WebSphere Education channel on youtube.com, thought i should publish it to my blog. The channel doesn't seem to be more active as i can see videos that were posted like 6 month old and not anything new , not sure if IBM had changed it's policy in publishing it to youtube.com. You can find some useful videos ranging on different topics in IBM WebSphere product family like Administration of IBM WebSphere Process Server, IBM WebSphere Application Server V6.1 Administration, IBM WebSphere Commerce Installation and Configuration and so on. You can also subscribe to the channel so that whenever a new education session is posted you will be notified.

Using OpenSSL tool to check SSL certificates for expiration dates

I am sure many of the Application or System administrators might have encoutered the issue of SSL certificates getting expired in the middle of the day causing application outages. The part of the reason is that the application server environments are getting complex day by day in terms of number of systems it's interacting over SSL like LDAP, WebServices, WebSever plugin, Siebel and even database connectivity in highly secured environments. It further complicates as these are disparate systems maintained by different groups in the enterprise having different expiration dates and different formats of keystore and truststore. So unless you have good enterprise wide policy of checking SSL expiration dates well ahead, there is a very good chance that you might run into application failures.

Being an Websphere Application Administrator, you will be the first point of contact when application goes down, even the SSL certs expired in one of the system that it's intreacting with like the LDAP wich is used for security authentication. In order to diagnose , identify and notify the repsective system owner, openSSL tool comes handy in such a way that you just need to know the hostname and the port where the services SSL port is listening on and you will be able to find the expiration dates without having to know the password for the keystore or the format or on how to access and view the keystore. The tool usually gets installed by default in unix systems and in windows you can download and install it.


Steps to check the SSL Certificate expiratio using openSSL tool:

1) openssl s_client -connect hostname:port > cert - this command will get the certificate and redirect it to the file.
2) openssl x509 -in cert -noout -enddate - show the expiration date of the downloaded certificate.


(e.g) To check the expiration for www14.software.ibm.com webserver host

$openssl s_client -connect www14.software.ibm.com:443 > cert

Loading 'screen' into random state - done
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0

$openssl x509 -in cert -noout -enddate

notAfter=Dec 8 13:00:22 2009 GMT

Note not only this works with webserver it will work with any service with SSL turned on.

You can also check other information about the certificates without redirecting to the file. Also you can check my other article for SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore as well.

$openssl s_client -connect www14.software.ibm.com:443

Loading 'screen' into random state - done
CONNECTED(000006DC)
depth=1 /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/C=US/ST=New York/L=Armonk/O=IBM/CN=www14.software.ibm.com
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
1 s:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC1TCCAj6gAwIBAgIDCiE+MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDgxMTA3MTMwMDIyWhcNMDkxMjA4MTMwMDIy
WjBgMQswCQYDVQQGEwJVUzERMA8GA1UECBMITmV3IFlvcmsxDzANBgNVBAcTBkFy
bW9uazEMMAoGA1UEChMDSUJNMR8wHQYDVQQDExZ3d3cxNC5zb2Z0d2FyZS5pYm0u
Y29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC4tUCL4Ar8Sx8pNEbRb+XQ
7CMa1fgh1k2p0zucagZeX/C8e3DMefU5UH3pJKjYZoYte6PWB92jwtw673Rj0Oq6
oDBljHbdtYo0H58C6QqSkjMLFEKyNBHOAZ1SGws11H5WCz+xiOZ1c5Xq11Jjl/BM
nu6PKbfKUUrBwIdTfK/heQIDAQABo4GuMIGrMA4GA1UdDwEB/wQEAwIE8DAdBgNV
HQ4EFgQUuB0DNKiqXdoH87tWzq+71OYK7FUwOgYDVR0fBDMwMTAvoC2gK4YpaHR0
cDovL2NybC5nZW90cnVzdC5jb20vY3Jscy9zZWN1cmVjYS5jcmwwHwYDVR0jBBgw
FoAUSOZo+SvSspXXR9gjIBBPM5iQn9QwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG
AQUFBwMCMA0GCSqGSIb3DQEBBQUAA4GBAExSdcrIMiwQqDOAelP4C6QiUeXgis2g
8ePc17lrt2NjRe1aRoSmMYq2U7Rc3203L3WFQ/SOFzfUSKWwBjx98IPFGOuKTMza
DYe1xknf+jjg8IwzhAtcg9Y06jabJBwxMXebSH2lazqFSD8ztBedBjLM7/zQ/Ttz
a7e4t98R0/fX
-----END CERTIFICATE-----
subject=/C=US/ST=New York/L=Armonk/O=IBM/CN=www14.software.ibm.com
issuer=/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
No client certificate CA names sent
---
SSL handshake has read 1688 bytes and written 322 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 00038028A696989D0973A59DEF091113DCCB885C585858584A80A84C0000298D
Session-ID-ctx:
Master-Key: 6A694E8816CE422DB3AD280BEC469ACBDBE1EB7BF116C5E3C600A1A68CC71B7864AE39D8A59CC1F07263C1AB1664238D
Key-Arg : None
Start Time: 1249945678
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)

java.util.ConcurrentModificationException during serialization of a synchronized LRUMap

As many of you might now already that LRUMap is not synchronized and is not thread-safe. If you wish to use this map from multiple threads concurrently, you must use appropriate synchronization. The simplest approach is to wrap this map using Collections.synchronizedMap(Map). But even after using this method lrumap=Collections.synchronizedMap(new LRUMap(10)) , particularly when serializing the Map if multiple threads are using it it throws the java.util.ConcurrentModificationException as shown below even though it's synchronized.

It looks like when serializing the Map attribute the map is iterated through a SequencedHashMap$OrderedIterator object to get each entry from the map and then serialize it. Even the LRUMap in this case is a synchronized Map but the iterator isn't, hence it is imperative that the user manually synchronize on the map when iterating over any of its collection views, not sure if this was missed in the serialization code of LRUMap. But if you use synchronized HashMap this problem doesn't seem to happen. Be cautious of using this LRUMap like adding to a HttpSession as an session attribute where serialization happens in a high availability cluster when session bounces between nodes and that you might run into the problem and might loose some session data. Note i was using commons-collections-2.1.jar, i didn't check if this was fixed in the latest releases.

java.util.ConcurrentModificationException
at org.apache.commons.collections.SequencedHashMap$OrderedIterator.next(Unknown Source)
at org.apache.jsp.serializemap_jsp._jspService(serializemap_jsp.java:97)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Jul 31, 2009 3:16:52 PM org.apache.catalina.core.StandardWrapperValve invoke

Poll: Which is the popular J2EE Application Server ?

Poll results for the Poll: Which is the popular J2EE Application Server ? , held between March 6th 2009 to Aug 6th 2009 and voted my about 90 voters.






You can still continue to vote though:

Which is the popular J2EE Application Server ?

WebSphere Portal Websites Gallery

Here are some of the websites created using WebSphere Portal. You can find all sorts of companies ranging from telecom, retail, government, etc.

corporate.homedepot.com	www.alcatel-lucent.com
www.usda.gov	www.ncaa.org
kb.palm.com	Fifth Third Bank
www.3m.com	portal.honeywell.com
www.gapinc.com	Air France Cargo
www.alltel.com	www.labcorp.com

Also you can search "/wps/portal" keyword search in google to all the websites provided they haven't changed the context root.

Verify firewall port assignments using netcat during WebSphere installation planning

During production installation of WebSphere, particularly where multiple node cluster is involved you will be overwhelmed with how many firewall changes need to be made across different vlans and open ports to make sure all the nodes, dmgr, websevers, databases, & ldap can communicate with each others without problems. Also at the same time make sure you only open the ports that are needed for security reasons. ACL firewall rules would become complex due to the different number of ports involved in WebSphere , by default on a single standalone node installation you can see below that there are about 18 ports or so involved,

Port Name Port
-------------------------------------------------------------------
BOOTSTRAP_ADDRESS 2809
SOAP_CONNECTOR_ADDRESS 8880
ORB_LISTENER_ADDRESS 9100
SAS_SSL_SERVERAUTH_LISTENER_ADDRESS 9401
CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS 9403
CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS 9402
WC_adminhost 9060
WC_defaulthost 9080
DCS_UNICAST_ADDRESS 9353
WC_adminhost_secure 9043
WC_defaulthost_secure 9443
SIP_DEFAULTHOST 5060
SIP_DEFAULTHOST_SECURE 5061
SIB_ENDPOINT_ADDRESS 7276
SIB_ENDPOINT_SECURE_ADDRESS 7286
SIB_MQ_ENDPOINT_ADDRESS 5558
SIB_MQ_ENDPOINT_SECURE_ADDRESS 5578
IPC_CONNECTOR_ADDRESS 9633

So in order to make sure all firewall port assignments are proper before installing you need to connect to the listening port and check from the client. Since you don't have websphere actually installed those ports won't be listening and makes it difficult to check and verify. In order to verify you can use the netcat or nc utility in unix or linux. Basically with nc or netcat utility you can listen on any partucular port TCP or UDP and make connections from the client and verify that these port communication is opened through the firewall and resolve your network issues quickly.

Use netcat or nc to listen on port 9080,
(e.g) nc -l 9080

and then you can connect to the port from another client machine using nc or any other tool like telnet to see if it accepts connection,

(e.g) nc 192.169.1.1 9080
you can type on the stdin which will be transferred and displayed across each machine, form that you can know that ports are opened.

Also use nc to port scan as well, (e.g) nc -v -z 192.168.1.1 9080-9084
the output will showing what ports are opened as below,
hostname [192.168.1.1] 9080 (?) open

How to find JVM is 32bit or 64bit ?

You might think it's something simple like typing java -version to find , yes it is for most Java versions, except for Sun JDK 32-bit version doesn't explicitly say which is causing confusion to many users which kind of prompted me to write this article.

If you run java -version in Sun HotSpot 1.6 VM , you will see the below output showing nothing about the bit on the 32-bit JVM where as in 64-bit JVM it explicitly says. so based on this you can almost be kind of sure it's a 32-bit JVM when no bit information shows up in the output.

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)

or
java -version -server from the jdk directory

java version "1.6.0"Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)

Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b16, mixed mode)

Whereas on the other hand IBM JDK versions seems to explicitly say what type of JVM it is for both 32-bit as well as 64-bit.

Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)

Also when you try to run with 64-bit model on the 32-bit JVM it might complain as below,

java -d64 -server -version
Running a 64-bit JVM is not supported on this platform.

Also you can look at the lib directory of the jre and see if there is any subdirectory ends with 64 like /opt/jdk1.6.0_11/jre/lib/amd64/ confirming that it's a 64-bit jvm. On a running jvm run Linux command: lsof -p pid of java and see where the libjvm.so process are loaded, which might indicate the type of JVM.

Note there is no such thing as WebSphere Application Server 64-bit or 32-bit version , basically it depends on what kinf of JVM (32-bit 0r 64-bit) you are running on.

Maximum heap size limit of java is smaller than you think

You might think that on a 32-bit OS, a process should be able to address address 2^32 = 4Gb of address space, however in practice some of the address space is used by the OS kernel and so is not available to the process. So there are limitations to how much a process can address and it can vary depending on the platform and the versions of JDK. There are programs like IBM Heap Analyzer requires large amount of memory while analyzing heap dumps taken from JVM with setting of max heap size of -Xmx 1400M(1.4GB) or more and while analyzing, your jvm might very well run out of memory, hence you might needed a 64-bit OS running 64-bit JVM to even analyze a heap dump taken in a 32-bit JVM. Here below is the table lists the max heap size a JVM can take in the respective platform and the different versions.

tr style="HEIGHT: 135pt">

S.No	JVM Version	OS	Bit	Xmx Max Heap Size	Error When JVM not able to allocate memory
1	Java(TM) SE Runtime Environment (build 1.6.0-b105) / Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)	Windows XP	32 bit	1612M	Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine.
2	Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2) Classic VM (build 1.4.2, J2RE 1.4.2 IBM Windows 32 build cn1420-20040626 (JIT enabled: jitc))	Windows XP	32 bit	1635M	[ Unable to allocate an initial java heap of 1724907520 bytes. ] [ **Out of memory, aborting** ] [ ] [ *** panic: JVMST016: Cannot allocate memory for initial java heap ] This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
3	Java(TM) SE Runtime Environment (build pwi3260sr2-20080818_01(SR2)) IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 Windows XP x86-32 jvmwi3260-20080816_22093 (JIT enabled, AOT enabled)	Windows XP	32 bit	1813M	JVMJ9VM015W Initialization error for library j9gc24(2): Failed to instantiate heap; 1814M requested Could not create the Java virtual machine.
4	java version "1.6.0_13"Java(TM) SE Runtime Environment (build 1.6.0_13-b03)Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)	Linux	32 bit	2678M	Error occurred during initialization of VMCould not reserve enough space for object heapCould not create the Java virtual machine.

JIT Compilation of Java code won't happen before 10000 invocations of the same code block

With the default settings of the Java 1.6 HotSpot VM running in a -server mode don't expect your java code to be compiled before 10000 invocations of same code block or the method. This may not be suitable for servers where your server load is less and might take a long time to reach 10000 invocations or where you have an environment with multiple cells and the cell gets flipped every day before reaching the threshold or the servers are getting restarted frequently. In that case you might loose the performance, where the bytecode not being compiled at all. So based on your requirements you can set a lower value of compile threshold from 10000 to get the code compiled faster and boost performance within a short period of time after the restart of the server.

Set the -XX:CompileThreshold=1500 to a lower value in the JVM option and see how your code performs and tune accordingly. Note it's not advisable to set the value below 1000 or -Xcomp (always force JIT compile) as the JVM won't even have enough profiling information before it can generate optimized code and might actually performance degradation rather than improvement. Also note running the JVM with -Xint (interpreted mode) is not advisable in production systems which severely affect the performance unless you are running in a debug code or isolating problems related to JIT.

set -XX:+PrintCompilation to see when the methods are getting complied in the jvm stdout.

Aged Timeout connection pooling setting in websphere might shrink the pool size below the specified Minimum Connection setting

If you have used websphere datasources you might have seen a jdbc connection pool setting called Aged Timeout specifies the interval in seconds before a physical connection is discarded irrespective of whether the connection was idle or used before. This property is usually set to a specified interval to maintain the freshness of the connection between the database and also to avoid any firewall sitting between your application and database terminate the connection after staying idle for a period of time, which most firewall does. Other instance where this property
might be useful is if you have an Oracle RAC (Real Application Clusters) kind of environment where you wanted to do some kind of rolling upgrade or scheduled maintenance across different nodes in the cluster , in that case DBA's might want to bring down each node gracefully after all connections are drained from the node instead of shutting down the node and terminate the connections abruptly. so this property comes in handy that after a specified time interval in the aged timeout property the connections are closed and when needed it gets created again where at this point the connections are established to a different node based on the Oracle RAC configuration change allowing them to redirect the connection to node other than the node scheduled for downtime. This is all good based on the requirements but the downside to this is the connection manager will simply close the connection based on the aged timeout when the time expires and will not recreate the connection even if the pool is below the minimum connection settings that will caused the pool to go shrink below the expected minimum size which might impact the performace of the application at certain times. Theoretically speaking you don't need to set a aged timeout particulary if you are using Oracle RAC with properly implemented high availability techniques either using Fast Connection Failover or Transparent Application Failover where tthese are totally capable to handle failover in case of an scheduled/plan outage or unplanned outage, but DBA's sometimes seems to be overly cautious, wanted application administrator to enforce these settings. There may be environments where you will be using Oracle RAC but none of those failover mechanisms been implemented and in that case insist you DBA's to implement any one of those failover mechanism.Note if you are using tomcat default's commons dbcp connection pooling there isn't a aged timeout parameter in the connection pooling settings, then you might have to rely on the database failover features.If you are using c3po another popular connection pooling datasource there is a maxConnectionAge property where it will close the connection after the specified interval and also create a connection at the same time if necessary in order to not to shrink the pool below the specified minimum pool size.

VisualVM remote application monitoring through SSH proxy

As many of you might already know Java VisualVM an all in one Java troubleshooting tool is available as a JDK tool in Sun JDK distributions starting from JDK 6 update 7 and Apple's Java for Mac OS X 10.5 Update 4. Also VisualVM a stand alone separately downloadable tool is also available at visualvm.dev.java.net. It's a visual tool enhances the capability of performance and monitoring analysis for Java SE platform for both production and development environments. You can find more detailed information in the VisualVM product site.

But our topic is how to make visualvm work through SSH Tunnel specifically in production systems where it's a normal practice to allow only SSH access on port 22 for the users or administrators, hence visualvm will not work with the default settings as it uses port 1099 to communicate to remote host via jstatd. For some reason, VisualVM doesn't seem to work through SSH Port Forwarding or SSH Proxy which seems to be a big blocker for many people which wish yo use this tool in this similar kind of setup.

The only way i came up so far to make the VisualVM work is to use to use a system proxifier like WideCap (Free) or ProxyCap (Shareware) and was able to connect to remotely a running java process over ssh proxy. Here are the steps detailing on how to configure ,

1) Download WideCap and install.
2)Open WideCap control panel and click on new proxy , provide hostname as localhost, port
8333, and select the socks version of your SSH that your client and server uses.





3)Add a new application and point to the location of java.exe and create new rule to get triggered when the java is started.




4) Enable WideCap.
5) run putty or any ssh client as SSH proxy daemon. (e.g) putty -D 8333 username@server or ssh -D 8333 user@host
6) Start VisualVM and add remote host and now you should be able to see the java process running on the remote server and you should be able to connect and monitor.



On a cautious note that widecap is not very consistent and it crashes outlook (at least for me ) and IE sometimes but on the other hand proxycap seem to work fine along with other standard windows applications without problems.

Also if you run into problems make sure all the instructions are followed in this link

WebSphere Portal Interview Questions for an Administrator

List of interview questions that i might ask someone for the job of WebSphere Portal Server Administrator. What is the difference between Application Server and Portal Server ? Application servers extend the ability of a Web server to handle Web application requests, and enables a server to generate a dynamic, customized response to a client request. A portal server extends the application server by providing a portlet container that can run portlets and to create portal a website that provides users with a single point of access to Web-based resources by aggregating those resources in one place. What are the steps involved in deploying themes and skins in a clustered production websphere portal environment ? Export the WebSphere wps.ear (Portal EAR) using wsadmin. Use EarExpander tool to expand the exported wps.ear file. Copy the updated themes and skins into ../themes/html, ../skins/html folder. Use EarExpander tool to collapse the EAR directory into an EAR file. Use wsadmin to update the wps.ear to complete the deployment of updated themes and skins. What changes needs to be done to view changes to your theme and skins JSPs without restarting the portal server ? You need to enable automatic JSP loading by setting reloadingEnable property to true in ibm-web-ext.xmi file of the wps.ear. What are the 3 different ways of installing a portlet application in WebSphere Portal ? Install a portlet using the portal administration page using Web Modules portlet. Install a portlet using xmlaccess tool. Pre-deploy a portlet as a standard EAR by installing the portlet WAR file in WAS console and then registering the portlet using xmlaccess. What is the purpose of XMLAccess configuration file Export.xml & ExportRelease.xml ? What is the difference ? & When will you use one over the other ? Export.xml exports the complete portal configuration and useful when transferring configurations between development installations. ExportRelease.xml exports the complete portal configuration from the release domain as required by the portal ReleaseBuilder tool and useful when tranaferring different release configurations between staging and production environments. List me the steps involved in building a release in WebSphere Portal ? If you have a completely new installation of the staging server and the production server: Install the staging server, then install the production server. Develop a release on the staging server. Build the release on the staging server. Empty portal contents on the production server by running the WPSconfig.sh|bat action-empty-portal task. Import that release onto the production server. Refer to Transferring a complete configuration for information. If you already have a production server without a staging system: Export the release of your production server. Install an empty staging server using one of the following two methods: Install the staging server with the flag -W emptyPortal.active=True. After installing and configuring the staging server, run the WPSconfig.sh|bat action-empty-portal task. Import the production release onto the staging server. Refer to Transferring a complete configuration for information. Develop and build a new release on the staging server. Export that new release from the staging server. Use ReleaseBuilder to generate the differential between the two releases. Import the differential onto the production server. What is the purpose of ReleaseBuilder tool in WebSphere Portal ? ReleaseBuilder enables management of release configurations independent of user configurations and used during staging of follow-on releases of WebSphere portals, configurations, and artifacts need to be moved between systems. What are the steps involved in editing WebSPhere Member Manager (wmm.xml) files on a federated node ? On the primary node of the WebSphere Portal cluster, check out the files using ./WPSconfig.sh check-out-wmm-cfg-files-from-dmgr task. Make any changes to the Member Manager files. The files can be edited in the portal_server_root/wmm directory on the WebSphere Portal node. When you have completed your changes, check the files back in using ./WPSconfig.sh check-in-wmm-cfg-files-to-dmgr. How to change the default portal URI /wps/portal after installation ? Set the property WpsContextRoot to the new URI and run ./WPSConfig.sh modify-servlet-path task. List different types of user registry supported by WebSphere Portal ? LDAP (includes LDAP with an optional database user registry). Database user registry. Custom User registry, (non-LDAP, non-database). What is LDAP realm support and why would I want to use it? A Realm allows you to group users from one or more LDAP trees of one user registry and expose them as a coherent user population to WebSphere Portal; this is also referred to as horizontal partitioning. Realms allow a flexible user management with various configuration options; for example, you can combine principals from one or more corporate LDAP tree. A realm must be mapped to a Virtual Portal to allow the realm's defined user population to login to the Virtual Portal. What is an Application group and why would I want to use it ? Application groups is a concept that allows you to define user groups within the database user registry with members (users or groups) contained in the LDAP user registry you configured. The benefit of application groups is that you can create Groups that are only used in WebSphere Portal particularly in scenarios where there is Read-only LDAP or special group setup specific to the portal. What are the two methods to install the empty staging WebSphere portal server ? Empty portal contents on the staging server by running the ./WPSconfig.sh action-empty-portal. Install portal with the ./install.sh -W emptyPortal.active="True" option. How do you stop portal traffic to the node being upgraded in 24x7 cluster upgrade process ? Locate the cluster member being upgraded, and change the value in the Configured weight column from a value to zero and make sure the plugin config information is propagated to webserver to stop traffic. How to set limits on searches for users and groups ? Setting a maximum number of search results maximumSearchResults="200" in wmm.xml What portal resources are scoped for virtual portal ? Portal pages. Portlet instances. Portal Search Engine search services and search collections. This includes the search content sources. What portal resources can't be seperated for virual portal ? Themes and skins. Vault segments and vault slots. Supported clients and markups. Composite applications and templates. Policies. How do you enable temporary and extended trace logging for WebSphere Portal ? Temporary traces can be set for a temporary period by using the administration portlet Enable Tracing or the IBM WebSphere Application Server administrative console and also using the Enable Tracing portlet in the portal administration page. To enable extended trace settings for a longer period of time, that is, for more than one session, switch them on in the WebSphere Application Server configuration, save the updates and restart the portal server. What are the different states of the syndication process ? Idle: No syndication is occurring. Pending: A request has been made to the syndicator, but it has yet to initiate a request to the syndication application. Queued: The syndicator has sent a request to the syndication application, but syndication is not yet active. Active: Syndication is occurring between the syndicator and subscriber. Disabled: Syndication is currently disabled. What are the two types of rendering portlets ? Local rendering portlet and Remote rendering portlet. You can also find answers to all of the questions in the WebSphere Portal 6.0 Information Center or check the WebSphere Portal Books