IBM WebSphere Commerce Websites Gallery

Here are some of the shopping websites created using IBM WebSphere Commerce Suite. You can find all sorts of top brand companies ranging from Electronics retailers, Garments retailers, government, etc.

www.hallmark.com	www.sonystyle.com
">www.abercrombie.com	www.basspro.com
shop.usps.com	www.lee.com
www.homedepot.ca	www.lenscrafters.com

Also you can search "/wcs/stores" keyword search in google to list all the websites developed using IBM Commerce Suite, provided they haven't changed the default context root.

Tips to get most out of IBM Support

I am sure many of you might already worked with IBM support to resolve problems related to the product , but here are some tips that you can follow to effectively reduce the problem resolution time and ease the process.

1) If you suspect if the problem is product related, start by opening a service request or PMR (Problem Management Record) either through online or by calling the IBM Support 1-800-IBM-SERV. You might need a IBM customer number.

2) Start with a simple description of the problem when you open a PMR.

3) Make sure you provide a clear description in such a way it represents the problem area so that your PMR gets assigned to the right team as there are several teams with different specialization within the same product , so that you will save time by getting the problem assigned to the correct team. If you know the team before hand then specify the team queue in the PMR.

4) Specify mode of communication in the PMR either by email or phone. If you think the problem is complex and need some explanation to do, then specify phone call back so that IBM support representative can call back and understand the problem, if you think the problem something trivial a little description in the PMR would make them understand then specify email.

5) Collect MustGather documents either manually or using http://www-01.ibm.com/software/support/isa/ IBM Support Assistant tool and then attach the the logs to PMR which will help in faster resolution to the PMR.

6) If possible, try to create a reproducible test case preferably outside to your custom application, so that IBM can use this test case to test it in the lab and provide a solution, Also it will also help to isolate that the problem is not related to your company's custom code or the environment.

7) Please use commonly used terminologies when updating the PMR or talking to IBM Support representative instead of the terminologies that you might use within your IT department as it might confuse the person handling the PMR as different companies use different ways to describe a problem. Clarify things if and when needed.

8) Specify the severity and priority in the PMR, Be reasonable and provide a business justification, set your expectations like when you need the next update and when the problem should be resolved, so that they can work towards it.

9) Please update the PMR with any new findings and updates as and when you come to know or also when you need progress updates from IBM.

10) Understand the escalation procedure and use it to escalate when things are not moving as reasonably expected.

11) Be friendly and appreciate the IBM Support representative work when problem gets resolved.

12) Follow the instructions from IBM even though it doesn't make sense sometimes , otherwise it would become a bottleneck to proceed to the next step in debugging the problem.

13) Besides doing all the above you can also self serve simultaneously by searching support knowledge base, infocenter , forums , etc, as there might be problems with an already known resolution.

Poll: On what OS your WebSphere AppServer is installed ?

Poll results for : On what OS your WebSphere AppServer is installed ? , held between Aug 16th 2009 to Sep 16th 2009 and voted my about 86 voters.





You can still continue to vote though:

Poll: On what OS your WebSphere AppServer is installed ?

IBM WebSphere related twitters

Here are some of the IBM WebSphere related twitters having interesting tweets on various topics related to IBM WebSphere Product Family.

IBM_WAS IBM WAS SupportIBM WebSphere Application Server support news, updates, and information.
IBM_Commerce IBM Commerce SupportIBM WebSphere Commerce support news, updates, and information.
IBM_WPS IBM WPS SupportIBM WebSphere Process Server support news, updates, and information.
IBM_Monitor IBM Monitor SupportIBM WebSphere Business Monitor Support page.
IBM_Modeler IBM Modeler SupportIBM WebSphere Business Modeler support.
WebSphereBUZZ WebSphere Buzz News.
WebSpherePortal WebSphere PortalYour location for interesting links associated with IBM WebSphere Portal.
WebSphereHelp WebSphere Help, Tips & Tricks.

Book on WebSphere Application Server 7.0 Administration Guide from Packt Publishing

WebSphere Application Server 7.0 Administration Guide from Packt Publishing written by Steven Robinson is one of the few books available on WebSphere 7 Administration. This book is very useful for Websphere 7 Administrators from mid-level to advanced skills to manage Websphere Application Server 7.0. This book covers all the topics from installation, configuration, monitoring to product maintenance. This book has a lot of well written examples with step by step instructions including screenshots of major aspects of server configuration. The chapters are easy to read and you should be able to complete the entire book in few weeks and become a complete WebSphere 7.0 Administration Expert. The book also talk about WebSphere Messaging in chapter 6 in detail on how to configure using the default or WebSphere MQ provider.I would definitely recommend this book. The book is available in both eBook and as a regular printed book. More details about the book can be found in publishers website

WebSphere Java process hangs and freezes

We recently had an issue where the websphere Java process got hung and freezes in 4 servers almost at the same time, where 3 server nodes are part of a cluster and the other one is a standalone. Restarting of websphere AppServer fixed the issue. This issue was still puzzling as to why all the servers got hung at the same time and even the one that is not part of the cluster got hung as well. We did some investigation and found the commonality among all these servers is that all the websphere installation directory is nfs mounted on a NAS (Network Attached storage) device. We suspected that either nfs mount or the NAS might have had problems as there was no better explanation for all the server to go down at the same time. We checked the OS /var/log/messages file and found these nfs service messages happened around the same time the server went down ,

Aug 20 04:09:45 appserver01 kernel: nfs: server nasserver01 OK
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:51 appserver01 kernel: nfs: server nasserver01 not responding, still trying
Aug 20 04:10:53 appserver01 kernel: nfs: server nasserver01 OK

These messages seems to be related to nfs timeout. As there were no problem with the NAS device itself , it was clear that nfs service was timing out might have caused the issue. We changed the nfs to use the TCP and nfs version 3 which is more reliable instead of UDP with some additional tuning parameters. Once remounting with new parameters the problem didn't happen so far. Here are the new setting for the nfs mount over TCP.

/etc/fstab:

nasserver01:/app/WebSphere /mnt/WebSphere (rw,noatime,hard,intr,tcp,nfsvers=3,retrans=5,rsize=8192,wsize=8192,timeo=14,addr=10.10.1.20)

In case if the problem still exists after the tuning , nfsstat or tcpdump traces can be used to analyze the problem.

WebSphere Education videos on youtube.com

I found this WebSphere Education channel on youtube.com, thought i should publish it to my blog. The channel doesn't seem to be more active as i can see videos that were posted like 6 month old and not anything new , not sure if IBM had changed it's policy in publishing it to youtube.com. You can find some useful videos ranging on different topics in IBM WebSphere product family like Administration of IBM WebSphere Process Server, IBM WebSphere Application Server V6.1 Administration, IBM WebSphere Commerce Installation and Configuration and so on. You can also subscribe to the channel so that whenever a new education session is posted you will be notified.

Poll: Which is the popular J2EE Application Server ?

Poll results for the Poll: Which is the popular J2EE Application Server ? , held between March 6th 2009 to Aug 6th 2009 and voted my about 90 voters.






You can still continue to vote though:

Which is the popular J2EE Application Server ?

Verify firewall port assignments using netcat during WebSphere installation planning

During production installation of WebSphere, particularly where multiple node cluster is involved you will be overwhelmed with how many firewall changes need to be made across different vlans and open ports to make sure all the nodes, dmgr, websevers, databases, & ldap can communicate with each others without problems. Also at the same time make sure you only open the ports that are needed for security reasons. ACL firewall rules would become complex due to the different number of ports involved in WebSphere , by default on a single standalone node installation you can see below that there are about 18 ports or so involved,

Port Name Port
-------------------------------------------------------------------
BOOTSTRAP_ADDRESS 2809
SOAP_CONNECTOR_ADDRESS 8880
ORB_LISTENER_ADDRESS 9100
SAS_SSL_SERVERAUTH_LISTENER_ADDRESS 9401
CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS 9403
CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS 9402
WC_adminhost 9060
WC_defaulthost 9080
DCS_UNICAST_ADDRESS 9353
WC_adminhost_secure 9043
WC_defaulthost_secure 9443
SIP_DEFAULTHOST 5060
SIP_DEFAULTHOST_SECURE 5061
SIB_ENDPOINT_ADDRESS 7276
SIB_ENDPOINT_SECURE_ADDRESS 7286
SIB_MQ_ENDPOINT_ADDRESS 5558
SIB_MQ_ENDPOINT_SECURE_ADDRESS 5578
IPC_CONNECTOR_ADDRESS 9633

So in order to make sure all firewall port assignments are proper before installing you need to connect to the listening port and check from the client. Since you don't have websphere actually installed those ports won't be listening and makes it difficult to check and verify. In order to verify you can use the netcat or nc utility in unix or linux. Basically with nc or netcat utility you can listen on any partucular port TCP or UDP and make connections from the client and verify that these port communication is opened through the firewall and resolve your network issues quickly.

Use netcat or nc to listen on port 9080,
(e.g) nc -l 9080

and then you can connect to the port from another client machine using nc or any other tool like telnet to see if it accepts connection,

(e.g) nc 192.169.1.1 9080
you can type on the stdin which will be transferred and displayed across each machine, form that you can know that ports are opened.

Also use nc to port scan as well, (e.g) nc -v -z 192.168.1.1 9080-9084
the output will showing what ports are opened as below,
hostname [192.168.1.1] 9080 (?) open

Aged Timeout connection pooling setting in websphere might shrink the pool size below the specified Minimum Connection setting

If you have used websphere datasources you might have seen a jdbc connection pool setting called Aged Timeout specifies the interval in seconds before a physical connection is discarded irrespective of whether the connection was idle or used before. This property is usually set to a specified interval to maintain the freshness of the connection between the database and also to avoid any firewall sitting between your application and database terminate the connection after staying idle for a period of time, which most firewall does. Other instance where this property
might be useful is if you have an Oracle RAC (Real Application Clusters) kind of environment where you wanted to do some kind of rolling upgrade or scheduled maintenance across different nodes in the cluster , in that case DBA's might want to bring down each node gracefully after all connections are drained from the node instead of shutting down the node and terminate the connections abruptly. so this property comes in handy that after a specified time interval in the aged timeout property the connections are closed and when needed it gets created again where at this point the connections are established to a different node based on the Oracle RAC configuration change allowing them to redirect the connection to node other than the node scheduled for downtime. This is all good based on the requirements but the downside to this is the connection manager will simply close the connection based on the aged timeout when the time expires and will not recreate the connection even if the pool is below the minimum connection settings that will caused the pool to go shrink below the expected minimum size which might impact the performace of the application at certain times. Theoretically speaking you don't need to set a aged timeout particulary if you are using Oracle RAC with properly implemented high availability techniques either using Fast Connection Failover or Transparent Application Failover where tthese are totally capable to handle failover in case of an scheduled/plan outage or unplanned outage, but DBA's sometimes seems to be overly cautious, wanted application administrator to enforce these settings. There may be environments where you will be using Oracle RAC but none of those failover mechanisms been implemented and in that case insist you DBA's to implement any one of those failover mechanism.Note if you are using tomcat default's commons dbcp connection pooling there isn't a aged timeout parameter in the connection pooling settings, then you might have to rely on the database failover features.If you are using c3po another popular connection pooling datasource there is a maxConnectionAge property where it will close the connection after the specified interval and also create a connection at the same time if necessary in order to not to shrink the pool below the specified minimum pool size.

How to find the websphere version of your installation ?

You might think that it's something trivial that everyone might know but unfortunately my blog gets hits looking for the command to find the version of the websphere almost everyday , hence i wrote this article and also learnt that you can write blog articles about things that are trivial as well and might get hits as good or ever better number of hits compared to non-trivial articles with less amount of time spent on writing one :) .

To find the version of WebSphere Application Sever,

• You can run versionInfo command to generate a report from data extracted from XML files in the properties/version folder. The report includes a list of changed components and installed or uninstalled maintenance packages. you can find more about the differnet
  

On Windows : run %WAS_HOME%\bin\versionInfo.bat (e.g) c:\WebSphere\AppServer\versionInfo.bat

On Unix : run $WAS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/AppServer/bin/versionInfo.sh

Sample Report generated by the versionInfo.bat command:

D:\WebSphere\AppServer\bin>versionInfo.bat
WVER0010I: Copyright (c) IBM Corporation 2002; All rights reserved.
WVER0011I: WebSphere Application Server Release 5.1
WVER0012I: VersionInfo reporter version 1.15.2.1, dated 4/28/04

--------------------------------------------------------------------------------
IBM WebSphere Application Server Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 2009-07-20T20:24:56-07:00

Installation
--------------------------------------------------------------------------------
Product Directory D:\WebSphere\AppServer
Version Directory ${product.dir}\properties\version
DTD Directory ${version.dir}\dtd
Log Directory D:\WebSphere\AppServer\logs\update
Backup Directory ${version.dir}\backup
TMP Directory C:\DOCUME~1\rkannan\LOCALS~1\Temp

Installation Platform
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1

Technology List
--------------------------------------------------------------------------------
BASE installed
JDK installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1.1.14
ID BASE
Build Level cf140715.02
Build Date 04/19/2007

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere SDK
Version 1.4.1
ID JDK
Build Level ${build.level}
Build Date ${dateStamp}

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

To find the version of WebSphere Portal Server,

• You can run WPVersionInfo.sh to gather version information of the websphere portal server installation.
  

On Windows : run %WPS_HOME%\bin\WPVersionInfo.bat (e.g) c:\WebSphere\PortalServer\bin\WPVersionInfo.bat

On Unix : run $WPS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/PortalServer/bin/WPVersionInfo.sh

Sample report of the command:

[root@server101 bin]# ./WPVersionInfo.sh
WVER0010I: Copyright IBM Corporation 2002, 2008; All rights reserved.
WVER0011I: WebSphere Portal
WVER0012I: VersionInfo reporter version 1.10.1.1, dated 20080530_1701
--------------------------------------------------------------------------------
IBM WebSphere Portal Product Installation Status Report
--------------------------------------------------------------------------------
... ...
Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Portal MultiPlatform
Version6.1.0.0
ID MP
Build Levelwp610_494_01 2008-06-19
Build Date 06/19/2008
Installed Product
... ...
--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------


To find the version of IBM JDK installed along with WebSphere Application Server,

• you can run java -fullversion to find the jdk version details

On Windows : run %WAS_HOME%\java\bin\java -fullversion (e.g) c:\WebSphere\AppServer\java\bin\java -fullversion

On Unix : run $WAS_HOME/java/bin/java -fullversion (e.g) /opt/WebSphere/AppServer/java/bin/java -fullversion

Sample output of the command:

D:\WebSphere\AppServer\java\bin>java -fullversion
java full version "J2RE 1.4.2 IBM Windows 32 build cn1420-20040626"

websphere jdbc connection pool aged timeout setting in websphere might shrink the pool size below the specified Minimum Connection setting

If you have used websphere datasources you might have seen a jdbc connection pool setting called Aged Timeout specifies the interval in seconds before a physical connection is discarded irrespective of whether the connection was idle or used before. This property is usually set to a specified interval to maintain the freshness of the connection between the database and also to avoid any firewall sitting between your application and database terminate the connection after staying idle for a period of time, which most firewall does. Other instance where this property might be useful is if you have an Oracle RAC (Real Application Clusters) kind of environment where you wanted to do some kind of rolling upgrade or scheduled maintenance across different nodes in the cluster , in that case DBA's might want to bring down each node gracefully after all connections are drained from the node instead of shutting down the node and terminate the connections abruptly. so this property comes in handy that after a specified time interval in the aged timeout property the connections are closed and when needed it gets created again where at this point the connections are established to a different node based on the Oracle RAC configuration change allowing them to redirect the connection to node other than the node scheduled for downtime. This is all good based on the requirements but the downside to this is the connection manager will simply close the connection based on the aged timeout when the time expires and will not recreate the connection even if the pool is below the minimum connection settings that will caused the pool to go shrink below the expected minimum size which might impact the performace of the application at certain times. Theoretically speaking you don't need to set a aged timeout particulary if you are using Oracle RAC with properly implemented high availability techniques either using Fast Connection Failover or Transparent Application Failover where tthese are totally capable to handle failover in case of an scheduled/plan outage or unplanned outage, but DBA's sometimes seems to be overly cautious, wanted application administrator to enforce these settings. There may be environments where you will be using Oracle RAC but none of those failover mechanisms been implemented and in that case insist you DBA's to implement any one of those failover mechanism. Note if you are using tomcat default's commons dbcp connection pooling there isn't a aged timeout parameter in the connection pooling settings, then you might have to rely on the database failover features. If you are using c3po another popular connection pooling datasource there is a maxConnectionAge property where it will close the connection after the specified interval and also create a connection at the same time if necessary in order to not to shrink the pool below the specified minimum pool size.

java.sql.SQLException: OALL8 is in an inconsistent state message in logs

You might see the following exception in the logs of your WebSphere or Tomcat Application Server,

2009-06-13 17:35:23,294 434385047 ERROR [com.ibm.ws.services.encryption.v1.provider.EncryptionServiceImpl] (web-1:) - org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: OALL8 is in an inconsistent state at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:238)

One of the known reason might be that your database got shutdown in one of the node particularly in a Oracle RAC kind of environment where multiple nodes are present. The error usually recovers once the connections are failed over but it seems like atleast few transactions are failing before it can recover. Also the message is usually preceeded by this message,

2009-06-13 17:35:22,405 434384158 ERROR [org.apache.axis2.engine.AxisEngine] (web-2:) - Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted
org.apache.axis2.AxisFault: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted

which indicates that the db is shutting down. If you want to have a gracful shutdown of db without even affecting few transactions then check my other article websphere jdbc connection pool aged timeout setting in websphere might shrink the pool size below the specified Minimum Connection setting in the blog.

How to find the version of websphere, java and portal that you are running ?

You might think that it's something trivial that everyone might know but unfortunately my blog gets hits looking for the command to find the version of the websphere almost everyday , hence i wrote this article and also learnt that you can write blog articles about things that are trivial as well and might get hits as good or ever better number of hits compared to non-trivial articles with less amount of time spent on writing one :) .

To find the version of WebSphere Application Sever,

• You can run versionInfo command to generate a report from data extracted from XML files in the properties/version folder. The report includes a list of changed components and installed or uninstalled maintenance packages. you can find more about the differnet
  

On Windows : run %WAS_HOME%\bin\versionInfo.bat (e.g) c:\WebSphere\AppServer\versionInfo.bat

On Unix : run $WAS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/AppServer/bin/versionInfo.sh

Sample Report generated by the versionInfo.bat command:

D:\WebSphere\AppServer\bin>versionInfo.bat
WVER0010I: Copyright (c) IBM Corporation 2002; All rights reserved.
WVER0011I: WebSphere Application Server Release 5.1
WVER0012I: VersionInfo reporter version 1.15.2.1, dated 4/28/04

--------------------------------------------------------------------------------
IBM WebSphere Application Server Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 2009-07-20T20:24:56-07:00

Installation
--------------------------------------------------------------------------------
Product Directory D:\WebSphere\AppServer
Version Directory ${product.dir}\properties\version
DTD Directory ${version.dir}\dtd
Log Directory D:\WebSphere\AppServer\logs\update
Backup Directory ${version.dir}\backup
TMP Directory C:\DOCUME~1\rkannan\LOCALS~1\Temp

Installation Platform
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1

Technology List
--------------------------------------------------------------------------------
BASE installed
JDK installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1.1.14
ID BASE
Build Level cf140715.02
Build Date 04/19/2007

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere SDK
Version 1.4.1
ID JDK
Build Level ${build.level}
Build Date ${dateStamp}

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

To find the version of WebSphere Portal Server,

• You can run WPVersionInfo.sh to gather version information of the websphere portal server installation.
  

On Windows : run %WPS_HOME%\bin\WPVersionInfo.bat (e.g) c:\WebSphere\PortalServer\bin\WPVersionInfo.bat

On Unix : run $WPS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/PortalServer/bin/WPVersionInfo.sh

Sample report of the command:

[root@server101 bin]# ./WPVersionInfo.sh
WVER0010I: Copyright IBM Corporation 2002, 2008; All rights reserved.
WVER0011I: WebSphere Portal
WVER0012I: VersionInfo reporter version 1.10.1.1, dated 20080530_1701
--------------------------------------------------------------------------------
IBM WebSphere Portal Product Installation Status Report
--------------------------------------------------------------------------------
... ...
Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Portal MultiPlatform
Version6.1.0.0
ID MP
Build Levelwp610_494_01 2008-06-19
Build Date 06/19/2008
Installed Product
... ...
--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------


To find the version of IBM JDK installed along with WebSphere Application Server,

• you can run java -fullversion to find the jdk version details

On Windows : run %WAS_HOME%\java\bin\java -fullversion (e.g) c:\WebSphere\AppServer\java\bin\java -fullversion

On Unix : run $WAS_HOME/java/bin/java -fullversion (e.g) /opt/WebSphere/AppServer/java/bin/java -fullversion

Sample output of the command:

D:\WebSphere\AppServer\java\bin>java -fullversion
java full version "J2RE 1.4.2 IBM Windows 32 build cn1420-20040626"

java.sql.SQLException: Io exception might be related to firewall or load balancer issues

You might see the following exception in the java SystemOut logs of WebSphere,

Caused by: java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA00
10E: SQL State = null, Error Code = 17,002
... 150 more
---- Begin backtrace for nested exception
java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA0010E: SQL St
ate = null, Error Code = 17,002
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java(Compiled Code))
at java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled Code))
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:501)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:544)
at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:788)
at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:765)
at com.ibm.ejs.j2c.poolmanager.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1431)
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1182)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))

If you see those exceptions it might be due to some network related, either the firewall ( FSWM - Firewall Services Module ) or your load balancer dropping connections or packets causing the exception to be thrown on the logs in the WebSphere with the strange message plus your connections might get lost and might also see an application related error. This exception might be seen when there was some change going on the FSWM or loadbalancer like acl updates where it's trying to reload and applying them, it usually affects the new connections existing pooled connections seems to work ok without problems.

Also if you are using oracle you might see the following error message in the oracle listener sqlnet.log,

TNS-12637: Packet receive failed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12537: TNS:connection closed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12547: TNS:lost contact
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12535: TNS:operation timed out
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12637: Packet receive failed

Here is the explanation from the oracle documentation,

TNS-12637 Packet receive failed

Cause: A process was unable to receive a packet from another process. Possible causes include:
The other process was terminated.
The machine on which the other process is running went down.
Some other communications error occurred.

To rectify it either update the software/firmware to the latest version in your FWSM or load balancer , check for any network bottlenecks, check load average of the machines and take necessary actions. If you are continously seeing the error then there is a possibility that there might be some kind of DoS attack on the db listener.

Synchronization issues related to Java collections when accessed concurrently

Being a java developer or an application server administrator interested in learning how the code works like me :) , you must have seen the javadoc http://java.sun.com/javase/6/docs/api/java/util/HashMap.html warning: Note that this implementation is not synchronized. If multiple threads access a hash map concurrently, and at least one of the threads modifies the map structurally, it must be synchronized externally. (A structural modification is any operation that adds or deletes one or more mappings; merely changing the value associated with a key that an instance already contains is not a structural modification.) This is typically accomplished by synchronizing on some object that naturally encapsulates the map. If no such object exists, the map should be "wrapped" using the Collections.synchronizedMap method. This is best done at creation time, to prevent accidental unsynchronized access to the map.

So here are some things that might happen depends on what kind of Map you are using like LRUMap or TreeMap etc. I have seen these behaviors in runtime where the Map is not synchronized but concurrently accessed, and though of publishing it in the blog which might help others to debug similar issues or correlate the symptoms to the failure they might have seen but not having a clue.



1) LRUMap Memory leak : If you are using a LRUMap and multiple threads are putting and getting values concurrently, some kind of memory leak happens and the object grows beyond the specified maximum size. Every time you access the LRUMap either by get or put, the object you are getting or putting had to be moved to the MRU(Most recently used) place within the Map. Also when the LRUMap is full it has to remove the LRU object to provide room for the new object. I believe this is accomplished by some kind of a link list data structure within the map. Hence there is a lot of overhead during the process and there is quite a chance to get the link list corrupted when multiples threads are get/put’ing concurrently particularly when the map is full.

I noticed the “size()” as returned by the object sometimes gets higher than the maximum size and sometimes show less than the maximum size when it’s supposedly be full depending on how and where the corruption happens within the link list .Further accessing the object after it’s corrupted, the “size()” is starting to get reduced every time a further corruption happens and leads into negative size (-1…-20) and keep increasing in the negative side. Note the “size()” that I am referring is the size returned by the counter that the object is using to track and not the actual object size. so I took a heap dump and found that any object that is added after Is never discarded at all , I guess because of the logic that size never gets bigger than the maximum size after corruption.It looks like one the object is corrupted it’s starting to leak and then increasing the overall size(byte size) and the behavior in erroneous.
I am attaching the lru.jsp that I used to simulate so that you can try and see how it behaves. The below picture shows how the object java.math.BigDecimal grows in consecutive heapdumps even though the max size is set to 2.

Note that this behavior happened on WebSphere 5.1 Application Server on JDK 1.4.2 not sure what might be the behavior on other version of JDK or AppServer. Please post in my comment if you are seeing any different behaviors for any other types of Map.

2) TreeMap High Cpu Utilization: This behavior happened particularly when the interator iterating through the treemap collection is not synchronized, where one thread is changing (add or delete) the element and then other iterating through it concurrently. I didn't do much analysis but it looks like the underlying data structure of the TreeMap some how gets corrupted and causes a circular reference between two elements causing the interator to go into an infinite loop which caues the CPU to spike and eventually the ApplicationServer will become unresponsive. You probably want to kill the process to reocver from this state. Here is the thread stacktrace that you would see if take a thread dump, Attaching the tree.jsp for you to test and see.

at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1570)
at java.util.TreeMap$NavigableSubMap$SubMapKeyIterator.next(TreeMap.java:1629)
at org.apache.jsp.tree2_jsp._jspService(tree2_jsp.java:120)

Also note many programmers may forget to synchronize on the collection when iterating eventhough the collection is synchronized. Hence, it is imperative that the user manually synchronize on the returned collection when iterating over it to prevent any non-deterministic behavior.

Collection c = Collections.synchronizedCollection(myCollection);
...
synchronized(c) {
Iterator i = c.iterator(); // Must be in the synchronized block
while (i.hasNext())
foo(i.next());
}

Live Popularity Of Enterprise Portal Server Market by Google Keword Search using Yahoo Pipes

I just came across this interesting mashup kind of a tool called Yahoo Pipes an interactive feed aggregator and manipulator. I was thinking to use the pipes to build something relevant to my websphere blog and then decided to build a pipe which basically plot a enterpise portal popularity graph based on the total number of realtime google search results returned for each of the popular portal servers names like WebSphere Portal, Weblogic Portal, Sun One, Sharepoint, Jboss and Oracle. You can see the source of the pipe here http://pipes.yahoo.com/pipes/pipe.edit?_id=269ed254b3df0a7426e34187c98c0d17 The results also kind of confirmed the truth with this news that IBM WebSphere Portal is the leader in Enterprise portal market.

If you are not seeing the graph below please click here , because it seems like google search blocks yahoo pipes for making repeated calls, you will see the graph snapshot that was taken on July 13th 2008.

Performance issues with 'Manage Users and Groups' portlet in WebSphere Portal

There are situations like when assigning a member to a group or creating a new member for a group, listing users for a group, etc may result in poor or slow performance or even browser timeout with the 'Manage Users and Groups' portlet. These especially happends when a large number of users in a group or large number of groups exists in the ldap server. To avoid such conditions you might want to modify these configuration parameters http://www-1.ibm.com/support/docview.wss?rs=688&ca=portall2&uid=swg21247210 of the 'Manage Users and Groups' portlet based on your needs.

Useful UNIX commands for WebSphere & Portal Administrator

These are some useful commands that i came across when administrating WebSphere Application Server or WebSphere Portal Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT.

Performance related Commands to monitor system performance
-------------------------------------------------------------------------
1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the CPU

%CPU PID USER COMMAND 71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere /AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=54859 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer

2) Find the last 10 process that use the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

%CPU PID USER COMMAND 2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer 0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer 0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/cust/IBM/WebSphere/AppServer

3)Find cpu usage
mpstat -P ALL or mpstat

Linux 2.4.21-27.ELsmp (machine1.www.com) 02/02/2008 11:55:00 AM CPU %user %nice %system %iowait %irq %soft %idle intr/s 11:55:00 AM all 1.51 0.00 0.41 0.04 0.01 0.09 97.94 184.92 11:55:00 AM 0 1.50 0.00 0.41 0.04 0.01 0.11 97.93 123.76 11:55:00 AM 1 1.51 0.00 0.42 0.04 0.00 0.07 97.95 61.16

4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc.
top

12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00 572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9% cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6% cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2% Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd, 295304k buff 3998132k active, 682312k inactive Swap: 2048276k av, 0k used, 2048276k free 1258592k cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 17719 root 16 0 306M 306M 3992 S 0.7 2.6 22:14 1 java 19358 root 16 0 2532 2532 896 R 0.5 0.0 0:00 1 top 17771 root 15 0 306M 306M 3992 S 0.4 2.6 3:48 0 java 18058 root 16 0 458M 458M 4032 S 0.1 3.9 12:55 0 java 1 root 15 0 512 512 452 S 0.0 0.0 1:19 0 init 2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0

5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &

Linux 2.4.21-27.ELsmp (machine1.web.com) 02/02/2008 12:04:34 PM CPU %user %nice %system %iowait %idle 12:04:36 PM all 0.25 0.00 0.00 0.00 99.75 12:04:38 PM all 0.00 0.00 0.50 0.25 99.25 12:04:40 PM all 7.00 0.00 7.25 0.00 85.75 12:04:42 PM all 21.05 0.00 1.50 0.00 77.44 12:04:44 PM all 3.00 0.00 0.50 0.00 96.50 Average: all 6.26 0.00 1.95 0.05 91.74

Disk Usage related Commands to monitor Disk space
--------------------------------------------------------
1)Disk Usage on all mounts
df -h -T

Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext3 2.9G 785M 2.0G 29% / /dev/sda7 ext3 57G 9.3G 45G 18% /cust /dev/sda6 ext3 2.0G 82M 1.8G 5% /lc none tmpfs 5.7G 0 5.7G 0% /dev/shm /dev/sda5 ext3 2.0G 33M 1.8G 2% /tmp /dev/sda3 ext3 2.0G 64M 1.8G 4% /var

2)prints just one line with the total size of the directory, the below example show the size of the /usr/IBM/Websphere WebSphere Portal Server directory.
du -ch | grep total

bash-2.05b# du -ch | grep total 6.3G total

Network related Commands
-------------------------------------
1)
a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers.
b) nslookup - Queries a name server for a host or domain lookup.
c) host - DNS lookup uility.

The below example shows the information like CNAME , Aliases, ipaddres , etc about www.google.com


a) dig <ip address or hostname>

bash-2.05b# dig www.google.com ; <<>> DiG 9.2.4 <<>> www.google.com ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 585119 IN CNAME www.l.google.com. www.l.google.com. 69 IN A 216.239.51.99 www.l.google.com. 69 IN A 216.239.51.104 ;; AUTHORITY SECTION: l.google.com. 60734 IN NS g.l.google.com. l.google.com. 60734 IN NS b.l.google.com. l.google.com. 60734 IN NS f.l.google.com. l.google.com. 60734 IN NS c.l.google.com. l.google.com. 60734 IN NS a.l.google.com. l.google.com. 60734 IN NS e.l.google.com. l.google.com. 60734 IN NS d.l.google.com. ;; Query time: 2 msec ;; SERVER: 10.5.50.184#53(10.5.50.184) ;; WHEN: Sun Feb 3 09:51:46 2008 ;; MSG SIZE rcvd: 196

b) host <ip address or hostname>

bash-2.05b# host www.google.com www.google.com is an alias for www.l.google.com. www.l.google.com has address 216.239.51.104 www.l.google.com has address 216.239.51.99

c) nslookup <ip address or hostname>

bash-2.05b# nslookup www.google.com Server: 10.5.50.184 Address: 10.5.50.184#53 Non-authoritative answer: www.google.com canonical name = www.l.google.com. Name: www.l.google.com Address: 216.239.51.99 Name: www.l.google.com Address: 216.239.51.104

2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a loopback.


infconfig -a

bash-2.05b# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:11:43:32:49:99 inet addr:10.2.150.74 Bcast:10.5.50.127 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0 TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3168219417 (3021.4 Mb) TX bytes:3724305382 (3551.7 Mb) Interrupt:17 eth1 Link encap:Ethernet HWaddr 00:11:43:32:49:9A inet addr:10.5.154.138 Bcast:10.5.54.127 Mask:255.255.255.128 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:18 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0 TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1454141744 (1386.7 Mb) TX bytes:1454141744 (1386.7 Mb)

3) To find more information about a network interface, the below example shows detail information about interface eth0.


bash-2.05b# ethtool eth0

Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: Not reported Advertised auto-negotiation: No Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: off Supports Wake-on: d Wake-on: d Link detected: yes

File related commands
----------------------
1) find text in the matching files and show the filename with the pattern. The below example shows the list of filenames and the location of the hostname in those files, which will be helpful when changing hostnames for your websphere installation.

find . -name '*.xml' -print | xargs grep 'hostName' /dev/null

bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.xml' -print | xargs grep 'hostName' /dev/null | more /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverNode01/serverindex.xml:<serverindex: ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5 .0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointRefs="NamedEndPoint_1130375480667 NamedE ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPoint_1130375480671 NamedEndPoint_113037548067 2 NamedEndPoint_1130375480673"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2.example.com/serverindex.xml:< serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2.example.com/serverindex.xml:< serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverCellManager01/serverindex.xml: <ser verindex:ServerIndex xmi:id="ServerIndex_1" endPointRefs="NamedEndPoint_1 NamedEndPoint_2" hostName="portalserver.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2011indchn2Node01/serverindex.xml:<serverindex: ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5 .0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointRefs="NamedEndPoint_1130375717654 NamedE ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPoint_1130375717658 NamedEndPoint_113037571765 9 NamedEndPoint_1130375717660"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2UnNode01/serverindex.xml:<serverinde x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas /5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2UnNode01/serverindex.xml:<serverinde x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas /5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serverindex:ServerI ndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5.0/serv erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode01Cell/nodes/portalserverNode01/serverindex.x ml:<serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/app server/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serveri ndex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/sche mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">

2) Soemtimes you might run out of diskspace and want to clean up some files, so this command will help you find all large files. In this example it will list all the files greater than 10 mb in size.

find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5

bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5-rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/tranlog/log1 -rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log2 -rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log1

3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to FTP,email or http upload.

To pack on the fly...
tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz

To unpack on the fly..
gunzip < FILE.tar.gz | tar -xvf -

To pack on the fly and email the file to IBM Support
tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR 09921,227,000" -b "SystemOut.log for server crash " WebSphere_Support@mainz.ibm.com

To find selective files and tar it up on the fly...
find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf System_logs.tar


4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin where you might want to write stdout to the file as well as to the console so that you have a copy even when the standard output disappears because of the screen buffer size.


ls -l | tee > filename
xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url http://localhost:9080/wps/config | tee filename
./wsadmin.sh | tee wsadmin.log

5) find files that is recently updated which might be useful to see what changes are made to the system recently (e.g) below command shows the files that are modified within the last 20 minutes(-mmin) and 20 days (-mtime)


find /usr/IBM -mmin -20
find /usr/IBM -mtime -20

/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs

6) Find a memory usage of the system.


free
cat /proc/meminfo

bash-2.05b# free total used free shared buffers cached Mem: 11819592 4938416 6881176 0 298148 1440200 -/+ buffers/cache: 3200068 8619524 Swap: 2048276 0 2048276 bash-2.05b# cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 12103262208 5056954368 7046307840 0 305303552 1474797568 Swap: 2097434624 0 2097434624 MemTotal: 11819592 kB MemFree: 6881160 kB MemShared: 0 kB Buffers: 298148 kB Cached: 1440232 kB SwapCached: 0 kB Active: 3718616 kB ActiveAnon: 2968776 kB ActiveCache: 749840 kB Inact_dirty: 789328 kB Inact_laundry: 237596 kB Inact_clean: 0 kB Inact_target: 949108 kB HighTotal: 11140992 kB HighFree: 6681628 kB LowTotal: 678600 kB LowFree: 199532 kB SwapTotal: 2048276 kB SwapFree: 2048276 kB Committed_AS: 6039100 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB

4) To repeat a command contimuously.

watch -d=10 -n 1 "netstat -a"

Every 1s: netstat -a| grep ESTABLISHED Sun Feb 3 21:48:04 2008 tcp 0 0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545 ESTABLISHED

SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore

If you are a WebSphere Administrator or any Application Server (WebLogic, Tomcat) administrator you might already know that managing the ssl certificates in a large complex environments becomes hectic and troublesome because of the different expiration dates of the certificates that websphere uses and also the ssl certificates of the external systems (like SAP , Siebel) that websphere applicaton server might connect to using a secure connection, multiple administrators in an organization renewing it and not keeping tracking of the expiration dates. The problem is , SSL certificate might expire resulting in servers will not initialize and your running servers will stop operating, becomes unresposive if not properly renewed on time. Hence this article will explain how you can monitor the expiration of the ssl certificates using a simple command and proactively monitor the expiration dates, setup your calendar and renew it on time to prevent any downtimes.

If you are using the latest version (as on Dec 2007) of WebSphere Application Server 6.1 and using default self signed there are provisions in the application server itself to effectively monitor the certificate expiration, notifying you and renewing it automatically. If you are using a perosnal self signed certificate or a personal certificate signed by a certificate authority (CA) like VeriSign or Thawte , which is most likely in all of your production environments then you might want to use the below command to find the expiration date and renew it.

If you are using a java keystore ( DummyServerKeyFile.jks , DummyServerTrustFile.jks , jssecerts , cacerts) use the below find command along with keytool or websphere's ikeyman command line option to find the expiration of the keys.

LINUX:


Using Ikeyman cmdline utility (ikeyman.sh):

find PATH -name *.jks | xargs -i bash -c '$WAS_HOME/java/bin/java -classpath $WAS_HOME/java/jre/lib/ext/ibmjceprovider.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmjcefw.jar:$WAS_HOME/AppServer/java/jre/lib/ext/US_export_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/local_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmpkcs.jar:$WAS_HOME/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw <password> | grep "Not After:.*yyy[y,y]" -B 3 && echo {}'

(e.g)
find /usr/IBM/WebSphere/AppServer/profiles/ -name *.jks | xargs -i bash -c '/usr/IBM/WebSphere/AppServer/java/bin/java -classpath /usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/AppServer//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw WebAS | grep "Not After:.*200[7,8]" -B 3 && echo {}'

Using Java Keytool utility:

find $WAS_HOME -name '*.jks' | xargs -i bash -c "echo {}; $WAS_HOME/java/jre/bin/keytool -list -v -keystore {} -storepass password | grep 'until: .*/.*/Y[Y,Y]' -A 3 -B 7"

(e.g)
find /usr/IBM/WebSphere/AppServer -name '*.jks' | xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS | grep 'until: .*/.*/0[7,8]' -A 3 -B 7"

Output:

/usr/IBM/WebSphere/AppServer/profiles/etc//WWWStageClientKey.jks
Alias name: verisign class 1 ca individual subscriber-persona not validated
Creation date: Dec 18, 2006
Entry type: trustedCertEntry

Owner: CN=VeriSign Class 1 CA Individual Subscriber-Persona Not Validated, OU="www.verisign.com/repository/RPA Incorp. By Ref.,LIAB.LTD(c)98", OU=VeriSign Trust Network, O="VeriSign, Inc."
Issuer: OU=Class 1 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
Serial number: d8b4feeaad218df5bf4756a9d29e17ffb
Valid from: 5/12/98 12:00 AM until: 5/12/08 11:59 PM
Certificate fingerprints:
         MD5:  CA:66:3C:FC:71:2B:BA:41:92:71:DD:72:AD:E5:65:65
         SHA1: 12:51:9A:E9:CD:77:7A:56:01:84:F1:FB:D5:42:15:22:2E:95:E7:1F
--
Creation date: Dec 18, 2006
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=US
Issuer: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=US
Serial number: 458709ef
Valid from: 12/18/06 9:36 PM until: 12/18/07 9:36 PM
Certificate fingerprints:
         MD5:  C2:4F:CC:SS:19:DC:E4:88:B0:2A:78:76:69:4D:DC:EF:47
         SHA1: 53:AB:D0:13:45:45:12:59:64:F6:C9:38:41:F2:C8:E3:37:05:73:95:F3:83

/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks

Alias name: www stage websphere server ca
Creation date: Dec 8, 2006
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=wadm.stage.www.com, OU=IT, O="www, Inc.", L=Fremont , ST=California, C=US
Issuer: OU=Secure Server Certification Authority, O="RSA Data Security, Inc.", C=US
Serial number: 1ea5f21b33ca65e551440d662399dffdfga382
Valid from: 12/5/06 12:00 AM until: 12/15/07 11:59 PM
Certificate fingerprints:
         MD5:  C2:22:E4:12:12:6D:3F:22:67:4E:37:48:12:3F:F3:1B:FE:26
         SHA1: CA:81:37:A7:E9:12:22:19:27:D9:3F:C5:79:E9:25:C8:1A:3C:1E:5C:17:02

As you can see in the above output the command will list the file name (/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks) , certificate label (www stage websphere server ca) , Entry Type (keyEntry - personal cert or trustedCertEntry - Signer Cert), and the expiration date (12/5/06 12:00 AM until: 12/15/07 11:59 PM) of
all the java keystore (*.jks) in the given directory that expired in 2007 (07) or 2008 (08). By which you shoule be able to indentify the certs, setup your calendat atleast 10 days before the expiration and renew it to proactively to preveny any problems.

Note you can also change the above command by passing differnt filename like cacerts and jssecerts and also pass different year infomration to find expirations during those years.

Sometimes you might get no output which means either there are no expiring ceritificates for the years that you passed or if you are expecting any expiring certificate to show and if not then the password for the keystore might be wrong, so run without the grep portion in the above command which will list all the certificates irrespective of the expiration date or it will complain if the password is wrong as below.

find /usr/IBM/WebSphere/AppServer -name '*.jks' | xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS

/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks
java.io.IOException: Keystore was tampered with, or password was incorrect


Also you can use the same command by modifying the -type parameter to "-type cms" for the webserver such as IBM HttpServer where it's using cms database ( kdb file).


find /usr/IBM/HttpServer -name *.kdb | xargs -i bash -c '/usr/IBM/WebSphere/Plugins/java/bin/java -classpath /usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/Plugins//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type cms -pw WebAS | grep "Not After:.*200[7,8]" -B 3 && echo {}'