Aged Timeout connection pooling setting in websphere might shrink the pool size below the specified Minimum Connection setting

If you have used websphere datasources you might have seen a jdbc connection pool setting called Aged Timeout specifies the interval in seconds before a physical connection is discarded irrespective of whether the connection was idle or used before. This property is usually set to a specified interval to maintain the freshness of the connection between the database and also to avoid any firewall sitting between your application and database terminate the connection after staying idle for a period of time, which most firewall does. Other instance where this property
might be useful is if you have an Oracle RAC (Real Application Clusters) kind of environment where you wanted to do some kind of rolling upgrade or scheduled maintenance across different nodes in the cluster , in that case DBA's might want to bring down each node gracefully after all connections are drained from the node instead of shutting down the node and terminate the connections abruptly. so this property comes in handy that after a specified time interval in the aged timeout property the connections are closed and when needed it gets created again where at this point the connections are established to a different node based on the Oracle RAC configuration change allowing them to redirect the connection to node other than the node scheduled for downtime. This is all good based on the requirements but the downside to this is the connection manager will simply close the connection based on the aged timeout when the time expires and will not recreate the connection even if the pool is below the minimum connection settings that will caused the pool to go shrink below the expected minimum size which might impact the performace of the application at certain times. Theoretically speaking you don't need to set a aged timeout particulary if you are using Oracle RAC with properly implemented high availability techniques either using Fast Connection Failover or Transparent Application Failover where tthese are totally capable to handle failover in case of an scheduled/plan outage or unplanned outage, but DBA's sometimes seems to be overly cautious, wanted application administrator to enforce these settings. There may be environments where you will be using Oracle RAC but none of those failover mechanisms been implemented and in that case insist you DBA's to implement any one of those failover mechanism.Note if you are using tomcat default's commons dbcp connection pooling there isn't a aged timeout parameter in the connection pooling settings, then you might have to rely on the database failover features.If you are using c3po another popular connection pooling datasource there is a maxConnectionAge property where it will close the connection after the specified interval and also create a connection at the same time if necessary in order to not to shrink the pool below the specified minimum pool size.

VisualVM remote application monitoring through SSH proxy

As many of you might already know Java VisualVM an all in one Java troubleshooting tool is available as a JDK tool in Sun JDK distributions starting from JDK 6 update 7 and Apple's Java for Mac OS X 10.5 Update 4. Also VisualVM a stand alone separately downloadable tool is also available at visualvm.dev.java.net. It's a visual tool enhances the capability of performance and monitoring analysis for Java SE platform for both production and development environments. You can find more detailed information in the VisualVM product site.

But our topic is how to make visualvm work through SSH Tunnel specifically in production systems where it's a normal practice to allow only SSH access on port 22 for the users or administrators, hence visualvm will not work with the default settings as it uses port 1099 to communicate to remote host via jstatd. For some reason, VisualVM doesn't seem to work through SSH Port Forwarding or SSH Proxy which seems to be a big blocker for many people which wish yo use this tool in this similar kind of setup.

The only way i came up so far to make the VisualVM work is to use to use a system proxifier like WideCap (Free) or ProxyCap (Shareware) and was able to connect to remotely a running java process over ssh proxy. Here are the steps detailing on how to configure ,

1) Download WideCap and install.
2)Open WideCap control panel and click on new proxy , provide hostname as localhost, port
8333, and select the socks version of your SSH that your client and server uses.





3)Add a new application and point to the location of java.exe and create new rule to get triggered when the java is started.




4) Enable WideCap.
5) run putty or any ssh client as SSH proxy daemon. (e.g) putty -D 8333 username@server or ssh -D 8333 user@host
6) Start VisualVM and add remote host and now you should be able to see the java process running on the remote server and you should be able to connect and monitor.



On a cautious note that widecap is not very consistent and it crashes outlook (at least for me ) and IE sometimes but on the other hand proxycap seem to work fine along with other standard windows applications without problems.

Also if you run into problems make sure all the instructions are followed in this link

WebSphere Portal Interview Questions for an Administrator

List of interview questions that i might ask someone for the job of WebSphere Portal Server Administrator. What is the difference between Application Server and Portal Server ? Application servers extend the ability of a Web server to handle Web application requests, and enables a server to generate a dynamic, customized response to a client request. A portal server extends the application server by providing a portlet container that can run portlets and to create portal a website that provides users with a single point of access to Web-based resources by aggregating those resources in one place. What are the steps involved in deploying themes and skins in a clustered production websphere portal environment ? Export the WebSphere wps.ear (Portal EAR) using wsadmin. Use EarExpander tool to expand the exported wps.ear file. Copy the updated themes and skins into ../themes/html, ../skins/html folder. Use EarExpander tool to collapse the EAR directory into an EAR file. Use wsadmin to update the wps.ear to complete the deployment of updated themes and skins. What changes needs to be done to view changes to your theme and skins JSPs without restarting the portal server ? You need to enable automatic JSP loading by setting reloadingEnable property to true in ibm-web-ext.xmi file of the wps.ear. What are the 3 different ways of installing a portlet application in WebSphere Portal ? Install a portlet using the portal administration page using Web Modules portlet. Install a portlet using xmlaccess tool. Pre-deploy a portlet as a standard EAR by installing the portlet WAR file in WAS console and then registering the portlet using xmlaccess. What is the purpose of XMLAccess configuration file Export.xml & ExportRelease.xml ? What is the difference ? & When will you use one over the other ? Export.xml exports the complete portal configuration and useful when transferring configurations between development installations. ExportRelease.xml exports the complete portal configuration from the release domain as required by the portal ReleaseBuilder tool and useful when tranaferring different release configurations between staging and production environments. List me the steps involved in building a release in WebSphere Portal ? If you have a completely new installation of the staging server and the production server: Install the staging server, then install the production server. Develop a release on the staging server. Build the release on the staging server. Empty portal contents on the production server by running the WPSconfig.sh|bat action-empty-portal task. Import that release onto the production server. Refer to Transferring a complete configuration for information. If you already have a production server without a staging system: Export the release of your production server. Install an empty staging server using one of the following two methods: Install the staging server with the flag -W emptyPortal.active=True. After installing and configuring the staging server, run the WPSconfig.sh|bat action-empty-portal task. Import the production release onto the staging server. Refer to Transferring a complete configuration for information. Develop and build a new release on the staging server. Export that new release from the staging server. Use ReleaseBuilder to generate the differential between the two releases. Import the differential onto the production server. What is the purpose of ReleaseBuilder tool in WebSphere Portal ? ReleaseBuilder enables management of release configurations independent of user configurations and used during staging of follow-on releases of WebSphere portals, configurations, and artifacts need to be moved between systems. What are the steps involved in editing WebSPhere Member Manager (wmm.xml) files on a federated node ? On the primary node of the WebSphere Portal cluster, check out the files using ./WPSconfig.sh check-out-wmm-cfg-files-from-dmgr task. Make any changes to the Member Manager files. The files can be edited in the portal_server_root/wmm directory on the WebSphere Portal node. When you have completed your changes, check the files back in using ./WPSconfig.sh check-in-wmm-cfg-files-to-dmgr. How to change the default portal URI /wps/portal after installation ? Set the property WpsContextRoot to the new URI and run ./WPSConfig.sh modify-servlet-path task. List different types of user registry supported by WebSphere Portal ? LDAP (includes LDAP with an optional database user registry). Database user registry. Custom User registry, (non-LDAP, non-database). What is LDAP realm support and why would I want to use it? A Realm allows you to group users from one or more LDAP trees of one user registry and expose them as a coherent user population to WebSphere Portal; this is also referred to as horizontal partitioning. Realms allow a flexible user management with various configuration options; for example, you can combine principals from one or more corporate LDAP tree. A realm must be mapped to a Virtual Portal to allow the realm's defined user population to login to the Virtual Portal. What is an Application group and why would I want to use it ? Application groups is a concept that allows you to define user groups within the database user registry with members (users or groups) contained in the LDAP user registry you configured. The benefit of application groups is that you can create Groups that are only used in WebSphere Portal particularly in scenarios where there is Read-only LDAP or special group setup specific to the portal. What are the two methods to install the empty staging WebSphere portal server ? Empty portal contents on the staging server by running the ./WPSconfig.sh action-empty-portal. Install portal with the ./install.sh -W emptyPortal.active="True" option. How do you stop portal traffic to the node being upgraded in 24x7 cluster upgrade process ? Locate the cluster member being upgraded, and change the value in the Configured weight column from a value to zero and make sure the plugin config information is propagated to webserver to stop traffic. How to set limits on searches for users and groups ? Setting a maximum number of search results maximumSearchResults="200" in wmm.xml What portal resources are scoped for virtual portal ? Portal pages. Portlet instances. Portal Search Engine search services and search collections. This includes the search content sources. What portal resources can't be seperated for virual portal ? Themes and skins. Vault segments and vault slots. Supported clients and markups. Composite applications and templates. Policies. How do you enable temporary and extended trace logging for WebSphere Portal ? Temporary traces can be set for a temporary period by using the administration portlet Enable Tracing or the IBM WebSphere Application Server administrative console and also using the Enable Tracing portlet in the portal administration page. To enable extended trace settings for a longer period of time, that is, for more than one session, switch them on in the WebSphere Application Server configuration, save the updates and restart the portal server. What are the different states of the syndication process ? Idle: No syndication is occurring. Pending: A request has been made to the syndicator, but it has yet to initiate a request to the syndication application. Queued: The syndicator has sent a request to the syndication application, but syndication is not yet active. Active: Syndication is occurring between the syndicator and subscriber. Disabled: Syndication is currently disabled. What are the two types of rendering portlets ? Local rendering portlet and Remote rendering portlet. You can also find answers to all of the questions in the WebSphere Portal 6.0 Information Center or check the WebSphere Portal Books

How to find the websphere version of your installation ?

You might think that it's something trivial that everyone might know but unfortunately my blog gets hits looking for the command to find the version of the websphere almost everyday , hence i wrote this article and also learnt that you can write blog articles about things that are trivial as well and might get hits as good or ever better number of hits compared to non-trivial articles with less amount of time spent on writing one :) .

To find the version of WebSphere Application Sever,

• You can run versionInfo command to generate a report from data extracted from XML files in the properties/version folder. The report includes a list of changed components and installed or uninstalled maintenance packages. you can find more about the differnet
  

On Windows : run %WAS_HOME%\bin\versionInfo.bat (e.g) c:\WebSphere\AppServer\versionInfo.bat

On Unix : run $WAS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/AppServer/bin/versionInfo.sh

Sample Report generated by the versionInfo.bat command:

D:\WebSphere\AppServer\bin>versionInfo.bat
WVER0010I: Copyright (c) IBM Corporation 2002; All rights reserved.
WVER0011I: WebSphere Application Server Release 5.1
WVER0012I: VersionInfo reporter version 1.15.2.1, dated 4/28/04

--------------------------------------------------------------------------------
IBM WebSphere Application Server Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 2009-07-20T20:24:56-07:00

Installation
--------------------------------------------------------------------------------
Product Directory D:\WebSphere\AppServer
Version Directory ${product.dir}\properties\version
DTD Directory ${version.dir}\dtd
Log Directory D:\WebSphere\AppServer\logs\update
Backup Directory ${version.dir}\backup
TMP Directory C:\DOCUME~1\rkannan\LOCALS~1\Temp

Installation Platform
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1

Technology List
--------------------------------------------------------------------------------
BASE installed
JDK installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1.1.14
ID BASE
Build Level cf140715.02
Build Date 04/19/2007

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere SDK
Version 1.4.1
ID JDK
Build Level ${build.level}
Build Date ${dateStamp}

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

To find the version of WebSphere Portal Server,

• You can run WPVersionInfo.sh to gather version information of the websphere portal server installation.
  

On Windows : run %WPS_HOME%\bin\WPVersionInfo.bat (e.g) c:\WebSphere\PortalServer\bin\WPVersionInfo.bat

On Unix : run $WPS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/PortalServer/bin/WPVersionInfo.sh

Sample report of the command:

[root@server101 bin]# ./WPVersionInfo.sh
WVER0010I: Copyright IBM Corporation 2002, 2008; All rights reserved.
WVER0011I: WebSphere Portal
WVER0012I: VersionInfo reporter version 1.10.1.1, dated 20080530_1701
--------------------------------------------------------------------------------
IBM WebSphere Portal Product Installation Status Report
--------------------------------------------------------------------------------
... ...
Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Portal MultiPlatform
Version6.1.0.0
ID MP
Build Levelwp610_494_01 2008-06-19
Build Date 06/19/2008
Installed Product
... ...
--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------


To find the version of IBM JDK installed along with WebSphere Application Server,

• you can run java -fullversion to find the jdk version details

On Windows : run %WAS_HOME%\java\bin\java -fullversion (e.g) c:\WebSphere\AppServer\java\bin\java -fullversion

On Unix : run $WAS_HOME/java/bin/java -fullversion (e.g) /opt/WebSphere/AppServer/java/bin/java -fullversion

Sample output of the command:

D:\WebSphere\AppServer\java\bin>java -fullversion
java full version "J2RE 1.4.2 IBM Windows 32 build cn1420-20040626"

websphere jdbc connection pool aged timeout setting in websphere might shrink the pool size below the specified Minimum Connection setting

If you have used websphere datasources you might have seen a jdbc connection pool setting called Aged Timeout specifies the interval in seconds before a physical connection is discarded irrespective of whether the connection was idle or used before. This property is usually set to a specified interval to maintain the freshness of the connection between the database and also to avoid any firewall sitting between your application and database terminate the connection after staying idle for a period of time, which most firewall does. Other instance where this property might be useful is if you have an Oracle RAC (Real Application Clusters) kind of environment where you wanted to do some kind of rolling upgrade or scheduled maintenance across different nodes in the cluster , in that case DBA's might want to bring down each node gracefully after all connections are drained from the node instead of shutting down the node and terminate the connections abruptly. so this property comes in handy that after a specified time interval in the aged timeout property the connections are closed and when needed it gets created again where at this point the connections are established to a different node based on the Oracle RAC configuration change allowing them to redirect the connection to node other than the node scheduled for downtime. This is all good based on the requirements but the downside to this is the connection manager will simply close the connection based on the aged timeout when the time expires and will not recreate the connection even if the pool is below the minimum connection settings that will caused the pool to go shrink below the expected minimum size which might impact the performace of the application at certain times. Theoretically speaking you don't need to set a aged timeout particulary if you are using Oracle RAC with properly implemented high availability techniques either using Fast Connection Failover or Transparent Application Failover where tthese are totally capable to handle failover in case of an scheduled/plan outage or unplanned outage, but DBA's sometimes seems to be overly cautious, wanted application administrator to enforce these settings. There may be environments where you will be using Oracle RAC but none of those failover mechanisms been implemented and in that case insist you DBA's to implement any one of those failover mechanism. Note if you are using tomcat default's commons dbcp connection pooling there isn't a aged timeout parameter in the connection pooling settings, then you might have to rely on the database failover features. If you are using c3po another popular connection pooling datasource there is a maxConnectionAge property where it will close the connection after the specified interval and also create a connection at the same time if necessary in order to not to shrink the pool below the specified minimum pool size.

java.sql.SQLException: OALL8 is in an inconsistent state message in logs

You might see the following exception in the logs of your WebSphere or Tomcat Application Server,

2009-06-13 17:35:23,294 434385047 ERROR [com.ibm.ws.services.encryption.v1.provider.EncryptionServiceImpl] (web-1:) - org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: OALL8 is in an inconsistent state at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:238)

One of the known reason might be that your database got shutdown in one of the node particularly in a Oracle RAC kind of environment where multiple nodes are present. The error usually recovers once the connections are failed over but it seems like atleast few transactions are failing before it can recover. Also the message is usually preceeded by this message,

2009-06-13 17:35:22,405 434384158 ERROR [org.apache.axis2.engine.AxisEngine] (web-2:) - Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted
org.apache.axis2.AxisFault: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted

which indicates that the db is shutting down. If you want to have a gracful shutdown of db without even affecting few transactions then check my other article websphere jdbc connection pool aged timeout setting in websphere might shrink the pool size below the specified Minimum Connection setting in the blog.

How to find the version of websphere, java and portal that you are running ?

You might think that it's something trivial that everyone might know but unfortunately my blog gets hits looking for the command to find the version of the websphere almost everyday , hence i wrote this article and also learnt that you can write blog articles about things that are trivial as well and might get hits as good or ever better number of hits compared to non-trivial articles with less amount of time spent on writing one :) .

To find the version of WebSphere Application Sever,

• You can run versionInfo command to generate a report from data extracted from XML files in the properties/version folder. The report includes a list of changed components and installed or uninstalled maintenance packages. you can find more about the differnet
  

On Windows : run %WAS_HOME%\bin\versionInfo.bat (e.g) c:\WebSphere\AppServer\versionInfo.bat

On Unix : run $WAS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/AppServer/bin/versionInfo.sh

Sample Report generated by the versionInfo.bat command:

D:\WebSphere\AppServer\bin>versionInfo.bat
WVER0010I: Copyright (c) IBM Corporation 2002; All rights reserved.
WVER0011I: WebSphere Application Server Release 5.1
WVER0012I: VersionInfo reporter version 1.15.2.1, dated 4/28/04

--------------------------------------------------------------------------------
IBM WebSphere Application Server Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 2009-07-20T20:24:56-07:00

Installation
--------------------------------------------------------------------------------
Product Directory D:\WebSphere\AppServer
Version Directory ${product.dir}\properties\version
DTD Directory ${version.dir}\dtd
Log Directory D:\WebSphere\AppServer\logs\update
Backup Directory ${version.dir}\backup
TMP Directory C:\DOCUME~1\rkannan\LOCALS~1\Temp

Installation Platform
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1

Technology List
--------------------------------------------------------------------------------
BASE installed
JDK installed

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Application Server
Version 5.1.1.14
ID BASE
Build Level cf140715.02
Build Date 04/19/2007

Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere SDK
Version 1.4.1
ID JDK
Build Level ${build.level}
Build Date ${dateStamp}

--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------

To find the version of WebSphere Portal Server,

• You can run WPVersionInfo.sh to gather version information of the websphere portal server installation.
  

On Windows : run %WPS_HOME%\bin\WPVersionInfo.bat (e.g) c:\WebSphere\PortalServer\bin\WPVersionInfo.bat

On Unix : run $WPS_HOME/bin/versionInfo.sh (e.g) /opt/WebSphere/PortalServer/bin/WPVersionInfo.sh

Sample report of the command:

[root@server101 bin]# ./WPVersionInfo.sh
WVER0010I: Copyright IBM Corporation 2002, 2008; All rights reserved.
WVER0011I: WebSphere Portal
WVER0012I: VersionInfo reporter version 1.10.1.1, dated 20080530_1701
--------------------------------------------------------------------------------
IBM WebSphere Portal Product Installation Status Report
--------------------------------------------------------------------------------
... ...
Installed Product
--------------------------------------------------------------------------------
Name IBM WebSphere Portal MultiPlatform
Version6.1.0.0
ID MP
Build Levelwp610_494_01 2008-06-19
Build Date 06/19/2008
Installed Product
... ...
--------------------------------------------------------------------------------
End Installation Status Report
--------------------------------------------------------------------------------


To find the version of IBM JDK installed along with WebSphere Application Server,

• you can run java -fullversion to find the jdk version details

On Windows : run %WAS_HOME%\java\bin\java -fullversion (e.g) c:\WebSphere\AppServer\java\bin\java -fullversion

On Unix : run $WAS_HOME/java/bin/java -fullversion (e.g) /opt/WebSphere/AppServer/java/bin/java -fullversion

Sample output of the command:

D:\WebSphere\AppServer\java\bin>java -fullversion
java full version "J2RE 1.4.2 IBM Windows 32 build cn1420-20040626"

Akamai IP in the access logs of webserver or appserver

If your website is akamized either by Edge Content Delivery or Dynamic Site Accelerator your webserver or appserver access logs will always shows the IP of the akamai edge server instead of the actual client ip, but with the exception if the client is using the origin server domain address to reach the server , which usaully happens when you are testing the server for problems isolating akamai. During any outage issues you will be curious to know whose ip is logged , where the ip came from and how many requests are made during the time. Unfortunately like many other isp's, akamai doesn't provide any reverse lookup on their servers to easily isloate ip's of akamai to that of others. so here is the little trick you can use to find whether a particular ip belongs to an akamai server. Just get the ip address from the logs and make a request using your browser like http://77.67.85.65/ , since akamai servers listens on port 80 you will usually get a 400 Bad Request http response with Invalid URL message and if you inspect the headers with the live http headers firefox plugin, you will notice that the "Server:" header will have a value of AkamaiGHost as show below which can confirm that the particular ip is from an akamai server, Note Akamai also give us the ability to get the client ip address in a custom IP header called True-Client-IP, with custom logging enabled you should be able to print the real client ip in the access log as well.

Invalid URL

The requested URL "/", is invalid.

Reference #9.3d55434d.1248071422.0

---------------------------------------------------------------------------------------------
http://77.67.85.65/

GET / HTTP/1.1
Host: 77.67.85.65
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-serial-no

HTTP/1.x 400 Bad Request
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 187
Expires: Mon, 20 Jul 2009 06:17:42 GMT
Date: Mon, 20 Jul 2009 06:17:42 GMT
Connection: close
----------------------------------------------------------

Understanding Akamai Headers to debug slowness or cache related problems

If your website is akamaized or basically cached by akamai, you would expect to see your web pages loaded faster, but in some cases you won't find the difference which might be related to your akamai settings or the response headers that your website is sending via akamai. Hence in order to debug and fine tune, it is necessary to understand the headers that akamai send along with the regular HTTP headers. To inspect those headers you can download the latest version of EdgeSuite Booster for Internet Explorer on Akamai’s portal (https://control.akamai.com/) under Support > Documentation > EdgeSuite > Tools or use Live HTTP Headers firefox plugin. Here is a sample of a response headers via akamai where the highlighted ones are akamai specific headers,

http://www.example.com/common/global.html

GET /common/global.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)
Accept: text/css,*/*;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-serial-no
Cache-Control: max-age=0

HTTP/1.x 200 OK
Content-Type: text/html
Etag: W/"26100-124692218130400"
Cache-Control: max-age=120
Date: Tue, 14 Jul 2009 18:01:27 GMT
X-Cache: TCP_MEM_HIT from a69-174-50-208 (AkamaiGHost/5.7.2-5726413) (-)
X-Cache-Key: S/L/2649/75069/6h/www.example.com/common/global.html
X-True-Cache-Key: /L/www.example.com/common/global.html
X-Serial: 2649
Connection: keep-alive
Vary: Accept-Encoding
X-Check-Cacheable: YES

The X-Cache headers are related to akamai and here are the different headers you might get to see and their respective meanings,

X-Cache HTTP Response Header
Returned from request using "Pragma: akamai-x-cache-on":

TCP_HIT: The object was fresh in cache and object from disk cache.
TCP_MISS: The object was not in cache, server fetched object from origin.
TCP_REFRESH_HIT: The object was stale in cache and we successfully refreshed with the origin on an If-Modified-Since request.
TCP_REFRESH_MISS: Object was stale in cache and refresh obtained a new object from origin in response to our IF-Modified-Since request.
TCP_REFRESH_FAIL_HIT: Object was stale in cache and we failed on refresh (couldn't reach origin) so we served the stale object.
TCP_IMS_HIT: IF-Modified-Since request from client and object was fresh in cache and served.
TCP_NEGATIVE_HIT: Object previously returned a "not found" (or any other negatively cacheable response) and that cached response was a hit for this new request.
TCP_MEM_HIT: Object was on disk and in the memory cache. Server served it without hitting the disk.
TCP_DENIED: Denied access to the client for whatever reason
TCP_COOKIE_DENY: Denied access on cookie authentication (if centralized or decentralized authorization feature is being used in configuration)

X-Cache-Key HTTP Response Header
Returned from request using "Pragma: akamai-x-get-cache-key"; the value is the internal cache key for the page or object on the Akamai server.

WebSphere Portal Books , Redbooks, Redpapers, Wiki, Articles, Infocenter, etc resources links

List of links for different resources on the topic of WebSphere Portal.

WebSphere Portal Press Books:

IBM WebSphere Portal Primer: Second Edition

Mastering IBM WebSphere Portal: Expert Guidance to Build and Deploy Portal Applications

Programming Portlets: From JSR 168 to IBM WebSphere Portal Extensions

JSR-168 Portlet Development Simplified, Second Edition: Learning How to Develop Effective, JSR-168, Portal Applications, Everything from the GenericPortlet to the Struts and JSF Apache Portlet Bridges

Exploring IBM Accelerators for WebSphere Portal


IBM WebSphere Portal Redbooks:

WebSphere Portal Version 6 Enterprise Scale Deployment Best Practices


IBM Lotus Wiki:

Managing WebSphere Portal 6.1 Environments

WebSphere Portal Family Wiki

IBM WebSphere Portal RedPapers:

WebSphere Portal Best Practices

IBM WebSphere Portal V6 Self Help Guide

IBM WebSphere Application Server V6.1 Security Handbook

IBM WebSphere Portal Infocenter:

IBM WebSphere Portal Version 6.1 Information Center

IBM WebSphere Portal Version 6.0 Information Center

Automating Deployment of IBM WebSphere Portal with ANT

Coming soon ! - Note , i have already developed ANT scripts but as soon as i find time i can write the article or ping me if someone interested to write for me :) based on the scripts that i can provide.

java.sql.SQLException: Io exception might be related to firewall or load balancer issues

You might see the following exception in the java SystemOut logs of WebSphere,

Caused by: java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA00
10E: SQL State = null, Error Code = 17,002
... 150 more
---- Begin backtrace for nested exception
java.sql.SQLException: Io exception: Got minus one from a read callDSRA0010E: SQL State = null, Error Code = 17,002DSRA0010E: SQL St
ate = null, Error Code = 17,002
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java(Compiled Code))
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java(Compiled Code))
at java.lang.reflect.Constructor.newInstance(Constructor.java(Compiled Code))
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:501)
at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:544)
at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:788)
at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:765)
at com.ibm.ejs.j2c.poolmanager.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1431)
at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java:1182)
at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java(Compiled Code))
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java(Compiled Code))

If you see those exceptions it might be due to some network related, either the firewall ( FSWM - Firewall Services Module ) or your load balancer dropping connections or packets causing the exception to be thrown on the logs in the WebSphere with the strange message plus your connections might get lost and might also see an application related error. This exception might be seen when there was some change going on the FSWM or loadbalancer like acl updates where it's trying to reload and applying them, it usually affects the new connections existing pooled connections seems to work ok without problems.

Also if you are using oracle you might see the following error message in the oracle listener sqlnet.log,

TNS-12637: Packet receive failed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12537: TNS:connection closed
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12547: TNS:lost contact
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12535: TNS:operation timed out
TNS for Linux: Version 10.2.0.3.0 - Production
TNS-12637: Packet receive failed

Here is the explanation from the oracle documentation,

TNS-12637 Packet receive failed

Cause: A process was unable to receive a packet from another process. Possible causes include:
The other process was terminated.
The machine on which the other process is running went down.
Some other communications error occurred.

To rectify it either update the software/firmware to the latest version in your FWSM or load balancer , check for any network bottlenecks, check load average of the machines and take necessary actions. If you are continously seeing the error then there is a possibility that there might be some kind of DoS attack on the db listener.

Synchronization issues related to Java collections when accessed concurrently

Being a java developer or an application server administrator interested in learning how the code works like me :) , you must have seen the javadoc http://java.sun.com/javase/6/docs/api/java/util/HashMap.html warning: Note that this implementation is not synchronized. If multiple threads access a hash map concurrently, and at least one of the threads modifies the map structurally, it must be synchronized externally. (A structural modification is any operation that adds or deletes one or more mappings; merely changing the value associated with a key that an instance already contains is not a structural modification.) This is typically accomplished by synchronizing on some object that naturally encapsulates the map. If no such object exists, the map should be "wrapped" using the Collections.synchronizedMap method. This is best done at creation time, to prevent accidental unsynchronized access to the map.

So here are some things that might happen depends on what kind of Map you are using like LRUMap or TreeMap etc. I have seen these behaviors in runtime where the Map is not synchronized but concurrently accessed, and though of publishing it in the blog which might help others to debug similar issues or correlate the symptoms to the failure they might have seen but not having a clue.



1) LRUMap Memory leak : If you are using a LRUMap and multiple threads are putting and getting values concurrently, some kind of memory leak happens and the object grows beyond the specified maximum size. Every time you access the LRUMap either by get or put, the object you are getting or putting had to be moved to the MRU(Most recently used) place within the Map. Also when the LRUMap is full it has to remove the LRU object to provide room for the new object. I believe this is accomplished by some kind of a link list data structure within the map. Hence there is a lot of overhead during the process and there is quite a chance to get the link list corrupted when multiples threads are get/put’ing concurrently particularly when the map is full.

I noticed the “size()” as returned by the object sometimes gets higher than the maximum size and sometimes show less than the maximum size when it’s supposedly be full depending on how and where the corruption happens within the link list .Further accessing the object after it’s corrupted, the “size()” is starting to get reduced every time a further corruption happens and leads into negative size (-1…-20) and keep increasing in the negative side. Note the “size()” that I am referring is the size returned by the counter that the object is using to track and not the actual object size. so I took a heap dump and found that any object that is added after Is never discarded at all , I guess because of the logic that size never gets bigger than the maximum size after corruption.It looks like one the object is corrupted it’s starting to leak and then increasing the overall size(byte size) and the behavior in erroneous.
I am attaching the lru.jsp that I used to simulate so that you can try and see how it behaves. The below picture shows how the object java.math.BigDecimal grows in consecutive heapdumps even though the max size is set to 2.

Note that this behavior happened on WebSphere 5.1 Application Server on JDK 1.4.2 not sure what might be the behavior on other version of JDK or AppServer. Please post in my comment if you are seeing any different behaviors for any other types of Map.

2) TreeMap High Cpu Utilization: This behavior happened particularly when the interator iterating through the treemap collection is not synchronized, where one thread is changing (add or delete) the element and then other iterating through it concurrently. I didn't do much analysis but it looks like the underlying data structure of the TreeMap some how gets corrupted and causes a circular reference between two elements causing the interator to go into an infinite loop which caues the CPU to spike and eventually the ApplicationServer will become unresponsive. You probably want to kill the process to reocver from this state. Here is the thread stacktrace that you would see if take a thread dump, Attaching the tree.jsp for you to test and see.

at java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1570)
at java.util.TreeMap$NavigableSubMap$SubMapKeyIterator.next(TreeMap.java:1629)
at org.apache.jsp.tree2_jsp._jspService(tree2_jsp.java:120)

Also note many programmers may forget to synchronize on the collection when iterating eventhough the collection is synchronized. Hence, it is imperative that the user manually synchronize on the returned collection when iterating over it to prevent any non-deterministic behavior.

Collection c = Collections.synchronizedCollection(myCollection);
...
synchronized(c) {
Iterator i = c.iterator(); // Must be in the synchronized block
while (i.hasNext())
foo(i.next());
}