Showing posts with label httpserver. Show all posts
Showing posts with label httpserver. Show all posts

Sunday, June 15, 2008

IBM HTTP Server child process core dumps

IBM HTTP Server child process core dumps failing to serve some or all of the requests. You might see the following Segmentation fault errors in the error.log of the server.

[Sun Feb 10 19:26:52 2008] [notice] IBM_HTTP_Server/6.1.0.9 Apache/2.0.47 (Unix) configured -- resuming normal operations
[Sun Feb 10 19:26:52 2008] [notice] CoreDumpDirectory not set; core dumps may not be written for child process crashes
[Sun Feb 10 19:27:20 2008] [notice] child pid 7755 exit signal Segmentation fault (11)
[Sun Feb 10 19:27:21 2008] [notice] child pid 7756 exit signal Segmentation fault (11)
[Sun Feb 10 19:27:23 2008] [notice] child pid 7758 exit signal Segmentation fault (11)
[Sun Feb 10 19:27:25 2008] [notice] child pid 7759 exit signal Segmentation fault (11)
[Sun Feb 10 19:27:26 2008] [notice] child pid 7760 exit signal Segmentation fault (11)
[Sun Feb 10 19:27:27 2008] [notice] child pid 7761 exit signal Segmentation fault (11)
[Sun Feb 10 19:27:37 2008] [notice] child pid 7762 exit signal Segmentation fault (11)
[Sun Feb 10 19:29:36 2008] [notice] child pid 7763 exit signal Segmentation fault (11)
[Sun Feb 10 19:29:49 2008] [notice] child pid 7771 exit signal Segmentation fault (11)
[Sun Feb 10 19:30:21 2008] [notice] child pid 7772 exit signal Segmentation fault (11)
[Sun Feb 10 19:30:27 2008] [notice] child pid 7773 exit signal Segmentation fault (11)
[Sun Feb 10 20:12:12 2008] [notice] child pid 8056 exit signal Segmentation fault (11)

In our case the problem appears to be that we had mistakenly specified the ResponseChunkSize value that is too high like 400000. This value is the number of 1024 byte pages so it is multipled by 1024 which exceeds an acceptable value. so setting it to a lower value of 4000 seems to have resolved the issue. It took several days before we could figure out the issue after working with IBM, and these are following steps you can do that might help debugging the issue if in your case is not related to the ResponseChunkSize.

Please do the following:

1. To set up system to obtain cores:

http://publib.boulder.ibm.com/httpserv/ihsdiag/coredumps.html

2. Make sure you have latest IHSDIAG for debugging IHS issues.

http://www-1.ibm.com/support/docview.wss?uid=swg24008409

3. Run IHSDIAG against core that is obtained:

http://publib.boulder.ibm.com/httpserv/ihsdiag/gather_crash_doc.html

4. Run IHSDIAG against good system to get system information:

http://publib.boulder.ibm.com/httpserv/ihsdiag/describeconfig.html

Friday, December 28, 2007

SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore

If you are a WebSphere Administrator or any Application Server (WebLogic, Tomcat) administrator you might already know that managing the ssl certificates in a large complex environments becomes hectic and troublesome because of the different expiration dates of the certificates that websphere uses and also the ssl certificates of the external systems (like SAP , Siebel) that websphere applicaton server might connect to using a secure connection, multiple administrators in an organization renewing it and not keeping tracking of the expiration dates. The problem is , SSL certificate might expire resulting in servers will not initialize and your running servers will stop operating, becomes unresposive if not properly renewed on time. Hence this article will explain how you can monitor the expiration of the ssl certificates using a simple command and proactively monitor the expiration dates, setup your calendar and renew it on time to prevent any downtimes.

If you are using the latest version (as on Dec 2007) of WebSphere Application Server 6.1 and using default self signed there are provisions in the application server itself to effectively monitor the certificate expiration, notifying you and renewing it automatically. If you are using a perosnal self signed certificate or a personal certificate signed by a certificate authority (CA) like VeriSign or Thawte , which is most likely in all of your production environments then you might want to use the below command to find the expiration date and renew it.

If you are using a java keystore ( DummyServerKeyFile.jks , DummyServerTrustFile.jks , jssecerts , cacerts) use the below find command along with keytool or websphere's ikeyman command line option to find the expiration of the keys.

LINUX:


Using Ikeyman cmdline utility (ikeyman.sh):

find PATH -name *.jks | xargs -i bash -c '$WAS_HOME/java/bin/java -classpath $WAS_HOME/java/jre/lib/ext/ibmjceprovider.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmjcefw.jar:$WAS_HOME/AppServer/java/jre/lib/ext/US_export_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/local_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmpkcs.jar:$WAS_HOME/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw <password> | grep "Not After:.*yyy[y,y]" -B 3 && echo {}'

(e.g)
find /usr/IBM/WebSphere/AppServer/profiles/ -name *.jks | xargs -i bash -c '/usr/IBM/WebSphere/AppServer/java/bin/java -classpath /usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/AppServer//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw WebAS | grep "Not After:.*200[7,8]" -B 3 && echo {}'

Using Java Keytool utility:

find $WAS_HOME -name '*.jks' | xargs -i bash -c "echo {}; $WAS_HOME/java/jre/bin/keytool -list -v -keystore {} -storepass password | grep 'until: .*/.*/Y[Y,Y]' -A 3 -B 7"

(e.g)
find /usr/IBM/WebSphere/AppServer -name '*.jks' | xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS | grep 'until: .*/.*/0[7,8]' -A 3 -B 7"

Output:


/usr/IBM/WebSphere/AppServer/profiles/etc//WWWStageClientKey.jks
Alias name: verisign class 1 ca individual subscriber-persona not validated
Creation date: Dec 18, 2006
Entry type: trustedCertEntry

Owner: CN=VeriSign Class 1 CA Individual Subscriber-Persona Not Validated, OU="www.verisign.com/repository/RPA Incorp. By Ref.,LIAB.LTD(c)98", OU=VeriSign Trust Network, O="VeriSign, Inc."
Issuer: OU=Class 1 Public Primary Certification Authority, O="VeriSign, Inc.", C=US
Serial number: d8b4feeaad218df5bf4756a9d29e17ffb
Valid from: 5/12/98 12:00 AM until: 5/12/08 11:59 PM
Certificate fingerprints:
MD5: CA:66:3C:FC:71:2B:BA:41:92:71:DD:72:AD:E5:65:65
SHA1: 12:51:9A:E9:CD:77:7A:56:01:84:F1:FB:D5:42:15:22:2E:95:E7:1F
--
Creation date: Dec 18, 2006
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=US
Issuer: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=US
Serial number: 458709ef
Valid from: 12/18/06 9:36 PM until: 12/18/07 9:36 PM
Certificate fingerprints:
MD5: C2:4F:CC:SS:19:DC:E4:88:B0:2A:78:76:69:4D:DC:EF:47
SHA1: 53:AB:D0:13:45:45:12:59:64:F6:C9:38:41:F2:C8:E3:37:05:73:95:F3:83

/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks
Alias name: www stage websphere server ca
Creation date: Dec 8, 2006
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=wadm.stage.www.com, OU=IT, O="www, Inc.", L=Fremont , ST=California, C=US
Issuer: OU=Secure Server Certification Authority, O="RSA Data Security, Inc.", C=US
Serial number: 1ea5f21b33ca65e551440d662399dffdfga382
Valid from: 12/5/06 12:00 AM until: 12/15/07 11:59 PM
Certificate fingerprints:
MD5: C2:22:E4:12:12:6D:3F:22:67:4E:37:48:12:3F:F3:1B:FE:26
SHA1: CA:81:37:A7:E9:12:22:19:27:D9:3F:C5:79:E9:25:C8:1A:3C:1E:5C:17:02

As you can see in the above output the command will list the file name (/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks) , certificate label (www stage websphere server ca) , Entry Type (keyEntry - personal cert or trustedCertEntry - Signer Cert), and the expiration date (12/5/06 12:00 AM until: 12/15/07 11:59 PM) of
all the java keystore (*.jks) in the given directory that expired in 2007 (07) or 2008 (08). By which you shoule be able to indentify the certs, setup your calendat atleast 10 days before the expiration and renew it to proactively to preveny any problems.

Note you can also change the above command by passing differnt filename like cacerts and jssecerts and also pass different year infomration to find expirations during those years.

Sometimes you might get no output which means either there are no expiring ceritificates for the years that you passed or if you are expecting any expiring certificate to show and if not then the password for the keystore might be wrong, so run without the grep portion in the above command which will list all the certificates irrespective of the expiration date or it will complain if the password is wrong as below.

find /usr/IBM/WebSphere/AppServer -name '*.jks' | xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS

/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks
java.io.IOException: Keystore was tampered with, or password was incorrect


Also you can use the same command by modifying the -type parameter to "-type cms" for the webserver such as IBM HttpServer where it's using cms database ( kdb file).


find /usr/IBM/HttpServer -name *.kdb | xargs -i bash -c '/usr/IBM/WebSphere/Plugins/java/bin/java -classpath /usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/Plugins//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type cms -pw WebAS | grep "Not After:.*200[7,8]" -B 3 && echo {}'