Useful UNIX commands for WebSphere & Portal Administrator

These are some useful commands that i came across when administrating WebSphere Application Server or WebSphere Portal Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT.

Performance related Commands to monitor system performance
-------------------------------------------------------------------------
1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the CPU

%CPU PID USER COMMAND 71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere /AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=54859 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer

2) Find the last 10 process that use the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

%CPU PID USER COMMAND 2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer 0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ 0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer 0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/cust/IBM/WebSphere/AppServer

3)Find cpu usage
mpstat -P ALL or mpstat

Linux 2.4.21-27.ELsmp (machine1.www.com) 02/02/2008 11:55:00 AM CPU %user %nice %system %iowait %irq %soft %idle intr/s 11:55:00 AM all 1.51 0.00 0.41 0.04 0.01 0.09 97.94 184.92 11:55:00 AM 0 1.50 0.00 0.41 0.04 0.01 0.11 97.93 123.76 11:55:00 AM 1 1.51 0.00 0.42 0.04 0.00 0.07 97.95 61.16

4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc.
top

12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00 572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9% cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6% cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2% Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd, 295304k buff 3998132k active, 682312k inactive Swap: 2048276k av, 0k used, 2048276k free 1258592k cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 17719 root 16 0 306M 306M 3992 S 0.7 2.6 22:14 1 java 19358 root 16 0 2532 2532 896 R 0.5 0.0 0:00 1 top 17771 root 15 0 306M 306M 3992 S 0.4 2.6 3:48 0 java 18058 root 16 0 458M 458M 4032 S 0.1 3.9 12:55 0 java 1 root 15 0 512 512 452 S 0.0 0.0 1:19 0 init 2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0

5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &

Linux 2.4.21-27.ELsmp (machine1.web.com) 02/02/2008 12:04:34 PM CPU %user %nice %system %iowait %idle 12:04:36 PM all 0.25 0.00 0.00 0.00 99.75 12:04:38 PM all 0.00 0.00 0.50 0.25 99.25 12:04:40 PM all 7.00 0.00 7.25 0.00 85.75 12:04:42 PM all 21.05 0.00 1.50 0.00 77.44 12:04:44 PM all 3.00 0.00 0.50 0.00 96.50 Average: all 6.26 0.00 1.95 0.05 91.74

Disk Usage related Commands to monitor Disk space
--------------------------------------------------------
1)Disk Usage on all mounts
df -h -T

Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext3 2.9G 785M 2.0G 29% / /dev/sda7 ext3 57G 9.3G 45G 18% /cust /dev/sda6 ext3 2.0G 82M 1.8G 5% /lc none tmpfs 5.7G 0 5.7G 0% /dev/shm /dev/sda5 ext3 2.0G 33M 1.8G 2% /tmp /dev/sda3 ext3 2.0G 64M 1.8G 4% /var

2)prints just one line with the total size of the directory, the below example show the size of the /usr/IBM/Websphere WebSphere Portal Server directory.
du -ch | grep total

bash-2.05b# du -ch | grep total 6.3G total

Network related Commands
-------------------------------------
1)
a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers.
b) nslookup - Queries a name server for a host or domain lookup.
c) host - DNS lookup uility.

The below example shows the information like CNAME , Aliases, ipaddres , etc about www.google.com


a) dig <ip address or hostname>

bash-2.05b# dig www.google.com ; <<>> DiG 9.2.4 <<>> www.google.com ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 585119 IN CNAME www.l.google.com. www.l.google.com. 69 IN A 216.239.51.99 www.l.google.com. 69 IN A 216.239.51.104 ;; AUTHORITY SECTION: l.google.com. 60734 IN NS g.l.google.com. l.google.com. 60734 IN NS b.l.google.com. l.google.com. 60734 IN NS f.l.google.com. l.google.com. 60734 IN NS c.l.google.com. l.google.com. 60734 IN NS a.l.google.com. l.google.com. 60734 IN NS e.l.google.com. l.google.com. 60734 IN NS d.l.google.com. ;; Query time: 2 msec ;; SERVER: 10.5.50.184#53(10.5.50.184) ;; WHEN: Sun Feb 3 09:51:46 2008 ;; MSG SIZE rcvd: 196

b) host <ip address or hostname>

bash-2.05b# host www.google.com www.google.com is an alias for www.l.google.com. www.l.google.com has address 216.239.51.104 www.l.google.com has address 216.239.51.99

c) nslookup <ip address or hostname>

bash-2.05b# nslookup www.google.com Server: 10.5.50.184 Address: 10.5.50.184#53 Non-authoritative answer: www.google.com canonical name = www.l.google.com. Name: www.l.google.com Address: 216.239.51.99 Name: www.l.google.com Address: 216.239.51.104

2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a loopback.


infconfig -a

bash-2.05b# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:11:43:32:49:99 inet addr:10.2.150.74 Bcast:10.5.50.127 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0 TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3168219417 (3021.4 Mb) TX bytes:3724305382 (3551.7 Mb) Interrupt:17 eth1 Link encap:Ethernet HWaddr 00:11:43:32:49:9A inet addr:10.5.154.138 Bcast:10.5.54.127 Mask:255.255.255.128 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:18 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0 TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1454141744 (1386.7 Mb) TX bytes:1454141744 (1386.7 Mb)

3) To find more information about a network interface, the below example shows detail information about interface eth0.


bash-2.05b# ethtool eth0

Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: Not reported Advertised auto-negotiation: No Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: off Supports Wake-on: d Wake-on: d Link detected: yes

File related commands
----------------------
1) find text in the matching files and show the filename with the pattern. The below example shows the list of filenames and the location of the hostname in those files, which will be helpful when changing hostnames for your websphere installation.

find . -name '*.xml' -print | xargs grep 'hostName' /dev/null

bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.xml' -print | xargs grep 'hostName' /dev/null | more /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverNode01/serverindex.xml:<serverindex: ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5 .0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointRefs="NamedEndPoint_1130375480667 NamedE ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPoint_1130375480671 NamedEndPoint_113037548067 2 NamedEndPoint_1130375480673"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2.example.com/serverindex.xml:< serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2.example.com/serverindex.xml:< serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverCellManager01/serverindex.xml: <ser verindex:ServerIndex xmi:id="ServerIndex_1" endPointRefs="NamedEndPoint_1 NamedEndPoint_2" hostName="portalserver.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2011indchn2Node01/serverindex.xml:<serverindex: ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5 .0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointRefs="NamedEndPoint_1130375717654 NamedE ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPoint_1130375717658 NamedEndPoint_113037571765 9 NamedEndPoint_1130375717660"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2UnNode01/serverindex.xml:<serverinde x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas /5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2UnNode01/serverindex.xml:<serverinde x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas /5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serverindex:ServerI ndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5.0/serv erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode01Cell/nodes/portalserverNode01/serverindex.x ml:<serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/app server/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com"> /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serveri ndex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/sche mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">

2) Soemtimes you might run out of diskspace and want to clean up some files, so this command will help you find all large files. In this example it will list all the files greater than 10 mb in size.

find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5

bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5-rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/tranlog/log1 -rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log2 -rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log1

3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to FTP,email or http upload.

To pack on the fly...
tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz

To unpack on the fly..
gunzip < FILE.tar.gz | tar -xvf -

To pack on the fly and email the file to IBM Support
tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR 09921,227,000" -b "SystemOut.log for server crash " WebSphere_Support@mainz.ibm.com

To find selective files and tar it up on the fly...
find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf System_logs.tar


4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin where you might want to write stdout to the file as well as to the console so that you have a copy even when the standard output disappears because of the screen buffer size.


ls -l | tee > filename
xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url http://localhost:9080/wps/config | tee filename
./wsadmin.sh | tee wsadmin.log

5) find files that is recently updated which might be useful to see what changes are made to the system recently (e.g) below command shows the files that are modified within the last 20 minutes(-mmin) and 20 days (-mtime)


find /usr/IBM -mmin -20
find /usr/IBM -mtime -20

/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs

6) Find a memory usage of the system.


free
cat /proc/meminfo

bash-2.05b# free total used free shared buffers cached Mem: 11819592 4938416 6881176 0 298148 1440200 -/+ buffers/cache: 3200068 8619524 Swap: 2048276 0 2048276 bash-2.05b# cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 12103262208 5056954368 7046307840 0 305303552 1474797568 Swap: 2097434624 0 2097434624 MemTotal: 11819592 kB MemFree: 6881160 kB MemShared: 0 kB Buffers: 298148 kB Cached: 1440232 kB SwapCached: 0 kB Active: 3718616 kB ActiveAnon: 2968776 kB ActiveCache: 749840 kB Inact_dirty: 789328 kB Inact_laundry: 237596 kB Inact_clean: 0 kB Inact_target: 949108 kB HighTotal: 11140992 kB HighFree: 6681628 kB LowTotal: 678600 kB LowFree: 199532 kB SwapTotal: 2048276 kB SwapFree: 2048276 kB Committed_AS: 6039100 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB

4) To repeat a command contimuously.

watch -d=10 -n 1 "netstat -a"

Every 1s: netstat -a| grep ESTABLISHED Sun Feb 3 21:48:04 2008 tcp 0 0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544 ESTABLISHED tcp 0 0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545 ESTABLISHED